Human-in-the-loop approval

Deep dive

How it works

Parts:

• Risk classes — read (auto), reversible write (auto within limits), irreversible or money/external effect (ask or deny).
• Policy table — rules by tool, argument thresholds and requester role; no match means deny.
• Escalation triggers — signals that hand the whole conversation to a person.
• Pending store — durable record per proposal: call, requester, deadline, state, idempotency key.
• Checkpoint — saved agent state; the run resumes on the decision, in any process.
• Approval queue — risk-ranked inbox showing action, arguments, diff and reason.
• Audit log — append-only: ask, decision, who, when.

1. The agent proposes a call (tool calling); the harness classifies it.
2. Auto: run it and log it. Deny: return an error result the model can read. Ask: continue.
3. Ask: derive the idempotency key from run, tool and exact arguments; store or find the pending record; checkpoint the run and stop.
4. The approval request shows the exact call, a diff or preview (amount, recipient, file diff, order text) and why the policy asked (rule, threshold, low confidence).
5. The human approves, edits (re-classified, never run unchecked) or rejects with a reason.
6. Approve: mark done, execute under the key, resume with the result; a second approve is a no-op. Reject or deadline passed: resume with a denial.
7. Scope: once, this session, or a standing rule with an expiry and a ceiling, logged like any decision.
8. Handoff: when a trigger fires, the agent writes a context summary (goal, steps done, tool results, what is blocked), assigns the queue and stops replying.

 proposal -> [classify] --Auto--> run -> log
                |--Deny--> error result to model
                '--Ask--> [pending: key, deadline] -> checkpoint, run stops
                                |  human: approve / edit / reject
                                v
      approve (once) -> run -> resume       reject / timeout -> denial -> resume

Escalation triggers for a whole conversation: an explicit request for a person; repeated failure (same error or no progress for N turns); negative or worsening sentiment; a topic reserved for humans (legal, safety); low confidence. Prefer measurable signals; self-reported model confidence is a weak trigger alone.

Same mechanism in other fields: support refunds (auto to a small amount, ask above, deny past a hard cap, manager role for the top band); coding agents (reads auto, shell commands and pushes ask; Claude Code evaluates deny, then ask, then allow, first match winning); payments and trades (four-eyes: maker proposes, a different checker releases); healthcare (a drafted order takes effect after a clinician signs); infrastructure (production changes wait for an approver); moderation (uncertain or severe cases go to a reviewer queue).

Code

Policy by tool and amount threshold, a pending store with idempotent approval, separation of duties, default-deny timeouts and an append-only audit log. Std only; time is an explicit now so the tests are deterministic. A real store persists records transactionally.

use std::collections::HashMap;

#[derive(Clone, Debug, PartialEq)]
pub struct Call { pub run: u64, pub tool: String, pub args: String, pub amount: u64 } // amount: cents moved, 0 if none

#[derive(Debug, PartialEq)]
pub enum Decision { Auto, Ask(&'static str), Deny(&'static str) }

/// amount <= auto_max runs unattended, <= ask_max needs a human, above is refused.
pub struct Rule { pub tool: &'static str, pub auto_max: u64, pub ask_max: u64 }

pub fn classify(rules: &[Rule], c: &Call) -> Decision {
    match rules.iter().find(|r| r.tool == c.tool) {
        None => Decision::Deny("tool not in policy"), // default deny
        Some(r) if c.amount <= r.auto_max => Decision::Auto,
        Some(r) if c.amount <= r.ask_max => Decision::Ask("over auto threshold"),
        Some(_) => Decision::Deny("over hard limit"),
    }
}

#[derive(Debug, PartialEq, Clone, Copy)]
pub enum State { Pending, Done, Rejected, Expired }
pub struct Pending { pub call: Call, pub requester: String, pub deadline: u64, pub state: State }

#[derive(Default)]
pub struct Store { items: HashMap<String, Pending>, audit: Vec<String> } // audit is append-only

impl Store {
    pub fn audit(&self) -> &[String] { &self.audit }
    fn log(&mut self, now: u64, msg: String) { self.audit.push(format!("t={now} {msg}")); }

    /// The key is derived from run + tool + exact arguments, so a retried proposal maps to the same record.
    pub fn submit(&mut self, c: Call, requester: &str, now: u64, ttl: u64) -> String {
        let key = format!("{}:{}:{}", c.run, c.tool, c.args);
        if !self.items.contains_key(&key) {
            self.log(now, format!("ask {key} by {requester}"));
            self.items.insert(key.clone(), Pending { call: c, requester: requester.into(), deadline: now + ttl, state: State::Pending });
        }
        key
    }

    /// Ok(true) = executed now, Ok(false) = already executed (no-op). Err = refused.
    pub fn approve(&mut self, key: &str, approver: &str, now: u64, exec: &mut dyn FnMut(&Call)) -> Result<bool, &'static str> {
        let p = self.items.get_mut(key).ok_or("unknown key")?;
        match p.state {
            State::Done => return Ok(false),
            State::Rejected | State::Expired => return Err("already closed"),
            State::Pending => {}
        }
        if p.requester == approver { return Err("requester cannot approve"); } // separation of duties
        if now > p.deadline { p.state = State::Expired; self.log(now, format!("expired {key}")); return Err("expired"); }
        p.state = State::Done; // real store: one transaction; also pass `key` to the tool as its idempotency key
        exec(&p.call);
        self.log(now, format!("approved {key} by {approver}"));
        Ok(true)
    }

    pub fn reject(&mut self, key: &str, approver: &str, now: u64) {
        if let Some(p) = self.items.get_mut(key) {
            if p.state == State::Pending { p.state = State::Rejected; self.log(now, format!("rejected {key} by {approver}")); }
        }
    }

    /// Timeout sweep: every pending record past its deadline is denied.
    pub fn expire(&mut self, now: u64) {
        let dead: Vec<String> = self.items.iter().filter(|(_, p)| p.state == State::Pending && now > p.deadline).map(|(k, _)| k.clone()).collect();
        for k in dead { self.items.get_mut(&k).unwrap().state = State::Expired; self.log(now, format!("expired {k}")); }
    }
}

#[cfg(test)]
mod tests {
    use super::*;
    fn call(tool: &str, amount: u64) -> Call { Call { run: 1, tool: tool.into(), args: format!("order=42 amt={amount}"), amount } }

    #[test]
    fn paths() {
        let rules = [Rule { tool: "refund", auto_max: 5_000, ask_max: 50_000 }];
        assert_eq!(classify(&rules, &call("refund", 4_999)), Decision::Auto);
        assert!(matches!(classify(&rules, &call("refund", 20_000)), Decision::Ask(_)));
        assert!(matches!(classify(&rules, &call("refund", 90_000)), Decision::Deny(_)));
        assert!(matches!(classify(&rules, &call("rm_rf", 0)), Decision::Deny(_)));

        let (mut s, mut runs) = (Store::default(), 0);
        let mut exec = |_: &Call| runs += 1;
        let k = s.submit(call("refund", 20_000), "agent-7", 100, 3_600);
        assert_eq!(k, s.submit(call("refund", 20_000), "agent-7", 101, 3_600)); // retry, same record
        assert_eq!(s.approve(&k, "agent-7", 200, &mut exec), Err("requester cannot approve"));
        assert_eq!(s.approve(&k, "maria", 200, &mut exec), Ok(true));
        assert_eq!(s.approve(&k, "maria", 201, &mut exec), Ok(false)); // second approve: no-op
        let k2 = s.submit(call("refund", 30_000), "agent-7", 100, 3_600);
        s.reject(&k2, "maria", 150);
        assert_eq!(s.approve(&k2, "maria", 160, &mut exec), Err("already closed"));
        let k3 = s.submit(call("refund", 40_000), "agent-7", 100, 3_600);
        assert_eq!(s.approve(&k3, "maria", 9_999, &mut exec), Err("expired")); // late approval = deny
        let k4 = s.submit(call("refund", 41_000), "agent-7", 100, 10);
        s.expire(500);
        assert_eq!(s.approve(&k4, "maria", 501, &mut exec), Err("already closed"));
        assert_eq!(runs, 1);
        assert_eq!(s.audit().len(), 8);
    }
}

Compiled and tested (cargo test --release): retried proposal, self-approval, double approve, reject, late approve and sweep expiry all pass.

Complexity and constants

• Classification is a scan of the rule table: O(rules), a few nanoseconds for a handful of rules.
• Store operations are O(1) by key; the audit log appends O(1).
• Cost is dominated by the human, not the machine.
• Size: one pending record is the call plus a few fields of state, plus the checkpoint.

Efficiency

• Cost: machine side, measured here on Apple M4 Pro, one thread, in memory: classify about 4 ns per call; a full classify + submit + approve + execute + two audit lines cycle about 1.7 million per second (self-measured, excludes durable writes, which add a transaction per decision). The real cost is human attention and the delay of the paused task. Bound: human latency.
• Levers:
  • Auto-allow below thresholds and for reads — removes those entries from the queue (share depends on traffic; measure) — when low-risk calls dominate.
  • Park, don't poll — a checkpointed run holds no process while waiting (LangGraph interrupts wait indefinitely on saved state) — any wait longer than seconds.
  • Batch similar asks (same tool, same requester) — one decision for many — when items are alike and individually low risk.
  • Risk-ranked queue with deadlines — irreversible and urgent first — when a backlog exists.
  • Standing rules with expiry — repeated asks vanish — for stable patterns.
• Naive → standard → state of the art:
  • Ask on every tool call · human time proportional to all calls, then rubber-stamping.
  • Static allow/ask/deny rules per tool and threshold · human time proportional to risky calls.
  • Risk-ranked queue, expiring standing rules, sampled audit of auto-approved calls · human time proportional to novel risky calls, with a measured miss rate.

Implementation notes

• Build the key from the exact arguments, not the model's call id: retries and double clicks must meet the same record.
• Pass the key to the downstream system as its own idempotency key (idempotency keys); the store's Done flag alone cannot cover a crash between executing and marking.
• Resume must not redo work. LangGraph resumes by re-running the node from its start, so side effects before interrupt() must be idempotent; write the approval step as its own node.
• Durable wait: persist state, resume on an external message (OpenAI Agents SDK serializes the paused run; Temporal delivers the decision to a workflow as a Signal). See durable workflow execution.
• Re-check the policy at execution time; state may have changed during the wait.
• Render arguments by code, not by the model; its explanation is a hint, not evidence (prompt injection defense).
• The approver needs rights over the action, not just a login (authorization models); rules can live in a policy engine.
• Keep the log tamper-evident (audit log); correlate with traces.

Pitfalls

• Approval fatigue: hundreds of identical asks breed approve-all clicking. Cut volume with thresholds, batching and ranking; sample auto-approved calls to measure the miss rate. EU AI Act Art. 14 lists awareness of automation bias among oversight measures.
• Approving a summary: the human saw prose, the system ran other arguments. Bind approval to the exact call.
• Stale approval: state changed during the wait. Short deadline; recheck at execution.
• Silent default-allow: a timeout or dead queue that lets the action pass.
• Self-approval: enforce requester != approver in code, including agents acting for the requester.
• Edit bypass: an edited call skips classification.
• Handoff without context: the human restarts the conversation; send summary and tool results.
• Approval is not containment: add sandboxing and least-privilege credentials.

Variants and alternatives

In production

• LangGraph interrupt() with a checkpointer and Command to resume.
• OpenAI Agents SDK: needs_approval (bool or per-call function), RunState serialization for long waits, per-call or sticky always_approve / always_reject.
• Claude Code permission rules (allow, ask, deny).
• Temporal Signals for human decisions in durable workflows.
• Four-eyes release in payments, clinician order sign-off, change approval, moderation review queues.

Implementations

Sources

• OWASP, Top 10 for LLM Applications 2025, LLM06 Excessive Agency (mitigation: require user approval for high-impact actions). page
• European Union, Regulation (EU) 2024/1689 (AI Act), Article 14 Human oversight. text
• NIST, AI Risk Management Framework 1.0, NIST AI 100-1, 2023. PDF
• LangChain, LangGraph: Interrupts. docs
• OpenAI, Agents SDK: Human-in-the-loop. docs
• Anthropic, Claude Code: permissions. docs
• Temporal, Workflow message passing: Signals, Queries, Updates. docs