Human-in-the-loop Agent Design: Which Steps Need Human Approval?

A Feishu message draft is already written: title, body, and attachment link are filled in. Only one step remains: send it. But the agent stops before send_message and waits for your confirmation.

An email can be generated automatically, but before it reaches a customer the UI must show the recipient, subject, body summary, and attachments. A CMS form can be filled in, but the submit_form call is blocked by policy and waits for the owner to approve it. These look like a simple “confirm” button in the UI, but the real boundary is runtime state. The agent’s RunState is saved, then execution resumes only after a person makes a decision. Where to pause, who can approve, and what happens on reject or timeout are not frontend details. They are part of the tool system’s safety boundary.

This guide gives you a practical risk matrix, an approval-point checklist, a pause/resume mechanism, and an audit-field template so approval moves from “add a confirmation button” to “make the run serializable, resumable, and auditable.”

Risk Matrix: Decide Which Actions Need Approval

Not every action needs approval. Read-only actions can run automatically. Deletes and payments should stop and wait for a person. A useful decision starts with these five dimensions:

Dimension	Classification rule	Example actions
External impact	Touches an external system or user	Send email, submit a form, call an external API, write to a collaboration system such as Feishu or Slack
Reversibility	Whether the action can be undone	Delete a record (irreversible), save a draft (reversible), pay money (partly reversible through compensation), send a message (irreversible)
Data sensitivity	Level of data involved	Query public data (public), modify internal records (internal), export user privacy data (sensitive), read production config (sensitive)
Money or permission threshold	Involves funds or permission changes	Payment, transfer, refund, permission change, batch operation, user-data deletion
Autonomy level	Allowed automation level	Read-only query (fully automatic), draft write (fully automatic), outbound message (confirmation), delete/payment (approval)

This matrix maps well to OWASP LLM06’s risk categories around excessive functionality, excessive permissions, and excessive autonomy. You can use it directly or tune the thresholds for your business.

External impact: any action that touches another system or person deserves attention. An email cannot be unsent. A form submission may trigger an order. An external API call may change someone else’s data.

Reversibility: deleting a record is irreversible, a draft can be edited any time, and a payment may be reversible only through a refund or compensation process.

Data sensitivity: public data can usually be read freely; internal data needs write controls; sensitive data such as user privacy or production config should require approval.

Money or permission threshold: anything that moves money or changes access should stop. Payments, transfers, refunds, permission changes, batch operations, and user-data deletion are all high-risk points.

Autonomy level: reads can be fully automatic. Draft writes can also be automatic because they are only drafts. Outbound actions need confirmation. Deletes and payments require approval.

The matrix is not a one-time configuration. Adjust it to the business. If sending an internal Feishu message is low-risk, you might lower it to confirmation. If payment touches real funds, keep it at approval plus two-person review.

Three Real Scenarios for Risk Classification

Case 1: Feishu message draft

When the agent writes a Feishu message draft, it can run automatically (L0). The draft is only saved in the draft box. It has not been sent, is reversible, has no external impact, and does not expose sensitive data. But when the agent calls send_message to send it to a customer, it needs approval (L2). Once sent, the message is irreversible, reaches an external user, and may contain sensitive or misleading content. Confirmation before an MCP tools/call fits this case.

Case 2: Email sending

When the agent generates email content, it can run automatically (L0). The content is just text and can be changed. But when the agent calls the email API to send it to a customer, it needs approval that shows recipient, subject, and attachments (L2). The approval UI must show evidence and a summary, not only a “confirm send” button.

Case 3: CMS form submission

When the agent fills a form, it can run automatically (L0). The form is filled but not submitted; the data is still local. When the agent calls the CMS API to submit the form, the policy should block it and wait for owner approval (L2). The trigger may be an automated guardrail such as “amount exceeds threshold” or a static policy such as “all CMS submissions require approval.”

Case 4: Production database deletion

When the agent queries a production database, it can run automatically (L0). It is a read, has no external impact, is reversible in practice, and does not modify data. When it calls a delete API against production, it needs strong human approval plus backup auditing (L3). Production deletion is irreversible, sensitive, and user-impacting. It should be constrained by policy rules, not left to guardrails alone.

These cases show the important pattern: different steps in the same task have different risk levels. Drafting can be automatic, sending requires approval, and production deletion requires two-person review. Risk classification should happen at the action level.

Approval Point Checklist: Be Specific About Action Types

Once you have the matrix, define approval levels. This checklist covers common action types:

L0 automatic: read operations such as database queries, vector search, and config reads; draft writes such as saving a draft or generating a preview. These actions have no external impact, are reversible, and do not touch sensitive data.

L1 confirmation: outbound messages or data, such as sending email, submitting forms, and calling external APIs; batch reads such as data export or bulk lookup. These actions have external impact but are still relatively controllable, so they need confirmation rather than strict approval.

L2 approval: delete or permission changes such as record deletion, permission updates, and bulk deletes; writes to collaboration systems such as Feishu messages, Slack messages, and CRM records. These are irreversible or have higher external impact, so the run must pause for approval.

L3 strong approval plus two-person review: payments and transfers such as paid orders and refunds; sensitive-data operations such as exporting user privacy data, changing production config, and deleting production databases. These touch money or sensitive data and need two-person review.

This checklist is backed by the OpenAI Agents SDK needs_approval flow and MCP tool-safety guidance. The MCP spec expects tool calls to be visible, rejectable, and confirmed for sensitive actions; that maps to L2 and L3.

You can tune the list for your business:

If sending a Feishu message is low-risk internal notification, lower it to L1 confirmation.
If deleting a record touches user data, keep it at L2 approval.
If payment risk is very high, raise it to L3 with two-person review and a required approval reason.

The list is not static. When business rules change, an action such as “send message” may move out of the approval list.

Approval Flow State Machine: Pause, Save State, Resume

Approval is not a UI pop-up. It is a paused run state. When a tool call needs approval, the agent’s RunState is saved and execution resumes after a decision.

State Transition Diagram

The approval state machine has this flow:

request -> pending -> approved/rejected/timeout -> resume/abort/compensate

The steps are:

request: the tool call creates an approval request. RunState contains the tool name, arguments, and context.
pending: the run waits for a human decision. The state is saved in a checkpoint and linked to a thread_id.
approved: approval passes. The run resumes from the checkpoint and calls the tool.
rejected: approval is denied. The run enters abort or convert-to-draft.
timeout: approval times out. The run escalates or auto-rejects.
resume/abort/compensate: continue execution, stop the task, or compensate completed steps.

Resume Modes

There are three resume modes:

approve: continue execution, call the tool, and proceed to the next step.
reject: stop or convert the action into a draft; do not call the tool.
edit: modify parameters and continue, for example by changing the recipient or content before asking for approval again.

Checkpoint and thread state are the technical background for state persistence. The already published LangGraph checkpoint/thread state article explains the mechanics. A checkpoint stores state at the pause point; thread state lets the run resume at the right execution point.

OpenAI Agents SDK HITL Code Example

This example shows the OpenAI Agents SDK approval flow. The API can change, so check the official docs before publishing production code:

from agents import Agent, Runner, function_tool


@function_tool(needs_approval=True)
def send_email(to: str, subject: str, body: str) -> str:
    return send_email_handler(to=to, subject=subject, body=body)


agent = Agent(
    name="EmailAgent",
    tools=[send_email],
    instructions="Draft the email, then wait for approval before sending.",
)

result = Runner.run_sync(agent, "Write a refund notice email for the customer")

if result.interruptions:
    state = result.to_state()

    for interruption in result.interruptions:
        print(f"Pending tool: {interruption.tool_name}")
        print(f"Arguments: {interruption.arguments}")

        decision = show_approval_ui(interruption)

        if decision == "approve":
            state.approve(interruption)
        elif decision == "reject":
            state.reject(interruption)

    result = Runner.run_sync(agent, state)

Key points:

needs_approval=True marks a tool as requiring approval.
interruptions contains pending tool calls that need a decision.
result.to_state() converts the paused result into serializable RunState.
state.approve() or state.reject() records the decision.
Runner.run_sync(agent, state) resumes from the pause point.

Field names may change after 2026-07, so confirm them in the official docs.

LangGraph interrupt/resume Code Example

This example shows LangGraph interrupt and Command(resume=...):

from langgraph.graph import StateGraph, MessagesState
from langgraph.checkpoint.memory import MemorySaver
from langgraph.types import Command, interrupt


def send_email_node(state: MessagesState):
    approved = interrupt({
        "action": "send_email",
        "summary": state["email_summary"],
    })

    if approved != "approved":
        return {"messages": ["Email sending was rejected; the message was saved as a draft"]}

    email_result = send_email(state["email_params"])
    return {"messages": [email_result]}


graph = StateGraph(MessagesState)
graph.add_node("send_email", send_email_node)
graph.add_edge("draft_email", "send_email")

checkpointer = MemorySaver()
app = graph.compile(checkpointer=checkpointer)

thread_id = "thread_123"
config = {"configurable": {"thread_id": thread_id}}

result = app.invoke(
    {"messages": ["Write a refund notice for the customer"]},
    config=config,
)

# After the graph pauses, the interrupt payload is returned to the caller.
# Show the approval UI and wait for a human decision.
decision = show_approval_ui(result["__interrupt__"])

if decision == "approve":
    app.invoke(Command(resume="approved"), config=config)
elif decision == "reject":
    app.invoke(Command(resume="rejected"), config=config)
elif decision == "edit":
    app.update_state(config, {"email_params": {"to": "new_customer@example.com"}})
    app.invoke(Command(resume="approved"), config=config)

Key points:

interrupt() pauses the graph.
Command(resume=...) resumes execution.
checkpoint + thread_id keeps state consistent.
approve/reject/edit are all supported resume paths.

The API can change, so check the LangGraph docs before using this in production.

Approval Evidence Fields: What to Store and How to Trace It

Approval is not only a decision; it is also a record. This is the minimum audit-log field set:

Field	Description	Example
tool_name	Tool name plus operation type	send_email / delete_record
tool_arguments	Full argument JSON	{“to”: “customer@example.com”, “subject”: “Refund notice”}
invoker_id	Caller identity, user or system	user@example.com / agent_run_abc123
request_time	Approval request time	2026-06-23T09:26:10Z
approver_id	Approver identity	on-call-engineer@example.com
decision_time	Decision time	2026-06-23T09:35:12Z
decision	Decision result	approved / rejected / timeout_auto_reject
evidence	Approval evidence such as screenshot or summary	”Recipient is correct and the content contains no sensitive data”
audit_trail_id	Link to the run logs	run_abc123_step_5_tool_3

These fields come from MCP tool-audit recommendations and OpenAI API approval items such as mcp_approval_request. Approval logs are part of observability; the already published agent monitoring/recovery article covers the broader logging practice.

How should RunState be serialized? In the OpenAI Agents SDK, a paused result can be converted with result.to_state(). LangGraph uses checkpoint + thread_id. Save the serialized state in a database or log system and link it to the audit_trail_id.

Audit logs have three main uses:

Incident tracing: when data leakage is discovered later, you can see who approved which operation and when.
Compliance evidence: enterprise environments need proof that high-risk actions had human approval.
Policy improvement: track which actions are frequently approved, rejected, or timed out, then tune the approval policy.

The field set is not fixed. You can add approval duration, approval channel such as email/Slack/Feishu, or whether two-person review was used. But the minimum set above should exist.

Reject and Timeout: What Happens When Approval Fails

Approval does not always pass. Reject and timeout paths need explicit handling so tasks do not hang.

Three Paths After Rejection

Path 1: continue with fallback. Use a lower-risk action. If sending an email is rejected, save it as a draft and continue. This fits actions that can be downgraded.

Path 2: convert to draft. Turn the action into a draft state. If CMS submission is rejected, save it as a draft and wait for manual editing before submitting again. This fits workflows that need human intervention.

Path 3: abort task. Stop the whole task. If production database deletion is rejected, the run must stop. This fits irreversible high-risk actions.

Choose by action type:

Reversible action: fallback or convert to draft.
Irreversible high-risk action: abort task.
Needs human intervention: escalate.

Two Paths After Timeout

Path 1: escalate to backup approver. If the primary approver has not responded after 30 minutes, send the request to the on-call engineer. This fits decisions that still require a person.

Path 2: auto-reject. If the approval has timed out for one hour, reject it automatically and stop the task. This fits lower-risk but time-sensitive work.

Choose by context:

High-risk action: escalate; do not auto-reject into execution.
Time-sensitive action: auto-reject so the task does not hang forever.
General case: escalate and give the approver more time.

Rolling Back Completed Steps

If the task stops after rejection, completed steps may need rollback. For example, the agent may have created an order before payment approval was rejected, so the order needs to be canceled.

Rollback strategies:

Checkpoint rollback: resume from the checkpoint before approval and discard later steps.
Compensating transaction: call a compensation API, such as canceling an order or undoing an email send where possible.
Manual intervention: notify a person to handle it, such as canceling an order manually.

Rollback is not always possible. An email that has already been sent cannot be unsent. In that case, record the audit log and handle the incident afterward.

Four Safety Boundaries: Policy, Guardrail, Approval, Audit

Approval is not a standalone safety mechanism. Policy, guardrail, approval, and audit need to work together. None replaces the others.

Four-Layer Responsibility Table

Layer	Responsibility	Example
Policy	Static rules that constrain tool scope	”Never delete the production database”; “Payment tools can only call the sandbox environment”
Guardrail	Automated checks that block abnormal input or output	Input validation, output sanitization, sensitive-information filtering, amount-threshold checks
Approval	Human decision for high-risk actions	Show recipient and content before sending email, confirm before deleting a record, approve payment
Audit	After-the-fact traceability	Approval logs, tool-call logs, state-change logs

The layers do not replace one another:

Policy cannot replace guardrails: policy is static and cannot inspect dynamic input and output.
Guardrails cannot replace approval: guardrails are automated checks and cannot handle decisions that require business judgment.
Approval cannot replace audit: approval is the decision, audit is traceability. You need both.
Audit cannot replace the first three layers: audit happens after the fact and cannot prevent the risk.

Combined examples:

Payment: policy limits maximum amount, guardrail validates parameters, approval requires two-person review, audit records the decision.
Email: policy limits allowed recipient domains, guardrail checks sensitive content, approval shows a summary, audit records the send.

MCP Tool Safety Boundary

MCP (Model Context Protocol) tool calls have their own safety boundary. For the 2025-06-18 spec, verify the version again before publishing:

tools/list shows available tools: users can see what the agent is able to call instead of accepting hidden risk.
tools/call confirmation before sensitive actions maps to the approval layer.
inputSchema validation maps to the guardrail layer.
timeout limits prevent tool calls from hanging forever.
audit logging maps to the audit layer.

Key reminder: MCP approval does not replace OAuth scopes or server-side authorization. MCP approval is confirmation before a tool call. OAuth scopes are API access permissions. Server-side authorization is business-logic permission checking. You need all three.

For example, a Feishu MCP server may have passed OAuth and include the send_message scope. That does not mean every message is safe. MCP approval should confirm the content before sending; server-side authorization should check that the recipient is allowed.

Approval cases for outbound messages, collaboration-system writes, and batch table changes are covered in the planned Feishu MCP research article.

Approval UI Design Notes

The approval UI is not just approve/reject buttons. It must show enough information for a person to make a decision.

Design principles:

Show the tool name and arguments so the approver knows what the agent wants to call and with which parameters.
Show the expected impact, such as “send an email to customer@example.com with the subject Refund notice.”
Show reversibility, such as “cannot be undone after sending” or “can be restored after deletion.”
Show data sensitivity, such as “contains user privacy data” or “public data.”
Separate cancel from reject. Cancel means abandoning this approval interaction; reject means denying the tool call and recording it in the audit log.

Core UI elements:

Tool name + operation type
Full arguments, collapsible if needed
Expected impact summary
Reversibility warning
Data-sensitivity label
Approval reason input, optional or required by risk level
Approve / reject / cancel buttons

Important reminder: UI does not replace server-side authorization. The UI is presentation; server-side authorization is the backend permission check. Even after an approver clicks confirm, the backend should still verify the caller, target object, and permission.

If the approval UI says “delete record ID=123” and the approver clicks confirm, the backend still needs to check whether that record belongs to the current user and whether the caller can delete it.

HITL is not an isolated pop-up. It belongs in the tool gateway, logging, and permission system. The planned MCP production architecture article covers that architecture.

OWASP LLM01/LLM06 Risk Mapping

OWASP LLM Top 10 defines security risks for LLM and agent systems. Version labels can change, so verify them before publication. Two risks matter directly for approval design:

Risk ID	Risk description	Approval response
LLM01 Prompt Injection	External input induces unauthorized function calls, data leakage, or external command execution	Require human approval for high-risk actions; do not rely only on prompt rules; show arguments and expected impact in the approval UI
LLM06 Excessive Agency	Excessive functionality, excessive permissions, and excessive autonomy are tool-system risks	Limit tool scope with policy, limit automation level with approval, and scope any “always allow this session” button

LLM01 explains why prompt injection can push a model away from the original instructions and into unauthorized tool calls. Approval should pause before high-risk actions and show arguments plus expected impact so a person can judge the action.

LLM06 explains why excessive autonomy is a tool-system risk. Approval is not a magic fix; it has to work with policy that limits tool scope and approval rules that limit automation. A “always allow this session” button must have narrow scope, otherwise prompt injection can abuse it.

NIST AI RMF Core Mapping

NIST AI RMF Core organizes AI risk management into four stages. A solo developer or small team can use a lightweight version without turning it into enterprise compliance:

Stage	Approval responsibility	Example
Govern	Define roles and risk rules	Define approver roles such as owner or on-call engineer; define L0-L3 approval levels; define reject and timeout policies
Map	Identify high-risk scenarios	Use the risk matrix to find high-risk actions such as delete, payment, and permission change; identify prompt-injection paths
Measure	Measure approval coverage and rejection rate	Track approval coverage for high-risk actions; track rejection and timeout rates; tune the policy
Manage	Incident response and recovery	Use approval logs for incident tracing; roll back completed steps; run compensation transactions

Govern defines approval rules. Map identifies high-risk scenarios. Measure checks whether the control works by tracking coverage, rejection rate, and timeout rate. Manage handles incidents and recovery.

For a personal project or small team, the lightweight version is enough: define approval levels, identify high-risk actions, track rejection rate, and record audit logs. You do not need a full enterprise compliance process, but the minimum set should exist.

Conclusion

Risk classification is the first step in approval design. Not every action needs approval: reads and draft writes can run automatically; deletes and payments must stop for a person. Use five dimensions: external impact, reversibility, data sensitivity, money or permission threshold, and autonomy level.

Approval is not a pop-up. It is serializable, resumable, and auditable system state. RunState is stored in a checkpoint, execution resumes from the pause point, and the audit log records both the decision and the execution chain.

The four safety boundaries do different jobs. Policy limits tool scope. Guardrails automatically check input and output. Approval gives humans a decision point for high-risk actions. Audit provides traceability. You need the combination.

OWASP LLM01 and LLM06 make prompt injection and excessive agency central risks for agent tool systems. Approval must work with policy and guardrails; it should not stand alone.

NIST AI RMF Core gives a risk-management frame. A small team can keep it lightweight: define approval levels, identify high-risk actions, track rejection rate, and record audit logs.

Next reading:

LangGraph checkpoint/thread state article, already published: the technical background for saving approval state.
Agent monitoring/recovery article, already published: approval logs as part of observability.
MCP production architecture article, planned: why HITL belongs in the tool gateway and permission system.
Feishu MCP research article, planned: approval scenarios for outbound messages and collaboration-system writes.