What is actually inside an AI agent

A practical breakdown of what actually sits inside an AI agent in 2026: the decision loop, tools, memory, checkpoints, policies, approvals, tracing, evals, and the line between a workflow and a full agent runtime.

20 Mar 2026· 13 min read· Technology

Best forEngineers building AI featuresFounders trying to separate hype from real architectureProduct teams evaluating agent frameworksDevelopers who want a practical model instead of marketing language

Editorial cover for a deep breakdown of the internal structure of an AI agent

read this first

Read this before you call everything an agent

The most common mistake is simple: people see a model with tool calling and immediately call it an agent. That is enough for a demo. It is not enough for a system that must keep state, recover from a bad step, ask for approval, and leave a clear trail after a failure.

The simplest useful definition is this: the model does not just answer once, it repeatedly chooses the next action based on what it just observed. That is the core loop. [1][2][3]

In production, the definition becomes stricter: a real agent runtime has identity, boundaries, a list of allowed tools, a memory policy, observability, evaluations, and often approval or rollback paths. [1][6][7][8]

That is why workflow and agent should not be collapsed into the same thing. In a workflow, the route is mostly defined in code. In an agent, a substantial part of the route is discovered at runtime. [1][2][3]

Memory is not just chat history. In production you usually need at least one of these: short-lived working state, checkpoints, thread or session state, and sometimes a separate store for durable facts. [5][6][9]

If you remove tracing, evals, and review surfaces, the system may still look autonomous, but it stops being transparent enough to inspect. That is where many teams confuse a slick demo with a dependable agent. [7][8]

Where the confusion around the word agent actually starts

The confusion begins because different communities use the word agent for different layers of the system. For some, an agent is any loop that can call tools. For others, an agent is already a governed runtime with policies, approvals, memory, and observability. Both are describing something real, but they are describing different scopes.

The DOU article and discussion show this split well. The article itself explains the base loop clearly, but the comments bring in an important objection: a serious agent is not just 'a brain in a loop', it is a managed execution unit with explicit operational boundaries. [1] That is exactly why simple demos and production systems feel like different categories of software.

Anthropic has recently stayed close to a simple lower bound: an agent is an LLM that autonomously uses tools in a loop. [2] That is a useful baseline. It separates an agent from a one-shot completion. But it is still only the lower bound, not the whole system.

In practical terms, you can frame it like this: the agent starts at the loop, but the useful product starts at the runtime around the loop. Once the system can search, read, write, execute code, and call APIs repeatedly, the main problems shift away from generation quality and into control, state, and failure handling.

The minimum engine that exists inside almost every agent

At the center of the system there is a repeated loop. A request or goal arrives. The model reads the current context. It chooses the next action. That action may be a normal answer, a structured output, or a tool call. Then the environment returns a new observation, and the loop repeats until a stop condition is met.

This is the part most people notice first. It is also the easiest part to explain. In Anthropic's Agent SDK material, the common feedback loop is described as 'gather context -> take action -> verify work -> repeat'. [3] For many production agents, that is a very accurate simplified picture, even though the runtime around the loop becomes much more elaborate.

A minimal implementation can genuinely be short. The DOU piece is right about that. A basic loop with tools, prompt state, and a max-step guard fits into a fairly small amount of code. [1] But short code does not mean a simple system. It usually means that complexity has been moved into separate services, tool contracts, memory handling, and operational safeguards.

Here is a deliberately stripped-down version of the mechanism. It is short on purpose so the boundaries stay obvious:

PYTHON

messages = [system_prompt, user_request]
steps = 0

while steps < MAX_STEPS:
    response = model.run(messages=messages, tools=TOOLS)

    if response.final_output:
        return response.final_output

    if response.tool_calls:
        for call in response.tool_calls:
            result = run_tool(call.name, call.arguments)
            messages.append({
                "role": "tool",
                "tool_call_id": call.id,
                "content": serialize(result),
            })
    else:
        messages.append(response.message)

    steps += 1

raise RuntimeError("Agent exceeded max steps")

If you only read this loop, you can see where an agent begins. But it is still not enough to explain why one agent stays stable, another becomes expensive, and a third quietly accumulates risk until it breaks under real usage.

The smallest agent model looks like this: there is a goal, the model chooses an action, calls a tool, gets a new observation, and either loops again or stops. [1][2][3]

Takeaway

The loop explains agent behavior. By itself, it does not explain quality, cost, or safety.

What the demo hides: the runtime stack around the loop

Once you move beyond toy agents, the key question is no longer whether the model can call a tool. The real question becomes: under which rules is it allowed to act, what state does it carry, what gets stored, what requires approval, and how will you inspect a bad run after it fails.

OpenAI's release on building agents is useful here because it is not framed as model calls alone. The announcement highlights built-in tools, orchestration, and integrated observability for tracing workflow execution. [4] That is an important signal: even platform vendors now frame agent building as a runtime problem, not just a prompt problem.

The same pattern shows up in framework docs. LangGraph's persistence docs talk about checkpoints, threads, replay, time travel, and fault tolerance. [6] Microsoft Agent Framework puts observability front and center and routes traces, logs, and metrics through OpenTelemetry. [7] These are not side features. They are the pieces that let teams debug, recover, review, and control long-running behavior.

A useful working model is this: goal and permissions at the top, planning policy below them, then tool access, then state and memory, and a control plane around all of it with tracing, evals, approvals, and rollback. The model matters a lot, but it is still only one layer inside the full structure.

A production agent usually looks like a layered runtime: role and permissions, planning policy, tool access, state and memory, with tracing, evals, approvals, and governance on top. [1][4][5][6][7][8]

Takeaway

The more freedom an agent gets, the more runtime scaffolding has to exist around the model.

Workflow versus agent runtime: the boundary is control

This distinction matters because teams often pull agent infrastructure into problems that are really just workflows. The result is a system that costs more, is harder to test, and still does not gain enough extra capability to justify the trade-off.

Workflow

The route is mostly defined in code. The model helps within already known steps, but orchestration remains explicit. This is usually the right choice when the sequence is known upfront, reproducibility matters, and failure modes should stay narrow. [1][2][4]

Agent runtime

The model determines a much larger share of the route while the system is already running. That is useful when the next step cannot be described in code in advance, but it demands a much stronger approach to state, tracing, approvals, and evaluations. [3][4][6][7][8]

Bad middle ground

The system looks autonomous, but has no clear permissions, no stop rules, no replayable history, and no reliable review surface. That is where teams end up with expensive demos that are hard to trust and even harder to debug.

Both workflows and agents can use tools. The practical difference is who decides the route at runtime and what control surfaces exist around that decision. [1][2][4]

Rule of thumb

If the path is already known, start with a workflow. Reach for an agent only when the route itself has to be discovered during execution.

Memory, state, and context: three things people keep collapsing into one

Comparison point	What it usually means	Why it matters
Current context window	The active prompt, recent messages, tool results, and working state of the current run.	This is the first place that breaks when an agent loops too long or accumulates too much noisy tool output. [1][5][9]
Checkpoints or thread state	Saved execution snapshots from which a run can be resumed, replayed, or inspected later.	This is what makes human approval, interruption, debugging, and recovery after failure practical. [6]
Long-term memory store	Longer-lived facts, user preferences, or summaries stored outside the current context window.	Useful when the agent needs continuity across sessions, but risky when the memory policy is vague or the data becomes stale. [5][6]

What makes an agent production-grade: observability, approvals, and evals

This layer turns an autonomous demo into an engineering system. Without it, production use becomes hard very quickly.

No observability. If you cannot inspect traces, tool call arguments, state transitions, and failure points, you cannot reliably understand behavior after release. [4][7]

No approval boundary. If the agent can trigger side effects without human review where review is required, the system becomes fragile fast.

No eval discipline. Anthropic is direct on this point: capability evals, regression suites, transcript review, and continuous checks stop being optional once the system matters. [8]

No stop rules or budget policy. Agents that can run freely without step limits or cost control can become expensive and unstable without making it obvious. [1][3]

No rollback or recovery path. If a step fails halfway through and the system cannot resume or replay from a known state, the operator ends up effectively blind. [6]

Summary

The more autonomy you grant, the more control plane you need around it. Otherwise the system is only agentic in the least useful sense: it acts on its own, but nobody can govern it well.

A practical checklist for teams building or buying AI agents

If you want one section you can drop into an architecture review, use this one.

Ask where the path is actually decided.

If the route is mostly fixed in code, call it a workflow. If the system discovers the route while it is already running, you are in agent territory and should budget for additional control layers right away.

Separate the model from the runtime.

A better model helps, but it does not replace memory policy, tool contracts, checkpoints, or observability. Treat those as first-class components, not as secondary details. [4][6][7]

Define memory policy early.

Be explicit about what lives only in the current context, what goes into checkpoints, and what can survive across sessions. If this stays vague, the agent will eventually carry the wrong thing forward. [5][6][9]

Install tracing before complexity grows.

If tracing arrives only after the agent becomes complex, the team starts reconstructing failures from logs and screenshots instead of reading a clean execution record. [4][7]

Treat evaluations like unit tests for behavior.

Agent quality drifts unless you keep a regression suite, read transcripts, and continuously run key scenarios. This becomes obvious as soon as models, prompts, or tools change. [8]

Bottom line

There is no magic inside an AI agent. There is a loop wrapped in state, rules, tools, and a control plane. Teams get into trouble when they ship the loop and forget everything that should sit around it.

FAQ

Can an LLM with tool calling already count as an agent?

At the minimum level, often yes. But in engineering practice that only describes the lower bound. A serious agent usually still needs state policy, tracing, approvals, recovery paths, and evaluation logic.

What do people most often miss inside an AI agent?

They usually focus on the model and the tools, but miss the runtime around them. Memory policy, checkpoints, observability, and approvals often matter more than one extra prompt trick.

When should a team choose a workflow instead of an agent?

A workflow is the better choice when the route is already known and should stay explicit in code. An agent is useful when the route itself must be discovered during execution and the added control-plane complexity is justified.

Why have evals become such a large part of agent engineering?

Because agents act across many steps, tools, and intermediate states. Without evals, teams often notice regressions only after they hit users or expensive production environments.

Sources

Primary sources used for this article. Verified on March 20, 2026.

Reviewed: 20 Mar 2026Applies to: LLM assistants with tool usageApplies to: Coding, research, and operations agentsApplies to: Single-agent and multi-agent systemsTested with: OpenAI Agents SDK docsTested with: OpenAI new tools for building agentsTested with: Anthropic agent engineering postsTested with: LangGraph persistence docsTested with: Microsoft Agent Framework observability docs