KEVINSHOULTZ
AboutWorkAgentsSecurityBlogContact

SYS.AGENTS // ORCHESTRATION

Systems of Agents, Not Prompts

Models are probabilistic. The systems around them don't have to be. Most people use agents one prompt at a time; the leverage is in architecting them — defined as data, composed like software, orchestrated as workflows, and held to gates they cannot skip. Deterministic systems, built from non-deterministic parts.

COMPOSED // ORCHESTRATED // VERIFIED
The Three Convictions
[conviction]

Agents Are Composed, Not Prompted

Definitions are data. Behavior is architecture.

Every agent — its instructions, its tools, its team — lives as a record in the database and is built through a single factory, not hand-assembled in code. That turns a prompt into an artifact: something you can query, review, test, and reuse. More than a hundred agents run this way. Each one is a vocabulary word; the value is in the composition.

[conviction]

The Workflow Is the Product

Determinism lives in the orchestration, not the model.

A chat window is a demo. Real work is a pipeline: explicit steps, bounded handoffs, defined failure paths, and a known end state. When a state machine owns the sequence, the model can be creative inside a step while the system around it stays predictable, observable, and repeatable. The model is probabilistic; the workflow doesn't have to be.

[conviction]

Verification Is the Contract

Done means a gate said so — not the agent.

Agents are confident narrators of their own success. So the definition of done is external: one validation target that runs the same checks CI runs, hooks the agent cannot route around, and a rule that a check which could not run is a failure, not a pass. Trust the gate, not the transcript.

Anatomy of an Agent System

Every layer of an agent system has a predecessor you already know. The discipline that made those layers reliable still applies — it just has to be applied on purpose, because the agent won't apply it for you.

01
was: Config files

Definitions

The Challenge

Prompts scattered through code drift silently. Nobody can say which version an agent actually ran, and nothing can be tested on its own.

The Practice

Agents, instructions, tools, and teams stored as data and loaded through a registry. One factory builds every agent, so construction is uniform and every definition is reviewable.

02
was: Cron jobs & batch pipelines

Orchestration

The Challenge

Agents chained ad hoc produce work that succeeds on Tuesday and wanders on Wednesday, with no record of which path it took.

The Practice

Explicit state machines. Each step bounded, each handoff defined, each run recorded. The model decides within a step; the workflow decides what happens next.

03
was: APIs & service accounts

Tools & Context

The Challenge

An agent is only as good — and only as dangerous — as the tools and context it is handed. Too little and it guesses. Too much and it wanders.

The Practice

Narrow, purpose-built tools with scoped credentials, and context curated per task: documentation written for agents to read, at every level of the codebase, not just for humans.

04
was: Code review

Verification

The Challenge

Agents produce plausible output faster than anyone can read it, and they report success whether or not it happened.

The Practice

Deterministic gates — lint, types, tests, and build behind one validation target, enforced at commit and at push — with exit codes that tell a failed check apart from one that never ran.

05
was: Runbooks & tribal knowledge

Operations

The Challenge

Every repository that invents its own conventions teaches agents a local dialect, and an agent moving between repositories carries the wrong one.

The Practice

One execution surface and one hook shape as the standard, decisions recorded alongside the command that enforces them, and enforcement status measured by running the check rather than claimed in a document.

In Practice
[in practice]

One Factory, 100+ Agents

Defined as data. Built one way.

Agents, their instructions, tools, and teams live in PostgreSQL and load through a registry; every one is constructed by the same factory. A new agent is a new record — reviewable and composable with the rest — not a new piece of glue code. Multiple model providers sit behind the same interface, so the architecture outlives any one vendor.

[in practice]

Agent Workers on AWS

Step Functions → ECS Fargate → S3.

Long-running agent work runs as ephemeral Fargate workers orchestrated by Step Functions, each holding least-privilege credentials, with results sealed to S3. The state machine owns the sequence and the failure paths; the agent owns only the step it was given.

[in practice]

Gates Agents Cannot Skip

The same checks locally that CI runs.

This site's own repository runs this way: one validation target that runs every step CI runs, a pre-commit hook that checks what was staged, and a pre-push hook that runs the full gate and blocks on anything short of a pass — including a check that could not run. The hooks themselves are proven by a probe that breaks the tree on purpose and requires them to refuse.

Why This Intersection Is Rare

Plenty of people can prompt an agent, and plenty can build a pipeline. Fewer have spent twenty-five years building systems where the answer had to be right — defense logistics at USTRANSCOM and Scott AFB, aviation, geospatial platforms where a wrong result carries a mission cost — and then turned that discipline on agents: more than a hundred of them, defined as data, orchestrated as workflows, and held to gates they cannot talk their way past.

The Bottom Line

Agents make the work faster. Architecture makes it right. The job now is building deterministic systems out of probabilistic parts — so the output on Wednesday matches the output on Tuesday, and there is a gate that proves it.

The other half: securing what agents can touchSee what the code proves
End of line
ContactPrivacyLinkedInGitHubRSS

© 2026 Kevin Shoultz. Built and shipped from the Midwest.