The Care Stack

Most of a healthcare AI agent should be ordinary code

Victor Wilson · · 3 min read

A recent analysis by Tomasz Tunguz looked at 14 AI agent workflows running in production. About 65% of the steps in those systems were plain deterministic code. They weren’t model calls. They were ordinary logic, API calls and data transformations.

If you’ve sat through a pitch for an AI platform that would run clinical data workflows on its own from start to finish, that number may be reassuring. It suggests full autonomy was oversold. It should also give you pause, because many teams build their agent systems the other way around. They start with the model and add guardrails later.

In healthcare data, getting that balance wrong is more than a cost problem. It can become a patient safety problem.

What the code should do

In a well-built agent workflow, the code handles the work that has one right answer.

Schema validation and data contracts. When you take in HL7v2 ADT feeds or FHIR bundles, you want strict schema checks, not a model’s opinion about whether a field looks valid.

Routing. Deciding which downstream system gets a payload is a lookup table.

Retries, rate limits and error handling. These are engineering problems with known solutions. You don’t want a model improvising your retry logic at 2 a.m. while an EHR feed is failing.

PHI handling and access control. HIPAA compliance isn’t a probability. Data is either de-identified correctly or it isn’t.

The rest is where a model earns its place. Reading unstructured clinical notes. Classifying ambiguous diagnosis codes. Summarizing a long patient history for care coordination. Pulling details out of referrals that still arrive by fax. In these tasks, understanding language beats writing rules.

Deciding step by step

The teams shipping reliable agents don’t build one long chain of model calls. They build a pipeline where each step is deliberately assigned to code or to a model. Three questions decide it:

  1. Is the set of possible outputs fixed? If so, use code. Mapping ICD-10 codes to risk categories doesn’t need a language model.
  2. Can a program check the answer? If so, use code for the check, even when a model produced the answer.
  3. Does the task truly need language understanding? Only then use a model. Even then, limit what it can return with structured output, such as a JSON schema, a function call or a fixed list of allowed values.

In practice, that might mean dbt for the deterministic transformations, an orchestrator like Dagster or Prefect running the pipeline, and model calls kept to specific steps with typed inputs and outputs. Each model step behaves like a function with a contract. It doesn’t wander through your clinical warehouse on its own.

Cost and reliability

Every unneeded model call spends money on work that a SQL query or a short Python function could do almost instantly, for free. Across the millions of claims, lab results and documents a health system handles each month, the gap between a well-split design and one that sends everything through a model gets large.

Cost isn’t the main reason, though. Speed and reliability are. Code runs in milliseconds and fails in predictable ways. Model calls add variable response times, depend on rate limits, and can change behavior when the provider updates the model without telling you. The fewer model calls you have, the less can go wrong at once.

The words shape the design

What you call the system affects how you build it. Call it an “AI agent,” and teams tend to put the model at the center and treat everything else as support. Call it an “AI-assisted pipeline,” and teams start with solid engineering and add a model only where it’s needed.

In healthcare, the second approach holds up better in a compliance review, mostly because you can test it. You can unit test a code step. You can wrap it in data quality checks with dbt tests or Great Expectations. Writing a reliable test for an autonomous agent’s reasoning about whether to approve a prior authorization is much harder.

The teams that do best over the next few years won’t be the ones with the most elaborate model integrations. They’ll be the ones disciplined enough to use a model only where nothing else works, with sound engineering everywhere else. Tunguz’s data points to roughly two-thirds code. In healthcare, I’d push the balance further toward code.

Before your next sprint planning, list every step in your agent workflows and mark each one as code or model. Ask the three questions above of every model step. Any step that could be a lookup, a rule or a schema check should become one.

Discussion

    Join the discussion