The Care Stack

The data engineer's job is changing

Victor Wilson · · 4 min read

For most of my career, a data engineer’s week looked about the same. Write SQL transformations. Fix the pipeline that broke overnight. Build a connector to a new source. Write the documentation later, if there was time.

That work still matters. But AI agents are getting good at much of it, and quickly.

I’ve spent about twenty years building data platforms for healthcare organizations, from on-premise warehouses to cloud platforms. I’ve watched the role change several times: ETL developers, then Hadoop specialists, then dbt modelers. This time feels different. The daily work itself is changing, and the tools are only part of it.

What agents can do today

Here’s where things stand in early 2026. Agents can:

  • Write SQL that runs and also fits your schema, your business rules and your naming conventions.
  • Design, deploy and monitor pipelines with little human input.
  • Watch data quality, find the likely cause of a problem, and suggest a fix.
  • Notice when an upstream schema changes and carry the change through downstream models.
  • Write documentation that matches what the code actually does, and keep it current.

A year ago, most of this was demo material. Now teams are running it in production.

Why healthcare gains the most

Healthcare data is complex and the stakes are high. A wrong HL7 mapping or a bad FHIR transformation can do more than break a dashboard. It can feed a clinical decision.

You might think that makes healthcare a poor fit for AI-assisted engineering. I think the opposite is true. The complexity that makes this data hard for people to handle consistently is what makes it a good fit for agents.

EHR integration work repeats, with small differences each time. An agent can learn your house conventions and apply them the same way every time.

Compliance requirements, from HIPAA to state law to payer rules, are rules. Agents apply rules more consistently than tired people do.

Clinical data models like OMOP need deep domain knowledge. That knowledge can be written down once and reused on every project.

From builder to architect

The data engineer role will survive. What goes away are its most repetitive parts.

The engineers who do well over the next few years won’t be the ones who write the most SQL. They’ll be the ones who:

  • Design the system: the architecture, the data contracts and the patterns agents work within.
  • Keep the context current: the data dictionary, business rules and quality expectations that make agents useful.
  • Review agent output, catch edge cases, and handle the work that needs real judgment.
  • Take on new problems: a first integration with a new system, a hard legacy migration, a design choice you’ll live with for ten years.

The best data engineers have always been architects who also code. AI makes that hard to miss.

What an AI-ready platform looks like

A platform built to work with agents differs from what most health systems are building today in four ways.

First, you describe what you want rather than how to build it. “I need a daily patient census table that joins Epic admissions with bed management data and applies our standard de-identification rules.” An agent works out the code, the schedule and the monitoring.

Second, pipelines repair themselves when they can. When a source schema changes or quality drops, an agent finds it, works out why, and either fixes it or sends a person a specific recommendation. Fewer 3 a.m. pages about a missing column.

Third, the platform keeps tuning itself. Agents look at query patterns, storage and compute costs, and make the change, such as resizing a Snowflake warehouse that’s too large, instead of only flagging it.

Fourth, and most important, agents know your data. When an agent knows that “MRN” in the old system is “patient_id” in your FHIR data, that department 4200 is the cardiac ICU, and that census always jumps on Monday mornings, it starts to be truly useful.

Where it still falls short

We aren’t there yet. A few things still need work.

Health systems need to know why an agent made each transformation choice, every time, and that audit trail isn’t always there. Regulators such as CMS and OCR haven’t caught up with AI-managed pipelines. Agents handle the common cases well and sometimes get the rare, important ones wrong with confidence. And most health system IT teams are still recovering from their last migration.

Still, the direction is clear. If you build a platform today with clean metadata, written business rules and modular design, you can adopt these tools as they mature. If you don’t, you’ll likely face another modernization in five years.

What I’d do now

If you lead a healthcare data team, here’s where I’d start:

  1. Fix your metadata. Get your data catalog, business glossary and lineage in order. That is the context agents work from, and it pays off even without agents.
  2. Hire for design. Your next data engineer should be someone who can design systems as well as build them. The building part is getting cheaper.
  3. Start with one small agent task. Documentation, quality checks or test writing are good choices. Let the team get used to working with an agent on low-risk work.
  4. Build in small pieces. Large pipelines are hard for agents and people to reason about. Small, well-documented models are what agents work with best.
  5. Stay calm. This shift will take years. You have time to plan it, but not time to ignore it.

This week, pick the ten tables your team gets the most questions about. Check whether each has a plain-language definition, a named owner and written business rules. Whatever is missing is your first project, and it’s the same work that will make an agent useful later.

Discussion

    Join the discussion