What I learned building a data platform with AI agents
Last week I built a working Snowflake data platform in two days, with AI agents doing most of the coding. It had ingestion, a full dbt project, a metrics layer, plain-English queries, dashboards and automated deployment. Twenty-eight pull requests, each one reviewed before it merged.
The speed is what people notice, but it isn’t the lesson. The lesson is how the work was split between me and the agents. That split made it fast, and it’s the part worth copying.
What I built
The project used World Bank health indicators: population health, health spending and health infrastructure, across most countries and several decades. Real public data, with real mess in it.
The stack:
- Ingestion: dlt pipelines pulling from the World Bank API.
- Transformation: a dbt project with staging models for cleaning and typing, intermediate models for joins and business logic, and marts for analysis.
- Metrics: five dbt models that define the key measures once: health spending per person, life expectancy, physician density, immunization coverage and maternal mortality.
- Snowflake: separate dev, test and prod schemas, external stages, and a Cortex semantic model so people can ask questions in plain English.
- Dashboards: Streamlit apps running inside Snowflake, with no outside hosting.
- Deployment: GitHub Actions running dbt build and tests on every merge to main.
That’s a normal platform, the kind a small team would usually plan a few weeks for.
How the work was split
It wasn’t “type a prompt, get code.” It felt more like being a senior architect working with a very fast engineer who never gets tired and sometimes needs to hear “no, not like that.”
I made the design calls. I chose dlt over Airbyte. I chose the three-schema setup. I defined what the metrics layer had to do. Those choices come from knowing what holds up in production and what turns into maintenance work a year later. The agents didn’t make them. They carried them out, quickly.
When I said “staging, intermediate, marts, with a separate metrics layer,” an agent set up the whole dbt project in minutes: folders, first models, project config, and a custom macro to route models to the right schemas. I reviewed it, found two naming problems, and they were fixed right away.
The more interesting moments were problems that needed both of us.
Streamlit deployment. The plan was to deploy with the Snowflake CLI. It failed with authentication errors, unclear documentation and version conflicts. I decided to skip the CLI and deploy with plain SQL, using CREATE STREAMLIT with staged files. An agent rewrote the deployment in about twenty minutes. A team could easily have spent half a day fighting the CLI before landing in the same place.
Number types. World Bank data mixes whole numbers, like population counts, with values that need four decimal places, like mortality ratios. I knew mixed types would cause trouble for Cortex queries later. We cast everything to DECIMAL(18,4) in staging. Predictable, and no surprises in the semantic layer.
Schema names. By default, dbt adds your custom schema name to the target schema, so you get names like PROD_STAGING. I wanted clean names. An agent overrode the generate_schema_name macro so models land exactly in STAGING, INTERMEDIATE or MARTS.
The pattern held all the way through. I caught the things that would hurt in month three of production. The agents handled the volume of work that would wear out a team.
Where the agents fell short
The limits matter as much as the speed.
They don’t know what matters. The World Bank list has dozens of indicators. I picked the five a health executive would care about. Left alone, an agent would have built all of them and called it thorough. To me, most of it was noise.
They make mistakes. One staging model quietly dropped records with no country code. Those were valid rows, global totals rather than errors. I caught it in review. Without review, the platform would have shipped with subtly wrong numbers.
They can’t do discovery. Working out what a VP of analytics actually needs, as opposed to what they ask for, takes conversation and experience.
They can’t make compliance calls. An agent can build a HIPAA-compliant design. Deciding what counts as enough protection for a specific organization takes legal advice and context the agent doesn’t have.
The setup works because it respects those limits. I didn’t ask the agents to make design calls, and they didn’t pretend they could.
What this means for healthcare teams
Health systems need modern data platforms, and most can’t staff for them. Good data engineers are hard to hire, margins are thin, and every new hire or vendor brings compliance overhead.
Working this way helps. A platform that used to take a quarter can be running much sooner, which changes the math on moving off a legacy warehouse. The person with healthcare knowledge, who knows a claims join is wrong if you ignore adjustment reason codes, is still there. Their time just goes further. After the build, agents can take on the steady work that usually stalls: new sources, tests, documentation and pipeline monitoring.
How I’d set it up
If you run a small data team, here’s what I’d copy:
- Spend your own hours on design and review. Let agents write the boilerplate SQL, the project setup and the CI configuration.
- Send every change through a pull request, and have a person review each one before it merges. Those 28 pull requests were 28 chances to catch a mistake before production.
- Run agents in parallel. One builds dbt models while another builds dashboards and a third sets up deployment. You review as work finishes. Most of the time savings come from this, more than from any single agent being fast.
- Write down decisions as well as code. Agents write code well but don’t remember why a choice was made. Keep a short decision log: why SQL-only deployment, why fixed decimal types, why that schema setup.
Pick one small, well-understood item from your backlog, such as a new data source or a set of missing tests, and try this setup on it. Review every pull request yourself. After a week, you’ll know where agents save your team time and where they still need a close eye.
Discussion