AWS published a new blueprint called the Agentic Data Operations Platform (ADOP) on the AWS Machine Learning Blog. It uses AI agents to write data pipelines. This helps teams skip weeks of slow, manual setup.
How ADOP builds your data pipelines
ADOP helps data teams move datasets across Bronze, Silver, and Gold data stages. It handles slow tasks like onboarding sources, transforming data, and running quality checks. It also updates semantic layers and coordinates workflows.
Instead of writing all code by hand, you use a main Data Onboarding Agent. It runs on Claude Code and Amazon Bedrock. This agent creates smaller helper agents for specific jobs:
- Metadata and ontology: Creating schema tags and finding data structures.
- Transformations and quality: Writing ETL code and setting up data-quality rules.
- Workflow orchestration: Building pipeline DAGs for Apache Airflow or AWS Step Functions.
When the agents finish, human engineers review the code. Then, a CI/CD pipeline deploys standard PySpark, SQL, DAGs, IAM policies, and Cedar security rules to staging and production.
Most importantly, the live production pipeline runs standard code without calling an AI model. You get AI speed during setup, but predictable stability in production.
Guardrails and security controls
AWS adds several safety layers to protect company data:
- No raw data exposure: Agents see only schema metadata, sample counts, and column statistics.
- Safe secrets: System credentials deploy at launch time. They never enter the agent prompt.
- Strict guardrails: Bedrock Guardrails filter content, check grounding, and block sensitive data.
- Isolated networks: Work happens inside protected networks with mandatory human reviews.
What you shouldn’t trust yet
AWS claims this blueprint cuts source onboarding time from weeks to hours. That number is a vendor design claim. Nobody outside AWS has verified it yet. AWS has not published total costs, output accuracy rates, or formal compliance certifications. Even with metadata-only profiling, an agent can still write bad code or miss subtle regulatory rules.
Why this design matters
AI models can be unpredictable in production. ADOP avoids that risk by using AI only at build time.
That is huge for regulated industries.
You can inspect, test, and version-control standard SQL or PySpark code before it touches real customer records. However, this workflow takes upfront work. You must maintain clear prompts, strict policies, and thorough reviews. If your initial checks are weak, the agent could repeat the same bug across many datasets.
How to test it yourself
Start small. Pick two or three engineer champions and choose one non-critical data source.
Build that pipeline twice. First build it by hand, and then build it with ADOP. Compare total engineering time, token and cloud costs, rework needs, and data quality failures. Check the generated code carefully for network leaks or exposed secrets.
If your pilot proves reliable, you can safely expand ADOP across your data stack.