# Agent-Powered Pipeline Development: Designed for Humans, Built for Production Prophecy’s latest release democratizes pipeline development with AI agents, keeps humans in the loop Ravi Uppala 17 Jun 2025 AI agents are having a moment. From writing code to booking travel, they promise to take on the grunt work so we can move faster and focus on what matters. But when it comes to building production-ready data pipelines, the story isn't so simple. ### AI agents are transforming work. But they aren’t one-size-fits-all The promise of agents is real. They boost productivity, act like intelligent helper functions, and take on repetitive, low-level tasks so humans don’t have to. They’re the factory robots of digital work. Relentless. Precise. Fast. In some industries, they're already making a huge difference: - In customer support, agents can instantly draft email replies, suggest responses, or triage tickets based on priority. - In software development, coding agents can scaffold entire applications from natural language prompts. But the success of agents always depends on the domain and the design of human-AI collaboration. ### Production pipelines are high-stakes. You can’t just hit "Go." We don’t believe agents should generate an entire pipeline and automatically deploy it to production. Why? Because data prep isn’t a one-shot task. The stakes are high. And the risks are very real: - An incorrect join might silently drop half your rows - An assumption about column types could throw off your entire model - Missed edge cases can lead to wrong metrics. Or worse, wrong business decisions. Data prep involves judgment. Nuance. Context. And too many interdependent steps to trust an opaque black box. ### The better path: Step-by-step assistance, with humans in the loop Think of building pipelines like assembling a car. Robots play a huge role. But there are checkpoints, quality gates, and human engineers reviewing critical steps. That’s how we’re building agents at Prophecy: To simplify, accelerate, and assist. But always with a human at the center, ready to inspect and refine. This approach empowers more users, scales trust, and helps organizations balance even faster self-service with governance. ### From first principles: How we designed our agents To design AI agents for data preparation, we started from the ground up: What does it actually take to create a production-ready pipeline? We broke the problem down into three universal steps: 1. **Find** the right data 2. **Understand** what that data is and how to use it 3. **Transform** it to make it AI- and analytics-ready This is true whether you're a data engineer or a business analyst. Whether you're on the finance team or customer service. For BI or AI use cases. With desktop files or enterprise databases. These steps never change. What does change is how efficiently (and confidently) you can move through them. That’s where agents come in. ### See it. Adjust it. Talk to it. The power of visual + natural language Prophecy’s unique approach keeps every step transparent, visible, and editable. As the agent works, you see its output rendered visually on the canvas. You can click into each transformation, inspect its logic, and view the data at that stage. And if something needs changing? Just ask. You can refine any step using natural language - not just at the start, but throughout the process. Want to rename a column? Filter rows differently? Update a join condition? _Say it in plain English, and the agent will respond. No black boxes._ This makes data prep accessible to more users, while keeping quality and governance intact. ### How Prophecy agents help you build pipelines: Step-by-step Let’s walk through how Prophecy agents support each step of the pipeline journey: #### Find the right data Locating the right dataset is often the most time-consuming and overlooked step in building data pipelines. Users struggle to: - Search across diverse sources: Siloed datasets and inconsistent metadata make it hard to know what’s available. - Semantic search gaps: Traditional keyword search is limited. Users need to find data based on meaning, not just names or descriptions. Prophecy provides search capabilities across all accessible datasets in your fabric - leveraging semantic matching and enabling you to search by title, description, or even column names. This drastically reduces time spent hunting for the right data, so your team can focus on higher value activities. Guardrails are still in place—the agents are limited by the permissions of the users. With our Databricks Unity Catalog integration, you can extend users access to the power of agents, without creating security or governance risks. #### Understand the data Even after users have found candidate datasets, the next hurdle is exploring and understanding what each offers: - Limited dataset insight: Users rarely have the full context—schema details, sample records, or rich descriptions—needed to assess whether a dataset is fit for purpose. - Comparing alternatives: Choosing between datasets is difficult without side-by-side comparisons of details completeness and relevance. With Prophecy, users can preview data, understand schema details, read descriptions, and even ask questions about any dataset to quickly evaluate its suitability. Comparing datasets helps users select the most relevant data sources for their purpose. With this, exploration to understand the data becomes a natural & transparent process—empowering all users to quickly make informed selections. #### Transform it for analytics or AI Once the right data is in hand, users often hit roadblocks in shaping it to fit their unique needs: - Transformation logic: Users may know what they want but struggle to translate business logic into data transformation steps. - Feedback and control: It’s hard to see the impact of each transformation or easily adjust logic as requirements evolve. Simply describe your transformation goals in natural language and Prophecy translates them into visual, editable pipeline steps. The agent is designed for maximum transparency. See real-time visual feedback and easily tweak steps—either by updating instructions or making direct changes in the pipeline canvas. You’re always in control, with guardrails on what questions the user can ask the agent and the flexibility to refine and iterate until the output is just right. Preview/restore: Exploring data and refining transformations is an iterative process. Users often try out different prompts and actions to achieve the desired results, and flexibility is essential. That’s why our solution includes Preview and Restore options: - Preview: Instantly see the effects of your changes—whether you’re experimenting with new prompts or adjusting transformation logic. This gives you confidence in every step before committing to changes. In the same vein, the ability to quickly preview an older state allows users to understand & then decide if they want to go to that state. - Restore: If you want to revisit or revert to a previous version of your work, simply restore your session to any earlier state in your chat history. This makes it easy to backtrack, try alternative approaches, or restart from a known good configuration. Data preparation is often a means to the end of performing analytics and tracking the analysis on dashboards. With Prophecy, users can create rich visualizations for their final output & publish to business apps, thus completing the task. ### The future is assistive When it comes to building production-ready pipelines, agents are most powerful not as replacements for people, but as tools that empower them to work faster and with greater confidence. That’s why we’ve designed agents that: - Work in real time, transparently showing their steps on a visual canvas - Invite feedback and updates through natural language - Operate under your governance, your standards, and your approval process This is how we deliver on the promise of governed, self-service data preparation. Faster pipelines. Greater accuracy. And a data platform that works the way you think. ### Ready to see Prophecy in action? [Request a demo](https://www.prophecy.ai/request-a-demo) and we’ll walk you through how Prophecy’s AI-powered visual data pipelines and high-quality open source code empowers everyone to speed data transformation.