AI agents in production: what we've learned
Building a multiagent system that works in a demo is easy. Building one that runs reliably in production is a different sport. Here are the hard-won lessons.
Zelvante Team
AI Engineering
Everyone is shipping AI agent demos. Very few are running them in production. The gap between 'works in a demo' and 'works in production' is where most agentic systems die. Here's what we've learned from deploying multiagent systems for real clients.
Lesson 1: Orchestration is the whole game
A single LLM call is easy. An agent that decides which tool to use, calls it, interprets the result, and decides what to do next — that's hard. The orchestration layer is where production systems live or die. You need: tool selection logic, error recovery, timeout handling, state management, and observability at every step.
Most agent failures aren't model failures — they're orchestration failures. The model didn't know what to do when the API timed out.
Lesson 2: RAG is not a feature, it's infrastructure
Retrieval-augmented generation sounds simple: chunk your data, embed it, search it. In production, the chunking strategy, embedding model, vector database, and retrieval pipeline all need to be tuned to your specific data and use case. A generic RAG setup will work for a demo and fail in production because it retrieves the wrong context 30% of the time.
- Chunk size matters more than embedding model choice (for most use cases)
- Hybrid search (keyword + vector) beats pure vector search by 15-20% in recall
- Re-ranking with a cross-encoder adds 50-100ms latency but improves precision significantly
- Your retrieval index needs the same governance as your database — access controls, audit logs, versioning
Lesson 3: Governance is non-negotiable
Agents that take actions (not just generate text) need guardrails. Who can approve a $10K spend? What happens when the agent is confident but wrong? How do you audit what the agent did and why? These aren't theoretical questions — they're the difference between a system that scales and one that causes a incident.
“The best agent governance framework we've seen: every action above a threshold requires human approval, and every action is logged with full reasoning trace.”
Lesson 4: Observability is 80% of the work
In a traditional app, you log requests and errors. In an agent system, you need to log: the prompt, the model's reasoning, the tool calls, the tool results, the final decision, and the confidence level. Without this, you can't debug failures, improve performance, or prove to compliance that the system is behaving correctly.
We build observability into every agent from day one. The cost is ~15% of total development time. The value is immeasurable — you can't improve what you can't see.
Lesson 5: Start narrow, then expand
The temptation with agents is to build a 'general purpose' system that can do anything. This always fails. Production agents that work are narrow: they do one thing, very well, with clear boundaries. Expand the scope only after the narrow use case is reliable.
If you're building an agentic system and want to talk to engineers who've shipped them to production, book a discovery call. We'll share what's worked, what hasn't, and what we'd do differently.
Want to talk to the team that wrote this?
Book a free 30-minute discovery call. Talk to a senior engineer, not a sales rep.
Book a discovery call