AI & Agents · Cloud, DevOps & LLMOps · Databases & Data Engineering

Azure AI Automation: Document Intelligence, RAG & Governed Agents

By David Gadyan · Published November 4, 2026

An end-to-end Azure and LLM platform for construction and manufacturing back offices. It turns messy PDFs into validated data with the source quoted for every field, answers staff questions only from company documents with citations, and runs an AI agent whose every write action waits for human approval. Every call is traced and costed, and prompt versions are measured on golden sets. English and German.

The problem

Teams that try to connect Azure AI services to real operations hit the same walls: many services to orchestrate (Document Intelligence, Blob storage, queues, search and the model), state that has to survive async steps and human approvals, tool calls that crash the agent loop, compliance requirements around GDPR, audit and identity, and no way to tell whether a prompt change made answers better or worse.

This platform is built for construction and manufacturing back offices, where the paperwork is supplier invoices, inspection reports, material certificates and drawing sheets, in English and German.

The product in use

Walkthrough: a flagged German invoice with field-level provenance, a cited answer, an agent request that waits for approval and resumes, the trace that ties it together, and an eval comparison between two configurations.

What I built

Dashboard: documents processed, agent runs, pending approvals and LLM cost, with cost per day by module and spend per model

One view of cost and quality. Documents processed, agent runs, pending approvals and model spend for the last 14 days, broken down by module and by model.

A German concrete invoice next to its extracted fields, each field linked to the exact place on the page it came from

Document intelligence with provenance. The pipeline extracts words with coordinates (PyMuPDF, Azure AI Document Intelligence, or vision OCR for scans), classifies the document and extracts a typed record with structured outputs. Every field quotes its source text, so a reviewer can click a value and see where on the page it came from.

Validation tab: the line items sum to 9,630.00 but the subtotal says 9,530, so the invoice is flagged for review

Rules before booking. Deterministic checks validate totals, dates and cross-field logic. Here the line items do not add up to the subtotal, so the invoice goes to review instead of into the books.

Operations agent opening a high-priority ticket for a failed concrete cube test, with each tool call shown

Approvals: a send-email action requested by the agent waits for a person to approve or reject it

A governed agent. A tool-calling agent looks up projects, suppliers, purchase orders and tickets and searches the knowledge base. Anything that writes (a ticket, an order status, an e-mail to a supplier) is parked for an approver; the worker executes it after approval and the run resumes. Role-based tool permissions, loop detection, token budgets, tool timeouts, a kill switch and a complete audit log sit around it.

Trace waterfall of one agent run: model calls, tool calls and the approved action, with time and cost per span

Evals: golden-set runs for retrieval, QA and extraction, and a comparison between two prompt and embedding configurations

Traced and measured. Every run is an OpenTelemetry trace with token counts and cost per model call, viewable in the app or in Application Insights. Prompts are versioned and golden sets measure retrieval recall, answer correctness and faithfulness, abstention and extraction accuracy. On the demo data, switching to a multilingual embedding model took QA correctness and abstention accuracy from 0.875 to 1.00; the eval view is what showed the gap.

How it runs

Local-first: one docker compose up runs the API, worker, UI, PostgreSQL with pgvector and Azurite, with no Azure resources needed. The same code deploys to Azure Container Apps with managed identity, Key Vault, Blob and Queue storage, PostgreSQL Flexible Server, Document Intelligence and Log Analytics, all defined in Bicep.

Stack by layer

Front end: React 19, Vite and TypeScriptTailwind CSS and shadcn/uimotion
Back end: Python 3.12 and FastAPISQLAlchemy and PydanticQueue worker for async steps and approvals
AI: Claude via the Anthropic SDK, with a self-hosted Ollama alternativeStructured outputs with a source quote for every fieldHybrid retrieval (vector + keyword, RRF fusion, rerank)Tool-calling agent, also exposed over MCPGolden-set evals with LLM judges
Data: PostgreSQL 16 with pgvector and full-text searchAzure AI Document Intelligence, PyMuPDF and vision OCRAzure Blob and Queue storage (Azurite locally)
Infrastructure: Azure Container Apps with managed identityBicep, Key Vault, Log AnalyticsOpenTelemetry to Postgres and Application InsightsDocker Compose and GitHub Actions
Security: Role-based tool permissions and a kill switchEvery agent write action approved by a personComplete audit logPrompt-injection hygiene, token budgets and tool timeouts

Outcome

Supplier invoices, inspection reports, material certificates and drawings become typed records with the exact source text quoted for every field; deterministic rules flag a German invoice with a wrong net amount instead of booking it.
Staff questions are answered only from company documents, with numbered citations that open the source passage and an explicit "insufficient context" answer when the documents do not cover it.
The agent can look things up freely, but creating tickets, changing orders or e-mailing suppliers waits for an approver; the same tools work from Claude Desktop or Claude Code over MCP with the same approvals.
Every document run, answer and agent run is a trace with tokens and cost per model call, and prompt changes are compared on golden-set numbers, not impressions.