AI Engineering in Production: What They Don't Tell You

Every week there's a new tutorial showing how to build a chatbot or deploy a fine-tuned model in 15 minutes. What these tutorials rarely cover is what happens after you go to production.

Having worked on AI systems in enterprise environments, here are the things I wish someone had told me earlier.

Latency Is Your First Surprise

Development environments are forgiving. In production, your users notice every millisecond. AI inference — especially for large language models — is expensive and slow. The gap between "it works" and "it works at acceptable latency under production load" is significant.

Key considerations:

• Model quantisation — INT8 or INT4 models trade a little quality for dramatically faster inference
• Caching — semantic caching can reduce inference calls by 40-60% for repeated or similar queries
• Batching — GPU utilisation dramatically improves with batched inference
• Async architecture — for non-real-time use cases, push inference to background jobs

Observability Is Different for AI

Traditional APM tools are not designed for AI workloads. You need additional instrumentation:

• Token consumption tracking — per-request, per-user, per-day
• Latency percentiles — p50, p95, p99 for both time-to-first-token and total completion time
• Quality metrics — hallucination rate, faithfulness scores (for RAG), user feedback signals
• Prompt logging — you need to be able to debug bad outputs, which requires the full prompt context

Prompt Injection Is a Real Threat

If users can influence the content that enters your prompts, you have a potential injection vulnerability. This is particularly acute in RAG systems where retrieved documents are inserted into prompts.

Mitigations include input sanitisation, output validation, and architectural patterns that separate user input from system instructions.

Cost Will Surprise You

A demo running 10 queries per minute and a production system running 10,000 queries per minute have very different cost profiles. Model costs scale with tokens, not with requests — long contexts are expensive.

Build cost tracking into your system from day one, set budget alerts, and design token efficiency into your prompts.

The Model Is Not Static

Model providers update their models, sometimes without adequate notice. Behaviour that worked in one model version may change. Build evaluation pipelines that you can run against new model versions before you cut over.

Closing Thoughts

AI engineering in production is infrastructure engineering, not just ML engineering. The skills that make you good at building reliable distributed systems — observability, resilience, cost management, security — transfer directly to AI systems.

The teams that ship reliable AI products are the ones that treat their AI systems with the same operational rigour as the rest of their stack.