Building Production-Grade AI Systems

The gap between a demo and a system

Almost any team can produce an AI demo that lands well in a boardroom. A clever prompt, a well-chosen example, and a model that responds fluently is enough to win the room. The hard part begins the moment that demo has to serve real traffic, touch regulated data, and produce answers that hold up under audit. At Baalvion Industries we build the Baalvion Operating System (BOS) — a multi-tenant trade infrastructure spanning commerce, finance, compliance, logistics, and intelligence across 198 markets and 180+ jurisdictions. When AI sits inside that platform, it is not a feature; it is load-bearing. It scores compliance risk, classifies customs declarations, summarises cross-border settlement exceptions, and surfaces market intelligence. None of those can fail quietly.

The discipline of taking a model from prototype to production is, in our experience, less about the model and more about everything wrapped around it: the evaluation harness that tells you whether a change helped or hurt, the observability that lets you debug a single bad answer six weeks later, the cost controls that keep inference economics sane at scale, the guardrails that keep the system inside policy, and the retrieval layer that grounds answers in facts. This is the work that turns a probabilistic component into something a regulated enterprise can depend on. It mirrors how we ship the rest of the BOS platform — infrastructure-grade, compliance-first, and auditable end to end.

Reliability: treat the model as an unreliable dependency

The first architectural decision is to stop treating the model as a trusted function call. It is a remote, non-deterministic dependency with variable latency, occasional malformed output, and provider-side rate limits and outages. We design around that reality the same way we design around any flaky network dependency: timeouts, bounded retries with jittered backoff, circuit breakers, and bulkheads that stop one degraded model from starving the rest of the system. A request to a language model should never be the thing that takes down a settlement workflow.

Structured output is enforced rather than hoped for. We constrain generations to JSON schemas, validate every response at the boundary, and reject or repair anything that does not conform — never trusting model output any more than we would trust raw user input. Where a task can be served by a smaller, cheaper, faster model with acceptable quality, we route to it and reserve frontier models for the genuinely hard cases. Idempotency keys protect against duplicate side effects when a retry fires after a response was actually produced. The goal is graceful degradation: if the intelligence layer is slow or unavailable, the governance and compliance suite falls back to deterministic rules rather than blocking the transaction.

Evals: the test suite for non-deterministic software

You cannot improve what you cannot measure, and you cannot ship changes safely to a system that has no regression suite. For AI, that suite is an evaluation harness. Before any prompt change, model upgrade, or retrieval tweak reaches production, it runs against a curated, versioned dataset of representative cases drawn from real traffic — including the failures, edge cases, and adversarial inputs we have seen in the field. We track exact-match and structural accuracy where the task allows it, and use rubric-based grading, often with an LLM-as-judge calibrated against human labels, where outputs are open-ended.

The non-negotiable principle is that evals gate deployment. A prompt that scores 4% lower on the compliance-classification set does not ship just because it reads more elegantly. We hold a human-labelled golden set as ground truth, watch for the LLM judge drifting away from it, and treat a divergence as a bug in the eval, not a licence to ignore it. This is the same posture we take everywhere in engineering: changes are proven against a baseline before they touch production, and the evidence is recorded. It is what lets us iterate on the AI solutions inside BOS without quietly regressing quality that customers in regulated finance depend on.

Observability: be able to explain any single answer

When a customs classification or a fraud score is questioned — and in regulated trade it will be — "the model decided" is not an acceptable answer. Every inference in BOS emits a structured trace: the input, the resolved prompt and template version, the retrieved context and its sources, the model and parameters used, token counts, latency, cost, and the validated output. Traces are tied to a tenant and a correlation ID so we can reconstruct exactly what the system saw and produced for a specific decision, months after the fact.

On top of per-request tracing we run aggregate monitoring: token throughput, p50/p95/p99 latency, error and retry rates, cost per tenant, and quality signals such as schema-rejection rate and guardrail-trip rate. We also watch for drift — shifts in input distribution or output patterns that often precede a quality problem long before a user complains. This observability is not bolted on at the end; it is part of the auditable, transparent posture that underpins our trust and security commitments, alongside AES-256 encryption, SOC 2 Type II, and ISO 27001 controls. An AI decision you cannot explain is a liability in any jurisdiction we operate in.

Cost control: inference economics that survive scale

Token-based inference has a cost structure unlike traditional compute: it scales with usage in ways that can quietly erode unit economics if left unmanaged. At hundreds of thousands of transactions, a careless prompt design or an unnecessary frontier-model call multiplies into real money. We treat cost as a first-class engineering constraint, not an afterthought for the finance team to discover later.

The levers are well understood and we apply them deliberately. Prompt caching reuses the expensive, stable portion of a context across requests. Semantic caching returns a stored answer when an equivalent question has already been answered. Model routing sends the bulk of easy traffic to small, cheap models. Context is trimmed to what retrieval proves is relevant rather than stuffing the window. Batch and asynchronous processing absorb non-interactive workloads at lower priority. Every one of these is measured against the eval suite, because the failure mode is obvious: cutting cost by silently cutting quality. Cost per successful, validated outcome — not cost per token — is the metric that matters.

Guardrails: keeping the system inside policy

A model that is helpful but occasionally off-policy is unacceptable in trade and finance. Guardrails operate on both sides of the model. On input, we screen for prompt injection, detect and redact sensitive data such as PII and payment details before it ever reaches a third-party provider, and enforce tenant-level access so one customer's context can never leak into another's prompt. On output, we validate against schema, run policy and safety classifiers, and apply business rules — a sanctioned-party hit is blocked deterministically, never left to model discretion.

Crucially, AI advises but does not get the final word on irreversible actions. In our compliance pathways the model's reasoning enriches a decision, but a deterministic rule engine and, where required, a human reviewer hold the authority to block. This layered design — probabilistic intelligence inside a deterministic, auditable shell — is how we reconcile the power of generative models with the compliance-first, KYC/AML and GDPR obligations that come with operating across 180+ jurisdictions. It is the same philosophy behind our AI agents: autonomy is bounded by explicit policy, and every action is logged.

Retrieval: grounding answers in facts, not memory

A model's parametric memory is frozen, generic, and prone to confident fabrication. For anything that depends on a specific tenant's data, a current tariff schedule, or a live shipment status, the answer must be grounded in retrieval. Retrieval-augmented generation (RAG) is the pattern, but the quality of a RAG system lives almost entirely in the retrieval half, not the generation half. Most "the AI hallucinated" incidents are, on inspection, retrieval failures: the right document was never fetched.

We invest accordingly. Hybrid retrieval combines dense vector similarity with lexical search so that both semantic matches and exact identifiers — a HS code, an invoice number, a counterparty name — are reliably found. Chunking is tuned to the document structure rather than applied blindly. We re-rank candidates before they enter the context window, and we attach source citations to every grounded claim so a user can verify it. Every retrieval respects tenant isolation, enforced down to the data layer, so retrieval can never become a cross-tenant leak. Done well, retrieval is what lets the AI market intelligence layer answer questions about live, proprietary trade data with citations a compliance officer can audit — the difference between a plausible sentence and a defensible answer.

From component to platform

None of these concerns — reliability, evals, observability, cost, guardrails, retrieval — is optional, and none is independent. A change to retrieval moves your eval scores; a cost optimisation can trip a guardrail; a reliability fallback needs its own observability. Production-grade AI is the discipline of holding all of them together as one system, versioned and measured, inside a platform that was built to be auditable in the first place. That is the lens Baalvion brings to AI: not as a standalone novelty, but as one more layer of infrastructure engineered to the same standard as the trade platform and governance suite it serves. If you are taking AI from prototype to production, that is the bar worth holding yourself to.