Audit trails for AI systems

When a regulator asks why your model declined a claim eighteen months ago, “we retrained the model since then” is not an answer. Neither is “the logs have rotated.” Auditability is the contract that lets AI systems live in regulated work — and most teams discover what their contract actually says the first time someone asks for evidence.

This post is the version of that conversation we wish we’d had earlier. It describes the four log streams every Neuralcraft deployment writes from day one, why each one exists, and the mistakes we’ve made trying to skip them.

The premise: replayability is the deliverable

In a regulated decision system, the model is not the artefact you ship. The replayable decision is. A regulator, an internal auditor, or the customer whose case it touched can ask, months or years later, three questions:

1. What did the system see?
2. What did it decide, and how confident was it?
3. Why is that decision defensible?

If you can’t answer all three from logs alone — without re-running the model, without “checking with the team that built it” — you don’t have an auditable system. You have a model with some logging.

The four streams

Every Neuralcraft deployment writes four append-only streams. They are versioned, immutable, and — this matters — separable, so a request for evidence can be served without exposing the rest.

1. Inputs

Every input the system saw, normalised and hashed. Not the raw payload from the upstream service — the canonical form the model received, after PII scrubbing, feature selection, and any preprocessing. Two reasons:

• The audit question is “what did the model see,” not “what did the API receive.” Those diverge the moment you add a feature flag.
• Storing the post-processing form is what lets you replay the decision later, even if the upstream schema has changed.

We hash inputs with a versioned scheme so we can prove a record hasn’t been tampered with without storing PII unnecessarily.

2. Decisions

The output, plus everything needed to defend it: the model version, the prompt template version (for LLM systems), the retrieval set used (for RAG), the calibrated confidence, and the abstention flag.

The abstention flag is the one teams forget. “The model declined to answer” is itself a decision — and it’s often the most consequential one in regulated work. Logging it explicitly, with the same rigor as the positive outputs, is what lets you defend a “we didn’t decide” stance later.

3. Explanations

Whatever explanation the system surfaced to the human in the loop — citations, attributions, retrieved spans, calibrated confidence bands. Logged verbatim, not regenerated.

This is the stream that catches the most teams off guard. Explanations that are regenerated on demand are not auditable: the model has changed, the retrieval index has changed, and the explanation a regulator sees in 2027 will not be the one your operator saw in 2026. Log what was shown, when it was shown.

4. Reviews

The human action on top of the decision: approve, override, abstain, escalate. With a free-text reason field that you do not parse — it is evidence, not data.

The review stream is what closes the loop. It’s also what lets you measure calibration drift over time: when operators start overriding the model more often, that is the earliest signal that something is wrong, and it’s a signal you only have if you logged the reviews.

What this looks like, structurally

type DecisionRecord = {
  decision_id: string;            // ULID, monotonic
  case_id: string;                // upstream identifier
  model_version: string;          // semver + commit sha
  prompt_template_version?: string;
  retrieval_set_hash?: string;    // RAG only
  inputs_hash: string;            // canonical-form hash
  output: unknown;                // schema-validated
  confidence: number;             // calibrated
  abstained: boolean;
  policy_version: string;         // which decision policy applied
  ts: string;                     // ISO 8601 UTC
};

The four streams reference each other by decision_id. They live in separate tables, often separate stores, with separate retention policies. A regulator asking for “all decisions in case CL-44912” gets a join across the four; an internal team asking “what’s our override rate this quarter” hits only the reviews stream.

Three mistakes we’ve made

Storing the API request, not the model input. We did this on the first healthcare deployment. Six months later we changed the upstream feature extractor and lost the ability to replay decisions from before the change. We now treat the canonical input form as the authoritative log entry.

Regenerating explanations. A tempting shortcut: store the inputs and the decision, regenerate the explanation on demand. This breaks the moment the model or retrieval index changes. Don’t do this. Log the explanation that was shown.

One unified log table. “It’s all just events, store it together.” This fights you the moment you have to serve a regulator request without exposing the rest of the system. Separate streams, separate retention, separate access control — from day one.

What this enables

Once the four streams are in place, a few things become possible that aren’t otherwise:

• Replay: re-run any historical decision, with the exact inputs and model version, to confirm the output the regulator is asking about.
• Calibration drift: track override rate over time, segmented by case type, to catch model degradation before it becomes an incident.
• Counterfactuals: ask “would today’s model have made the same call?” for any historical case, which is what you need when changing models in production.
• Right-to-explanation: serve the explanation that was actually shown, to the customer whose case it touched, on request.

None of this is exotic. The hard part is committing to it on day one, before there’s a regulator asking, before there’s an incident — when the team is small and the temptation is to ship the model and add the logs later. There is no later. The logs are part of the model.

The cost of getting this wrong

The cost is not “we get fined.” The cost is that an AI system that cannot defend its own decisions cannot stay in regulated work. Either it gets pulled, or it gets quietly downgraded to a “decision support” label that exempts it from the audit it can’t pass — at which point it stops mattering.

The teams we’ve seen succeed in this space have, without exception, treated audit infrastructure as a peer of the model. The model is the easier half.