How to build audit-ready AI systems for regulated workflows

Audit-readiness is not a posture you assemble the week before an inspection. It is a property of the system: for any past decision, you can show what the system saw, what it did, who approved it, and that none of those records has changed since.

The auditor's question

Every regulated AI deployment eventually meets the same question. It comes from an internal auditor, an external one, a supervisor or a court, and it sounds like this:

“On 4 May at 14:22, your system told an analyst to flag this payment. Show me why.”

There are two honest ways to answer. You can reconstruct the decision: produce the inputs, retrieved documents, prompt, model version, output and reviewer action exactly as they were recorded at the time, with proof that the records are intact. Or you can re-execute it: run the same inputs through the same pinned components and show that the same result comes out. The first is always possible if you designed for it. The second is harder than most teams expect with language models.

What you cannot do is answer from memory, from a tracing dashboard that samples a fraction of traffic, or from a vendor's assurance that the model usually behaves. What follows is about making the honest answer cheap.

What the rules ask for, as of September 2026

The EU AI Act

Regulation (EU) 2024/1689 puts the logging duty on the system itself. Article 12(1) says high-risk AI systems “shall technically allow for the automatic recording of events (logs) over the lifetime of the system.”^[1] Article 12(2) ties those logs to three purposes: identifying situations where the system may present a risk or undergo a substantial modification, supporting post-market monitoring, and letting deployers monitor its operation under Article 26(5).^[1]

The Act spells out a minimum log for only one category, remote biometric identification (Annex III, point 1(a)). There, Article 12(3) requires the period of each use, the reference database checked, the input data that led to a match, and the identity of the people who verified the result.^[1] It makes a good template for any system: when, against what, on which input, confirmed by whom.

Retention sits with both parties. Providers must keep the automatically generated logs under their control for a period appropriate to the intended purpose, “of at least six months,” unless other Union or national law provides otherwise, in particular data protection law (Article 19(1)). Deployers carry the same duty for logs under their control (Article 26(6)).^[1] Both articles add that financial institutions keep these logs as part of the documentation they already maintain under financial services law. Separately, Article 18 requires providers to keep technical and quality management documentation for ten years after the system is placed on the market; that is a different record from the logs.^[1]

Human oversight is the other half. Under Article 14(4), the people assigned to oversight must be enabled, as appropriate and proportionate, to understand the system's capacities and limitations, remain aware of automation bias, correctly interpret its output, decide to disregard, override or reverse that output, and interrupt the system through a “stop” button or a similar procedure.^[1] Article 26(2) requires deployers to assign that oversight to people with the necessary competence, training, authority and support.^[1] The reviewer's actions therefore belong in the same log as the model's.

The dates moved. The duties did not.

Article 113, as adopted, applied most of the Act from 2 August 2026.^[1] The Commission proposed a Digital Omnibus on AI on 19 November 2025 to revise that timetable for high-risk systems.^[3] It was adopted as Regulation (EU) 2026/1744 on 8 July 2026, published in the Official Journal on 24 July and in force from 27 July 2026.^[2] The amended Article 113 applies Chapter III, Sections 1 to 3 (classification, requirements, and the obligations of providers and deployers, which is where Articles 12, 14, 19 and 26 sit) from 2 December 2027 for systems classed as high-risk under Annex III, and from 2 August 2028 for those classed as high-risk through the product legislation in Annex I.^[2]

The omnibus does not amend the text of Articles 12, 14, 19 or 26.^[2] Only the start dates moved. That is less relief than it sounds. Logging, retention and oversight are architecture decisions, and retrofitting tamper-evident logging into a running system is slower and riskier than building it in. The amended Article 111(2) also covers systems already on the market before those dates only if their design changes significantly afterwards, with systems intended for public authorities required to comply by 2 August 2030.^[2] For a system whose model and prompts change every few months, relying on that is a question to put to counsel, not an engineering plan.

Standards and frameworks

ISO/IEC 42001:2023 specifies requirements for establishing, implementing, maintaining and continually improving an AI management system.^[4] It is a management system standard, so an assessment against it looks at whether your organisation runs a controlled process for its AI systems, not at whether one model is right.

The NIST AI Risk Management Framework (AI RMF 1.0, January 2023) is voluntary. Its “accountable and transparent” characteristic spans “how and when deployment, post-deployment, or end user decisions were made and by whom.”^[5] NIST's Generative AI Profile (NIST AI 600-1, July 2024) is more concrete. It suggests recording underlying foundation models, their versions and data provenance in AI system inventories (action GV-1.6-003), and documenting each instance where content is generated, modified or shared to give a tamper-proof history, alongside version control across the life cycle (MS-2.8-003).^[7] NIST says AI RMF 1.0 is being revised under the White House AI Action Plan, so check for a newer version before you map controls to it.^[6]

Financial services

Banks in the United States have a new baseline this year. On 17 April 2026 the Federal Reserve, OCC and FDIC issued revised model risk management guidance, SR 26-2, which supersedes SR 11-7.^[8] A footnote puts generative and agentic AI models outside its scope because they are “novel and rapidly evolving,” while saying that a bank's risk management and governance practices should still guide the controls for tools the guidance does not cover.^[8] In plain terms: the model risk guidance will not tell you how to govern an LLM workflow, and your examiners will still expect you to have decided.

In the EU, financial entities subject to DORA (Regulation (EU) 2022/2554) must meet its technical standards on ICT risk management. Article 12 of Delegated Regulation (EU) 2024/1774 requires documented logging procedures that identify the events to log and their retention period, protect logs “against tampering, deletion, and unauthorised access,” detect failures of the logging system, and synchronise clocks to a documented reliable time source.^[9] There is little reason to design an AI decision log to a lower standard than any other ICT log in the same bank.

None of these texts prescribes a data structure. Read together, they converge on four properties: record automatically, retain for a defined period, protect records from tampering, and make it possible to reconstruct who decided what, when, with which version of which component. The rest of this essay is about building those four properties.

Make the log tamper-evident

Teams often call their audit log append-only when they mean the application never issues an UPDATE. That is a convention, not a guarantee. A database administrator, a migration script or an attacker with the right credentials can still change a row, and nothing in the row will show it. What an auditor can rely on is tamper evidence: any change to a past record becomes detectable.

The simplest construction is a hash chain. Each entry stores the SHA-256 hash of the entry before it, and its own hash is computed over its contents plus that previous hash. Change one byte of entry 3 and its hash no longer matches; re-hash it and entry 4 now points at a hash that no longer exists. To hide the edit, a tamperer has to recompute every hash after it. Hashing needs identical bytes every time, so serialise entries canonically. RFC 8785, the JSON Canonicalization Scheme, exists for exactly this: sorted keys and fixed serialisation of strings and numbers so that hashing and signing are repeatable.^[10]

Try all three options. An edit alone fails at the altered entry. Re-hashing that entry moves the failure to the next link. Rewriting the whole tail passes every internal check, and that is the lesson: a hash chain proves internal consistency, not authenticity. Whoever controls the entire store can rebuild it. The defence is to publish the chain head somewhere the operator cannot rewrite, such as a signed checkpoint sent to the auditor, a write-once bucket in a separate account, or a public transparency log. Once the head is out of your hands, a rewritten history no longer matches it.

import hashlib
import json

GENESIS = "0" * 64


def canonical(obj: dict) -> bytes:
    # Sorted keys, compact separators, UTF-8. Equivalent to RFC 8785 for
    # ASCII keys, strings, booleans and integers below 2**53. Use a JCS
    # library if you hash floats.
    return json.dumps(
        obj, sort_keys=True, separators=(",", ":"), ensure_ascii=False
    ).encode()


def entry_hash(body: dict, prev: str) -> str:
    return hashlib.sha256(canonical({**body, "prev": prev})).hexdigest()


def append(log: list[dict], body: dict) -> dict:
    prev = log[-1]["hash"] if log else GENESIS
    entry = {**body, "prev": prev, "hash": entry_hash(body, prev)}
    log.append(entry)
    return entry


def verify(log: list[dict], checkpoint: str | None = None) -> int | None:
    """Index of the first entry that fails, or None if the chain verifies."""
    prev = GENESIS
    for i, entry in enumerate(log):
        body = {k: v for k, v in entry.items() if k not in ("prev", "hash")}
        if entry["prev"] != prev or entry_hash(body, prev) != entry["hash"]:
            return i  # this entry and everything after it are untrusted
        prev = entry["hash"]
    if checkpoint is not None and prev != checkpoint:
        return len(log)  # consistent, but not the history the auditor holds
    return None

The verifier walks forward from the first entry and stops trusting at the first failure, because every later entry chains from a link it can no longer vouch for. The optional checkpoint argument is the external head; without it, a full rewrite goes unnoticed.

When to reach for a Merkle tree

A hash chain's proofs grow with the log. Crosby and Wallach showed in 2009 that where a classic hash chain might need an 800 MB trace to prove that one event is in a log of 80 million events, their tree-based history returned a 3 KB proof with the same guarantees.^[11] They also made the point every audit design should start from: “Tamper-evidence requires auditing.” If nobody ever checks, nothing is ever detected.^[11]

Certificate Transparency made the tree approach work at internet scale. RFC 9162, version 2.0 of the protocol, which obsoletes the original RFC 6962, builds append-only logs from Merkle trees and defines two proofs: an inclusion proof that a given entry is in the tree, and a consistency proof that a newer tree is an append-only extension of an older one.^[12] It also warns that a misbehaving log can show different views of itself to different clients, which is why independent parties compare what they see.^[12] Sigstore's Rekor applies the same idea to software supply chains as a transparency log that answers inclusion-proof queries, and its next version moves to a tile-based log backend.^[13]

Our default for a single-tenant regulated workflow is a hash chain with signed checkpoints anchored outside the operator's control. We move to a Merkle log when a third party needs to verify single entries without receiving the whole log, or when several parties write to the same record.

One tension to settle early: Articles 19 and 26 defer to data protection law, and an immutable log sits awkwardly next to an erasure request. The usual pattern is to keep payloads in a separate store and chain only their digests. When retention ends or erasure applies, delete the payload and append a deletion event; the chain still verifies. Whether a digest of personal data is itself personal data is a question for your data protection officer, and keyed hashes are worth raising with them.

Pin everything that can drift

Reconstruction needs records. Re-execution needs pins. Four things drift underneath a language model workflow:

• The model. A provider alias can resolve to a new snapshot overnight. Record the model that answered, not only the one you asked for.
• The corpus. Documents are edited, re-chunked and re-embedded. Record the IDs, versions and content hashes of what was retrieved and passed to the model.
• The prompt and policy. Templates, tool schemas, thresholds and routing rules change with every release. Version them like code and record the version per decision.
• Inference itself. Even with every input fixed, hosted inference can vary.

In September 2025, Thinking Machines Lab sampled 1,000 completions at temperature 0 from Qwen3-235B with the same prompt and got 80 unique completions. All of them were identical for the first 102 tokens and diverged at token 103.^[14] Their analysis attributes most endpoint nondeterminism to kernels that are not batch-invariant: server load changes the batch size, and the batch size changes the arithmetic. With batch-invariant kernels, all 1,000 completions matched.^[14] Seeds help without guaranteeing anything; the OpenTelemetry conventions describe requests with the same seed as “more likely” to return the same result.^[15]

Notice the case where the model, corpus and prompt are unpinned but inference is deterministic: five identical outputs, none of them the decision that was made. Consistency is not fidelity. The practical rule is to treat reconstruction as the audit guarantee and re-execution as an engineering tool. Every decision must be reconstructible from records. Re-execution with pinned components is how you regression-test a model or prompt change before release, and where a hosted model cannot be made deterministic, you compare outcome distributions on an evaluation set instead of expecting identical text.

Put the control flow in code

The same thinking applies to orchestration. We build agent workflows as explicit graphs: the model makes judgements inside nodes, but the permitted transitions between steps are code. That makes the trace checkable. An auditor, or a test, can confirm that every adverse action passed through the human approval node, because the graph has no other path to it. In a free-form agent loop, the only record of why a step happened is the model's own account of itself.

Instrument model calls with a shared vocabulary

You do not need to invent attribute names for model calls. OpenTelemetry's semantic conventions for generative AI define spans for inference, retrieval and tool execution, with attributes such as gen_ai.operation.name, gen_ai.provider.name, gen_ai.request.model, gen_ai.response.model, gen_ai.request.temperature, gen_ai.request.seed, gen_ai.response.id and token usage counts.^[15] The conventions are still marked as in development and are now maintained in their own repository, so pin the version your instrumentation emits.^[15]

Two details matter for audit. First, gen_ai.response.model is the model that actually answered; the specification's example values are gpt-4 for the request and gpt-4-0613 for the response.^[15] That resolved name is your model pin. Second, the conventions say instrumentations should not capture instructions, inputs or outputs by default, and recommend that production systems store content externally and record references on the spans.^[15] That is the shape of a good evidence store: content lives in access-controlled, content-addressed storage, and telemetry carries the hashes.

import { SpanKind, SpanStatusCode, trace } from "@opentelemetry/api";
import { createHash } from "node:crypto";
import { llm } from "./llm-client"; // your model client
import type { Passage } from "./evidence"; // { id, version, sha256, text }

const tracer = trace.getTracer("screening-service", "12.0.0");
const sha256 = (s: string) =>
  createHash("sha256").update(s, "utf8").digest("hex");

// From configuration. PROVIDER is a gen_ai.provider.name value such as
// "openai". The request may name an alias; the response names the snapshot.
const PROVIDER = process.env.LLM_PROVIDER!;
const REQUEST_MODEL = process.env.LLM_MODEL!;

export async function recommend(
  caseId: string,
  prompt: string,
  passages: Passage[],
) {
  return tracer.startActiveSpan(
    `chat ${REQUEST_MODEL}`,
    {
      kind: SpanKind.CLIENT,
      attributes: {
        "gen_ai.operation.name": "chat",
        "gen_ai.provider.name": PROVIDER,
        "gen_ai.request.model": REQUEST_MODEL,
        "gen_ai.request.temperature": 0,
        "gen_ai.request.seed": 7,
        "gen_ai.request.max_tokens": 400,
        // Content stays in the evidence store; spans carry references.
        "screening.case.id": caseId,
        "screening.prompt.template": "sanctions-screen@3.2",
        "screening.prompt.sha256": sha256(prompt),
        "screening.retrieval.chunk_sha256": passages.map((p) => p.sha256),
      },
    },
    async (span) => {
      try {
        const res = await llm.chat({
          model: REQUEST_MODEL,
          prompt,
          temperature: 0,
          seed: 7,
          maxTokens: 400,
        });
        span.setAttributes({
          // The pin: the snapshot that actually answered.
          "gen_ai.response.model": res.model,
          "gen_ai.response.id": res.id,
          "gen_ai.response.finish_reasons": [res.finishReason],
          "gen_ai.usage.input_tokens": res.usage.inputTokens,
          "gen_ai.usage.output_tokens": res.usage.outputTokens,
          "screening.output.sha256": sha256(res.text),
        });
        return { ...res, traceId: span.spanContext().traceId };
      } catch (err) {
        span.recordException(err as Error);
        span.setAttribute("error.type", (err as Error).name);
        span.setStatus({ code: SpanStatusCode.ERROR });
        throw err;
      } finally {
        span.end();
      }
    },
  );
}

Telemetry is not the audit log, though. Traces are sampled, kept for weeks and stored in systems built for operators rather than for evidence. Use spans to run the system, and write one ledger entry per decision that carries the trace ID and the same content hashes. The ledger is what you retain, anchor and hand to an auditor.

Build evidence chains a reviewer can walk

A decision record helps an auditor only if every reference in it resolves to something that still exists in the form it had at the time. We model the record as a chain a reviewer can walk backwards from the outcome to the input.

The difference between the two modes is the difference between a pointer and a copy. A URL, a branch name or a model alias tells you where to look today. A version plus a content hash tells you what was there then, and lets anyone check that the copy you produce is the one that was used.

The review link deserves the most care. Article 14 is about people being able to interpret output, resist automation bias and override the system.^[1] You can only show that happened if the record captures what the reviewer was shown, what they decided, whether they changed the recommendation, and why. We put the evidence inline in the review screen so the reviewer approves the recommendation and the passages behind it together, and that approval becomes a ledger entry in its own right.

Agree the evidence contract before you ship

Before a regulated AI system goes live, we write down, with the auditor or compliance owner, what evidence the system will produce and how it will be checked. The contract is short and specific:

1. Decision classes. Which decisions the system makes alone, which need approval and by which role, and which it must refuse.
2. Records. The fields logged per decision, where they live, who can read them and how long they are kept, set against the six-month floor for high-risk systems and against data protection limits.
3. Integrity. How the log is chained, how often checkpoints are signed, where they are anchored, and who verifies them.
4. Time. Which reference clock every component synchronises to, since a log with drifting clocks cannot order events.
5. Change control. Which evaluation suite gates a model, prompt or policy change, who owns it, and how the versions in production are recorded.
6. Reconstruction. The procedure for rebuilding a past decision from records, and a scheduled drill in which the auditor picks a decision and the team rebuilds it while they watch.

The contract becomes the specification. It sets the graph topology, the refusal rules, the retention jobs and the evaluation suite. Engineering backwards from it produces a system that is audit-ready by construction. Retrofitting the same properties later means migrating historical logs that nobody can prove were never changed.

What this buys you

The omnibus gave EU high-risk systems more time, and SR 26-2 left generative AI to each bank's own governance. Neither change reduces what an auditor will ask. The systems that answer “why did it do that?” in minutes are the ones built so that every past decision can be walked from output back to input, with proof that nothing on the path has changed.

That is how we build AI agents (opens in a new tab) for financial services (opens in a new tab) and for legal and compliance teams (opens in a new tab), where the auditor is a standing constraint rather than an annual event.