Deterministic Python vs LLM Reasoning: Docker Bench

Core

• Risk management breaks when “LLM reasoning” invents numbers, skips constraints, or silently changes assumptions.
• This benchmark uses deterministic Python as ground truth and measures divergence from LLM-produced outputs.
• You’ll build a repeatable harness with Docker and Docker Compose to run, log, and compare results consistently.

Introduction

In trading and risk systems, “reasoning” isn’t valuable if it isn’t auditable. A risk check that sometimes returns the right number is still a broken control: you need repeatable calculations, explicit assumptions, and logs that let you explain every decision after the fact.

This post builds a deterministic Python vs LLM reasoning benchmark with Docker: deterministic Python functions define ground truth, while LLM outputs (or any “AI reasoning” output you capture) are treated as a candidate answer that must be validated. The goal isn’t to dunk on LLMs; it’s to create a harness that makes failures measurable, reproducible, and actionable.

You’ll implement three trading-adjacent test cases where narratives are tempting but math is non-negotiable, define what “pass/fail” means, and produce structured logs for hallucinations—then run everything in Docker Compose so results don’t drift with local environments.

Hook: Why “LLM reasoning” fails hardest in Risk Management

Risk management is where LLM-style “reasoning” tends to fail in the most dangerous ways: not with obvious nonsense, but with plausible-looking outputs that are subtly wrong. The failure mode isn’t just arithmetic mistakes; it’s constraint violations and assumption drift—changing units, ignoring rounding rules, or inventing missing inputs.

In a trading system, those errors can propagate into order sizing, margin usage, and exposure reporting. The output may look coherent to a human reviewer skimming a dashboard, but it won’t be consistent under replay or audit.

In finance, a control that can’t be reproduced on demand is functionally indistinguishable from a control that doesn’t exist—because you can’t prove it worked when it mattered.

What makes risk calculations brittle for LLM outputs

• Hard invariants: leverage limits, max position size, and margin requirements must hold for every run.
• Units and precision: currency, basis points, and quantity rounding rules are easy to “hand-wave” incorrectly.
• Edge cases: zero/negative prices, missing fields, and partial fills are common in real feeds.

What the benchmark is trying to answer

• When an LLM produces a numeric answer, how often does it match deterministic ground truth within a defined tolerance?
• When it doesn’t match, can we classify the failure (math error vs constraint violation vs assumption drift)?
• Can we reproduce the failure exactly (same inputs, same environment, same output capture)?

Define the benchmark: deterministic Python as the ground truth

The benchmark design is simple: deterministic Python functions compute “correct” outputs from structured inputs. Any LLM output is treated as an untrusted candidate that must be parsed, validated, and compared. This keeps the benchmark aligned with the repeated practitioner demand: 100% deterministic Python as the reference, and no prompts, no LLM reasoning inside the correctness path.

Ground truth rules

• Inputs are JSON fixtures checked into source control.
• Ground truth is computed by pure functions (no network, no time, no randomness).
• Comparison uses explicit tolerances and explicit invariants (e.g., “post-trade leverage must be <= limit”).

Benchmark outputs

Each test case produces:

1. Ground truth numeric values and invariant checks.
2. Parsed candidate values (from an LLM output capture, or any external system output).
3. A verdict plus a structured log record when there’s divergence.

The following JSON config defines tolerances, invariants, and where to find fixtures and captured outputs. It’s intentionally boring: the point is to make runs repeatable and easy to diff.

{
  "benchmark_name": "deterministic-python-vs-llm-reasoning",
  "fixtures_dir": "fixtures",
  "llm_outputs_dir": "llm_outputs",
  "report_dir": "reports",
  "tolerances": {
    "abs": 1e-9,
    "rel": 1e-6
  },
  "invariants": {
    "max_leverage": 3.0,
    "max_position_notional_usd": 250000.0,
    "min_equity_usd": 0.0
  },
  "require_fields": [
    "account.equity_usd",
    "account.margin_used_usd",
    "order.side",
    "order.qty",
    "order.price_usd"
  ]
}
  

Test cases from a trading system architecture: narratives vs math

These test cases mirror a common trading system architecture: an order request comes in, risk checks compute post-trade exposure, and the system either approves or rejects. The benchmark focuses on the boundary where LLMs often get used incorrectly: turning a narrative (“we’re safe, leverage is fine”) into a numeric decision without deterministic computation.

Case 1: Post-trade leverage check (the classic foot-gun)

Ground truth: leverage = (existing_notional + order_notional) / equity. Approve only if leverage <= max_leverage and equity is non-negative.

Where narratives go wrong: LLM outputs often “explain” leverage qualitatively, or compute with the wrong denominator (balance vs equity), or ignore existing exposure.

Case 2: Position notional limit (units and rounding)

Ground truth: position_notional = abs(position_qty * mark_price). Reject if it exceeds max_position_notional_usd. Rounding rules should be explicit.

Where narratives go wrong: mixing mark price vs order price, or treating qty as USD notional.

Case 3: Margin usage delta (constraint propagation)

Ground truth: margin_used increases by order_notional / leverage_limit (simplified), and must remain <= equity. Even if the LLM gets the delta right, it may fail the invariant check.

Minimal, working benchmark runner

The script below loads fixtures, computes deterministic ground truth, parses captured “LLM outputs” (stored as JSON), compares results, and writes a JSONL report for any mismatch. This is the core of the deterministic Python vs LLM reasoning benchmark with Docker: the correctness path is pure Python math, and the LLM side is just an input artifact.

import json
import math
import os
from dataclasses import dataclass
from datetime import datetime, timezone
from typing import Any, Dict, Tuple


def load_json(path: str) -> Any:
    with open(path, "r", encoding="utf-8") as f:
        return json.load(f)


def get_required(d: Dict[str, Any], dotted: str) -> Any:
    cur: Any = d
    for part in dotted.split("."):
        if not isinstance(cur, dict) or part not in cur:
            raise KeyError(f"Missing required field: {dotted}")
        cur = cur[part]
    return cur


def approx_equal(a: float, b: float, abs_tol: float, rel_tol: float) -> bool:
    return math.isclose(a, b, abs_tol=abs_tol, rel_tol=rel_tol)


@dataclass(frozen=True)
class Verdict:
    ok: bool
    reason: str


def compute_ground_truth(fx: Dict[str, Any], invariants: Dict[str, float]) -> Dict[str, Any]:
    equity = float(get_required(fx, "account.equity_usd"))
    existing_notional = float(fx.get("account", {}).get("existing_notional_usd", 0.0))

    side = str(get_required(fx, "order.side")).lower()
    qty = float(get_required(fx, "order.qty"))
    price = float(get_required(fx, "order.price_usd"))

    if qty < 0 or price < 0:
        raise ValueError("qty and price_usd must be non-negative")

    order_notional = qty * price
    post_notional = existing_notional + order_notional
    leverage = (post_notional / equity) if equity != 0 else float("inf")

    max_lev = float(invariants["max_leverage"])
    max_pos = float(invariants["max_position_notional_usd"])

    # Simplified position notional: use post_notional as a proxy for a single-asset account
    position_notional = abs(post_notional)

    approved = True
    reasons = []

    if equity  max_lev:
        approved = False
        reasons.append("leverage_limit_exceeded")

    if position_notional > max_pos:
        approved = False
        reasons.append("position_notional_exceeded")

    return {
        "order_side": side,
        "order_notional_usd": order_notional,
        "post_notional_usd": post_notional,
        "post_trade_leverage": leverage,
        "position_notional_usd": position_notional,
        "approved": approved,
        "reasons": reasons,
    }


def parse_llm_output(obj: Dict[str, Any]) -> Dict[str, Any]:
    # Expect the LLM output capture to already be structured JSON.
    # If you have raw text, convert it to JSON outside this benchmark.
    return {
        "post_trade_leverage": float(obj["post_trade_leverage"]),
        "approved": bool(obj["approved"]),
        "order_notional_usd": float(obj.get("order_notional_usd", 0.0)),
        "reasons": list(obj.get("reasons", [])),
    }


def compare(gt: Dict[str, Any], cand: Dict[str, Any], tol: Dict[str, float]) -> Tuple[Verdict, Dict[str, Any]]:
    diffs: Dict[str, Any] = {}

    if not approx_equal(gt["post_trade_leverage"], cand["post_trade_leverage"], tol["abs"], tol["rel"]):
        diffs["post_trade_leverage"] = {"gt": gt["post_trade_leverage"], "cand": cand["post_trade_leverage"]}

    if gt["approved"] != cand["approved"]:
        diffs["approved"] = {"gt": gt["approved"], "cand": cand["approved"]}

    # Optional: compare order notional if provided
    if "order_notional_usd" in cand and cand["order_notional_usd"] != 0.0:
        if not approx_equal(gt["order_notional_usd"], cand["order_notional_usd"], tol["abs"], tol["rel"]):
            diffs["order_notional_usd"] = {"gt": gt["order_notional_usd"], "cand": cand["order_notional_usd"]}

    if diffs:
        # Classify a common hallucination shape: approving while leverage is over the limit.
        reason = "mismatch"
        if (gt["approved"] is False) and (cand["approved"] is True):
            reason = "constraint_violation_candidate_approved"
        return Verdict(False, reason), diffs

    return Verdict(True, "match"), diffs


def main() -> None:
    config = load_json("benchmark_config.json")
    fixtures_dir = config["fixtures_dir"]
    llm_dir = config["llm_outputs_dir"]
    report_dir = config["report_dir"]
    os.makedirs(report_dir, exist_ok=True)

    tol = config["tolerances"]
    invariants = config["invariants"]

    report_path = os.path.join(report_dir, "report.jsonl")
    total = 0
    failed = 0

    with open(report_path, "w", encoding="utf-8") as rep:
        for name in sorted(os.listdir(fixtures_dir)):
            if not name.endswith(".json"):
                continue
            total += 1
            fx_path = os.path.join(fixtures_dir, name)
            out_path = os.path.join(llm_dir, name)

            fx = load_json(fx_path)
            for req in config["require_fields"]:
                _ = get_required(fx, req)

            gt = compute_ground_truth(fx, invariants)

            if not os.path.exists(out_path):
                failed += 1
                rep.write(json.dumps({
                    "ts": datetime.now(timezone.utc).isoformat(),
                    "case": name,
                    "verdict": "fail",
                    "reason": "missing_llm_output_capture",
                    "ground_truth": gt
                }) + "\n")
                continue

            cand_raw = load_json(out_path)
            cand = parse_llm_output(cand_raw)
            verdict, diffs = compare(gt, cand, tol)

            if not verdict.ok:
                failed += 1
                rep.write(json.dumps({
                    "ts": datetime.now(timezone.utc).isoformat(),
                    "case": name,
                    "verdict": "fail",
                    "reason": verdict.reason,
                    "diffs": diffs,
                    "ground_truth": gt,
                    "candidate": cand
                }) + "\n")

    summary = {"total": total, "failed": failed, "passed": total - failed, "report": report_path}
    print(json.dumps(summary, indent=2))


if __name__ == "__main__":
    main()
  

Example fixtures and captured outputs

To make this concrete, store one fixture and one captured candidate output per case. The fixture is the input to deterministic Python; the candidate output is what an LLM (or any AI layer) claimed the answer was. Both are versioned artifacts, which is what makes the benchmark reproducible.

• fixtures/case1.json includes equity, existing notional, and an order.
• llm_outputs/case1.json contains the candidate leverage and approve/reject decision.

Reporting: what to log when an AI hallucination appears

When a mismatch happens, the benchmark should emit logs that help you answer two questions quickly: (1) what exactly diverged, and (2) what kind of failure is it? “The model hallucinated” is not a useful incident report.

Minimum fields to log

• Case ID and input fixture hash (or filename + git commit).
• Ground truth values and invariants.
• Candidate output values as parsed (not raw text).
• Diffs with gt vs candidate.
• Failure classification: math error, constraint violation, missing field, unit mismatch.

Why structured JSONL beats screenshots and free-form text

JSONL lets you post-process failures: count failure types, group by invariant, and trend over time. It also makes it easy to ship into whatever log store you already use.

To run the benchmark and inspect failures, use a simple CLI flow. This is also how you’ll run it inside containers later.

# Local run (outside Docker)
python -V
python benchmark.py

# Inspect only failures
cat reports/report.jsonl | jq -c 'select(.verdict=="fail") | {case, reason, diffs}'

# Count failure reasons
cat reports/report.jsonl | jq -r '.reason' | sort | uniq -c | sort -nr
  

Failure classification that maps to real controls

In risk management, the most important class is usually “candidate approved but should reject.” That’s a control failure. A numeric mismatch that still rejects might be less severe, but it’s still a sign you can’t trust the numbers for reporting.

Practical categories to start with:

1. constraint_violation_candidate_approved: candidate approves while any invariant fails under ground truth.
2. math_mismatch: candidate rejects/approves correctly but numeric fields diverge beyond tolerance.
3. missing_llm_output_capture: you can’t evaluate what you didn’t record.
4. schema_error: candidate output can’t be parsed into required numeric fields.

Reproducible runs with Docker: Dockerfile + Docker Compose

Deterministic math is only half the story; your benchmark runs must also be deterministic. Docker helps by pinning the runtime (Python version, dependencies, locale) and by making it easy to run the exact same harness in CI.

The role of Docker in this benchmark workflow

• Pin Python and dependencies so floating environments don’t change results.
• Mount fixtures and output captures as volumes for clean separation of code vs data.
• Produce reports in a host-mounted directory so CI artifacts are easy to collect.

Dockerfile: pinned Python runtime

This Dockerfile builds a small image that runs the benchmark. It copies the harness code and expects fixtures and LLM output captures to be mounted at runtime (so you can swap datasets without rebuilding).

FROM python:3.12-slim

ENV PYTHONDONTWRITEBYTECODE=1 \
    PYTHONUNBUFFERED=1

WORKDIR /app

# Optional: jq is useful when debugging inside the container
RUN apt-get update \
  && apt-get install -y --no-install-recommends ca-certificates \
  && rm -rf /var/lib/apt/lists/*

COPY benchmark.py /app/benchmark.py
COPY benchmark_config.json /app/benchmark_config.json

# Create expected directories (will be overridden by bind mounts in compose)
RUN mkdir -p /app/fixtures /app/llm_outputs /app/reports

CMD ["python", "benchmark.py"]
  

Docker Compose: repeatable execution with mounted datasets

This Compose file runs the benchmark container with three bind mounts: fixtures, captured candidate outputs, and reports. That keeps the run reproducible and makes outputs easy to inspect on the host.

services:
  benchmark:
    build:
      context: .
    container_name: deterministic-benchmark
    volumes:
      - ./fixtures:/app/fixtures:ro
      - ./llm_outputs:/app/llm_outputs:ro
      - ./reports:/app/reports
    working_dir: /app
    command: ["python", "benchmark.py"]
  

Run it the same way everywhere

With Docker Compose, the run is a single command, and the report artifact is always written to ./reports/report.jsonl.

• Local dev: run and iterate on fixtures quickly.
• CI: run on every change to risk math or parsing logic.

Optional: build/tag/push for CI runners or Kubernetes jobs

If you want this benchmark to run in a remote CI environment or as a scheduled job in a cluster, treat it like any other containerized workload: build, tag, push to a registry, then run it where you need it. Even if you don’t deploy it to Kubernetes today, having a pushed image makes the benchmark easier to run consistently across teams.

# Build and tag
docker build -t ghcr.io/your-org/deterministic-benchmark:0.1.0 .

# Push (requires 'docker login ghcr.io' and permissions)
docker push ghcr.io/your-org/deterministic-benchmark:0.1.0

# Example: run the pushed image with mounted datasets
docker run --rm \
  -v "$PWD/fixtures:/app/fixtures:ro" \
  -v "$PWD/llm_outputs:/app/llm_outputs:ro" \
  -v "$PWD/reports:/app/reports" \
  ghcr.io/your-org/deterministic-benchmark:0.1.0
  

Conclusion

Risk management is where “LLM reasoning” is least forgivable: you need invariant-preserving math, not plausible narratives. A deterministic benchmark makes that concrete by defining ground truth in Python, treating LLM outputs as untrusted candidates, and logging mismatches in a way that supports audits and incident review.

Build this harness, add your own fixtures from real post-mortems, and run it in Docker Compose on every change. If you’re currently letting an LLM produce numbers that drive approvals, limits, or reporting, use this deterministic Python vs LLM reasoning benchmark with Docker to quantify the gap—and to prove when it’s safe to automate versus when it must stay deterministic.