Hindcast: Multi-Agent Causal Simulation
Case study 01A multi-agent forecasting system that cannot read the future, replays byte for byte, and lands its event log in a live, Terraform-managed AWS lake.
Milestones M0 through M17 shipped · live on AWS in us-west-2 · first full study run
Co-built with Utkarsh Singh
Ask a model whether a regulator will clear a merger before the deadline and it will hand you a probability. It will not tell you who has to act, what each of them wants, or what each of them could actually see, and you cannot replay the answer or prove it was not quietly informed by what happened next. Hindcast models the why instead. It compiles a question into a typed causal graph of actors and forces, simulates it with LLM agents over 24 steps from many seeds, and scores the spread of outcomes against what really happened. The name is the method: a hindcast is a forecast made for a period that has already happened, using only what was known before it, then checked against the record.
Every backtested forecaster has the same credibility problem: from the outside you cannot tell whether it was clever or whether it saw the answer. So retrieval runs through a function that takes an as-of timestamp and has no default for it, the date filter lives inside Postgres rather than in a query someone might edit, and the simulation's database role has no grant on the outcomes table at all. It is checked adversarially. 500 post-resolution documents planted in the corpus and searched at all 180 cutoffs, through the vector, keyword, and rerank paths, return zero, and a positive control proves they come back the moment the filter is lifted. The harder leaks were the honest-looking ones: Wikipedia revisions rendered through today's templates, so a 2025 article carried the 2026 final standings, and news pages dated by what they claimed while the stored text was fetched months later. Both were found and repaired in place, with 200,122 documents re-dated and 340 purged.
Above that sits the simulation. Each actor sees a partial, noisy, lagged view of the world set by its position in the graph, and a pure-Python arbiter with no model call in it folds their decisions into the next state, which is what lets 25 of 25 stored runs reproduce their event-log hash byte for byte in fresh processes. Every model call goes through one door, where the Claude Code CLI, the Anthropic API, Claude Platform on AWS, and Amazon Bedrock sit behind a single interface. Bedrock Rerank is admitted only because it can reorder the time-locked pool and never widen it. Prompt injection through the corpus was measured before it was closed: a planted document posing as the operator was obeyed on 20 of 30 scenarios, and once evidence was quoted in markers it cannot forge, on none.
The event log has a home on AWS: a KMS-encrypted S3 lake of Parquet, catalogued by Glue and queried through Athena, deployed as 21 Terraform resources in us-west-2 with no drift on the plan that followed. Least privilege is tested rather than declared. One command writes three events as the writer role, asks that same role to delete what it just wrote and fails unless AWS refuses, then reads the rows back through Athena as a separate analyst role and checks that they match. The first live run found a defect no mock could have: the analyst role was missing glue:GetPartition. The production tiers stay in the code as switches and are off in the live profile, among them Object Lock, cross-region recovery, GuardDuty, Config, and an Aurora sandbox, so what is deployed has a standing cost of one KMS key.
Then the whole pipeline ran with a model deciding every turn: 36 development scenarios, 360 simulations, 66,231 model calls. Cascade, the simulation engine at its core, scored a Brier of 0.2197, a single model given the same evidence 0.2069, and the base rate 0.2500, and at 36 scenarios none of those differences is distinguishable from zero. The report says exactly that, along with the arithmetic that separating them would take around 2,000 scenarios, and the 125 held-out test scenarios stay unspent. The edges are written down too. The lake holds only the smoke test's synthetic events until the export of the real event log is built, inference runs through the Claude Code CLI, which cannot pin temperature, and the 12-cell ablation grid has not been run.
Architecture
- Scenario ledger
- 180 resolved binary questions from Polymarket, Manifold, and a curated file, selected under hard rules (a 0.50 base rate, no domain above 25%) and sealed behind a SHA-256 manifest that is re-hashed before any label is read. A 40 / 125 development and test split was declared before any forecast existed, and the simulation's database role has no grant on the outcomes table at all.
- Evidence corpus
- 1,998,127 embedded chunks from 317,780 documents across CC-NEWS, Wikipedia, government press releases, and SEC EDGAR, each one dated by when its text was knowable rather than by the date it claims. Wikipedia is read as each revision stood before the cutoff, not through today's templates.
- Chronofence retrieval
- Hybrid retrieval, vector distance fused with a time-locked keyword pool and recency, behind a SECURITY DEFINER function that cannot be called without an as-of timestamp. The cutoff is enforced inside Postgres, so nothing published on or after it can come back through the vector, keyword, or rerank path.
- Lathe and Loom
- A compiler that turns a question into a typed causal graph of actors, objectives, and world factors through draft, critique, and repair, with a validator that rejects graphs restating the outcome; 155 of the 165 scored scenarios compiled. Then a 24-step kernel where each actor sees a partial, noisy, lagged view set by its position in the graph, and a pure-Python arbiter with no model call and no I/O folds their decisions into the next state.
- One model door
- Every model call goes through one module, with the Claude Code CLI, the Anthropic API, Claude Platform on AWS, and Amazon Bedrock behind a single interface, so caching, cost metering, tracing, and replay are written once and switching provider is configuration. Bedrock Rerank may only permute the time-locked pool; Bedrock Guardrails audits compiled graphs after the fact and never edits one. CI fails if a second call site appears.
- Chorus and Assay
- Seeded replicates run in lockstep and are batched one submission per step across the wave, then scored from the definitions: Brier, Murphy decomposition, calibration, mid-rank AUC, and a paired bootstrap against the base rate, a single model given the same evidence, and the market's own price at each cutoff.
- Strata determinism
- An append-only event log, content-addressed record and replay of every model call, and one RNG per run drawn in a fixed order. 25 stored runs re-run in fresh processes reproduce their event-log hash byte for byte, and any outcome walks back to its root cause in well under a second.
- AWS analytics plane
- A KMS-encrypted S3 lake of Parquet events, catalogued by Glue and queried only through an Athena workgroup with enforced encryption and a 10 GiB scan limit. A writer role may append and is explicitly denied every delete; an analyst role may only query. 21 resources in us-west-2 from three Terraform roots and 15 modules, checked offline by 193 mock-provider tests, TFLint, and Checkov.