Skip to main content
Harshwardhan Patil

Projects

Systems that show their working.

Four builds where the interesting part isn't the model. It's everything around it that makes the output checkable. Numbers below are measured, and where something is built but unrun, or specified but unbuilt, it says so.

Hindcast: Multi-Agent Causal Simulation

Case study 01

A multi-agent forecasting system that cannot read the future, replays byte for byte, and lands its event log in a live, Terraform-managed AWS lake.

Milestones M0 through M17 shipped · live on AWS in us-west-2 · first full study run

Co-built with Utkarsh Singh

Python 3.12PostgreSQL 16 + pgvectorTerraformAWS S3 / Glue / AthenaAWS KMS + IAMAmazon BedrockClaude Code CLILangGraphbge-small-en-v1.5LangfuseDocker Composepytest + HypothesisCheckov + TFLintGitHub Actions

Ask a model whether a regulator will clear a merger before the deadline and it will hand you a probability. It will not tell you who has to act, what each of them wants, or what each of them could actually see, and you cannot replay the answer or prove it was not quietly informed by what happened next. Hindcast models the why instead. It compiles a question into a typed causal graph of actors and forces, simulates it with LLM agents over 24 steps from many seeds, and scores the spread of outcomes against what really happened. The name is the method: a hindcast is a forecast made for a period that has already happened, using only what was known before it, then checked against the record.

Every backtested forecaster has the same credibility problem: from the outside you cannot tell whether it was clever or whether it saw the answer. So retrieval runs through a function that takes an as-of timestamp and has no default for it, the date filter lives inside Postgres rather than in a query someone might edit, and the simulation's database role has no grant on the outcomes table at all. It is checked adversarially. 500 post-resolution documents planted in the corpus and searched at all 180 cutoffs, through the vector, keyword, and rerank paths, return zero, and a positive control proves they come back the moment the filter is lifted. The harder leaks were the honest-looking ones: Wikipedia revisions rendered through today's templates, so a 2025 article carried the 2026 final standings, and news pages dated by what they claimed while the stored text was fetched months later. Both were found and repaired in place, with 200,122 documents re-dated and 340 purged.

Above that sits the simulation. Each actor sees a partial, noisy, lagged view of the world set by its position in the graph, and a pure-Python arbiter with no model call in it folds their decisions into the next state, which is what lets 25 of 25 stored runs reproduce their event-log hash byte for byte in fresh processes. Every model call goes through one door, where the Claude Code CLI, the Anthropic API, Claude Platform on AWS, and Amazon Bedrock sit behind a single interface. Bedrock Rerank is admitted only because it can reorder the time-locked pool and never widen it. Prompt injection through the corpus was measured before it was closed: a planted document posing as the operator was obeyed on 20 of 30 scenarios, and once evidence was quoted in markers it cannot forge, on none.

The event log has a home on AWS: a KMS-encrypted S3 lake of Parquet, catalogued by Glue and queried through Athena, deployed as 21 Terraform resources in us-west-2 with no drift on the plan that followed. Least privilege is tested rather than declared. One command writes three events as the writer role, asks that same role to delete what it just wrote and fails unless AWS refuses, then reads the rows back through Athena as a separate analyst role and checks that they match. The first live run found a defect no mock could have: the analyst role was missing glue:GetPartition. The production tiers stay in the code as switches and are off in the live profile, among them Object Lock, cross-region recovery, GuardDuty, Config, and an Aurora sandbox, so what is deployed has a standing cost of one KMS key.

Then the whole pipeline ran with a model deciding every turn: 36 development scenarios, 360 simulations, 66,231 model calls. Cascade, the simulation engine at its core, scored a Brier of 0.2197, a single model given the same evidence 0.2069, and the base rate 0.2500, and at 36 scenarios none of those differences is distinguishable from zero. The report says exactly that, along with the arithmetic that separating them would take around 2,000 scenarios, and the 125 held-out test scenarios stay unspent. The edges are written down too. The lake holds only the smoke test's synthetic events until the export of the real event log is built, inference runs through the Claude Code CLI, which cannot pin temperature, and the 12-cell ablation grid has not been run.

2,533tests passing in CI2.0Mtime-locked chunks over 317,780 documents0 / 500planted future documents retrieved25 / 25runs replay byte-identically21AWS resources under Terraform, zero drift3 / 3rows round-tripped through S3, Glue and Athena1,027Checkov checks passed, 0 failed66,231model calls in the first full study
Hindcast simulation architectureA question passes through four stages in sequence: Chronofence for time-locked retrieval, a Compiler that turns the question and its evidence into a typed causal graph, per-actor Agents operating under information asymmetry, and a pure-Python Arbiter that resolves each step. Chronofence is fed by a PostgreSQL and pgvector corpus holding only pre-cutoff evidence. The Compiler and the Agents reach models through one door that fronts four providers and Amazon Bedrock. The Arbiter returns control to the Agents for each of the 24 simulation steps, and its event log lands in a live AWS lake of S3, Glue, and Athena.Chronofencetime-locked retrievalCompilerquestion → causal graphAgentsact under asymmetryArbiterresolves, no LLMPostgreSQL + pgvectorpre-cutoff evidence onlyOne model door4 providers + BedrockAWS event lakeS3 · Glue · Athena, liveevent log24-step loop
Every stage here has run with a model deciding every turn, and the arbiter is deliberately the one with no model call in it. The AWS lake is deployed and its round trip is proven with three synthetic events; exporting the real event log into it is the next step.

Architecture

Scenario ledger
180 resolved binary questions from Polymarket, Manifold, and a curated file, selected under hard rules (a 0.50 base rate, no domain above 25%) and sealed behind a SHA-256 manifest that is re-hashed before any label is read. A 40 / 125 development and test split was declared before any forecast existed, and the simulation's database role has no grant on the outcomes table at all.
Evidence corpus
1,998,127 embedded chunks from 317,780 documents across CC-NEWS, Wikipedia, government press releases, and SEC EDGAR, each one dated by when its text was knowable rather than by the date it claims. Wikipedia is read as each revision stood before the cutoff, not through today's templates.
Chronofence retrieval
Hybrid retrieval, vector distance fused with a time-locked keyword pool and recency, behind a SECURITY DEFINER function that cannot be called without an as-of timestamp. The cutoff is enforced inside Postgres, so nothing published on or after it can come back through the vector, keyword, or rerank path.
Lathe and Loom
A compiler that turns a question into a typed causal graph of actors, objectives, and world factors through draft, critique, and repair, with a validator that rejects graphs restating the outcome; 155 of the 165 scored scenarios compiled. Then a 24-step kernel where each actor sees a partial, noisy, lagged view set by its position in the graph, and a pure-Python arbiter with no model call and no I/O folds their decisions into the next state.
One model door
Every model call goes through one module, with the Claude Code CLI, the Anthropic API, Claude Platform on AWS, and Amazon Bedrock behind a single interface, so caching, cost metering, tracing, and replay are written once and switching provider is configuration. Bedrock Rerank may only permute the time-locked pool; Bedrock Guardrails audits compiled graphs after the fact and never edits one. CI fails if a second call site appears.
Chorus and Assay
Seeded replicates run in lockstep and are batched one submission per step across the wave, then scored from the definitions: Brier, Murphy decomposition, calibration, mid-rank AUC, and a paired bootstrap against the base rate, a single model given the same evidence, and the market's own price at each cutoff.
Strata determinism
An append-only event log, content-addressed record and replay of every model call, and one RNG per run drawn in a fixed order. 25 stored runs re-run in fresh processes reproduce their event-log hash byte for byte, and any outcome walks back to its root cause in well under a second.
AWS analytics plane
A KMS-encrypted S3 lake of Parquet events, catalogued by Glue and queried only through an Athena workgroup with enforced encryption and a 10 GiB scan limit. A writer role may append and is explicitly denied every delete; an analyst role may only query. 21 resources in us-west-2 from three Terraform roots and 15 modules, checked offline by 193 mock-provider tests, TFLint, and Checkov.

Pipewright: Data Pipeline Platform

Case study 02

The operational half of data work, from connector to lineage to write-back, as one product instead of nine scripts.

Sixteen phases shipped across two roadmaps · running locally end to end

Next.js 16React 19TypeScriptFastAPISQLAlchemy 2.0AlembicPostgreSQLpandas + pyarrowDocker ComposepytestVitestGitHub Actions

Most of what makes data work hard is not the transformation. It is everything around it: where did this column come from, which run produced this table, did the schema change under us last Tuesday, who gets told when the publish fails. That work usually lives in a drawer of disconnected scripts and a spreadsheet nobody trusts. Pipewright is an attempt to put it in one place and treat it as product surface rather than plumbing.

The bet the platform is built on is that analysts export to Excel because Excel lets them see and touch the data, and engineers hate the export because it forks the truth. So the grid edits like a spreadsheet, on millions of rows, but every cell edit is a step in a recipe rather than a mutation. Underneath it is a canonical type lattice and a relational IR with pandas and SQL backends, proven identical over a differential corpus, which is what lets a spreadsheet formula get type inference, column lineage, and pushdown to the source without being a special case. The planner splits a query tree into what the source can run and what the platform runs, and refuses to approximate that boundary.

Getting data in is its own problem, and the part I am most attached to. File reading is a pipeline where every stage reports what it decided, how confident it is, and the evidence behind it; a stage that genuinely cannot tell whether 03/04/2025 is March or April returns ambiguous and blocks instead of picking one. Each decision is stored against the file's column fingerprint, so next month's version of that file reads the same way rather than being inferred again from different rows. A 72-file corpus of deliberately awful spreadsheets backs all of it, and it found seven real bugs, including every import silently truncating at 5,000 rows and identifiers losing their leading zeros.

The same honesty rule runs through the connector catalogue. There are 211 connectors, which is a meaningless number on its own, so every one carries a verification tier, its verified_by field cites a test file, and a test checks that the citation resolves and that the named file actually drives that connector. Fifty are earned against real systems. The other 161 were written from vendor documentation and never executed here, and they say so on the card, in the config form, in the connection test, and in the warnings attached to a run. Nine engines the platform genuinely cannot drive each name the interface it can read instead.

Structurally it is a modular monolith: one FastAPI gateway is the only public deployable, and 24 domain packages sit behind it with real boundaries, cross-service calls going through registered hooks rather than imports. Permission is decided in exactly one place for the whole API and fails closed on an unrecognised write path. The repository is deliberate about its edges, and the handoff document keeps a list of them: write-back has only been run against SQLite because there is no server here for the Postgres and MySQL paths, notebook Python cells are disabled on macOS because the sandbox probe cannot enforce a memory limit, and streaming and CDC are a different execution model rather than another line in the catalogue.

24bounded service packages211connectors, 50 verified against real systems173transformation tools in the library5,943Python tests652web tests30forward-only migrations

Architecture

Studio
A Next.js App Router workspace where the grid edits like a spreadsheet: two-axis virtualisation, Excel keybindings, clipboard, column profiling, and a formula engine that compiles to the same intermediate representation everything else runs on, so a typed formula gets lineage and pushdown for free.
Type lattice and IR
A canonical type system and a relational IR with pandas and SQL backends, proven identical over a differential corpus, plus a pushdown planner that splits a query tree into what the source can run and what the platform runs, and refuses to approximate the boundary.
Ingestion intelligence
File reading as a pipeline where every stage reports what it decided, how confident it is, and the evidence behind it. A stage that genuinely cannot decide returns ambiguous and blocks rather than defaulting. Each decision is saved against the file's column fingerprint, so next month's file reads the same way instead of being re-inferred from different data.
Connector catalogue
211 connectors from four generators, each carrying a verification tier where verified_by cites a test file and a test checks that the citation resolves and that the file actually drives that connector. 50 are earned against real systems; the other 161 say so on the card, in the config form, in the connection test, and in the run's warnings.
Domain services
24 Python packages with real boundaries behind a single FastAPI gateway: workflows, lineage, observability, access, governance, connectors, reporting, intelligence, enterprise, quality, write-back and the rest. Cross-service calls go through registered hooks rather than imports, and permission is decided in exactly one place for the whole API.
Write-back and operations
Row identity resolution, staged change sets, a dry run inside a rolled-back transaction, a blast-radius guard, and migration export. Around it sit Postgres-leased schedules, incident grouping, notifications, a system-status surface, and a GitHub Actions pipeline that runs exactly the same verification path as the local make verify.

Metric Movement Diagnostics Platform

Case study 03

Answers the question every dashboard raises and none of them answer: why did that number move?

Shipped · validated against a known +52.1% spike

PythonPostgreSQLClaude APIBlock BootstrapStreamlitPlotlypytestPlaywright

A dashboard tells you revenue jumped 52.1% month over month. It will never tell you why, and the analyst who has to explain it by Monday spends the weekend slicing the same table by hand. This platform does that investigation.

Underneath is a PostgreSQL star schema over 112,000-plus transactions, which is what makes systematic slicing possible in the first place. On top of it runs a block bootstrap at 2,000 iterations, resampling in blocks so temporal structure survives, to establish which apparent drivers are real and which are variance wearing a convincing costume. I validated the whole approach against that +52.1% month-over-month spike specifically, because a diagnostic tool that cannot explain a movement you already understand has no business explaining one you do not.

The reasoning layer is a three-agent Claude pipeline: an Investigator that proposes candidate explanations, an Analyst that tests them against the statistical evidence, and a Critic that fact-checks every claim against the source data before a human ever sees it. Every hand-off is JSON-schema constrained, so the stages compose instead of drifting, and the agents sit behind a hard boundary where aggregates cross and raw records and PII never do. Correctness is enforced by 63 pytest tests, and the Streamlit and Plotly interface is verified end to end with Playwright.

112K+transactions in the star schema2,000bootstrap iterations per diagnosis+52.1%MoM spike used as the validation case63automated tests

Architecture

Warehouse
A PostgreSQL star schema over 112,000-plus transactions, modelled so a metric can be sliced systematically along every dimension rather than whichever ones someone thought to index.
Statistical engine
A block bootstrap at 2,000 iterations. Resampling in blocks preserves temporal structure, which is what separates a driver that is real from one that is variance in a convincing costume.
Reasoning pipeline
Three Claude agents in sequence: an Investigator proposing candidate explanations, an Analyst testing them against the statistical evidence, and a Critic that fact-checks every claim against source data before a human sees it. Hand-offs are JSON-schema constrained so stages compose instead of drifting.
Data boundary
Aggregates cross into the model context; raw records and PII never do. The boundary is a property of the pipeline, not a prompt instruction.
Interface
A Streamlit and Plotly application, verified end to end with Playwright and backed by 63 pytest tests.

Credit Card Fraud Decisioning System

Case study 04

Fraud modelling stops at a score. Fraud operations start there, so this one is built around the queue rather than the AUC.

Complete · eight-stage pipeline with a Streamlit review surface

PythonScikit-learnPandasJupyterStreamlitMatplotlib

A fraud model that is 99% accurate is not an achievement, it is a description of the class balance. What a fraud team actually needs to know is different: how many transactions land in the review queue tomorrow, whether the analysts can clear them, and which mistakes cost the most. So this project treats fraud as a decisioning problem and the classifier as one component inside it.

The pipeline runs in eight stages, each with a job it does not share. A data audit validates schemas and compares train/test structure before any feature exists, because a structural mismatch discovered during modelling gets absorbed instead of fixed. Feature engineering builds transaction, customer, merchant, temporal, and behavioural signals with preprocessing kept deliberately separate from the model fit, which is the boundary that keeps leakage out. Modelling compares supervised baselines against stronger approaches and anomaly-based components in a champion/challenger frame.

Then it stops being a modelling project. Scores are converted into a three-band operating policy of approve, review, and high-risk, chosen by evaluating review volume against capture rate, which is a capacity question rather than a statistical one. Error analysis looks specifically at where the system fails and which segments generate operational drag. The Streamlit dashboard is built for the person running the queue: threshold grids and policy comparison, analyst capacity simulation, alert prioritisation, false-positive and false-negative breakdowns, and diagnostics on the transactions sitting right at the threshold edge, where the policy is actually decided.

Architecture

Data audit
Schema validation, missingness inspection, and train/test structure comparison before a single feature is built. This is the stage that catches the problems modelling would otherwise absorb.
Feature engineering
Transaction, customer, merchant, temporal, and behavioural signals, kept leak-safe by holding preprocessing and modelling responsibilities apart rather than blurring them into one fit.
Modelling
Supervised baselines against more advanced approaches plus anomaly-based components, framed as champion/challenger so a replacement has to earn the slot.
Threshold strategy
Scores converted into an operating policy: approve, review, and high-risk bands chosen against review volume, capture trade-offs, and analyst capacity rather than a single F1-optimal cut.
Error analysis
Where the system fails and who pays for it: false positives, false negatives, edge cases, and the segments that quietly create operational drag.
Review surface
A Streamlit dashboard for KPI and trend monitoring, threshold grids, analyst capacity simulation, model comparison, alert-queue prioritisation, and error concentration.

More on GitHub

The repositories behind the case studies above, plus smaller experiments and analysis notebooks, live on GitHub.

github.com/harshkvpatil98