An end-to-end data product built the way I build them in production: ingestion from messy sources, entity resolution, a provenance-tracked warehouse layer, quality gates, and an application on top. The domain below is synthetic airport slot allocation — a structural stand-in for the regulated-asset platform I run day to day.
Four things that are hard to show on a CV, shown here instead.
Reads a generated JSON extract; no backend. Two independent sources report some of the same metrics and are plotted side by side — a deliberate design choice explained in ADR-002.
| Carrier | Group | Slots | Share | Revenue | EBITDA margin | Load factor | Annual fee |
|---|
The demo pipeline in etl/pipeline_demo.py is a
reduced version of one I operate daily. Every property below is deliberate
and exercised by the code.
python data/build_demo_db.py python etl/pipeline_demo.py --dry-run python etl/pipeline_demo.py python etl/pipeline_demo.py --simulate-outage python etl/news_updater.py
[DEGR] change_detect upstream unreachable;
last-good data retained
source degraded — exiting cleanly
Reconciling third-party names to an internal master is the unglamorous work that decides whether every downstream number is right. The demo warehouse ships deliberately messy aliases.
| Tier | Rule | Action |
|---|---|---|
| 1 exact | canonical / legal / known alias, same market | auto-link |
| 2 fuzzy | normalised score ≥ threshold, same market | link + log for review |
| 3 unmatched | no confident match | kept unlinked, never guessed |
collision: ARL / "Aurora Air" keep Aurora Air (1.00) demote Aurora Airways (0.86) → unlinked
The holdings view. Time of day runs left to right, and every block is capacity a carrier actually holds in that window. Unallocated capacity is drawn rather than omitted, because a gap in the plan is information: it is where the next award has room.
Every award, with the detail folded away. Click a row to open it. Summary tables that hide their provenance are how a wrong number survives review, so the expansion carries the assumptions, the source reference and the arithmetic rather than a prettier restatement of the row.
| Award | Airport | Year | Mechanism | Slots | Price / slot | Proceeds |
|---|
Two halves of the same idea. The scheduled job polls with ETag revalidation and writes a snapshot; the cards below are fetched live in your browser right now, and fall back to that snapshot if the request fails. Source is Wikipedia’s public REST API, openly licensed and deliberately unrelated to any client domain.
Descriptions only — these are client systems, so no code, data, schema or identifiers are reproduced.
Subscription web platform serving structured market data to enterprise users. ~34k LOC Python, 40+ routes, two databases with a one-way additive sync, 15 SQL views.
Change-detecting daily refresh across a commercial API and 19 public regulator sources, with per-source caching, selective detail fetch, downstream sync, freshness snapshots and provenance manifests.
Consolidated seven commercial and public sources into ~470k observations spanning 300+ entities and 100+ markets, including forecast series extracted from 51 unstructured PDF reports.
NL→SQL over a 20-table schema with retrieval across a full-text corpus, a curated exemplar library, and a regression harness of 87 cases with automated scoring and an error-pattern taxonomy.
Simulators across five national markets implementing ascending-clock, sealed-bid, hybrid and combinatorial formats — activity rules, eligibility points, bid increments, winner determination.
Winner-determination and assignment-stage optimiser producing per-participant distributions across candidate packages, with audit output for every round.
build_demo_db.py from a seeded
RNG. No real figures, entities or geographies.