Case study 05 / DeepChamp, the business side
Running a subscription app on data and AI.
DeepChamp is a live subscription app run by a small team, and every day I need to know what is actually limiting its growth. So I built a warehouse the team can trust, a decision summary that says when it can't be, a weekly constraint engine, and a daily agent that ranks the options with Jev, TypeSafe's calibrated judgment model, and can ship a fix but can't move money on its own.
Structure and method only. No business figures are published, and every value on this page is illustrative.
- 43sources, each with a freshness contract
- 342warehouse definitions: 189 models, 149 assertions, and 4 operations
- 7 daystrailing re-read for purchases that arrive late
- 7weekly constraints, and a met target is never the binding one
- 1binding constraint a week, named with its arithmetic
- 0money moves without the owner's approval
The problem
A small team running a live subscription app has to know, every day, what is really holding growth back.
The answer is spread across Apple, Stripe, the paywall provider, two ad platforms, and an attribution provider, and it is late, partial, and sometimes in conflict. Privacy changes blind channel attribution. Apple restates purchases days after the fact. A reinstall makes an old customer look new. And the person deciding how much to spend is also the person who built the app, so the system has to tell them plainly when a number can't be trusted yet, instead of handing over a confident wrong one.
What made it hard, and how I solved it
Most of the work is deciding when not to trust a number.
- 01
Late and restated data
Purchases reach the warehouse days after they happen, and an attribution feed keeps revising the last few days. Today's total is a draft.
A bounded daily close re-reads a trailing seven-day window every run, idempotent by event key, and a parity model compares the result with Apple's own API and flags any day that should have matured and did not match. A full reconciliation follows.
Today's number is a draft.
- 02
Identity across stores
One person is an App Store transaction, a Stripe customer, a paywall user, and an install, and a reinstall can make an old customer look new.
Identity keys are pseudonymized before they reach the warehouse and joined into one person graph. Purchases are keyed to the first purchase of their original transaction, so restores and pre-install purchases never count as new.
A restore is not a new customer.
- 03
Attribution that can't see everything
Privacy changes hide part of what paid spend drives, so last-touch attribution undercounts and cannot be the only view.
The dashboard shows a spend-response regression next to last-touch, and labels the regression not decision-grade until it has enough clean days to trust.
Two imperfect views beat one confident one.
- 04
The cost of asking
A warehouse that recomputes everything gets expensive without anyone noticing.
Every job has a maximum bytes-billed cap, a bootstrap dry-runs each step and stops if the estimate is over the cap or unknown, and a cheap query finds which partitions to read before the real one runs. Outputs are partitioned, clustered, and incremental where they can be, and a cost table raises an alert.
Estimate first, then spend.
- 05
Trusting numbers enough to move money
A dashboard that shows a zero when a feed is down will happily recommend the wrong thing.
A source-health registry blocks the build when a required source is failed, missing, degraded, or reconciling. Every figure carries an as-of time and goes stale after 36 hours, and anything that cannot be computed shows BLOCKED with the reason.
Never a silent zero.
- 06
Letting an AI near the money
An agent that can measure, decide, and spend is one wrong assumption from an expensive mistake.
Jev ranks and advises. Deterministic gates decide what any budget move is allowed to be. The daily agent can ship one focused code change through a pull request with tests, but a money move waits for the owner's explicit yes.
Advice is cheap. Authority is narrow.
How it works
Trust the data, decide with gates, then act and measure. Each layer has a narrow job, and each can say no.
From raw events to a decision that moves money
Data flows left to right into one summary, is ranked and gated, and comes out as a fix a person approves. What gets measured feeds the next week's ranking.
Ingestion
Read-only connectors write provider envelopes with idempotent merges, checkpoints, and a dead-letter table that stores hashes only. Fast lanes run every ten minutes for payments and attribution and hourly for the rest. A bounded daily close runs over a three-day window each morning and a full reconciliation follows, and late purchases are re-read for seven days. The source-health registry gives every source a status and a freshness target.
The warehouse
Raw provider envelopes feed a core layer (transactions, subscription state, product events, costs, the person graph) and a marts layer built for decisions. 189 models and 149 assertions cover it, and every required source that is not healthy blocks the build. Queries are cheap by design: partitioned and clustered outputs, incremental materializations, dry runs, and a hard billing cap on every job.
The decision summary
One published summary answers what changed and whether to trust it. It carries an integrity status, an as-of time and age on every section, and BLOCKED states when cost data is incomplete: missing spend leaves payback unknown rather than zero. It covers proceeds and renewable revenue week over week, contribution after costs, per-cohort and per-channel economics, and a per-version funnel.
Incrementality, next to last-touch
Last-touch attribution is shown as reported. Beside it, a regression of daily new users on same-day paid spend over the last 14 complete days, with a 28-day robustness fit, gives a baseline, a marginal cost per user, and a range for the return. It is marked not decision-grade under ten days or a weak fit, and the code says it is observational and needs a holdout to confirm.
The constraint engine
Seven constraints are checked every week in a fixed order: checkout completion, paid-close attribution, decision-grade variable costs, Apple settlement, source health, support capacity, and measured learning. The binding one is the highest-ranked constraint that is not ready and still has a gap. A target that is met is never binding: a database assertion and two tests enforce it, and the summary re-selects the next open one if it ever slips.
Jev in the loop
Each candidate action is scored on typed questions: its effect on week-over-week growth, the strength of the evidence, time to impact, production risk, and whether it addresses the binding constraint. Composite weights live in code, and priority is quality times estimated monthly value divided by effort. The ranking is cached on a fingerprint of the candidates, context, questions, and weights, so it re-runs only when an input changes, and an outage shows the last good ranking marked stale.
The decision matrix
Before any budget move, deterministic gates must pass: providers fresh, snapshot integrity passing, enough purchases in the sample, seven-day economics at break-even or better, a total lifetime-loss budget, cash cover, and a dispute rate under its limit. Jev's evidence and posture must also be decisive. It advises, and the gates decide. When its own loss rule says pull back, the verdict follows the rule.
The daily loop
An agent runs every morning. A readiness gate checks freshness, unhealthy sources, schedulers, and query cost first. It then measures each link from spend to install, paywall, conversion, plan mix, renewal, and cash, ranks candidates with Jev, names one binding constraint with the arithmetic, ships at most one focused fix through a pull request with tests, and writes a dated report. An append-only log and a ledger record every experiment with a success metric and the date it will be judged.
Guardrails
Every paywall and lifecycle test has a holdout and is never judged before its date or before enough conversions per arm. Every change carries rollback steps and a before-and-after snapshot. Spend has a floor the owner set. Anything the permission layer refuses is logged as blocked with the manual steps. Money moves need the owner's explicit approval, every time.
Reporting and cash
A publisher writes Google Sheets workbooks straight from the warehouse, and reads every write back to confirm it. A separate finance view sets card balances and due dates against the App Store's payout calendar: proceeds do not count as cash until they are paid out, and the recommended daily spend is the smallest of the economics, production, and cash-cover limits. It is display-only and never writes a budget.
| Figure | Integrity | Value, with its as-of time |
|---|---|---|
| Proceeds, week over week | PASS | +x.x% |
| Renewable revenue, week over week | PASS | +x.x% |
| Contribution after all costs | BLOCKED | n of 7 days without decision-grade costs |
| Return by cohort and channel | WARN | x.x× last-touch, x.x× to x.x× incremental |
| Binding constraint | PASS | one named, with its arithmetic |
Stories from running it
Three times the system was wrong in a way I could only see by running it.
- 01
The app version that only looked best
One version ranked first on day-one value per new user, and the ranking proposed adopting its funnel. The cause was an existing subscriber's purchase, restored on a reinstall and counted as new, plus one outlier in a small sample.
Purchases are now keyed to the first purchase of their original transaction, pre-install purchases and restores are excluded, and small samples are also shown without their top purchase.
Corrected, that version was the worst.
- 02
A HOLD that should have said pull back
The decision matrix showed HOLD while its own lifetime-loss gate was blocked. The verdict could fall back to HOLD even though the policy says to pull back when returns collapse or the loss goes over budget.
The pull-back triggers now turn HOLD into PULL BACK, bounded by the spend floor, with the reason recorded. The matrix also states whether it agrees with the Jev ranking.
The verdict follows the rule.
- 03
Purchases that arrived late
The ledger undercounted recent purchases against Apple. The close read the revenue feed once, for the last day, but that feed attributes events hours or days later.
Every run now re-reads a trailing seven-day window, idempotent by event key, and a parity model flags matured days that still disagree. The dashboard covered the gap from the paywall provider until the first re-read.
Re-read the window, don't trust the edge.
What I'd do next
- Replace the observational regression with a geo or time holdout. The code itself asks for one.
- Calibrate the decision-matrix and policy thresholds against labeled outcomes. They are conservative starting points today.
- Finish cost allocation, so contribution after all costs stops showing BLOCKED.
- Automate the cash view, which still leans on hand-maintained card and payout dates.
- Rank constraints by the size of their gap in money, not by a fixed order.
- BigQuery
- Dataform
- TypeScript
- Python
- Cloud Run
- Google Sheets API
- TypeSafe Jev