Case study 02 / Working system, broadcasting 24/7
Auto DJ: an AI that analyzes, plans, mixes, and grades a set.
Point it at a music library. It measures every track, turns the measurements into words for a calibrated judgment model, plans a set that can never break a harmonic rule, renders each transition the way a DJ would, and grades the result from the audio before anything ships.
Private repository. Walkthrough on request. Architected and shipped by me, directing a team of coding agents.
- 0.3 msmedian beat offset between decks after closed-loop sync, down from 1.5 ms
- 92%key agreement with Mixed In Key within a legal move, on 113 tracks
- 23×real time: a 5.4-hour video graded and rendered in 14.1 minutes on one Spot VM
- 28 to 3transitions at risk of drifting out of sync, first planner versus the loop search
- 0.823mean transition grade over 216 rendered transitions, up from 0.810
- 106tests over about 20,900 lines of Python in 89 modules
The problem
Turn a folder of songs into a set a working DJ would sign off on: every blend in time, in key, and on phrase, with taste, for hours, and proven to sound right.
A DJ makes a hundred small calls a minute on top of millisecond timing: when to bring the next song in, where its bass takes over, which song should come next, and whether the room is still with them. Auto DJ has to make every one of those calls, for a five-hour set, with nobody at the decks.
What made it hard
Six things have to be right at once, and each one breaks the mix on its own.
- 01
Millisecond timing
Kicks more than about 5 ms apart flam. A neural beat tracker lands within tens of milliseconds.
- 02
Harmony
Clashing keys ruin a blend even when the beats are perfect, and single key detectors often disagree.
- 03
The shape of a song
Mixing on phrase means knowing where the verse, build, and drop start, to the bar.
- 04
Taste
Among the technically legal ways to mix two songs, a few feel right. That is hard to write as a rule.
- 05
Order over hours
Hundreds of songs, every neighbour legal, one tempo, and still a shape to the night.
- 06
Knowing it worked
A plan can look right and still flam, clash, or stack two basslines once it is rendered.
How we solved it
One fix per hard part, each owned by a component with a narrow contract and its own test.
- 01
Millisecond timing
Fit one straight-line grid through the tracker's beats, nudge it onto the kick transients, then render each blend, measure it, and correct it until it locks.
Median offset between decks: 0.3 ms.
- 02
Harmony
Three key profiles vote. The Camelot rule (same key, one step, or the relative key) is enforced in the candidate query and checked again on the keys playing in the blend bars.
No song pair is ever planned outside the rule.
- 03
The shape of a song
Segment on 4, 8, and 16-bar phrase lines, find drops where the song gets suddenly louder and fuller, and let Jev label each section.
Cues and blends land on real phrase lines.
- 04
Taste
Jev chooses among options that already pass every rule, and its answer counts for 30%.
Judgment where it helps, never where it can break something.
- 05
Order over hours
Score every legal pair once, seed the loop with the hardest-to-place songs first, then search the whole cycle at once.
Transitions at risk of drifting fell from 28 to 3.
- 06
Knowing it worked
Grade every rendered transition from its own audio. A transition that will not lock within 2 ms is banned and the set is re-planned.
Nothing ships on the plan alone.
How it works
Nine stages, one plan. The same plan drives the audio render, the Rekordbox cues, and the grade, so they can never disagree.
How it works, end to end
Audio becomes measurements, measurements become words for Jev, and hard rules bound every choice. The gold badges mark where Jev makes a decision. A transition that fails the render gate sends its pair back to the planner, never to be used again.
Analyze: a grid you can mix on
A neural beat tracker finds beats and downbeats. Auto DJ fits one straight line through them, snaps the tempo to the whole BPM a producer used, and nudges the grid onto the kick drum. Three key detectors vote on the key. Stem separation says, bar by bar, whether drums, bass, and vocals are playing.
Structure: where the song goes
Sections start on phrase lines, so the segmentation prefers boundaries every 4, 8, and 16 bars. The drop is where the song suddenly gets louder, fuller, and busier. Jev reads the measurements as words and labels each section.
Plan: legal, then good
Candidates come from a query that only returns key-legal, tempo-safe songs. Each legal pair gets a value from key, tempo, phrase fit, and energy, plus Jev's choice. For a five-hour loop, every legal pair is scored once and the whole cycle is searched at once, shaping the night toward a target energy curve.
Render: mix the way a DJ does
Volume, a three-band EQ, and one reverb, nothing more. In the house drop-swap, the outgoing song's bass is gone within a quarter bar, the incoming song's bass arrives on its drop downbeat, so two basslines never play at once. Each deck is rendered, measured against the other, and corrected until the beats lock.
Rendering in the cloud
A five-hour set with video is far too much for a laptop. In the cloud it has to be fast, cheap, and impossible to leave running.
Plan locally, render in the cloud
The laptop plans; one Spot VM, as large as the region will give, mixes, gates, grades, and renders in parallel; the verified master stays in the cloud and only the stream copy comes down to a tiny 24/7 relay. A safety loop makes sure nothing is left running.
- A 5.4-hour video in 14.1 minutesGraded and rendered on one 180-vCPU Spot VM at 23× real time, every one of 587,381 frames accounted for. 33 to 39 minutes end to end.
- A 5.4-hour mix in 8.3 minutesEvery blend within 2 ms and 3 LU, mastered at 48 kHz / 24-bit to −12 LUFS and −1 dBTP.
- About $2.20 a renderAt list price, and about $0.60 of that is the download. The 24 GB master never leaves the cloud.
| Edit list | 7,420 cuts, timed to the music |
|---|---|
| Video chunks | 653, about 30 seconds each |
| Frames | 587,381, checked against the timeline |
| Grade and render | 14.1 min on 180 vCPUs: 23× real time, 7.8 vCPU-seconds per second of video |
| Mix | 8.3 min, then loudness, loop trim, and lossless and MP3 encodes |
| End to end | 33 to 39 min, upload to verified master |
| What comes down | the 5.3 GB stream copy only; the 24 GB master stays in the cloud |
| Cost | about $2.20 at list price, about $0.60 of it download |
| Step | Before: one 32-vCPU VM | Now: up to 180 vCPUs |
|---|---|---|
| Video | 5 h in 45 min (6.7× real time) | 5.4 h in 14.1 min (23× real time) |
| Mix | 5 h in 12 min | 5.4 h in 8.3 min |
| Saved plan to live stream | about 4 h | about 1 to 1.5 h |
One big machine, not a cluster
Chunked ffmpeg on a single Spot VM keeps every intermediate on one local disk and needs no scheduler or container registry, so going faster means asking for a bigger machine. A capacity ladder asks for 180 vCPUs first and steps down through 90, 60, 56, and 32 when the region is out of capacity, waiting on quota instead of failing. The run limit is sized from the measured work, so a stuck VM dies soon after it should have finished.
Cut where nothing changes
Video chunks are about 30 seconds, each one ffmpeg graph encoded once and uploaded the moment it is done; they join without re-encoding and the audio is muxed bit-identically. The mix is cut only where one song plays alone, so the parallel render matches a one-at-a-time render. A 12-song mix rendered in the cloud gave the same sync verdicts, level rides, loudness, and peaks as the laptop.
Do the expensive work once
Each source is color graded once and cached for every chunk that uses it. The stream copy is encoded once, at a constant 2 Mbps with x264's slowest preset, so the 24/7 broadcast only has to copy bytes.
Download is the real cost
The library lives in a same-region bucket with Autoclass, so reads are free and files nobody touches get cheaper on their own. Only the 5.3 GB stream copy comes down; the 24 GB master stays in the cloud.
Fail loudly, resume quietly
A heartbeat reports progress, CPU, and memory; a VM that goes silent for 15 minutes is replaced. If Spot reclaims the VM, a new one skips every chunk already uploaded. Every chunk's frame count is checked, the total must match the timeline exactly, and a blend that misses the gate is swapped and re-rendered automatically.
Nothing left running
VMs delete themselves, and a janitor runs before and after every render. A budget alert triggers a kill switch that deletes every render VM and blocks new renders; scratch storage expires after 14 days, and AI song generation has its own spend caps. Rendering runs in its own project with narrow custom roles, and the toolchain is pinned and checksum-verified.
The 24/7 broadcast
A free-tier e2-micro VM relays the finished stream copy to YouTube and Twitch around the clock. Because nothing is re-encoded, the smallest machine is enough, and the whole broadcast costs about $40 a month, almost all of it egress. An ingest checker confirms each platform is actually receiving, and a watchdog restarts the relay if it stops.
Saved plan to live, in about an hour
Going from a saved plan to a live stream used to take about four hours on 32 cores. It now takes about one to one and a half.
| Machine | vCPUs | 20 min of video in | Speed | Per hour of video |
|---|---|---|---|---|
| c3d-highcpu-180 | 180 | 45 s | 26.5× real time | $0.062 |
| c2d-highcpu-32 | 32 | 3.4 min | 6.0× real time | $0.103 |
| n4-highcpu-32 | 32 | 3.7 min | 5.4× real time | $0.129 |
| c4d-highcpu-16 | 16 | 4.5 min | 4.5× real time | $0.057 |
| c3d-highcpu-16 | 16 | 6.0 min | 3.3× real time | $0.044 |
How a mix gets graded
Two grades, for two decisions: one picks the set before anything is rendered, and one decides what ships after.
How a mix gets graded
Two grades. Before rendering, every legal pair gets a value that decides the set order: code-owned signals, Jev's choices, and the transition's own rule score, minus a risk term. After rendering, every transition is graded from its own audio, and the gate decides whether it ships.
Before rendering
Every legal pair gets a value. Code scores what it can measure: how smooth the key move is, how far the next song is stretched, whether the verses line up and the mix lands on a build, and how close the next song sits to the night's target energy. Jev adds the taste: which legal song should come next, and whether each run of three songs flows. The transition's own rule score and a sync-risk penalty finish it, and the loop search picks the order with the best total.
After rendering
Each transition is graded from the audio that was made: four bars of the outgoing song, the blend, and four bars of the incoming song into its drop. Beat lock and harmony weigh the most. Anything that fails the gate is banned from future plans and the set is planned again. Across 216 transitions, the whole-loop plan averaged 0.823, against 0.810 for planning one song at a time.
One thing we tested and dropped: asking Jev to re-score these measured facts added nothing. Its strength is judgment from words, so grading the audio stays in code.
The Jev decision system
Jev, TypeSafe's calibrated judgment model, makes the taste calls. It never sees an option the rules would reject, and low confidence never acts on its own.
Where Jev decides
Four decision points, each a typed question over options the rules already allow. Every answer comes back as probabilities with a confidence, and code decides how much it counts.
| Decision | Typed question | How the answer is used | When it is not trusted |
|---|---|---|---|
| Section labels | Choice among nine section types, plus yes-or-no: does the drop kick in here, does this lead into it | Pooled with the rule-based label: | A cue section under 0.2 confidence goes to a Needs review playlist |
| Key tie-break | Choice among the detectors' own candidates, never a free answer | Overrides the vote | Only at 0.6 confidence or more |
| Next song | Choice among the top 24 legal next songs | 30% of the pair value, next to 55% code and a lookahead | On any error the planner uses its own score |
| Flow of three | Score: is this three-song run weak, fine, or great | Steers five rounds of loop search away from weak runs | Unrated runs take the average |
| Transition window | Choice among up to four legal windows within reach of the best | 30% of the pick; the rule score keeps 70% | Fewer than two options or any error: the rule pick stands |
How confidence gates a decision
Measurements go in as words, Jev answers typed questions with probabilities and a confidence, and code decides what that answer may change. Low confidence never acts on its own: it falls back to the rule answer, sends the track to review, or leaves the plan as it was.
- Rules first. Every option Jev sees already passes the key, tempo, bass, and sync rules.
- Code owns the weights.Jev's share is 30% or pooled at 70/30, and it is set in code, not by the model.
- Proven by test. A property test swaps in a random critic and checks that every hard guarantee still holds.
- Cached and pinned. Every question and answer is cached by content, so a re-plan never pays twice.
Results
Mixes that lock, sets that stay in key, and a system that plays and broadcasts around the clock.
- A mastered mixLoudness matched to -12 LUFS under a -1 dBTP true-peak limiter, every blend locked or re-planned.
- Rekordbox cuesHot cues A to H and 8 and 4-bar countdowns to each drop, so a human can play the same set by hand.
- A 24/7 broadcastPre-rendered mixes loop to YouTube and Twitch behind ingest checks and a watchdog.
The songs can come from my AI music pipeline, which generates and quality-gates them before they are ever analyzed.