
ReadWithMe(experiment)
Synchronized audiobook player × ebook reader — same place in both.

Synchronized audiobook player × ebook reader — same place in both.
Player above is Alice’s Adventures in Wonderland: the Standard Ebooks text with two LibriVox recordings, a solo reading and a full-cast dramatic reading, both force-aligned. Switch narrations; the sentence address stays put.
Syncs an audiobook to a real EPUB with its formatting, so text scrolls and
highlights sentence-by-sentence with narration. Every sentence has two
coordinates: a text address (book, chapter, paragraph, sentence) stable
across editions, and a time span on one audio file. Alignment produces the
second from the first. Scroll-sync, deep links, switching editions in place and
playlists are all lookups once that table exists.
Abridgments cut text, narrators ad-lib, and rips carry intros and credits, so
alignment is a full outer join. A sentence with no audio is unvoiced;
audio with no sentence is unmatchedAudio. Both are rows.
Each chapter is aligned three ways, all with MMS forced alignment (TorchAudio CTC). They differ in what pins the text to the audio.
Two clean-up passes follow. close-gaps pulls a low-confidence start that sits
more than a second after the previous sentence’s end back to that end.
onset reads the audio and pulls a start back to where uncovered speech begins.
The final start is Whisper’s, after onset. Where Whisper sits more than 0.5 s
from rigid, and sparse agrees with rigid within a second, the row takes rigid’s
start and end instead: two independent joins outvote the third.
Rigid alone is accurate on a single narrator and fails on full-cast productions, which drop dialogue tags the text still has. A rigid join must spend audio on every word, so one unspoken tag pushes the following lines late. Anchored passes contain the same error between two recognized words.
Each gold set is one chapter whose sentence starts were checked by hand: the labeler corrected the wrong starts and accepted the rest. A start counts as correct within 0.5 s; a sentence the run fails to place counts as wrong. The gold chapters come from commercial audiobooks, so they are described here, not named.
| Chapter | Rigid | Vote | Rigid worst | Vote worst |
|---|---|---|---|---|
| Novel A, full cast, ch 31 | 93.5% | 97.1% | 39.0 s | 4.1 s |
| Novel A, full cast, ch 1 | 91.2% | 98.0% | 45.6 s | 3.2 s |
| Novel B, single narrator, ch 12 | 99.0% | 99.0% | 12.7 s | 2.5 s |
Accepted rows were shown against one run’s guess, so they lean towards that
run; the board reports explicitly corrected rows apart from accepted ones.
align eval keeps each (gold set, variant) run with a stamp of its flags, text
hash and code version, and re-runs only what went stale.
Where a start is still wrong, the reader opens a waveform over the audio bar. Each sentence is a region on it. Tab moves to the next sentence, the arrow keys step an edge to the next pause, and an edge can be dragged; each edit plays back at once and can be undone.
Every alignment embeds a content hash of the text it aligned against.
align qa runs after each alignment and checks the hash, monotonic order,
duration sanity and speaking-rate residuals.
When residuals look wrong, align doctor drops distinctive-sentence probes
(chosen by corpus IDF) across the chapter, places each one independently against
cached emissions, and reads Δ = found − expected.
Placing a probe means CTC in a window with no anchor, and there Viterbi spreads
a short target across lookalike fragments and reports a great score for it: a
38-character sentence placed across 1,537 seconds at score 0.95. The probe
search (scan_range) slides subwindows of 2.5× the expected duration D at
0.5×D stride, rejects placements longer than 3×D, and keeps the best
valid score.
The pattern of deltas names the failure.
Alignments are content-addressed sidecars: coordinates only. Audio and book text stay with the user.