← Experiments

ReadWithMe(experiment)

Synchronized audiobook player × ebook reader — same place in both.

Player above is Alice’s Adventures in Wonderland: the Standard Ebooks text with two LibriVox recordings, a solo reading and a full-cast dramatic reading, both force-aligned. Switch narrations; the sentence address stays put.

Syncs an audiobook to a real EPUB with its formatting, so text scrolls and highlights sentence-by-sentence with narration. Every sentence has two coordinates: a text address (book, chapter, paragraph, sentence) stable across editions, and a time span on one audio file. Alignment produces the second from the first. Scroll-sync, deep links, switching editions in place and playlists are all lookups once that table exists.

The join

Abridgments cut text, narrators ad-lib, and rips carry intros and credits, so alignment is a full outer join. A sentence with no audio is unvoiced; audio with no sentence is unmatchedAudio. Both are rows.

s1s2s3s4s5introa1a2a3a4unvoicedunmatchedAudio
Figure 1 Alignment as a full outer join. Text with no audio is unvoiced; audio with no text (a narrator intro) is unmatchedAudio. Both are rows.

Three passes

Each chapter is aligned three ways, all with MMS forced alignment (TorchAudio CTC). They differ in what pins the text to the audio.

  • Rigid walks the chapter in 300-word windows with a 60-word lookahead, each anchored at the end of the last placed sentence. A running estimate of frames per character sizes the next window, and a clamp forbids starting before the anchor.
  • Sparse runs a Wav2Vec2 recognizer over the audio first, matches its words to the text with a banded fuzzy alignment, and force-aligns only between those anchors.
  • Whisper is the same join with anchors from Whisper word timestamps.

Two clean-up passes follow. close-gaps pulls a low-confidence start that sits more than a second after the previous sentence’s end back to that end. onset reads the audio and pulls a start back to where uncovered speech begins.

chapteraudio +textsparsewhisperrigidclose-gapsonsetvotesidecar
Figure 2 Each chapter is aligned three ways. Two of the passes get a clean-up post-pass; the vote picks each sentence start from them.

The vote

The final start is Whisper’s, after onset. Where Whisper sits more than 0.5 s from rigid, and sparse agrees with rigid within a second, the row takes rigid’s start and end instead: two independent joins outvote the third.

wrstimekeep w(a)
wrstimetake r(b)
Figure 3 The vote on one sentence start. (a) The passes roughly agree: keep the Whisper onset. (b) Whisper sits away from rigid and sparse backs rigid within 1 s: take rigid start and end.

Rigid alone is accurate on a single narrator and fails on full-cast productions, which drop dialogue tags the text still has. A rigid join must spend audio on every word, so one unspoken tag pushes the following lines late. Anchored passes contain the same error between two recognized words.

Goaway,saidAlice.Go awaynext linenext linelate(a)
Goaway,saidAlice.Go awaynext lineanchoranchor(b)
Figure 4 A full-cast edition drops the dialogue tag. (a) A rigid join places every word in order, so the tag pushes the next line late. (b) Anchors on recognized words pin both sides, so the error stays between two anchors.

Measured

Each gold set is one chapter whose sentence starts were checked by hand: the labeler corrected the wrong starts and accepted the rest. A start counts as correct within 0.5 s; a sentence the run fails to place counts as wrong. The gold chapters come from commercial audiobooks, so they are described here, not named.

Chapter Rigid Vote Rigid worst Vote worst
Novel A, full cast, ch 31 93.5% 97.1% 39.0 s 4.1 s
Novel A, full cast, ch 1 91.2% 98.0% 45.6 s 3.2 s
Novel B, single narrator, ch 12 99.0% 99.0% 12.7 s 2.5 s

Accepted rows were shown against one run’s guess, so they lean towards that run; the board reports explicitly corrected rows apart from accepted ones. align eval keeps each (gold set, variant) run with a stamp of its flags, text hash and code version, and re-runs only what went stale.

Correcting by hand

Where a start is still wrong, the reader opens a waveform over the audio bar. Each sentence is a region on it. Tab moves to the next sentence, the arrow keys step an edge to the next pause, and an edge can be dragged; each edit plays back at once and can be undone.

The sync editor on chapter 1 of Alice: a waveform with one region per sentence, a start stepped to a pause and an end edge dragged

Drift and repair

Every alignment embeds a content hash of the text it aligned against. align qa runs after each alignment and checks the hash, monotonic order, duration sanity and speaking-rate residuals.

When residuals look wrong, align doctor drops distinctive-sentence probes (chosen by corpus IDF) across the chapter, places each one independently against cached emissions, and reads Δ = found − expected.

Placing a probe means CTC in a window with no anchor, and there Viterbi spreads a short target across lookalike fragments and reports a great score for it: a 38-character sentence placed across 1,537 seconds at score 0.95. The probe search (scan_range) slides subwindows of 2.5× the expected duration D at 0.5×D stride, rejects placements longer than 3×D, and keeps the best valid score.

audio range, about 1500 sscore 0.95, 1537 s(a)
same range2.5DD(b)
Figure 5 CTC in an unanchored window. (a) Viterbi spreads a short sentence across lookalike fragments and still scores well. (b) Sliding subwindows of 2.5 times the expected duration, rejecting spans over 3 times, keep the placement tight.

The pattern of deltas names the failure.

chapter time0(a) shift
chapter time0(b) drift
chapter time0(c) missing audio
Figure 6 What the doctor reads from distinctive-sentence probes: (a) a constant shift, (b) rate drift, (c) a missing stretch of audio.

Sidecars

Alignments are content-addressed sidecars: coordinates only. Audio and book text stay with the user.