commit de1f5468b20d2a497f00f7de244d084b431866b4 truex <truex@equwal.com> 2026-09-25 02:15:08 -0700 Follow the audiobook through the SubRead suite The reader reads every page of a PDF as it opens: the text layer first, else OCR, the current page and its neighbors before the rest, with the tokens cached. The cues are aligned again as pages finish. The clock of the player drives the follower: highlight and turn pages, turn pages only, or off. A tap on a word sends the text to the dictionary app and pauses the player until the dictionary closes. On Android the menu asks SubRead to make the subtitles from the audiobook and the book text, keeps the result for the PDF, and shares it as a file. The web keeps an audio element as its clock. The README describes the new scope. The LICENSE file holds the AGPL-3.0 text, the same as the other members of the suite.
README.md | 211 +++++++++++++++++----- index.html | 81 +++++++-- src/main.ts | 557 ++++++++++++++++++++++++++++++++++++++++++++++------------ src/style.css | 60 ++++++- 4 files changed, 736 insertions(+), 173 deletions(-)
diff --git a/README.md b/README.md index 30f4e32..24ee73b 100644 --- a/README.md +++ b/README.md @@ -1,27 +1,95 @@ # Scan Subread -A reader for scanned books. It runs in a browser and as an Android app. - -You load a scanned PDF (page images, no text layer). The app renders the -pages and runs OCR to get a box for each character. Load a subtitle file -(SRT or WebVTT) and an audio file: the app aligns each subtitle cue to -the OCR text and highlights the region of the page that shows the -current cue while the audio plays. Tap a cue in the list to seek the -audio. Tap a word on the page: the status line shows the text under the -tap. - -## Use +A PDF reader that follows the audiobook. It works with scans and with +text PDFs, in a browser and as an Android app, and it is a member of the +SubRead suite: + +- [SubRead Overlay](https://github.com/equwal/subread-overlay) reads the + position of the audiobook player and gives it to this app. Install it + and allow notification access. +- [SubRead Dictionary](https://github.com/equwal/subread-dictionary), or + any app in the text selection menu (Takoboto, AnkiDroid, a translator), + looks the words up. +- [SubRead](https://github.com/equwal/subread-android) makes the subtitle + file (`.srt`) from the audiobook and the text of the book. + +You load a PDF and the `.srt` of its audiobook. The app reads the text +of each page, aligns each subtitle cue to it, and follows the player: it +marks the line that the narrator reads and turns the page. Tap a word, +and the text from that word to the end of the line goes to the +dictionary of your choice. The player pauses while the dictionary is +open. + +## How to use it 1. Open the menu (the "Menu" button on a phone; the left panel on a desktop). -2. Load a PDF with "PDF (scanned)". Pick the OCR language. -3. Press "OCR". The first run downloads the OCR engine and the language - data (see Notes). The status line shows the progress. -4. Load the subtitles and the audio in the "Audiobook" section. - -OCR text is noisy. The character boxes that Tesseract returns for Japanese -overlap and are sometimes shifted by half a character. When a tap picks -the wrong character, tap a little to the left or the right. +2. Load a PDF. The app reads the pages at once: the current page first, + then its neighbors, then the rest. "Pages" in the menu shows how far + it is. +3. Load the subtitles, or on Android press "Make subtitles with SubRead": + pick the audiobook, and SubRead makes the `.srt` from the audio and + the text of the pages read so far. The result loads at once, is kept + for this PDF, and can be shared as a file. +4. Start the player. The status line shows its position. The page + follows. + +### The three follow modes + +- **Highlight and turn pages**: the line of the current cue is marked on + the page, and the page turns when the cue is on another page. +- **Turn pages only**: the page turns, no mark. Use this when the + alignment is not good enough for a mark, for example with a noisy scan. +- **Off**: the page does not move. The status line and the cue list + still show where the player is. + +A cue that the alignment did not find in the book has no page. In the +two follow modes the page then follows the nearest matched cue before it, +up to five cues back. The same cue does not turn the page twice: you can +turn pages by hand while a long cue plays, and the page follows again at +the next cue. + +### The top bar + +Prev and Next turn the page. `▶` and `⏸` control the player. "Audio → +page" moves the player to the first cue on the page you look at, the +same as "Move the audio to this page" in the KOReader plugin. + +### Text layer or OCR + +The app reads the text layer of a page first. When the page has fewer +than 10 characters of text, it is a scan: the app renders the page and +runs OCR (tesseract.js) in the language of the "OCR language" setting. +"Force OCR" skips the text layer, for a PDF whose text layer is wrong. +The status line says which one was used for each page. + +Both give the same tokens: one box per character with a line id and a +word id. The alignment, the mark, the page turn and the lookup work the +same way on a scan and on a text PDF. + +The tokens of each page are kept in IndexedDB, under the file name, the +file size, the page, the OCR language and the source. A book that was +read once aligns at once the next time. "Clear the page cache" removes +them. + +### Lookups + +A tap on a character sends the text from that character to the end of +its line (at most 40 characters) to the dictionary. When the line ends +within 4 characters, the next line of the page follows, so a word that +wraps is whole. "Dictionary" lists every app in the text selection menu; +"Ask each time" shows the Android chooser. "Pause on lookup" pauses the +player before the dictionary opens and starts it again when the +dictionary closes. + +On the web there is no dictionary app: the tap copies the text to the +clipboard and shows it in the status line. + +### The web + +The browser has no SubRead Overlay. The "Audio (web only)" field plays +the audio in the page, and the page follows that clock. Everything else +is the same. The web build is for development and for a desktop reader. ## Run @@ -34,13 +102,15 @@ npm run dev ``` Open the URL that Vite prints. Try the fixtures in `fixtures/`: -`sample-jpn.pdf` with the "Japanese, horizontal" OCR language, -`sample-jpn.srt` and `silence.wav` (or the `sample-eng` pair with -English). +`sample-eng-2p.pdf` (a two-page scan) or `sample-eng-2p-text.pdf` (the +same text as a text layer) with `sample-eng-2p.srt` and `silence.wav`. +One cue of that file crosses from page 1 to page 2. `sample-jpn.pdf` +and `sample-jpn-text.pdf` go with `sample-jpn.srt` and the "Japanese, +horizontal" OCR language. -The first OCR run downloads the tesseract worker, its WebAssembly core and -the language data from jsDelivr. The browser caches the language data in -IndexedDB after that. +The first OCR run downloads the tesseract worker, its WebAssembly core +and the language data from jsDelivr. The browser caches the language +data in IndexedDB after that. ## Check @@ -66,15 +136,42 @@ npm run format:check `npm test` runs the unit and property tests. They need no browser and no network. `npm run test:e2e` runs real OCR on the fixture images in Node -and checks that every cue is highlighted on the correct line. It needs -network on the first run to fetch language data into `.tessdata/`. +and checks that every cue is marked on the correct line and page, and +reads the text-layer PDFs with pdf.js in Node and checks that the text +layer gives every character in order. It needs network on the first run +to fetch language data into `.tessdata/`. `npm run fixtures` rebuilds the fixtures from `fixtures/sample-text.ts`. +The Japanese text-layer PDF needs `C:\Windows\Fonts\yumin.ttf`; without +it, that one file is skipped. ## Android Capacitor wraps the web build in an Android WebView. The app is the same -code. `capacitor.config.ts` holds the app id, the app name and the web +code. The native part is one plugin, +`android/app/src/main/java/com/equwal/scansubread/SubReadPlugin.java`, +the bridge to the suite: + +- `playerState`, `play`, `pause`, `seek`: the content provider of SubRead + Overlay, `content://space.subread.overlay.player/state` (the debug + build of the overlay has the authority + `space.subread.overlay.debug.player`; both are tried). The app reads + the player every 2 seconds (every 5 seconds after an error) and + computes the position between two reads from the reported position, + the time of the report and the speed, the same as the overlay itself. + A reported position more than 3 seconds from the expected one counts + as a seek. +- `lookup`: `Intent.ACTION_PROCESS_TEXT` with the text. The call + resolves when the dictionary closes. +- `dictionaries`, `setDictionary`: the apps that take that intent, and + the chosen one. +- `makeSubtitles`, `pickAudio`: the intent API of SubRead + (`space.subread.app.action.ALIGN`). The book text goes to SubRead + through a FileProvider. When SubRead is not installed, the menu links + to its releases. +- `shareText`: the share sheet, for the finished `.srt`. + +`capacitor.config.ts` holds the app id, the app name and the web directory. The `android/` project is the Capacitor template and is committed. @@ -121,8 +218,8 @@ adb install -r android/app/build/outputs/apk/debug/app-debug.apk Notes: - The WebView serves the app from `https://localhost`. The file fields - open the Android file chooser. Put your PDFs in `Download` to find - them fast. + open the Android file chooser. Put your PDFs and `.srt` files in + `Download` to find them fast. - The first OCR run needs network. tesseract.js fetches its worker, its WebAssembly core and the language data from jsDelivr over HTTPS. The `INTERNET` permission is in the manifest. No cleartext traffic setting @@ -131,25 +228,31 @@ Notes: - Pinch zoom is on (`zoomEnabled` in `capacitor.config.ts`). The page fits the screen width in portrait and the screen height in landscape. Zoom in and scroll to read small print. +- The page cache and the subtitles live in the WebView's IndexedDB. + Uninstalling the app deletes them. ## Dependencies Runtime: -- `pdfjs-dist`: renders PDF pages to a canvas. +- `pdfjs-dist`: renders PDF pages to a canvas and reads the text layer. - `tesseract.js`: OCR in a Web Worker. Returns a box for each symbol. - `fast-diff`: character-level Myers diff. The alignment is built on it. - `idb`: a thin Promise wrapper around IndexedDB. It replaces the callback and event plumbing of the raw IndexedDB API. -- `@capacitor/core`, `@capacitor/android`: the Android shell. +- `@capacitor/core`, `@capacitor/android`: the Android shell and the + plugin bridge. Development: - `vite`, `typescript`: build and strict type check. - `vitest`, `fast-check`: unit tests and property tests. - `prettier`: formatting. -- `sharp`, `pdf-lib`, `tsx`: build the synthetic fixtures (render text to - a page image, wrap it in an image-only PDF, write a silent WAV). +- `sharp`, `pdf-lib`, `@pdf-lib/fontkit`, `tsx`: build the synthetic + fixtures (render text to a page image, wrap it in an image-only PDF, + write the same text as a text-layer PDF, write a silent WAV). + `@pdf-lib/fontkit` lets pdf-lib embed a TrueType font for the + Japanese text-layer page. The SRT/VTT parser is written by hand (`src/subtitles.ts`). The `subtitle` package imports the Node `stream` module at load time, so it does not run @@ -157,17 +260,43 @@ in a browser bundle. ## Layout +- `src/player-state.ts`: pure parser of the state line of SubRead + Overlay. A port of `player_state.lua` from the KOReader plugin. +- `src/play-clock.ts`: pure clock: the position now from a report of + the player. A port of `PlayClock.kt` from SubRead Overlay. +- `src/follower.ts`: pure reducer: a change of the cue to a mark and a + page turn, in the three follow modes. +- `src/clock-source.ts`: the two time sources, the overlay (Android) + and the audio element (web). +- `src/text-layer.ts`: pure conversion of pdf.js text items to tokens. +- `src/ocr-tokens.ts`: pure conversion of a tesseract result to tokens. +- `src/align.ts`: pure alignment of subtitle cues to the page text. +- `src/subtitles.ts`: pure SRT/VTT parser and active-cue lookup. - `src/hit-test.ts`: pure tap on the page. Hit test, scan string, the lookup text. -- `src/align.ts`: pure alignment of subtitle cues to OCR text. -- `src/subtitles.ts`: pure SRT/VTT parser and active-cue lookup. -- `src/ocr-tokens.ts`: pure conversion of a tesseract result to tokens. -- `src/book-text.ts`: pure OCR tokens to book text. +- `src/book-text.ts`: pure tokens to the book text for SubRead. +- `src/subread.ts`: the typed side of the plugin, with the web fallback. +- `src/token-cache.ts`: IndexedDB store for the page tokens and the + subtitles (thin). - `src/ocr.ts`, `src/pdf.ts`: thin browser wrappers around the libraries. - `src/main.ts`: UI wiring. -- `test/`: unit and property tests. `test/e2e/`: OCR end-to-end test. -- `fixtures/`, `scripts/make-fixtures.ts`: synthetic scanned pages. +- `test/`: unit and property tests. `test/e2e/`: OCR and text-layer + end-to-end tests. +- `fixtures/`, `scripts/make-fixtures.ts`: synthetic scanned pages and + text-layer PDFs. + +## Scope limits + +- The alignment runs over the whole book each time a page is read. It + takes about 5 s for 200k characters at 10% OCR noise, and well under + 1 s for a chapter. +- A vertical font in a text layer (`dir: 'ttb'` in pdf.js) lays the + characters down the page. This path has unit tests with mocked items, + but no fixture PDF: pdf-lib does not write vertical text. +- One audio file for one book. A book in many audio files is not + handled; SubRead has the same limit. +- No iOS build. ## Licence -AGPL-3.0. +AGPL-3.0. See `LICENSE`. diff --git a/index.html b/index.html index 164d4f0..783f041 100644 --- a/index.html +++ b/index.html @@ -12,13 +12,61 @@ <button id="prev">Prev</button> <span id="page-label">No PDF</span> <button id="next">Next</button> + <button id="play" title="Play or pause the audiobook" disabled>▶</button> + <button id="sync-page" title="Move the audio to the first cue on this page" disabled> + Audio → page + </button> </header> <aside id="panel"> <h1>Scan Subread</h1> - <p class="hint">Load a scanned PDF, run OCR, then tap a word on the page.</p> + <p class="hint"> + Load a PDF and the subtitles of its audiobook. The page follows the player. Tap a word to + look it up. + </p> + + <label>PDF (scan or text) <input id="pdf-file" type="file" accept="application/pdf" /></label> + <label + >Subtitles (.srt or .vtt) <input id="sub-file" type="file" accept=".srt,.vtt,text/*" + /></label> + + <section id="make" hidden> + <div class="row"> + <button id="make-subs" disabled>Make subtitles with SubRead</button> + <select id="sub-lang" title="Narration language"> + <option value="auto">auto</option> + <option value="ja">ja</option> + <option value="en">en</option> + </select> + </div> + <div id="make-status" class="hint"></div> + </section> + + <div id="srt-row" class="row" hidden> + <a id="srt-download" download>Download the .srt</a> + <button id="srt-share" hidden>Share the .srt</button> + </div> + + <section id="audio-row"> + <label>Audio (web only) <input id="audio-file" type="file" accept="audio/*" /></label> + <audio id="audio" controls></audio> + </section> + + <div id="player-status" class="hint">Player: not read yet.</div> + <div id="pages-status" class="hint"></div> + <div id="status" aria-live="polite"></div> + <progress id="progress" max="1" value="0" hidden></progress> + + <fieldset id="follow"> + <legend>Follow the audio</legend> + <label + ><input type="radio" name="follow" value="highlight" checked /> Highlight and turn + pages</label + > + <label><input type="radio" name="follow" value="pages" /> Turn pages only</label> + <label><input type="radio" name="follow" value="off" /> Off</label> + </fieldset> - <label>PDF (scanned) <input id="pdf-file" type="file" accept="application/pdf" /></label> <label >OCR language <select id="lang"> @@ -28,19 +76,22 @@ <option value="jpn+eng">Japanese + English</option> </select> </label> - <label>Pages to OCR <input id="max-pages" type="number" min="1" value="3" /></label> - <button id="run" disabled>OCR</button> - <div id="status" aria-live="polite"></div> - <progress id="progress" max="1" value="0" hidden></progress> - - <section id="audiobook"> - <h2>Audiobook</h2> - <label>Audio (optional) <input id="audio-file" type="file" accept="audio/*" /></label> - <label - >Subtitles (optional) <input id="sub-file" type="file" accept=".srt,.vtt,text/*" - /></label> - <audio id="audio" controls></audio> - </section> + <label class="check" + ><input id="force-ocr" type="checkbox" /> Force OCR (the text layer of the PDF is + wrong)</label + > + <label + >Dictionary + <select id="dict"> + <option value="">Ask each time</option> + </select> + </label> + <label class="check" + ><input id="pause-lookup" type="checkbox" checked /> Pause on lookup</label + > + <div class="row"> + <button id="clear-cache">Clear the page cache</button> + </div> <ol id="cues"></ol> </aside> diff --git a/src/main.ts b/src/main.ts index 3e217cb..2aeb67f 100644 --- a/src/main.ts +++ b/src/main.ts @@ -1,17 +1,38 @@ -// UI wiring. Alignment logic lives in align.ts, the tap in hit-test.ts. +// UI wiring. The logic lives in the pure modules: align.ts, follower.ts, +// hit-test.ts, text-layer.ts, player-state.ts, play-clock.ts. import { alignCuesToTokens, type OcrToken, type TokenSpan } from './align'; +import { bookText } from './book-text'; +import { LocalAudioClock, OverlayClock, type ClockSource, type ClockState } from './clock-source'; +import { follow, type FollowMode } from './follower'; import { lookupText, tokenAt } from './hit-test'; -import { createOcr } from './ocr'; +import { createOcr, type Ocr } from './ocr'; import { loadPdf, type PdfDoc } from './pdf'; +import { isAndroid, SubRead } from './subread'; import { cueIndexAt, parseSubtitles, type Cue } from './subtitles'; - -/** Page width in pixels for OCR. Boxes are stored in this scale. */ +import { + bookKey, + clearPages, + getPage, + getSrt, + pageKey, + putPage, + putSrt, + type PageEntry, +} from './token-cache'; + +/** Page width in pixels for OCR and the text layer. Boxes are stored in this scale. */ const OCR_WIDTH = 1600; /** How far from a character a tap may land, as a share of the page width. */ const TAP_TOLERANCE = 0.015; +/** Milliseconds to wait after a page is read before the cues are aligned again. */ +const ALIGN_DEBOUNCE = 300; + +const OVERLAY_RELEASES = 'https://github.com/equwal/subread-overlay/releases/latest'; +const SUBREAD_RELEASES = 'https://github.com/equwal/subread-android/releases/latest'; + function el<T extends HTMLElement>(id: string): T { const e = document.getElementById(id); if (!e) throw new Error(`Missing element #${id}`); @@ -23,46 +44,118 @@ const ui = { panel: el<HTMLElement>('panel'), pdfFile: el<HTMLInputElement>('pdf-file'), subFile: el<HTMLInputElement>('sub-file'), + make: el<HTMLElement>('make'), + makeSubs: el<HTMLButtonElement>('make-subs'), + subLang: el<HTMLSelectElement>('sub-lang'), + makeStatus: el<HTMLDivElement>('make-status'), + srtRow: el<HTMLDivElement>('srt-row'), + srtDownload: el<HTMLAnchorElement>('srt-download'), + srtShare: el<HTMLButtonElement>('srt-share'), + audioRow: el<HTMLElement>('audio-row'), audioFile: el<HTMLInputElement>('audio-file'), - lang: el<HTMLSelectElement>('lang'), - maxPages: el<HTMLInputElement>('max-pages'), - run: el<HTMLButtonElement>('run'), + audio: el<HTMLAudioElement>('audio'), + playerStatus: el<HTMLDivElement>('player-status'), + pagesStatus: el<HTMLDivElement>('pages-status'), status: el<HTMLDivElement>('status'), progress: el<HTMLProgressElement>('progress'), - audio: el<HTMLAudioElement>('audio'), + follow: el<HTMLFieldSetElement>('follow'), + lang: el<HTMLSelectElement>('lang'), + forceOcr: el<HTMLInputElement>('force-ocr'), + dict: el<HTMLSelectElement>('dict'), + pauseLookup: el<HTMLInputElement>('pause-lookup'), + clearCache: el<HTMLButtonElement>('clear-cache'), cues: el<HTMLOListElement>('cues'), prev: el<HTMLButtonElement>('prev'), next: el<HTMLButtonElement>('next'), + play: el<HTMLButtonElement>('play'), + syncPage: el<HTMLButtonElement>('sync-page'), pageLabel: el<HTMLSpanElement>('page-label'), viewer: el<HTMLDivElement>('viewer'), page: el<HTMLDivElement>('page'), overlay: el<HTMLDivElement>('overlay'), }; -interface PageSize { - width: number; - height: number; -} - const state = { pdf: null as PdfDoc | null, - /** Size of each OCR'd page at OCR scale. Index = page. */ - pageSizes: [] as (PageSize | undefined)[], + file: null as File | null, + /** The tokens of each page that is read. Index = page. */ + pages: [] as (PageEntry | undefined)[], + /** The tokens of the pages read so far, in page order. */ tokens: [] as OcrToken[], cues: [] as Cue[], spans: [] as TokenSpan[], + /** The page of each cue, or null for an unmatched cue. */ + cuePages: [] as (number | null)[], currentPage: -1, + /** The cue of the last clock state, or -1. */ activeCue: -1, - busy: false, + /** The cue whose box is drawn on the page, or -1. */ + markedCue: -1, + clock: null as ClockState | null, renderSeq: 0, + readSeq: 0, + alignTimer: null as ReturnType<typeof setTimeout> | null, + srt: null as string | null, + /** True while the player waits for a dictionary lookup that this app paused. */ + lookupPaused: false, }; -function setStatus(text: string): void { - ui.status.textContent = text; +// --- Settings --- + +const settings = { + get(key: string, fallback: string): string { + return localStorage.getItem(`scan-subread.${key}`) ?? fallback; + }, + set(key: string, value: string): void { + localStorage.setItem(`scan-subread.${key}`, value); + }, +}; + +function followMode(): FollowMode { + const checked = ui.follow.querySelector<HTMLInputElement>('input:checked'); + return (checked?.value as FollowMode | undefined) ?? 'highlight'; +} + +function loadSettings(): void { + const mode = settings.get('follow', 'highlight'); + for (const input of ui.follow.querySelectorAll<HTMLInputElement>('input')) { + input.checked = input.value === mode; + } + ui.lang.value = settings.get('lang', 'eng'); + ui.subLang.value = settings.get('subLang', 'auto'); + ui.forceOcr.checked = settings.get('forceOcr', '0') === '1'; + ui.pauseLookup.checked = settings.get('pauseLookup', '1') === '1'; } -function updateRunButton(): void { - ui.run.disabled = state.busy || !state.pdf; +ui.follow.addEventListener('change', () => { + settings.set('follow', followMode()); + state.markedCue = -1; + drawBoxes(); + if (state.clock) applyClock({ ...state.clock, seeked: true }); +}); +ui.pauseLookup.addEventListener('change', () => + settings.set('pauseLookup', ui.pauseLookup.checked ? '1' : '0'), +); +ui.subLang.addEventListener('change', () => settings.set('subLang', ui.subLang.value)); +ui.lang.addEventListener('change', () => { + settings.set('lang', ui.lang.value); + void readBook(); +}); +ui.forceOcr.addEventListener('change', () => { + settings.set('forceOcr', ui.forceOcr.checked ? '1' : '0'); + void readBook(); +}); + +function setStatus(text: string, link?: { href: string; text: string }): void { + ui.status.textContent = text; + if (link) { + const a = document.createElement('a'); + a.href = link.href; + a.target = '_blank'; + a.rel = 'noopener'; + a.textContent = link.text; + ui.status.append(' ', a); + } } // --- Drawer --- @@ -74,93 +167,244 @@ function setMenu(open: boolean): void { ui.menu.addEventListener('click', () => setMenu(!document.body.classList.contains('menu-open'))); -// --- Loading inputs --- +// A tap outside the drawer closes the drawer. +document.addEventListener('click', (e) => { + const target = e.target as Node; + if ( + document.body.classList.contains('menu-open') && + !ui.panel.contains(target) && + !ui.menu.contains(target) + ) { + setMenu(false); + } +}); + +// --- The PDF --- ui.pdfFile.addEventListener('change', async () => { const file = ui.pdfFile.files?.[0]; if (!file) return; try { + state.readSeq++; await state.pdf?.destroy(); state.pdf = await loadPdf(await file.arrayBuffer()); + state.file = file; + state.pages = []; state.tokens = []; state.spans = []; - state.pageSizes = []; + state.cuePages = []; state.activeCue = -1; - setStatus(`PDF loaded: ${state.pdf.numPages} pages. Run OCR, then tap words on the page.`); + state.markedCue = -1; + ui.makeSubs.disabled = false; + setStatus(`PDF loaded: ${state.pdf.numPages} pages.`); await showPage(0); + const saved = await getSrt(bookKey(file)); + if (saved) loadSubtitles(saved, 'the last SubRead run'); + void readBook(); } catch (err) { setStatus(`Cannot open PDF: ${String(err)}`); } - updateRunButton(); }); -ui.subFile.addEventListener('change', async () => { - const file = ui.subFile.files?.[0]; - if (!file) return; - state.cues = parseSubtitles(await file.text()); +/** The next page to read: the current page, its neighbors, then the rest in order. */ +function nextPage(pending: Set<number>): number { + const current = state.currentPage; + for (const p of [current, current + 1, current - 1]) if (pending.has(p)) return p; + return Math.min(...pending); +} + +/** The tokens of one page: from the cache, else the text layer, else OCR. */ +async function readPage( + pdf: PdfDoc, + file: File, + page: number, + ocr: () => Promise<Ocr>, +): Promise<PageEntry> { + const lang = ui.lang.value; + const force = ui.forceOcr.checked; + if (!force) { + const cached = await getPage(pageKey(file, page, lang, 'text')); + if (cached) return cached; + } + const cachedOcr = await getPage(pageKey(file, page, lang, 'ocr')); + if (cachedOcr) return cachedOcr; + if (!force) { + const text = await pdf.pageText(page, OCR_WIDTH); + if (text.tokens) { + const entry: PageEntry = { ...text, tokens: text.tokens, source: 'text' }; + await putPage(pageKey(file, page, lang, 'text'), entry); + return entry; + } + } + const canvas = await pdf.renderPage(page, OCR_WIDTH); + const tokens = await (await ocr()).recognize(canvas, page); + const entry: PageEntry = { tokens, width: canvas.width, height: canvas.height, source: 'ocr' }; + await putPage(pageKey(file, page, lang, 'ocr'), entry); + return entry; +} + +/** + * Reads every page of the book: the current page first, then its + * neighbors, then the rest. The cues are aligned again as pages finish, + * so the follow starts before the whole book is read. + */ +async function readBook(): Promise<void> { + const { pdf, file } = state; + if (!pdf || !file) return; + const seq = ++state.readSeq; + state.pages = []; + state.tokens = []; + ui.pagesStatus.textContent = `Pages: 0/${pdf.numPages} read.`; + const pending = new Set(Array.from({ length: pdf.numPages }, (_, i) => i)); + // The OCR worker starts on the first page that needs it. + const ocr: { started?: Promise<Ocr> } = {}; + const getOcr = (): Promise<Ocr> => { + ocr.started ??= createOcr(ui.lang.value, (p) => { + ui.progress.value = p; + }); + return ocr.started; + }; + const counts = { text: 0, ocr: 0 }; + try { + while (pending.size > 0) { + const page = nextPage(pending); + pending.delete(page); + ui.progress.hidden = false; + ui.progress.value = 0; + setStatus(`Reading page ${page + 1} of ${pdf.numPages}...`); + const entry = await readPage(pdf, file, page, getOcr); + if (seq !== state.readSeq) return; // Another book or another setting took over. + counts[entry.source]++; + state.pages[page] = entry; + state.tokens = state.pages.flatMap((p) => p?.tokens ?? []); + setStatus(`Page ${page + 1}: ${entry.source === 'text' ? 'text layer' : 'OCR'}.`); + ui.pagesStatus.textContent = + `Pages: ${counts.text + counts.ocr}/${pdf.numPages} read ` + + `(${counts.text} text layer, ${counts.ocr} OCR).`; + if (page === state.currentPage) drawBoxes(); + scheduleAlign(); + } + } catch (err) { + if (seq === state.readSeq) setStatus(`Cannot read the pages: ${String(err)}`); + } finally { + if (ocr.started) await (await ocr.started).terminate().catch(() => undefined); + if (seq === state.readSeq) ui.progress.hidden = true; + } +} + +/** True when every page of the book is read. */ +function allPagesRead(): boolean { + const pdf = state.pdf; + return !!pdf && state.pages.filter((p) => p).length === pdf.numPages; +} + +ui.clearCache.addEventListener('click', async () => { + await clearPages(); + setStatus('Page cache cleared.'); +}); + +// --- Subtitles --- + +/** Load subtitles from a file or from SubRead. `source` names where they came from. */ +function loadSubtitles(text: string, source: string): void { + state.cues = parseSubtitles(text); + state.srt = text; state.spans = []; + state.cuePages = []; state.activeCue = -1; - setStatus(`${state.cues.length} cues loaded from ${file.name}.`); - alignIfReady(); + state.markedCue = -1; + setStatus(`${state.cues.length} cues loaded from ${source}.`); + ui.syncPage.disabled = state.cues.length === 0; renderCueList(); -}); + align(); + offerSrt(text); +} -ui.audioFile.addEventListener('change', () => { - const file = ui.audioFile.files?.[0]; +ui.subFile.addEventListener('change', async () => { + const file = ui.subFile.files?.[0]; if (!file) return; - ui.audio.src = URL.createObjectURL(file); + loadSubtitles(await file.text(), file.name); }); -// --- OCR and alignment --- +function offerSrt(text: string): void { + const name = `${(state.file?.name ?? 'book').replace(/\.pdf$/i, '')}.srt`; + ui.srtRow.hidden = false; + ui.srtDownload.hidden = isAndroid; + ui.srtShare.hidden = !isAndroid; + if (!isAndroid) { + URL.revokeObjectURL(ui.srtDownload.href); + ui.srtDownload.href = URL.createObjectURL(new Blob([text], { type: 'text/plain' })); + ui.srtDownload.download = name; + } + ui.srtShare.onclick = () => { + SubRead.shareText({ name, text }).catch((err) => setStatus(String(err))); + }; +} -ui.run.addEventListener('click', async () => { - const pdf = state.pdf; - if (!pdf || state.busy) return; - state.busy = true; - updateRunButton(); - ui.progress.hidden = false; - const pageCount = Math.min(pdf.numPages, Math.max(1, Number(ui.maxPages.value) || 1)); +ui.makeSubs.addEventListener('click', async () => { + const file = state.file; + if (!file) return; + let audio; try { - setStatus(`Loading OCR language "${ui.lang.value}"...`); - const ocr = await createOcr(ui.lang.value, (p) => { - ui.progress.value = p; - }); - try { - state.tokens = []; - state.pageSizes = []; - for (let p = 0; p < pageCount; p++) { - setStatus(`OCR page ${p + 1} of ${pageCount}...`); - ui.progress.value = 0; - const canvas = await pdf.renderPage(p, OCR_WIDTH); - state.pageSizes[p] = { width: canvas.width, height: canvas.height }; - state.tokens.push(...(await ocr.recognize(canvas, p))); - } - } finally { - await ocr.terminate(); - } - setStatus(`Done. ${state.tokens.length} characters read. Tap a word on the page.`); - alignIfReady(); - renderCueList(); - state.activeCue = -1; - if (state.spans.length > 0) await setActiveCue(0); - } catch (err) { - setStatus(`OCR failed: ${String(err)}`); - } finally { - state.busy = false; - ui.progress.hidden = true; - updateRunButton(); + audio = await SubRead.pickAudio(); + } catch { + return; + } + const read = state.pages.filter((p) => p).length; + ui.makeStatus.textContent = allPagesRead() + ? `SubRead makes the subtitles for ${audio.name}...` + : `Only ${read} of ${state.pdf?.numPages ?? 0} pages are read. The subtitles cover those pages. SubRead runs...`; + const result = await SubRead.makeSubtitles({ + audio: audio.uri, + bookText: bookText(state.tokens), + language: ui.subLang.value, + }); + if (result.error === 'not_installed') { + ui.makeStatus.replaceChildren('SubRead is not installed. '); + const a = document.createElement('a'); + a.href = SUBREAD_RELEASES; + a.target = '_blank'; + a.rel = 'noopener'; + a.textContent = 'Get SubRead'; + ui.makeStatus.append(a); + return; } + if (result.error !== undefined || result.srt === undefined) { + ui.makeStatus.textContent = `SubRead made no subtitles: ${result.error ?? 'no file'}.`; + return; + } + const rate = result.matchRate !== undefined && result.matchRate >= 0 ? result.matchRate : null; + ui.makeStatus.textContent = + `${result.cues ?? '?'} cues, language ${result.language ?? '?'}` + + (rate === null ? '.' : `, ${Math.round(rate * 100)}% of the lines found in the book.`) + + (rate !== null && rate < 0.8 + ? ' Under 80% usually means another edition or the wrong language.' + : ''); + await putSrt(bookKey(file), result.srt); + loadSubtitles(result.srt, 'SubRead'); }); -/** Align the cues to the OCR text when both are present. */ -function alignIfReady(): void { +// --- Alignment --- + +function scheduleAlign(): void { + if (state.alignTimer) clearTimeout(state.alignTimer); + state.alignTimer = setTimeout(align, ALIGN_DEBOUNCE); +} + +/** Align the cues to the pages read so far. */ +function align(): void { + state.alignTimer = null; if (state.cues.length === 0 || state.tokens.length === 0) return; state.spans = alignCuesToTokens( state.cues.map((c) => c.text), state.tokens, ); - const matched = state.spans.filter((s) => s.matched).length; + state.cuePages = state.spans.map((s) => (s.matched ? state.tokens[s.start]!.page : null)); + const matched = state.cuePages.filter((p) => p !== null).length; setStatus(`${matched}/${state.cues.length} cues matched to the page text.`); + renderCueList(); + // The cue of now may be matched now. Apply the clock state again. + if (state.clock) applyClock({ ...state.clock, seeked: true }); } // --- Cue list --- @@ -173,35 +417,108 @@ function renderCueList(): void { li.title = `${cue.start.toFixed(2)} s`; const span = state.spans[i]; if (span && !span.matched) li.classList.add('unmatched'); - li.addEventListener('click', () => { - ui.audio.currentTime = cue.start; - void setActiveCue(i); - }); + if (i === state.activeCue) li.classList.add('active'); + li.addEventListener('click', () => void clock.seek(cue.start * 1000)); return li; }), ); } -async function setActiveCue(i: number): Promise<void> { - if (i === state.activeCue) return; +function markCueInList(i: number): void { const items = ui.cues.children; items[state.activeCue]?.classList.remove('active'); - state.activeCue = i; const item = items[i]; if (item) { item.classList.add('active'); item.scrollIntoView({ block: 'nearest' }); } - const span = state.spans[i]; - if (!span) { - drawBoxes(); - return; +} + +// --- The follow --- + +function clockText(ms: number): string { + const s = Math.floor(ms / 1000); + const pad = (n: number): string => String(n).padStart(2, '0'); + const h = Math.floor(s / 3600); + return `${h > 0 ? `${h}:` : ''}${pad(Math.floor((s % 3600) / 60))}:${pad(s % 60)}`; +} + +function showPlayerStatus(s: ClockState): void { + ui.playerStatus.replaceChildren(); + ui.play.disabled = s.error === 'no_overlay' || (!isAndroid && s.error === 'no_player'); + ui.play.textContent = s.playing ? '⏸' : '▶'; + const name = isAndroid ? 'Player' : 'Audio'; + if (s.error === 'no_overlay') { + ui.playerStatus.append('SubRead Overlay not installed. '); + const a = document.createElement('a'); + a.href = OVERLAY_RELEASES; + a.target = '_blank'; + a.rel = 'noopener'; + a.textContent = 'Get it'; + ui.playerStatus.append(a); + } else if (s.error === 'no_notification_access') { + ui.playerStatus.textContent = 'Allow notification access in SubRead Overlay.'; + } else if (s.error === 'no_player') { + ui.playerStatus.textContent = isAndroid ? 'No player.' : 'No audio file.'; + } else if (s.positionMs === null) { + ui.playerStatus.textContent = `${name}: no position.`; + } else { + ui.playerStatus.textContent = `${name}: ${clockText(s.positionMs)}, ${s.playing ? 'playing' : 'paused'}`; } - const first = state.tokens[Math.min(span.start, state.tokens.length - 1)]; - if (first && first.page !== state.currentPage) await showPage(first.page); +} + +/** One state of the clock: mark the cue of now and turn the page. */ +function applyClock(s: ClockState): void { + state.clock = s; + showPlayerStatus(s); + if (s.positionMs === null) return; + const cue = cueIndexAt(state.cues, s.positionMs / 1000); + // The last cue stays marked in the silence between two cues. + if (cue < 0) return; + const out = follow({ + cuePages: state.cuePages, + mode: followMode(), + currentPage: state.currentPage, + previousCue: state.activeCue, + cue, + seeked: s.seeked, + }); + const changed = cue !== state.activeCue || s.seeked; + if (cue !== state.activeCue) markCueInList(cue); + state.activeCue = cue; + if (!changed || followMode() === 'off') return; + state.markedCue = out.highlightCue ?? -1; + if (out.turnToPage !== null) void showPage(out.turnToPage); else drawBoxes(); } +const clock: ClockSource = isAndroid ? new OverlayClock(SubRead) : new LocalAudioClock(ui.audio); +clock.start(applyClock); + +ui.play.addEventListener('click', () => { + const action = state.clock?.playing ? clock.pause() : clock.play(); + action.catch((err) => setStatus(String(err))); +}); + +ui.syncPage.addEventListener('click', () => { + const i = state.cuePages.indexOf(state.currentPage); + if (i < 0) { + setStatus('No cue of the subtitles is on this page.'); + return; + } + const ms = state.cues[i]!.start * 1000; + clock.seek(ms).then( + () => setStatus(`Audio moved to ${clockText(ms)}.`), + (err) => setStatus(String(err)), + ); +}); + +ui.audioFile.addEventListener('change', () => { + const file = ui.audioFile.files?.[0]; + if (!file) return; + ui.audio.src = URL.createObjectURL(file); +}); + // --- Page view --- async function showPage(index: number): Promise<void> { @@ -209,7 +526,7 @@ async function showPage(index: number): Promise<void> { if (!pdf || index < 0 || index >= pdf.numPages) return; const seq = ++state.renderSeq; state.currentPage = index; - ui.pageLabel.textContent = `Page ${index + 1} / ${pdf.numPages}`; + ui.pageLabel.textContent = `${index + 1} / ${pdf.numPages}`; // Render at device resolution, up to the OCR width, so zoom stays sharp. const dpr = window.devicePixelRatio || 1; const width = Math.round(Math.min(OCR_WIDTH, Math.max(300, ui.viewer.clientWidth - 16) * dpr)); @@ -220,11 +537,11 @@ async function showPage(index: number): Promise<void> { drawBoxes(); } -/** Draw one box per text line for the active cue on the current page. */ +/** Draw one box per text line for the marked cue on the current page. */ function drawBoxes(): void { ui.overlay.replaceChildren(); - const span = state.spans[state.activeCue]; - const size = state.pageSizes[state.currentPage]; + const span = state.spans[state.markedCue]; + const size = state.pages[state.currentPage]; if (!span || !size) return; const lines = new Map<number, OcrToken['bbox']>(); @@ -242,7 +559,7 @@ function drawBoxes(): void { } for (const box of lines.values()) { const div = document.createElement('div'); - div.className = span.matched ? 'box' : 'box unmatched'; + div.className = 'box'; div.style.left = `${(100 * box.x0) / size.width}%`; div.style.top = `${(100 * box.y0) / size.height}%`; div.style.width = `${(100 * (box.x1 - box.x0)) / size.width}%`; @@ -256,37 +573,53 @@ ui.next.addEventListener('click', () => void showPage(state.currentPage + 1)); // --- Tap on a word --- -/** Tap on the page: show the text under the tap. */ -ui.page.addEventListener('click', (e) => { +/** Tap on the page: send the text under the tap to the dictionary. */ +ui.page.addEventListener('click', async (e) => { const canvas = ui.page.querySelector('canvas'); - const size = state.pageSizes[state.currentPage]; + const size = state.pages[state.currentPage]; if (!canvas || !size) return; const rect = canvas.getBoundingClientRect(); const x = ((e.clientX - rect.left) / rect.width) * size.width; const y = ((e.clientY - rect.top) / rect.height) * size.height; const t = tokenAt(state.tokens, state.currentPage, x, y, TAP_TOLERANCE * size.width); if (t < 0) return; - setStatus(`Tapped: "${lookupText(state.tokens, t)}"`); -}); - -// A tap outside the drawer closes the drawer. -document.addEventListener('click', (e) => { - const target = e.target as Node; - if ( - document.body.classList.contains('menu-open') && - !ui.panel.contains(target) && - !ui.menu.contains(target) - ) { - setMenu(false); + const text = lookupText(state.tokens, t); + setStatus(`Lookup: "${text}"`); + const pause = ui.pauseLookup.checked && state.clock?.playing === true && !state.lookupPaused; + try { + if (pause) { + state.lookupPaused = true; + await clock.pause(); + } + const { closed } = await SubRead.lookup({ text }); + if (closed && state.lookupPaused) await clock.play(); + } catch (err) { + setStatus(`Lookup failed: ${String(err)}`); + } finally { + if (pause) state.lookupPaused = false; } }); -// --- Playback sync --- +// --- Dictionaries (Android) --- -ui.audio.addEventListener('timeupdate', () => { - const i = cueIndexAt(state.cues, ui.audio.currentTime); - // Keep the last cue lit during a pause between cues. - if (i >= 0) void setActiveCue(i); +async function loadDictionaries(): Promise<void> { + const { apps, chosen } = await SubRead.dictionaries(); + for (const app of apps) { + const option = document.createElement('option'); + option.value = app.component; + option.textContent = app.label; + ui.dict.append(option); + } + ui.dict.value = apps.some((a) => a.component === chosen) ? chosen : ''; +} + +ui.dict.addEventListener('change', () => { + SubRead.setDictionary({ component: ui.dict.value }).catch((err) => setStatus(String(err))); }); -updateRunButton(); +// --- Start --- + +loadSettings(); +ui.make.hidden = !isAndroid; +ui.audioRow.hidden = isAndroid; +if (isAndroid) void loadDictionaries().catch((err) => setStatus(String(err))); diff --git a/src/style.css b/src/style.css index c585608..c8bd484 100644 --- a/src/style.css +++ b/src/style.css @@ -31,7 +31,8 @@ body { #page-label { flex: 1; text-align: center; - font-size: 0.9rem; + font-size: 0.85rem; + white-space: nowrap; } #panel { @@ -85,16 +86,65 @@ body.menu-open #panel { min-height: 1.2em; } -#audiobook { +#topbar button { + white-space: nowrap; + font-size: 0.85rem; + padding: 0.25rem 0.45rem; +} + +#play { + min-width: 2.5rem; +} + +.row { + display: flex; + flex-wrap: wrap; + align-items: center; + gap: 0.5rem; + font-size: 0.85rem; +} + +#panel label.check, +#follow label { + flex-direction: row; + align-items: center; + gap: 0.4rem; +} + +#follow { + display: flex; + flex-direction: column; + gap: 0.3rem; + margin: 0; + padding: 0.4rem 0.6rem; + border: 1px solid #8884; + border-radius: 4px; + font-size: 0.85rem; +} + +#follow legend { + padding: 0 0.2rem; +} + +#make, +#audio-row { display: flex; flex-direction: column; gap: 0.4rem; font-size: 0.85rem; } -#audiobook h2 { - font-size: 0.95rem; - margin: 0; +#make-status, +#player-status, +#pages-status, +#status { + word-break: break-word; +} + +#status a, +#make-status a, +#player-status a { + color: inherit; } #audio {