commit b38c234bf808ebcc7b1f8ab8331117003d60aef0 truex <truex@equwal.com> 2026-09-24 12:21:59 -0700 Make subtitles from the audiobook and the OCR text The Audiobook section of the drawer has two paths. The server path uploads the audio and the OCR text to subread.space, starts the job, polls it and loads the SRT. A 402 answer shows a link to buy credits or sign in. The job id stays in localStorage, so the app resumes the poll after a restart. The in-app path runs the copied engine with the OCR lines as paragraphs. Both results go into an IndexedDB cache keyed by the audio file, then down the same path as a hand-picked SRT. On Android the requests run through the native HTTP plugin, because the WebView origin gets no CORS headers from the server. On the web the Vite dev proxy forwards /api.
README.md | 104 +++++++++++++- index.html | 50 ++++++- src/audiobook.ts | 393 +++++++++++++++++++++++++++++++++++++++++++++++++++++ src/cloud.ts | 174 ++++++++++++++++++++++++ src/http.ts | 106 +++++++++++++++ src/main.ts | 26 ++-- src/srt-cache.ts | 34 +++++ src/style.css | 26 +++- test/cloud.test.ts | 155 +++++++++++++++++++++ test/http.test.ts | 41 ++++++ 10 files changed, 1084 insertions(+), 25 deletions(-)
diff --git a/README.md b/README.md index 46ccdc0..2cb3f7f 100644 --- a/README.md +++ b/README.md @@ -9,10 +9,12 @@ for each character. Tap a word on the page, and the app shows the dictionary entries for the longest word that starts at that character, after Japanese deinflection (食べました → 食べる). -Optional: load the audiobook and its subtitle file (SRT or WebVTT). The -app aligns each subtitle cue to the OCR text and highlights the region of -the page that shows the current cue while the audio plays. Tap a cue in -the list to seek the audio. +Optional: load the audiobook. The app makes the subtitles from the +audiobook and the OCR text, on the subread.space server or in the app +itself (see "Audiobook"). You can also load a subtitle file (SRT or +WebVTT) by hand. The app aligns each subtitle cue to the OCR text and +highlights the region of the page that shows the current cue while the +audio plays. Tap a cue in the list to seek the audio. ## Use @@ -154,6 +156,74 @@ Notes: - Dictionaries live in the WebView's IndexedDB. Uninstalling the app deletes them. +## Audiobook + +The "Audiobook" section of the menu turns an audiobook and the OCR text +into subtitles. The subread.space service takes only epubs, not scanned +PDFs, so the app sends the OCR text instead: one line per OCR line, +pages in order, one blank line between pages (`src/book-text.ts`). A +space goes between words, but not between two CJK words. Two paths make +the subtitles. Both need a PDF with OCR done and an audio file. + +1. "Make subtitles on subread.space" uploads the audio and the OCR text + to the server (`POST /api/uploads`), starts the job with the + narration language, and polls the job every 5 seconds. The status + line shows the stage and the progress. "Stop" cancels the job. When + the job succeeds, the app downloads the SRT and loads it. The server + handles a 10-hour book in minutes. One job costs one credit. The + first anonymous job is free. When the server answers 402, the status + line shows the reason and a link to <https://subread.space>, where + you buy credits or sign in. The job id stays in `localStorage`, so + the app resumes the poll after a restart. +2. "Make subtitles in this app" runs the SubPlz browser engine from the + subread.space frontend (`src/engine/`): ffmpeg.wasm decodes the audio + in two-minute chunks, whisper-tiny (transformers.js on the ONNX + runtime) transcribes them, and the aligner matches the transcript to + the OCR lines. This is slow on a phone: the speech model runs in + WebAssembly at about real time or slower, and the screen must stay + on. The transcript is saved after each chunk, so a stopped job + resumes with the same audio file. The engine assets are not in the + app: transformers.js, the ONNX runtime, the ffmpeg core (32 MB) and + the model (about 120 MB) load from the "Engine assets URL", default + `https://subread.space/vendor/`. The browser caches them. + +The narration language is a Whisper code (ja or en). The app sets it +from the OCR language after each OCR run. The result SRT is kept in +IndexedDB under the audio file name and size, so the next time you pick +the same audio file the subtitles load at once. + +Sign in: type your email and press "Send link". The link opens +subread.space in the browser. The app has no deep link, so paste the +link (or the token in it) into "Paste the sign-in link or token" and +press "Verify". The account line shows the email and the credits. On the +web the identity is the `subplz_device` cookie of the dev proxy. On +Android the requests run through Capacitor's native HTTP plugin +(`src/http.ts`), because the WebView origin `https://localhost` gets no +CORS headers from the server. The native cookie store keeps the cookie +across restarts. The multipart upload crosses the native bridge as +base64, so the whole audio file is in memory once during the upload. + +"Advanced" holds two settings for a self-hosted server: the server URL +(default `https://subread.space`; empty on the web, where the Vite dev +proxy forwards `/api`) and the engine assets URL. + +Known limit on Android: the in-app path loads the engine assets from +another origin. That needs CORS headers on `/vendor/` of the server, +which subread.space does not send yet. Until the server adds +`add_header Access-Control-Allow-Origin *;` for `/vendor/` in nginx, the +in-app path fails on the phone with a load error, and the server path is +the one to use. On the web the Vite proxy makes `/vendor` same-origin, +so the in-app path runs in the browser pane end to end. + +License note: `src/engine/*.js` is copied from the `frontend/engine/` +directory of [equwal/subplz-web](https://github.com/equwal/subplz-web) +(AGPL-3.0, same author). The files carry a header comment that says so. +`align.js` and `align.worker.js` are verbatim. `asr.js`, `media.js` and +`job.js` are adapted: the paragraphs come from the OCR lines, the assets +load from a configurable base URL, and the ffmpeg wrapper comes from the +`@ffmpeg/ffmpeg` package (its worker must be same-origin). The `.d.ts` +files next to them give strict TypeScript the used signatures. + ## Local audio (Android only) The lookup popup can play a recorded pronunciation of each word. The @@ -199,6 +269,11 @@ Runtime: parser; the browser has no zip API. - `idb`: a thin Promise wrapper around IndexedDB. It replaces the callback and event plumbing of the raw IndexedDB API. +- `@ffmpeg/ffmpeg`: the ffmpeg.wasm worker wrapper for the in-app + path. It replaces a copy of the same files from the reference's + `frontend/vendor/ffmpeg/`. The 32 MB core is not bundled. +- `@capacitor/core`, `@capacitor/android`: the Android shell, the + LocalAudio plugin bridge and the native HTTP plugin. Development: @@ -234,6 +309,14 @@ any deinflection. - `src/align.ts`: pure alignment of subtitle cues to OCR text. - `src/subtitles.ts`: pure SRT/VTT parser and active-cue lookup. - `src/ocr-tokens.ts`: pure conversion of a tesseract result to tokens. +- `src/book-text.ts`: pure OCR tokens to book text and paragraphs. +- `src/cloud.ts`: pure client for the subread.space job API over an + injected transport. +- `src/http.ts`: the transports: `fetch` on the web, native HTTP on + Android. +- `src/srt-cache.ts`: IndexedDB store for finished subtitles (thin). +- `src/engine/`: the SubPlz browser engine, copied (see "Audiobook"). +- `src/audiobook.ts`: UI wiring of the "Audiobook" section. - `src/ocr.ts`, `src/pdf.ts`: thin browser wrappers around the libraries. - `src/main.ts`: UI wiring. - `test/`: unit and property tests. `test/e2e/`: OCR end-to-end test. @@ -242,7 +325,8 @@ any deinflection. ## Scope limits -- Web only. No backend. +- No backend of its own. The subread.space server makes the subtitles + on the server path; the app talks to it as a client. - OCR runs on the first N pages (the "Pages to OCR" field). Tokens are kept in memory. They are not saved between sessions; dictionaries are. - The scan string is the tapped character and the characters after it on @@ -251,7 +335,12 @@ any deinflection. - Alignment runs once over the whole book. It takes about 5 s for 200k characters at 10% OCR noise, and well under 1 s for a chapter. - No audiobook pause control, no iOS build. -- The fixtures' `silence.wav` is silent. It only drives the clock. +- The fixtures' `silence.wav` is silent. It only drives the clock. Both + subtitle paths fail or produce nonsense on it: the server reports + that no subtitle file was produced, and whisper-tiny hallucinates one + segment. +- The in-app path on Android waits for CORS headers on the server's + `/vendor/` (see "Audiobook"). ## Next steps @@ -266,3 +355,6 @@ any deinflection. sentence, with a setting for the pause length and a key to continue. 5. iOS: `npx cap add ios`. The web code is the same. It needs a Mac with Xcode. +6. Server: CORS headers on `/vendor/`, so the in-app path runs on + Android. A streaming native upload, so a 10-hour audiobook does not + sit in memory as base64 during the upload. diff --git a/index.html b/index.html index 96b28be..32ef7dc 100644 --- a/index.html +++ b/index.html @@ -53,11 +53,51 @@ </p> </section> - <label>Audio (optional) <input id="audio-file" type="file" accept="audio/*" /></label> - <label - >Subtitles (optional) <input id="sub-file" type="file" accept=".srt,.vtt,text/*" - /></label> - <audio id="audio" controls></audio> + <section id="audiobook"> + <h2>Audiobook</h2> + <label>Audio (optional) <input id="audio-file" type="file" accept="audio/*" /></label> + <label + >Subtitles (optional) <input id="sub-file" type="file" accept=".srt,.vtt,text/*" + /></label> + <label + >Narration language + <select id="ab-lang"> + <option value="ja">Japanese (ja)</option> + <option value="en">English (en)</option> + </select> + </label> + <div class="row"> + <button id="ab-cloud" disabled>Make subtitles on subread.space</button> + <button id="ab-local" disabled>Make subtitles in this app</button> + <button id="ab-stop" hidden>Stop</button> + </div> + <div id="ab-status" class="hint">Load a PDF, run OCR and pick an audio file.</div> + <progress id="ab-progress" max="1" value="0" hidden></progress> + <p class="hint"> + The server is fast and takes one credit per book. In this app the speech model runs in + WebAssembly: about real time or slower on a phone, and the screen must stay on. + </p> + <div id="ab-account" class="hint">Account: not checked yet.</div> + <label>Sign in (email) <input id="ab-email" type="email" autocomplete="email" /></label> + <div class="row"> + <button id="ab-send">Send link</button> + </div> + <label>Paste the sign-in link or token <input id="ab-token" type="text" /></label> + <div class="row"> + <button id="ab-verify">Verify</button> + </div> + <details> + <summary>Advanced</summary> + <label + >Server URL <input id="ab-server" type="url" placeholder="https://subread.space" + /></label> + <label + >Engine assets URL + <input id="ab-assets" type="url" placeholder="https://subread.space/vendor/" + /></label> + </details> + <audio id="audio" controls></audio> + </section> <ol id="cues"></ol> </aside> diff --git a/src/audiobook.ts b/src/audiobook.ts new file mode 100644 index 0000000..2683642 --- /dev/null +++ b/src/audiobook.ts @@ -0,0 +1,393 @@ +// The "Audiobook" section of the drawer: make subtitles from the audiobook +// and the OCR text, on subread.space or in this app. The result is an SRT +// that goes down the same path as a hand-picked subtitle file. + +import { bookFile, paragraphs, bookText, whisperCode } from './book-text'; +import type { OcrToken } from './align'; +import { + accountLine, + ApiError, + cancelJob, + fetchSrt, + getAccount, + pollUntilDone, + requestSignIn, + SITE, + startJob, + tokenFrom, + upload, + verifySignIn, + type Http, + type JobOut, +} from './cloud'; +import type { Job, JobStatus } from './engine/job.js'; +import { createHttp, DEFAULT_SERVER, defaultAssets } from './http'; +import { audioKey, getSrt, putSrt } from './srt-cache'; + +const POLL_MS = 5000; + +const KEYS = { + server: 'subread.server', + assets: 'subread.assets', + job: 'subread.job', +}; + +/** The server job that may still run after the app was closed. */ +interface PendingJob { + id: string; + audioKey: string; + audioName: string; +} + +export interface AudiobookHooks { + tokens(): readonly OcrToken[]; + /** The tesseract language of the last OCR run. */ + ocrLang(): string; + /** Load finished subtitles into the reader. */ + loadSrt(srt: string, source: string): void; +} + +export interface Audiobook { + /** Call after an OCR run: enables the buttons and sets the language. */ + onTokens(): void; +} + +function el<T extends HTMLElement>(id: string): T { + const e = document.getElementById(id); + if (!e) throw new Error(`Missing element #${id}`); + return e as T; +} + +function read(key: string): string | null { + try { + return localStorage.getItem(key); + } catch { + return null; + } +} + +function write(key: string, value: string | null): void { + try { + if (value === null) localStorage.removeItem(key); + else localStorage.setItem(key, value); + } catch { + // Storage is off. The setting lives for this session only. + } +} + +/** A URL that ends with one slash. */ +function withSlash(url: string): string { + return url.endsWith('/') ? url : `${url}/`; +} + +export function setupAudiobook(hooks: AudiobookHooks): Audiobook { + const ui = { + audioFile: el<HTMLInputElement>('audio-file'), + audio: el<HTMLAudioElement>('audio'), + lang: el<HTMLSelectElement>('ab-lang'), + cloud: el<HTMLButtonElement>('ab-cloud'), + local: el<HTMLButtonElement>('ab-local'), + stop: el<HTMLButtonElement>('ab-stop'), + status: el<HTMLDivElement>('ab-status'), + progress: el<HTMLProgressElement>('ab-progress'), + account: el<HTMLDivElement>('ab-account'), + email: el<HTMLInputElement>('ab-email'), + send: el<HTMLButtonElement>('ab-send'), + token: el<HTMLInputElement>('ab-token'), + verify: el<HTMLButtonElement>('ab-verify'), + server: el<HTMLInputElement>('ab-server'), + assets: el<HTMLInputElement>('ab-assets'), + }; + + const state = { + audio: null as File | null, + busy: false, + /** Stops the poll loop of the server job. */ + abort: null as AbortController | null, + /** The in-app job, while it runs. */ + job: null as Job | null, + http: null as Http | null, + }; + + // --- Settings --- + + function server(): string { + return (read(KEYS.server) ?? DEFAULT_SERVER).replace(/\/+$/, ''); + } + + function assets(): string { + return withSlash(read(KEYS.assets) ?? defaultAssets()); + } + + function http(): Http { + state.http ??= createHttp(server()); + return state.http; + } + + ui.server.value = read(KEYS.server) ?? ''; + ui.server.placeholder = DEFAULT_SERVER || '(dev proxy)'; + ui.assets.value = read(KEYS.assets) ?? ''; + ui.assets.placeholder = defaultAssets(); + + ui.server.addEventListener('change', () => { + write(KEYS.server, ui.server.value.trim() || null); + state.http = null; + void refreshAccount(); + }); + ui.assets.addEventListener('change', () => { + write(KEYS.assets, ui.assets.value.trim() || null); + }); + + // --- Status --- + + function setStatus(text: string, link?: { href: string; label: string }): void { + ui.status.replaceChildren(text); + if (link) { + const a = document.createElement('a'); + a.href = link.href; + a.target = '_blank'; + a.rel = 'noopener'; + a.textContent = link.label; + ui.status.append(' ', a); + } + } + + function setProgress(value: number | null): void { + ui.progress.hidden = value === null; + if (value !== null) ui.progress.value = value; + } + + function updateButtons(): void { + const ready = !state.busy && state.audio !== null && hooks.tokens().length > 0; + ui.cloud.disabled = !ready; + ui.local.disabled = !ready; + ui.stop.hidden = !state.busy; + } + + function setBusy(busy: boolean): void { + state.busy = busy; + updateButtons(); + if (!busy) setProgress(null); + } + + // --- Account --- + + async function refreshAccount(): Promise<void> { + try { + const account = await getAccount(http()); + ui.account.textContent = `Account: ${accountLine(account)}`; + } catch (err) { + ui.account.textContent = `Account: cannot reach the server (${String(err)}).`; + } + } + + ui.send.addEventListener('click', async () => { + ui.send.disabled = true; + try { + const r = await requestSignIn(http(), ui.email.value); + ui.account.textContent = r.dev_link + ? `Account: no mail server. Paste this link below: ${r.dev_link}` + : `Account: link sent to ${r.email}. Open it, then paste the link or token below.`; + } catch (err) { + ui.account.textContent = `Account: ${String(err)}`; + } finally { + ui.send.disabled = false; + } + }); + + ui.verify.addEventListener('click', async () => { + ui.verify.disabled = true; + try { + const account = await verifySignIn(http(), tokenFrom(ui.token.value)); + ui.token.value = ''; + ui.account.textContent = `Account: ${accountLine(account)}`; + } catch (err) { + ui.account.textContent = `Account: ${String(err)}`; + } finally { + ui.verify.disabled = false; + } + }); + + // --- Audio file and the SRT cache --- + + ui.audioFile.addEventListener('change', async () => { + const file = ui.audioFile.files?.[0]; + if (!file) return; + state.audio = file; + ui.audio.src = URL.createObjectURL(file); + updateButtons(); + const cached = await getSrt(audioKey(file)).catch(() => undefined); + if (cached) { + hooks.loadSrt(cached, 'the cache'); + setStatus(`Subtitles for ${file.name} loaded from the cache.`); + } else { + setStatus(`${file.name}: no subtitles yet.`); + } + }); + + /** Keep the SRT, and show it when its audio file is the picked one. */ + async function finish(srt: string, key: string, name: string, source: string): Promise<void> { + await putSrt(key, srt).catch(() => {}); + if (state.audio && audioKey(state.audio) === key) { + hooks.loadSrt(srt, source); + setStatus(`Subtitles ready from ${source}.`); + } else { + setStatus(`Subtitles for ${name} are ready. Pick that audio file to load them.`); + } + } + + // --- Server job --- + + function jobLine(job: JobOut): string { + const pct = Math.round(job.progress * 100); + return `Server job ${job.status}: ${job.stage} (${pct}%)`; + } + + /** Poll a server job to its end and load the SRT. */ + async function follow(pending: PendingJob): Promise<void> { + const abort = new AbortController(); + state.abort = abort; + try { + const job = await pollUntilDone(http(), pending.id, { + intervalMs: POLL_MS, + signal: abort.signal, + onStatus: (j) => { + setStatus(jobLine(j)); + setProgress(j.progress); + }, + }); + write(KEYS.job, null); + if (job.status === 'succeeded') { + const srt = await fetchSrt(http(), job.id); + await finish(srt, pending.audioKey, pending.audioName, 'subread.space'); + } else { + setStatus(`Server job ${job.status}: ${job.error ?? job.stage}`); + } + } finally { + state.abort = null; + } + } + + ui.cloud.addEventListener('click', async () => { + const audio = state.audio; + if (!audio || state.busy) return; + setBusy(true); + try { + setStatus(`Uploading ${audio.name} and the OCR text...`); + setProgress(0); + const job = await upload(http(), audio, bookFile(hooks.tokens())); + const pending: PendingJob = { id: job.id, audioKey: audioKey(audio), audioName: audio.name }; + write(KEYS.job, JSON.stringify(pending)); + try { + await startJob(http(), job.id, ui.lang.value); + } catch (err) { + write(KEYS.job, null); + if (err instanceof ApiError && err.status === 402) { + setStatus(`No credit: ${err.message}`, { href: SITE, label: 'Buy credits or sign in.' }); + return; + } + throw err; + } + await follow(pending); + void refreshAccount(); + } catch (err) { + if ((err as Error).name !== 'AbortError') setStatus(`Server job failed: ${String(err)}`); + } finally { + setBusy(false); + } + }); + + // --- In-app job --- + + function phaseLine(s: JobStatus): string { + let line = `${s.phase}: ${s.detail}`; + if (s.eta) line += `, about ${Math.round(s.eta / 60)} min left`; + return line; + } + + ui.local.addEventListener('click', async () => { + const audio = state.audio; + if (!audio || state.busy) return; + setBusy(true); + let wakeLock: WakeLockSentinel | null = null; + try { + setStatus('preparing: loading the engine'); + setProgress(0); + const { Job, Cancelled } = await import('./engine/job.js'); + const job = new Job({ + audio: [audio], + paragraphs: paragraphs(bookText(hooks.tokens())), + language: ui.lang.value, + assets: assets(), + onStatus: (s) => { + setStatus(phaseLine(s)); + setProgress(s.indeterminate ? null : s.fraction); + }, + }); + state.job = job; + try { + wakeLock = (await navigator.wakeLock?.request('screen')) ?? null; + } catch { + // Not granted. The user keeps the screen on by hand. + } + try { + const result = await job.run(); + const pct = Math.round(result.matchRate * 100); + await finish(result.srt, audioKey(audio), audio.name, `this app (${pct}% matched)`); + } catch (err) { + setStatus(err instanceof Cancelled ? 'Stopped.' : `In-app job failed: ${String(err)}`); + } finally { + job.close(); + } + } catch (err) { + setStatus(`Cannot load the engine: ${String(err)}`); + } finally { + state.job = null; + await wakeLock?.release().catch(() => {}); + setBusy(false); + } + }); + + ui.stop.addEventListener('click', async () => { + if (state.job) { + state.job.cancel(); + return; + } + const raw = read(KEYS.job); + state.abort?.abort(); + if (raw) { + const pending = JSON.parse(raw) as PendingJob; + write(KEYS.job, null); + try { + await cancelJob(http(), pending.id); + setStatus('Server job canceled.'); + } catch (err) { + setStatus(`Cancel failed: ${String(err)}`); + } + } + setBusy(false); + }); + + // --- Start --- + + const raw = read(KEYS.job); + if (raw) { + const pending = JSON.parse(raw) as PendingJob; + setStatus(`Resuming the server job for ${pending.audioName}...`); + setBusy(true); + follow(pending) + .catch((err: unknown) => { + if ((err as Error).name !== 'AbortError') setStatus(`Server job failed: ${String(err)}`); + }) + .finally(() => setBusy(false)); + } + void refreshAccount(); + updateButtons(); + + return { + onTokens() { + ui.lang.value = whisperCode(hooks.ocrLang()); + updateButtons(); + }, + }; +} diff --git a/src/cloud.ts b/src/cloud.ts new file mode 100644 index 0000000..00f6411 --- /dev/null +++ b/src/cloud.ts @@ -0,0 +1,174 @@ +// Client for the subread.space job API. Pure over an injected `Http`, so the +// flow is testable with a fake transport. +// +// Flow: POST /api/uploads (audio + book.txt) -> POST /api/jobs/{id}/start +// -> poll GET /api/jobs/{id} -> GET /api/jobs/{id}/files/srt. +// Identity is the `subplz_device` cookie that the server sets on the first +// response. The transport keeps it (browser cookie jar or native cookie store). + +export interface HttpRequest { + method: 'GET' | 'POST' | 'DELETE'; + /** Path under the server base, such as "/api/account". */ + path: string; + /** JSON body. */ + json?: unknown; + /** Multipart body: each file goes in the `files` field. */ + files?: File[]; +} + +export interface HttpResponse { + status: number; + /** The body as text. JSON is parsed by the caller. */ + text: string; +} + +export type Http = (req: HttpRequest) => Promise<HttpResponse>; + +/** A non-2xx answer. `status` 402 means: no credit. */ +export class ApiError extends Error { + constructor( + public readonly status: number, + message: string, + ) { + super(message); + this.name = 'ApiError'; + } +} + +export interface JobOut { + id: string; + /** draft, queued, running, succeeded, failed or canceled. */ + status: string; + stage: string; + progress: number; + language: string; + error: string | null; + artifacts: { kind: string; filename: string; url: string }[]; +} + +export interface AccountOut { + id: string; + signed_in: boolean; + email: string | null; + credits: number; + free_credits: number; + cloud_allowed: boolean; + billing_enabled: boolean; + email_sign_in_available: boolean; +} + +export const DONE = new Set(['succeeded', 'failed', 'canceled']); + +/** Where to buy credits or sign in. */ +export const SITE = 'https://subread.space'; + +/** Read the JSON of a 2xx answer. Throw ApiError with the server's `detail` otherwise. */ +async function call<T>(http: Http, req: HttpRequest): Promise<T> { + const res = await http(req); + if (res.status < 200 || res.status >= 300) { + let detail = `HTTP ${res.status}`; + try { + const body = JSON.parse(res.text) as { detail?: unknown }; + if (typeof body.detail === 'string') detail = body.detail; + else if (body.detail !== undefined) detail = JSON.stringify(body.detail); + } catch { + // Not JSON. Keep the status line. + } + throw new ApiError(res.status, detail); + } + return JSON.parse(res.text) as T; +} + +export async function upload(http: Http, audio: File, book: File): Promise<JobOut> { + const out = await call<{ job: JobOut }>(http, { + method: 'POST', + path: '/api/uploads', + files: [audio, book], + }); + return out.job; +} + +export function startJob(http: Http, id: string, language: string): Promise<JobOut> { + return call<JobOut>(http, { method: 'POST', path: `/api/jobs/${id}/start`, json: { language } }); +} + +export function getJob(http: Http, id: string): Promise<JobOut> { + return call<JobOut>(http, { method: 'GET', path: `/api/jobs/${id}` }); +} + +export function cancelJob(http: Http, id: string): Promise<JobOut> { + return call<JobOut>(http, { method: 'POST', path: `/api/jobs/${id}/cancel`, json: {} }); +} + +export function deleteJob(http: Http, id: string): Promise<unknown> { + return call<unknown>(http, { method: 'DELETE', path: `/api/jobs/${id}` }); +} + +export interface PollOptions { + intervalMs: number; + onStatus?: (job: JobOut) => void; + /** Injectable for tests. Defaults to setTimeout. */ + sleep?: (ms: number) => Promise<void>; + signal?: AbortSignal; +} + +const defaultSleep = (ms: number): Promise<void> => + new Promise((resolve) => setTimeout(resolve, ms)); + +/** + * Poll the job until it is succeeded, failed or canceled. Each answer goes + * to `onStatus`. An aborted signal stops the loop with an AbortError. + */ +export async function pollUntilDone(http: Http, id: string, opts: PollOptions): Promise<JobOut> { + const sleep = opts.sleep ?? defaultSleep; + for (;;) { + if (opts.signal?.aborted) throw new DOMException('Polling stopped.', 'AbortError'); + const job = await getJob(http, id); + opts.onStatus?.(job); + if (DONE.has(job.status)) return job; + await sleep(opts.intervalMs); + } +} + +/** The finished SRT. The server may redirect to a presigned URL. The transport follows it. */ +export async function fetchSrt(http: Http, id: string): Promise<string> { + const res = await http({ method: 'GET', path: `/api/jobs/${id}/files/srt` }); + if (res.status < 200 || res.status >= 300) throw new ApiError(res.status, `HTTP ${res.status}`); + return res.text; +} + +export function getAccount(http: Http): Promise<AccountOut> { + return call<AccountOut>(http, { method: 'GET', path: '/api/account' }); +} + +export function requestSignIn( + http: Http, + email: string, +): Promise<{ sent: boolean; email: string; dev_link?: string }> { + return call(http, { method: 'POST', path: '/api/auth/request', json: { email } }); +} + +export function verifySignIn(http: Http, token: string): Promise<AccountOut> { + return call<AccountOut>(http, { method: 'POST', path: '/api/auth/verify', json: { token } }); +} + +/** + * The token out of what the user pasted: a sign-in link + * (https://subread.space/?login=TOKEN) or the bare token. + */ +export function tokenFrom(input: string): string { + const s = input.trim(); + try { + const url = new URL(s); + return url.searchParams.get('login') ?? s; + } catch { + return s; + } +} + +/** One line for the account area of the drawer. */ +export function accountLine(a: AccountOut): string { + const who = a.signed_in && a.email ? `Signed in as ${a.email}.` : 'Not signed in.'; + const credits = a.billing_enabled ? ` Credits: ${a.credits} (${a.free_credits} free).` : ''; + return who + credits; +} diff --git a/src/http.ts b/src/http.ts new file mode 100644 index 0000000..bb721f9 --- /dev/null +++ b/src/http.ts @@ -0,0 +1,106 @@ +// The two transports behind `Http` (see cloud.ts). +// +// Web: `fetch` with relative paths. In development the Vite proxy forwards +// /api to the server, so the cookie is same-origin. +// +// Android: the WebView origin is https://localhost and the server sends no +// CORS headers, so a browser fetch fails. `CapacitorHttp.request` runs the +// request natively. Its cookies live in the WebView cookie store, which +// persists across restarts. A multipart body goes over the bridge as +// base64 entries, the same shape Capacitor's own fetch patch uses. + +import { Capacitor, CapacitorHttp } from '@capacitor/core'; +import type { Http, HttpRequest } from './cloud'; + +/** Default server per platform: the dev proxy on the web, the real host in the app. */ +export const DEFAULT_SERVER = Capacitor.isNativePlatform() ? 'https://subread.space' : ''; + +/** Default engine asset base per platform: the dev proxy on the web, the real host in the app. */ +export function defaultAssets(): string { + return Capacitor.isNativePlatform() + ? 'https://subread.space/vendor/' + : `${location.origin}/vendor/`; +} + +/** Bytes per `String.fromCharCode` call. More overflows the argument list. */ +const CHUNK = 0x8000; + +/** Read a file as base64. The whole file is in memory while it is encoded. */ +export async function fileToBase64(file: Blob): Promise<string> { + const bytes = new Uint8Array(await file.arrayBuffer()); + let binary = ''; + for (let i = 0; i < bytes.length; i += CHUNK) { + binary += String.fromCharCode(...bytes.subarray(i, i + CHUNK)); + } + return btoa(binary); +} + +export interface FormEntry { + key: string; + value: string; + type: 'base64File'; + contentType: string; + fileName: string; +} + +/** The multipart entries that the native HTTP plugin writes as `files` parts. */ +export async function formEntries(files: readonly File[]): Promise<FormEntry[]> { + const out: FormEntry[] = []; + for (const file of files) { + out.push({ + key: 'files', + value: await fileToBase64(file), + type: 'base64File', + contentType: file.type || 'application/octet-stream', + fileName: file.name, + }); + } + return out; +} + +export function webHttp(base: string): Http { + return async (req: HttpRequest) => { + const init: RequestInit = { method: req.method, credentials: 'same-origin' }; + if (req.files) { + const form = new FormData(); + for (const f of req.files) form.append('files', f, f.name); + init.body = form; + } else if (req.json !== undefined) { + init.headers = { 'Content-Type': 'application/json' }; + init.body = JSON.stringify(req.json); + } + const res = await fetch(base + req.path, init); + return { status: res.status, text: await res.text() }; + }; +} + +export function nativeHttp(base: string): Http { + return async (req: HttpRequest) => { + const headers: Record<string, string> = {}; + let data: unknown; + let dataType: 'formData' | undefined; + if (req.files) { + data = await formEntries(req.files); + dataType = 'formData'; + } else if (req.json !== undefined) { + headers['Content-Type'] = 'application/json'; + data = JSON.stringify(req.json); + } + const res = await CapacitorHttp.request({ + url: base + req.path, + method: req.method, + headers, + ...(data !== undefined ? { data } : {}), + ...(dataType ? { dataType } : {}), + responseType: 'text', + readTimeout: 120_000, + connectTimeout: 30_000, + }); + const text = typeof res.data === 'string' ? res.data : JSON.stringify(res.data); + return { status: res.status, text }; + }; +} + +export function createHttp(base: string): Http { + return Capacitor.isNativePlatform() ? nativeHttp(base) : webHttp(base); +} diff --git a/src/main.ts b/src/main.ts index 4efdffa..4ce3358 100644 --- a/src/main.ts +++ b/src/main.ts @@ -1,6 +1,7 @@ // UI wiring. Alignment logic lives in align.ts, lookup logic in lookup.ts. import { alignCuesToTokens, type OcrToken, type TokenSpan } from './align'; +import { setupAudiobook } from './audiobook'; import { deleteDictionary, importDictionary, @@ -40,7 +41,6 @@ const ui = { menu: el<HTMLButtonElement>('menu'), panel: el<HTMLElement>('panel'), pdfFile: el<HTMLInputElement>('pdf-file'), - audioFile: el<HTMLInputElement>('audio-file'), subFile: el<HTMLInputElement>('sub-file'), dictFile: el<HTMLInputElement>('dict-file'), dicts: el<HTMLUListElement>('dicts'), @@ -125,21 +125,26 @@ ui.pdfFile.addEventListener('change', async () => { updateRunButton(); }); -ui.subFile.addEventListener('change', async () => { - const file = ui.subFile.files?.[0]; - if (!file) return; - state.cues = parseSubtitles(await file.text()); +/** Load subtitles from a file or from a finished job. `source` names where they came from. */ +function loadSubtitles(text: string, source: string): void { + state.cues = parseSubtitles(text); state.spans = []; state.activeCue = -1; - setStatus(`${state.cues.length} cues loaded.`); + setStatus(`${state.cues.length} cues loaded from ${source}.`); alignIfReady(); renderCueList(); -}); +} -ui.audioFile.addEventListener('change', () => { - const file = ui.audioFile.files?.[0]; +ui.subFile.addEventListener('change', async () => { + const file = ui.subFile.files?.[0]; if (!file) return; - ui.audio.src = URL.createObjectURL(file); + loadSubtitles(await file.text(), file.name); +}); + +const audiobook = setupAudiobook({ + tokens: () => state.tokens, + ocrLang: () => ui.lang.value, + loadSrt: loadSubtitles, }); // --- Dictionaries --- @@ -312,6 +317,7 @@ ui.run.addEventListener('click', async () => { await ocr.terminate(); } setStatus(`Done. ${state.tokens.length} characters read. Tap a word to look it up.`); + audiobook.onTokens(); alignIfReady(); renderCueList(); state.activeCue = -1; diff --git a/src/srt-cache.ts b/src/srt-cache.ts new file mode 100644 index 0000000..17b391f --- /dev/null +++ b/src/srt-cache.ts @@ -0,0 +1,34 @@ +// Finished subtitles per audio file, in IndexedDB. Thin browser layer. +// +// Key: audio file name and size. A book is aligned once; the next time the +// same audio file is picked, the subtitles load from here. + +import { openDB, type DBSchema, type IDBPDatabase } from 'idb'; + +interface Schema extends DBSchema { + srt: { key: string; value: string }; +} + +let dbPromise: Promise<IDBPDatabase<Schema>> | undefined; + +function db(): Promise<IDBPDatabase<Schema>> { + dbPromise ??= openDB<Schema>('scan-subread-srt', 1, { + upgrade(d) { + d.createObjectStore('srt'); + }, + }); + return dbPromise; +} + +/** The cache key of an audio file. */ +export function audioKey(file: Pick<File, 'name' | 'size'>): string { + return `${file.name}|${file.size}`; +} + +export async function getSrt(key: string): Promise<string | undefined> { + return (await db()).get('srt', key); +} + +export async function putSrt(key: string, srt: string): Promise<void> { + await (await db()).put('srt', srt, key); +} diff --git a/src/style.css b/src/style.css index 8b99469..96a7ad3 100644 --- a/src/style.css +++ b/src/style.css @@ -102,28 +102,46 @@ body.menu-open #panel { flex: 1; } -#local-audio { +#local-audio, +#audiobook { display: flex; flex-direction: column; gap: 0.4rem; font-size: 0.85rem; } -#local-audio h2 { +#local-audio h2, +#audiobook h2 { font-size: 0.95rem; margin: 0; } -#local-audio .row { +#local-audio .row, +#audiobook .row { display: flex; + flex-wrap: wrap; gap: 0.5rem; } #local-audio code, -#la-status { +#la-status, +#ab-status, +#ab-account { word-break: break-all; } +#ab-status a { + color: inherit; +} + +#audiobook details { + font-size: 0.85rem; +} + +#audiobook details label { + margin-top: 0.4rem; +} + #audio { width: 100%; } diff --git a/test/cloud.test.ts b/test/cloud.test.ts new file mode 100644 index 0000000..19d4b58 --- /dev/null +++ b/test/cloud.test.ts @@ -0,0 +1,155 @@ +import { describe, expect, it } from 'vitest'; +import { + accountLine, + ApiError, + fetchSrt, + pollUntilDone, + startJob, + tokenFrom, + upload, + type Http, + type HttpRequest, + type JobOut, +} from '../src/cloud'; +import { parseSubtitles } from '../src/subtitles'; + +const job = (status: string, extra: Partial<JobOut> = {}): JobOut => ({ + id: 'job_1', + status, + stage: status, + progress: 0, + language: 'ja', + error: null, + artifacts: [], + ...extra, +}); + +const json = (status: number, body: unknown) => ({ status, text: JSON.stringify(body) }); + +/** A fake transport: one canned answer per call, in order, plus the log of requests. */ +function fakeHttp(answers: { status: number; text: string }[]): { http: Http; log: HttpRequest[] } { + const log: HttpRequest[] = []; + const http: Http = async (req) => { + log.push(req); + const next = answers.shift(); + if (!next) throw new Error(`No canned answer for ${req.method} ${req.path}`); + return next; + }; + return { http, log }; +} + +const noSleep = async (): Promise<void> => {}; + +describe('pollUntilDone', () => { + it('polls queued, running, then returns the succeeded job', async () => { + const { http, log } = fakeHttp([ + json(200, job('queued')), + json(200, job('running', { progress: 0.5 })), + json(200, job('succeeded', { progress: 1 })), + ]); + const seen: string[] = []; + const done = await pollUntilDone(http, 'job_1', { + intervalMs: 5000, + sleep: noSleep, + onStatus: (j) => seen.push(j.status), + }); + expect(done.status).toBe('succeeded'); + expect(seen).toEqual(['queued', 'running', 'succeeded']); + expect(log.map((r) => r.path)).toEqual([ + '/api/jobs/job_1', + '/api/jobs/job_1', + '/api/jobs/job_1', + ]); + }); + + it('returns a failed job with its error', async () => { + const { http } = fakeHttp([json(200, job('failed', { error: 'No speech found.' }))]); + const done = await pollUntilDone(http, 'job_1', { intervalMs: 0, sleep: noSleep }); + expect(done.status).toBe('failed'); + expect(done.error).toBe('No speech found.'); + }); + + it('stops when the signal is aborted', async () => { + const ctrl = new AbortController(); + const { http } = fakeHttp([json(200, job('running'))]); + await expect( + pollUntilDone(http, 'job_1', { + intervalMs: 0, + sleep: async () => ctrl.abort(), + signal: ctrl.signal, + }), + ).rejects.toMatchObject({ name: 'AbortError' }); + }); +}); + +describe('startJob', () => { + it('sends the language and returns the job', async () => { + const { http, log } = fakeHttp([json(200, job('queued'))]); + const out = await startJob(http, 'job_1', 'ja'); + expect(out.status).toBe('queued'); + expect(log[0]).toMatchObject({ + method: 'POST', + path: '/api/jobs/job_1/start', + json: { language: 'ja' }, + }); + }); + + it('throws ApiError 402 with the server detail when there is no credit', async () => { + const { http } = fakeHttp([json(402, { detail: 'No credit left.' })]); + const err = await startJob(http, 'job_1', 'ja').catch((e: unknown) => e); + expect(err).toBeInstanceOf(ApiError); + expect((err as ApiError).status).toBe(402); + expect((err as ApiError).message).toBe('No credit left.'); + }); +}); + +describe('upload', () => { + it('posts both files in the `files` field', async () => { + const audio = new File(['x'], 'book.wav', { type: 'audio/wav' }); + const book = new File(['a\n'], 'book.txt', { type: 'text/plain' }); + const { http, log } = fakeHttp([json(200, { job: job('draft'), detected: {} })]); + const out = await upload(http, audio, book); + expect(out.id).toBe('job_1'); + expect(log[0]?.files?.map((f) => f.name)).toEqual(['book.wav', 'book.txt']); + }); +}); + +describe('fetchSrt', () => { + it('returns SRT that the subtitle parser reads', async () => { + const srt = '1\n00:00:01,000 --> 00:00:03,500\n朝の光\n\n'; + const { http } = fakeHttp([{ status: 200, text: srt }]); + const cues = parseSubtitles(await fetchSrt(http, 'job_1')); + expect(cues).toEqual([{ start: 1, end: 3.5, text: '朝の光' }]); + }); +}); + +describe('tokenFrom', () => { + it('reads the token out of a sign-in link or takes the bare token', () => { + expect(tokenFrom('https://subread.space/?login=abc123')).toBe('abc123'); + expect(tokenFrom(' abc123 \n')).toBe('abc123'); + }); +}); + +describe('accountLine', () => { + const base = { + id: 'acc', + signed_in: false, + email: null, + credits: 1, + free_credits: 1, + cloud_allowed: true, + billing_enabled: true, + email_sign_in_available: true, + }; + + it('shows the sign-in state and the credits', () => { + expect(accountLine(base)).toBe('Not signed in. Credits: 1 (1 free).'); + expect(accountLine({ ...base, signed_in: true, email: 'a@b.c', credits: 3 })).toBe( + 'Signed in as a@b.c. Credits: 3 (1 free).', + ); + }); + + it('hides credits when billing is off', () => { + expect(accountLine({ ...base, billing_enabled: false })).toBe('Not signed in.'); + }); +}); diff --git a/test/http.test.ts b/test/http.test.ts new file mode 100644 index 0000000..a58ae99 --- /dev/null +++ b/test/http.test.ts @@ -0,0 +1,41 @@ +import { describe, expect, it } from 'vitest'; +import { fileToBase64, formEntries } from '../src/http'; + +describe('formEntries', () => { + it('encodes each file as one base64 `files` part', async () => { + const files = [ + new File([new Uint8Array([0, 1, 2, 255])], 'a.wav', { type: 'audio/wav' }), + new File(['hi\n'], 'book.txt', { type: 'text/plain' }), + ]; + const entries = await formEntries(files); + expect(entries).toEqual([ + { + key: 'files', + value: 'AAEC/w==', + type: 'base64File', + contentType: 'audio/wav', + fileName: 'a.wav', + }, + { + key: 'files', + value: 'aGkK', + type: 'base64File', + contentType: 'text/plain', + fileName: 'book.txt', + }, + ]); + }); + + it('falls back to octet-stream for a file without a type', async () => { + const [entry] = await formEntries([new File(['x'], 'raw')]); + expect(entry?.contentType).toBe('application/octet-stream'); + }); +}); + +describe('fileToBase64', () => { + it('round trips bytes', async () => { + const bytes = new Uint8Array(300).map((_, i) => i % 256); + const b64 = await fileToBase64(new Blob([bytes])); + expect(Uint8Array.from(atob(b64), (c) => c.charCodeAt(0))).toEqual(bytes); + }); +});