README.md (6064 bytes)
1 # SubRead for Android 2 3 Times an audiobook against its ebook, entirely on the phone, and writes the 4 `.srt` that [Hoshi Reader](https://github.com/HuangAntimony/Hoshi-Reader-Android)'s 5 read-along uses. Nothing is uploaded. 6 7 It also makes **videos + subtitles which can be played with any video player or 8 uploaded to YouTube**: an `.mp4` of the book's cover and the audio, with the 9 `.srt` beside it under the same name. 10 11 <p> 12 <img src="docs/screenshots/1-pick.png" width="300" alt="The first screen: what the app takes, how it works, and the two files to pick"> 13 <img src="docs/screenshots/2-outputs.png" width="300" alt="The outputs: subtitles, and video + subtitles with its size and frame rate"> 14 </p> 15 16 It is the same method as [subread.space](https://subread.space) and 17 [SubPlz](https://github.com/kanjieater/SubPlz): a small speech model transcribes 18 the audio, roughly; the transcript is aligned against the book; the subtitles 19 take their *timing* from the transcript and their *words* from the book. 20 21 ## Other apps 22 23 A reader or an audiobook player can ask SubRead for the `.srt` of a book and get 24 the file back: it starts SubRead with `space.subread.app.action.ALIGN`, the two 25 files as extras, and reads the subtitles from the result. The other app needs no 26 code of SubRead and no speech model of its own. The contract, with the extras 27 and the answer, is in [docs/intent-api.md](docs/intent-api.md). 28 29 ## Layout 30 31 | Path | What | 32 |---|---| 33 | `core/` | The alignment engine. Plain Kotlin, no Android - builds and tests with only a JDK | 34 | `app/` | The Android app: audio decoding, the whisper.cpp bridge, the long-running job, the UI | 35 | `third_party/whisper.cpp` | Pinned submodule | 36 | `tools/make_golden.py` | Generates test fixtures by running the reference Python implementation | 37 38 ## The engine 39 40 `core/` is a port of SubPlz's aligner (`ats.align`, `subplz.align.shift_align`), 41 checked stage by stage against the original's real output on real Whisper-tiny 42 transcripts: every cue across the fixtures comes out identical. 43 44 What is new is `AnchoredAligner`. The reference hands Biopython one chapter at a 45 time and needs gigabytes to do it, so it has to guess first which chapter of the 46 book each chapter of audio is. This aligns the whole book at once instead, by 47 anchoring on stretches unique to both texts and solving exactly only between 48 anchors: a 19-hour audiobook against its full text in a few seconds and a few 49 megabytes. `BookAligner` then uses the alignment itself to leave out text nobody 50 narrated - front matter, notes - instead of matching chapters. 51 52 ```bash 53 ./gradlew :core:test 54 ``` 55 56 ## The video 57 58 The picture of the video does not change, so almost none of it is encoded. 59 The device's H.264 encoder makes one key frame. `H264Still` writes each frame 60 after it by hand: a P slice in which each macroblock is skipped, about ten 61 bytes. One minute of these frames is written again and again with new time 62 stamps. AAC audio is copied; other audio is encoded to AAC once. 63 64 The result for a 10-hour book is about 60 MB of video at 720p and one frame a 65 second. The frame rate adds about 20 bytes for each frame. The size of the 66 picture changes only the key frame, one each minute. The sizes on offer are 67 those the device has an encoder for (1080p in software, more with a hardware 68 encoder). No Android encoder makes 4K or 8K H.264 today. 69 70 ## Building the app 71 72 CI builds it (`.github/workflows/build.yml`): the APK is an artifact of every 73 push to `main`. Locally it needs the Android SDK, NDK 29 and CMake 3.31. Without 74 an SDK, Gradle leaves `:app` out and `:core` still builds. 75 76 The speech model (`app/src/main/assets/models/ggml-tiny-q8_0.bin`, 43 MB, MIT 77 licence) is in the repository. It is the file of the whisper.cpp project, and 78 the workflow checks its hash. It is in git and not downloaded by the build, so 79 that a build needs no network: F-Droid asks for that, and the app itself has no 80 network permission. 81 82 The release key is not in the repository; CI reads it from Actions secrets. 83 84 ## Releases 85 86 Each release is a tag (`vMAJOR.MINOR.PATCH`) and a GitHub release with the signed 87 APK attached. Set `versionName` (the tag without the `v`) and `versionCode` in 88 `app/build.gradle.kts` first, and write `fastlane/metadata/android/en-US/changelogs/<versionCode>.txt`: 89 90 ```bash 91 ./gradlew :app:assembleRelease 92 git tag -a v0.2.0 -m "what changed" && git push origin main --tags 93 gh release create v0.2.0 app/build/outputs/apk/release/app-release.apk --notes "what changed" 94 ``` 95 96 `versionCode` must go up every release, or Android refuses the update. 97 98 Attach the APK two times: as `SubRead-<version>.apk`, and as `SubRead.apk`. Hoshi Reader 99 installs SubRead from `releases/latest/download/SubRead.apk`, a link that needs no version. 100 101 ### Google Play 102 103 ```bash 104 ./gradlew :app:bundleRelease -PplayStore=true 105 ``` 106 107 `-PplayStore=true` leaves the Ko-fi link out. Each other build has it. The texts, 108 the graphics and the answers to the Console's questions are in `play/`. 109 110 ### F-Droid 111 112 The build has what F-Droid asks for: free licences only, no network at build 113 time, no Google dependency list in the APK, and the version as plain numbers in 114 the build file. F-Droid reads the listing from `fastlane/metadata/android/`, and 115 finds new releases by their tags. `fdroid/space.subread.app.yml` is the recipe 116 for a merge request to [fdroiddata](https://gitlab.com/fdroid/fdroiddata). 117 F-Droid signs its builds with its own key, so an install from F-Droid and one 118 from GitHub or Google Play do not update each other. 119 120 ## Requirements 121 122 Android 8+, a 64-bit ARM processor with ARMv8.2 half-precision and dot-product 123 instructions (anything from 2018 on). Transcription runs at a small multiple of 124 real time, so a long book takes hours. The job runs only while the app is open 125 and keeps the screen on. An interrupted job continues from its last finished chunk. 126 127 ## Licence 128 129 AGPL-3.0: see `LICENSE`. You may use, change and host this, and you must give 130 your users the source of what you host. `NOTICE` has the licences of the work 131 this is built on (SubPlz, whisper.cpp, Whisper).