README.md (6675 bytes)
1 # Subrep for Android 2 3 [Subrep](https://honjimaku.com/subrep/) on a phone: live captions of the 4 sound of the device, made on the device by Whisper, shared by link. The 5 captions on the phone are free. Cloud captions, from a larger model, cost 6 hours that you buy on subread.space. 7 8 The app captures the sound of the apps on the phone (a video, a game, a 9 browser) or the microphone. Whisper, on the phone, turns it into captions. 10 The captions go to your share link on `honjimaku.com`, where the viewers read 11 along and mine with Yomitan, and to 12 [SubRead Overlay](https://github.com/equwal/subread-overlay) over the player, 13 where a tap on a word opens the dictionary. With captions on the phone, no 14 sound leaves the phone. 15 16 ## Screenshots 17 18 <p> 19 <img src="fastlane/metadata/android/en-US/images/phoneScreenshots/1.png" width="260" alt="The settings: the speech model, the language, the sound and the share link"> 20 </p> 21 22 The picture is from a Viwoods AiPaper Reader. 23 24 ## How to use it 25 26 1. Choose where the captions are made: on this phone (free), or in the 27 cloud. For the cloud, buy hours with a button under the choice, then come 28 back to the app. The cloud needs no model: go to step 3. 29 2. Choose the speech model and press "Download". `base` (82 MB) keeps up on 30 most phones. `small` (264 MB) reads better and comes late. `tiny` is for 31 an old phone. 32 3. Enter the language: `ja`, `en`, `ru`, ... or `auto`. A set language is 33 faster and more exact. 34 4. Choose the sound: the apps on this device (Android 10 or later), or the 35 microphone. 36 5. Press "Start the captions". Android asks for the capture of the screen 37 ("Share entire screen"), and once for the microphone permission. That 38 permission covers the sound of the apps too. 39 6. Start the video. The captions come to the app, to the link and to the 40 overlay. The notification has a "Stop" button. 41 42 The share link stays the same across restarts. "New link" makes another one, 43 and the old one stops. 44 45 ## Cloud captions 46 47 The cloud uses Whisper large-v3 turbo on DeepInfra, through subread.space. It is 48 more exact than the models that a phone can run, and it uses less battery. 49 50 | Pack | Price | 51 |---|---| 52 | 20 hours | $4.99 | 53 | 80 hours | $16.99 | 54 | 200 hours | $39.00 | 55 56 Each piece of speech costs its length, rounded up to the next second. 57 Silence costs nothing: the app sends only speech. The hours do not expire. 58 59 The hours belong to the account of this app on subread.space. The screen 60 shows its id (`acct_...`). If you install the app again, the app gets a new 61 account: write to the contact address on subread.space with the old id and 62 the receipt, and we move the hours. 63 64 What goes where: each piece of speech goes to subread.space, which gives it 65 to DeepInfra for the text. subread.space keeps neither the sound nor the 66 text. DeepInfra says that it does not store them ([data privacy](https://docs.deepinfra.com/account/data-privacy)). 67 The app sends nothing while the phone makes the captions. 68 69 ## What the phone cannot capture 70 71 - An app that refuses the capture of its sound: a video app with DRM, and 72 some music apps. Android gives silence. Use the microphone. 73 - Sound from another device over Bluetooth. A phone is a Bluetooth source, 74 not a speaker. 75 76 ## How it works 77 78 `CaptureService` reads 16 kHz mono PCM from `AudioRecord` (with an 79 `AudioPlaybackCaptureConfiguration` for the apps, or the microphone). 80 `Segmenter` cuts it into pieces at pauses, with the rules of the desktop 81 app: a block of 32 ms is speech above the noise floor times 3, a pause of 82 0.65 s closes a piece, 11 s is the most. `Engine` gives each piece to a 83 `Recognizer` on one thread: whisper.cpp (`Whisper`), or the cloud 84 (`CloudRecognizer`, which posts 16-bit PCM to 85 `https://subread.space/api/captions/transcribe`). Whisper runs with the encoder window cut to the length of the 86 piece, and drops the invented lines that `Hallucinations` lists. Each piece 87 is one final caption. 88 89 The relay protocol is that of `subrep.share`: a websocket to 90 `wss://honjimaku.com/subrep/pub/<room>`, a hello with the room secret, then 91 `{"type":"final","text":...,"tr":"","ts":...}` for each caption. 92 93 The overlay protocol is the `line` method of the content provider 94 `content://space.subread.overlay.player`. 95 96 `adb shell dumpsys activity service com.honjimaku.subrep/.CaptureService` 97 prints the state, for a bug report. 98 99 ## Build 100 101 whisper.cpp is a submodule: 102 103 ``` 104 git submodule update --init 105 ./gradlew :app:testDebugUnitTest :app:assembleDebug 106 ``` 107 108 The native library is built for 64-bit ARM with ARMv8.2 half-precision and 109 dot-product instructions (each phone since 2018). NDK 29 and CMake 3.31.6 110 from the Android SDK. 111 112 For a release, set `SUBREP_KEYSTORE_FILE`, `SUBREP_KEYSTORE_PASSWORD` and 113 `SUBREP_KEY_ALIAS` (default `subrep`), then `./gradlew :app:assembleRelease`. 114 115 ## More projects 116 117 - [SubRead](https://subread.space/): read along with an audiobook, in the browser. 118 Also [for Android](https://github.com/equwal/subread-android/releases/latest), 119 [for YouTube](https://github.com/equwal/subread-extension/releases/latest) 120 and [for KOReader](https://github.com/equwal/subread.koplugin). 121 - [SubRead Overlay](https://github.com/equwal/subread-overlay/releases/latest): subtitle lines over any Android media player. 122 - [SubRead Dictionary](https://github.com/equwal/subread-dictionary/releases/latest): a pop-up dictionary for Android that reads Yomitan dictionaries. 123 - [SubRead Anki](https://github.com/equwal/subread-anki): one tap makes an Anki card from any Android app. 124 - [Book Simulator](https://booksimulator.com/): a reading room for Aozora Bunko and Project Gutenberg books. 125 - [honjimaku.com](https://honjimaku.com/): subtitles for Japanese audiobooks. 126 - [sbm Sync](https://sbmsync.com/): your bookmarks, the same on every device, 127 with [sbm](https://github.com/equwal/sbm) for dmenu, 128 [sbm for Android](https://github.com/equwal/sbm-android/releases/latest) 129 and the [sbm add-on](https://github.com/equwal/sbm-extension/releases/latest) for Firefox and Chrome. 130 - [Rebind](https://github.com/equwal/rebind/releases): remap the hardware buttons of e-ink readers and Android, 131 with [Ink Recents](https://github.com/equwal/ink-recents/releases/latest), 132 [Ink Dim](https://github.com/equwal/ink-dim/releases/latest) 133 and [Ink Update](https://github.com/equwal/ink-update/releases/latest). 134 - [dickt.store](https://dickt.store/): language-learning tools, flashcards and web toys. 135 - [hentaibun.online](https://hentaibun.online/): learn kanbun and kobun. 136 - [Recently Written](https://recentlywritten.com/): the blog, and a list of [all projects](https://recentlywritten.com/projects.html). 137 138 ## Licence 139 140 AGPL-3.0. whisper.cpp is MIT.