commit b77f1bde4e330ec1135f4f640edd5c004fd72e08
equwal <truex@equwal.com>
2026-09-22 16:09:30 -0700
Add cloud captions with hours from subread.space
The captions on the phone stay free. A new first step chooses where the
captions are made: on this phone, or in the cloud (Whisper large-v3
turbo on Groq, through subread.space). The cloud needs no model, gives
better captions and uses less battery.
- Cloud.kt: the client of /api/captions. The first answer of the server
sets the device cookie; the app keeps it in Store.device, and the
hours belong to that account. It reads the balance and the packs,
opens the Stripe payment page in the browser, and posts each piece
of speech as 16-bit PCM.
- Recognizer: one interface for Whisper and CloudRecognizer.
LocalEngine is now Engine, because it runs both. A lost piece shows
an error and the next piece can work. No hours left stops the worker.
- The screen shows the hours left, the account id, the three packs
(20 h $4.99, 80 h $16.99, 200 h $39), and says where the sound goes.
It reads the balance again when the user comes back from the payment.
- Pcm.toBytes: the reverse of Pcm.toFloat.
- Version 0.2.0 (3). README: prices, account, what goes where.
Tests: CloudTest (11) against a local HTTP server of the JDK, with a
property test for the PCM round trip, and a test that the engine stops
when the hours are gone. 30 unit tests pass. Lint: 0 errors.
README.md | 49 ++++-
app/build.gradle.kts | 4 +-
app/src/main/cpp/whisper_jni.c | 2 +-
.../kotlin/com/honjimaku/subrep/CaptureService.kt | 14 +-
app/src/main/kotlin/com/honjimaku/subrep/Cloud.kt | 120 +++++++++++++
.../honjimaku/subrep/{LocalEngine.kt => Engine.kt} | 34 ++--
app/src/main/kotlin/com/honjimaku/subrep/Feed.kt | 11 +-
.../kotlin/com/honjimaku/subrep/MainActivity.kt | 97 +++++++++-
app/src/main/kotlin/com/honjimaku/subrep/Pcm.kt | 11 ++
.../main/kotlin/com/honjimaku/subrep/Recognizer.kt | 25 +++
app/src/main/kotlin/com/honjimaku/subrep/Store.kt | 10 ++
.../main/kotlin/com/honjimaku/subrep/Whisper.kt | 6 +-
app/src/main/res/values/strings.xml | 30 +++-
.../test/kotlin/com/honjimaku/subrep/CloudTest.kt | 199 +++++++++++++++++++++
14 files changed, 568 insertions(+), 44 deletions(-)
diff --git a/README.md b/README.md
index 97a51b6..c781475 100644
--- a/README.md
+++ b/README.md
@@ -1,33 +1,62 @@
# Subrep for Android
[Subrep](https://honjimaku.com/subrep/) on a phone: live captions of the
-sound of the device, made on the device by Whisper, shared by link.
+sound of the device, made on the device by Whisper, shared by link. The
+captions on the phone are free. Cloud captions, from a larger model, cost
+hours that you buy on subread.space.
The app captures the sound of the apps on the phone (a video, a game, a
browser) or the microphone. Whisper, on the phone, turns it into captions.
The captions go to your share link on `honjimaku.com`, where the viewers read
along and mine with Yomitan, and to
[SubRead Overlay](https://github.com/equwal/subread-overlay) over the player,
-where a tap on a word opens the dictionary. No sound leaves the phone.
+where a tap on a word opens the dictionary. With captions on the phone, no
+sound leaves the phone.
## How to use it
-1. Choose the speech model and press "Download". `base` (82 MB) keeps up on
+1. Choose where the captions are made: on this phone (free), or in the
+ cloud. For the cloud, buy hours with a button under the choice, then come
+ back to the app. The cloud needs no model: go to step 3.
+2. Choose the speech model and press "Download". `base` (82 MB) keeps up on
most phones. `small` (264 MB) reads better and comes late. `tiny` is for
an old phone.
-2. Enter the language: `ja`, `en`, `ru`, ... or `auto`. A set language is
+3. Enter the language: `ja`, `en`, `ru`, ... or `auto`. A set language is
faster and more exact.
-3. Choose the sound: the apps on this device (Android 10 or later), or the
+4. Choose the sound: the apps on this device (Android 10 or later), or the
microphone.
-4. Press "Start the captions". Android asks for the capture of the screen
+5. Press "Start the captions". Android asks for the capture of the screen
("Share entire screen"), and once for the microphone permission. That
permission covers the sound of the apps too.
-5. Start the video. The captions come to the app, to the link and to the
+6. Start the video. The captions come to the app, to the link and to the
overlay. The notification has a "Stop" button.
The share link stays the same across restarts. "New link" makes another one,
and the old one stops.
+## Cloud captions
+
+The cloud uses Whisper large-v3 turbo on Groq, through subread.space. It is
+more exact than the models that a phone can run, and it uses less battery.
+
+| Pack | Price |
+|---|---|
+| 20 hours | $4.99 |
+| 80 hours | $16.99 |
+| 200 hours | $39.00 |
+
+Each piece of speech costs its length, rounded up to the next second.
+Silence costs nothing: the app sends only speech. The hours do not expire.
+
+The hours belong to the account of this app on subread.space. The screen
+shows its id (`acct_...`). If you install the app again, the app gets a new
+account: write to the contact address on subread.space with the old id and
+the receipt, and we move the hours.
+
+What goes where: each piece of speech goes to subread.space, which gives it
+to Groq for the text. Neither keeps the sound or the text. The app sends
+nothing while the phone makes the captions.
+
## What the phone cannot capture
- An app that refuses the capture of its sound: a video app with DRM, and
@@ -41,8 +70,10 @@ and the old one stops.
`AudioPlaybackCaptureConfiguration` for the apps, or the microphone).
`Segmenter` cuts it into pieces at pauses, with the rules of the desktop
app: a block of 32 ms is speech above the noise floor times 3, a pause of
-0.65 s closes a piece, 11 s is the most. `LocalEngine` gives each piece to
-whisper.cpp on one thread, with the encoder window cut to the length of the
+0.65 s closes a piece, 11 s is the most. `Engine` gives each piece to a
+`Recognizer` on one thread: whisper.cpp (`Whisper`), or the cloud
+(`CloudRecognizer`, which posts 16-bit PCM to
+`https://subread.space/api/captions/transcribe`). Whisper runs with the encoder window cut to the length of the
piece, and drops the invented lines that `Hallucinations` lists. Each piece
is one final caption.
diff --git a/app/build.gradle.kts b/app/build.gradle.kts
index d73fef6..9427c2b 100644
--- a/app/build.gradle.kts
+++ b/app/build.gradle.kts
@@ -25,8 +25,8 @@ android {
// The sound of other apps can be captured from Android 10 (API 29). Older devices get the microphone.
minSdk = 26
targetSdk = 36
- versionCode = 2
- versionName = "0.1.1"
+ versionCode = 3
+ versionName = "0.2.0"
// Each phone worth running Whisper on is 64-bit ARM; the speech library is built for
// ARMv8.2 (see src/main/cpp/CMakeLists.txt).
diff --git a/app/src/main/cpp/whisper_jni.c b/app/src/main/cpp/whisper_jni.c
index 317cc05..542de8a 100644
--- a/app/src/main/cpp/whisper_jni.c
+++ b/app/src/main/cpp/whisper_jni.c
@@ -1,5 +1,5 @@
// The narrowest bridge to whisper.cpp: load a model, transcribe one piece of 16 kHz mono float
-// PCM, read the text back. One transcription runs at a time (LocalEngine has one worker).
+// PCM, read the text back. One transcription runs at a time (Engine has one worker).
#include <jni.h>
#include <string.h>
diff --git a/app/src/main/kotlin/com/honjimaku/subrep/CaptureService.kt b/app/src/main/kotlin/com/honjimaku/subrep/CaptureService.kt
index 89b3ebc..577a896 100644
--- a/app/src/main/kotlin/com/honjimaku/subrep/CaptureService.kt
+++ b/app/src/main/kotlin/com/honjimaku/subrep/CaptureService.kt
@@ -34,7 +34,7 @@ class CaptureService : Service() {
private val http = OkHttpClient.Builder().pingInterval(20, TimeUnit.SECONDS).build()
private val main = Handler(Looper.getMainLooper())
private val captions = HandlerThread("captions").apply { start() }
- private var engine: LocalEngine? = null
+ private var engine: Engine? = null
private var relay: Socket? = null
private var overlay: OverlayFeed? = null
private var record: AudioRecord? = null
@@ -58,8 +58,9 @@ class CaptureService : Service() {
foreground(screen)
Feed.start(store.share)
+ val cloud = store.cloud
val model = Model.byId(store.model)
- if (!model.downloaded(this)) {
+ if (!cloud && !model.downloaded(this)) {
Feed.fail("Download the ${model.label} model first.")
stopAll()
return
@@ -90,8 +91,13 @@ class CaptureService : Service() {
.also { it.start() }
}
if (store.overlay && Overlay.installed(this)) overlay = OverlayFeed(this, Handler(captions.looper))
- engine = LocalEngine(
- model.file(this), lang,
+ val open: () -> Recognizer = if (cloud) {
+ { CloudRecognizer(Cloud(http, store), Feed::cloudLeft) }
+ } else {
+ { Whisper.open(model.file(this)) }
+ }
+ engine = Engine(
+ open, lang,
onCaption = ::onCaption,
onLanguage = { found -> relay?.send(Messages.status(true, found, title)) },
onState = { Feed.engine(it) },
diff --git a/app/src/main/kotlin/com/honjimaku/subrep/Cloud.kt b/app/src/main/kotlin/com/honjimaku/subrep/Cloud.kt
new file mode 100644
index 0000000..3b17f46
--- /dev/null
+++ b/app/src/main/kotlin/com/honjimaku/subrep/Cloud.kt
@@ -0,0 +1,120 @@
+package com.honjimaku.subrep
+
+import okhttp3.HttpUrl.Companion.toHttpUrl
+import okhttp3.MediaType.Companion.toMediaType
+import okhttp3.OkHttpClient
+import okhttp3.Request
+import okhttp3.RequestBody.Companion.toRequestBody
+import okhttp3.Response
+import org.json.JSONObject
+import java.io.IOException
+
+/**
+ * Cloud captions on subread.space. The account is the device cookie of the site, kept in
+ * [Store.device]: the first answer of the server makes it. Hours bought with [checkout] go to
+ * that account.
+ *
+ * The device token is read and written through two functions, so a test needs no [Store].
+ */
+class Cloud(
+ private val http: OkHttpClient,
+ private val readDevice: () -> String,
+ private val writeDevice: (String) -> Unit,
+ private val base: String = BASE,
+) {
+
+ constructor(http: OkHttpClient, store: Store) : this(http, { store.device }, { store.device = it })
+
+ /** The piece is lost, but the next can work: the network, or the speech service. */
+ class PieceFailed(message: String) : Exception(message)
+
+ /** The account has no hours for this piece. */
+ class OutOfHours : Exception("No cloud hours left. Buy hours, or choose the phone.")
+
+ data class Pack(val id: String, val name: String, val hours: Int, val price: String)
+
+ data class State(val accountId: String, val secondsLeft: Long, val available: Boolean, val packs: List<Pack>)
+
+ data class Answer(val text: String, val secondsLeft: Long)
+
+ /** The balance and the packs for sale. Throws [IOException] without the network. */
+ fun state(): State = call(Request.Builder().url("$base/api/captions").build()).use { response ->
+ if (!response.isSuccessful) throw IOException("subread.space answered ${response.code}")
+ parseState(response.body!!.string())
+ }
+
+ /** The address of the payment page for [packId]. It opens in the browser. */
+ fun checkout(packId: String): String {
+ val body = JSONObject().put("pack_id", packId).toString().toRequestBody(JSON)
+ return call(Request.Builder().url("$base/api/captions/checkout").post(body).build()).use { response ->
+ val text = response.body!!.string()
+ if (!response.isSuccessful) throw IOException(detail(text) ?: "subread.space answered ${response.code}")
+ JSONObject(text).getString("url")
+ }
+ }
+
+ /** The text of [pcm], 16 kHz mono 16-bit little-endian PCM. */
+ fun transcribe(pcm: ByteArray, language: String): Answer {
+ val request = Request.Builder()
+ .url("$base/api/captions/transcribe".toHttpUrl().newBuilder().addQueryParameter("lang", language.ifEmpty { "auto" }).build())
+ .post(pcm.toRequestBody(PCM))
+ .build()
+ val response = try {
+ call(request)
+ } catch (e: IOException) {
+ throw PieceFailed("Cloud: no connection (${e.message}).")
+ }
+ return response.use {
+ val text = it.body!!.string()
+ when (it.code) {
+ 200 -> JSONObject(text).let { json -> Answer(json.getString("text"), json.getLong("seconds_left")) }
+ 402 -> throw OutOfHours()
+ else -> throw PieceFailed("Cloud: ${detail(text) ?: "the server answered ${it.code}"}.")
+ }
+ }
+ }
+
+ private fun call(request: Request): Response {
+ val device = readDevice()
+ val response = http.newCall(
+ if (device.isEmpty()) request else request.newBuilder().header("Cookie", "$COOKIE=$device").build(),
+ ).execute()
+ deviceFrom(response.headers("Set-Cookie"))?.let { if (it != device) writeDevice(it) }
+ return response
+ }
+
+ companion object {
+ const val BASE = "https://subread.space"
+ const val COOKIE = "subplz_device"
+ private val JSON = "application/json".toMediaType()
+ private val PCM = "application/octet-stream".toMediaType()
+
+ /** The device token in the Set-Cookie headers of an answer, or null. */
+ fun deviceFrom(setCookies: List<String>): String? = setCookies.firstNotNullOfOrNull { header ->
+ header.substringBefore(';').trim().takeIf { it.startsWith("$COOKIE=") }
+ ?.substringAfter('=')?.trim('"')?.takeIf { it.isNotEmpty() }
+ }
+
+ fun parseState(text: String): State {
+ val json = JSONObject(text)
+ val packs = json.getJSONArray("packs")
+ return State(
+ accountId = json.optString("account_id"),
+ secondsLeft = json.getLong("seconds_left"),
+ available = json.getBoolean("available"),
+ packs = (0 until packs.length()).map { i ->
+ packs.getJSONObject(i).run { Pack(getString("id"), getString("name"), getInt("hours"), getString("price_display")) }
+ },
+ )
+ }
+
+ /** The "detail" of a FastAPI error answer, or null. */
+ private fun detail(text: String): String? = runCatching { JSONObject(text).getString("detail") }.getOrNull()
+
+ /** "12 h 5 min" for [seconds]. */
+ fun hours(seconds: Long): String {
+ val minutes = seconds / 60
+ return if (minutes >= 60) "${minutes / 60} h ${minutes % 60} min" else "$minutes min"
+ }
+ }
+}
diff --git a/app/src/main/kotlin/com/honjimaku/subrep/LocalEngine.kt b/app/src/main/kotlin/com/honjimaku/subrep/Engine.kt
similarity index 74%
rename from app/src/main/kotlin/com/honjimaku/subrep/LocalEngine.kt
rename to app/src/main/kotlin/com/honjimaku/subrep/Engine.kt
index 179455f..ad8bcfe 100644
--- a/app/src/main/kotlin/com/honjimaku/subrep/LocalEngine.kt
+++ b/app/src/main/kotlin/com/honjimaku/subrep/Engine.kt
@@ -1,17 +1,16 @@
package com.honjimaku.subrep
-import android.os.SystemClock
-import java.io.File
import java.util.concurrent.LinkedBlockingDeque
import kotlin.concurrent.thread
/**
- * Speech recognition on the phone: the sound goes through the [Segmenter], and each piece goes
- * to Whisper on one worker thread. A piece that waits while Whisper is behind by more than
- * [QUEUE_LIMIT] pieces is dropped, the oldest first: a late caption helps nobody.
+ * Speech recognition: the sound goes through the [Segmenter], and each piece goes to the
+ * [Recognizer] (Whisper on the phone, or the cloud) on one worker thread. A piece that waits
+ * while the recognizer is behind by more than [QUEUE_LIMIT] pieces is dropped, the oldest first:
+ * a late caption helps nobody.
*/
-class LocalEngine(
- private val model: File,
+class Engine(
+ private val open: () -> Recognizer,
private val language: String,
private val onCaption: (Caption) -> Unit,
private val onLanguage: (String) -> Unit,
@@ -67,7 +66,7 @@ class LocalEngine(
private fun work() {
val whisper = try {
- Whisper.open(model)
+ open()
} catch (e: Exception) {
onState("${STATE_ERROR}${e.message}")
return
@@ -77,18 +76,33 @@ class LocalEngine(
}
onState(STATE_READY)
var lastLanguage = ""
+ var failed = false
try {
while (!Thread.currentThread().isInterrupted) {
val piece = queue.takeFirst()
if (piece === END) break
- val started = SystemClock.elapsedRealtime()
+ val started = System.nanoTime()
val text = try {
Hallucinations.collapse(whisper.transcribe(piece, language)).trim()
} catch (e: WhisperUnavailable) {
+ failed = true
onState("${STATE_ERROR}${e.message}")
continue
+ } catch (e: Cloud.PieceFailed) {
+ // One lost piece, for example a short drop of the network. The next can work.
+ failed = true
+ onState("${STATE_ERROR}${e.message}")
+ continue
+ } catch (e: Cloud.OutOfHours) {
+ // Each next piece would fail the same way.
+ onState("${STATE_ERROR}${e.message}")
+ break
+ }
+ if (failed) {
+ failed = false
+ onState(STATE_READY)
}
- speed = (SystemClock.elapsedRealtime() - started) / 1000f / (piece.size / 16_000f)
+ speed = (System.nanoTime() - started) / 1e9f / (piece.size / 16_000f)
if (language == "auto") {
val found = whisper.detectedLanguage
if (found.isNotEmpty() && found != lastLanguage) {
diff --git a/app/src/main/kotlin/com/honjimaku/subrep/Feed.kt b/app/src/main/kotlin/com/honjimaku/subrep/Feed.kt
index c596c13..49d53b1 100644
--- a/app/src/main/kotlin/com/honjimaku/subrep/Feed.kt
+++ b/app/src/main/kotlin/com/honjimaku/subrep/Feed.kt
@@ -25,7 +25,7 @@ object Feed {
@Volatile var running = false
private set
- /** [LocalEngine.STATE_LOADING], [LocalEngine.STATE_READY], or an error with its reason. */
+ /** [Engine.STATE_LOADING], [Engine.STATE_READY], or an error with its reason. */
@Volatile var engine = ""
private set
@@ -59,6 +59,10 @@ object Feed {
@Volatile var downloadError = ""
private set
+ /** The seconds of cloud captions that the account has, -1 while unknown. */
+ @Volatile var cloudSecondsLeft = -1L
+ private set
+
fun add(listener: Listener) = listeners.add(listener)
fun remove(listener: Listener) = listeners.remove(listener)
@@ -151,6 +155,11 @@ object Feed {
changed()
}
+ fun cloudLeft(seconds: Long) {
+ cloudSecondsLeft = seconds
+ changed()
+ }
+
/** The last lines, and the partial line after them. */
fun text(): String {
val lines = synchronized(lock) { finals.toList() }
diff --git a/app/src/main/kotlin/com/honjimaku/subrep/MainActivity.kt b/app/src/main/kotlin/com/honjimaku/subrep/MainActivity.kt
index 6b06da1..5ed26ab 100644
--- a/app/src/main/kotlin/com/honjimaku/subrep/MainActivity.kt
+++ b/app/src/main/kotlin/com/honjimaku/subrep/MainActivity.kt
@@ -34,6 +34,10 @@ import kotlin.concurrent.thread
class MainActivity : Activity(), Feed.Listener {
private lateinit var store: Store
+ private val http = OkHttpClient()
+ private lateinit var whereGroup: RadioGroup
+ private lateinit var cloudStatus: TextView
+ private lateinit var packButtons: LinearLayout
private lateinit var content: LinearLayout
private lateinit var modelGroup: RadioGroup
private lateinit var modelStatus: TextView
@@ -68,6 +72,8 @@ class MainActivity : Activity(), Feed.Listener {
super.onResume()
Feed.add(this)
onChange()
+ // After a payment in the browser, the user comes back here: show the new hours.
+ refreshCloud()
}
override fun onPause() {
@@ -80,6 +86,31 @@ class MainActivity : Activity(), Feed.Listener {
title(getString(R.string.app_name))
note(getString(R.string.about))
+ step(R.string.step_where, "")
+ whereGroup = RadioGroup(this).apply {
+ addView(RadioButton(this@MainActivity).apply {
+ id = PHONE_ID
+ text = getString(R.string.where_phone)
+ setTextColor(Color.BLACK)
+ })
+ addView(RadioButton(this@MainActivity).apply {
+ id = CLOUD_ID
+ text = getString(R.string.where_cloud)
+ setTextColor(Color.BLACK)
+ })
+ check(if (store.cloud) CLOUD_ID else PHONE_ID)
+ }
+ content.addView(whereGroup, wide())
+ cloudStatus = TextView(this).apply {
+ setTextColor(Color.BLACK)
+ textSize = 15f
+ setTextIsSelectable(true)
+ }
+ content.addView(cloudStatus, wide())
+ packButtons = LinearLayout(this).apply { orientation = LinearLayout.VERTICAL }
+ content.addView(packButtons, wide())
+ note(getString(R.string.cloud_privacy), top = 4)
+
step(R.string.step_model, getString(R.string.step_model_why))
modelGroup = RadioGroup(this)
for (model in Model.entries) {
@@ -179,6 +210,7 @@ class MainActivity : Activity(), Feed.Listener {
private fun chosenModel(): Model = Model.entries[(modelGroup.checkedRadioButtonId - MODEL_ID).coerceIn(0, Model.entries.size - 1)]
private fun save() {
+ store.cloud = whereGroup.checkedRadioButtonId == CLOUD_ID
store.model = chosenModel().id
store.lang = langField.text.toString()
store.title = titleField.text.toString()
@@ -220,7 +252,11 @@ class MainActivity : Activity(), Feed.Listener {
private fun startCapture() {
save()
- if (!chosenModel().downloaded(this)) return toast(R.string.need_model)
+ if (store.cloud) {
+ if (Feed.cloudSecondsLeft == 0L) return toast(R.string.need_hours)
+ } else if (!chosenModel().downloaded(this)) {
+ return toast(R.string.need_model)
+ }
val missing = buildList {
if (checkSelfPermission(Manifest.permission.RECORD_AUDIO) != PackageManager.PERMISSION_GRANTED) add(Manifest.permission.RECORD_AUDIO)
if (Build.VERSION.SDK_INT >= 33 && checkSelfPermission(Manifest.permission.POST_NOTIFICATIONS) != PackageManager.PERMISSION_GRANTED) {
@@ -254,15 +290,62 @@ class MainActivity : Activity(), Feed.Listener {
statusView.text = status()
linesView.text = Feed.text()
showModel()
+ cloud?.let { showCloud(it.copy(secondsLeft = Feed.cloudSecondsLeft.takeIf { s -> s >= 0 } ?: it.secondsLeft)) }
+ }
+
+ /** The last answer of subread.space about the cloud hours, or null before the first. */
+ private var cloud: Cloud.State? = null
+
+ private fun refreshCloud() {
+ val app = Cloud(http, store)
+ thread(name = "cloud") {
+ val state = runCatching { app.state() }.getOrNull()
+ runOnUiThread {
+ cloud = state
+ if (state == null) {
+ cloudStatus.text = getString(R.string.cloud_offline)
+ packButtons.removeAllViews()
+ } else {
+ Feed.cloudLeft(state.secondsLeft)
+ showCloud(state)
+ }
+ }
+ }
+ }
+
+ private fun showCloud(state: Cloud.State) {
+ if (!state.available) {
+ cloudStatus.text = getString(R.string.cloud_closed)
+ packButtons.removeAllViews()
+ return
+ }
+ cloudStatus.text = getString(R.string.cloud_left, Cloud.hours(state.secondsLeft), state.accountId)
+ if (packButtons.childCount == state.packs.size) return
+ packButtons.removeAllViews()
+ for (pack in state.packs) {
+ packButtons.addView(button(resources.getQuantityString(R.plurals.cloud_buy, pack.hours, pack.hours, pack.price)) { buy(pack) })
+ }
+ }
+
+ private fun buy(pack: Cloud.Pack) {
+ val app = Cloud(http, store)
+ thread(name = "checkout") {
+ val url = runCatching { app.checkout(pack.id) }
+ runOnUiThread {
+ url.onSuccess { open(it) }.onFailure {
+ Toast.makeText(this, getString(R.string.cloud_buy_failed, it.message), Toast.LENGTH_LONG).show()
+ }
+ }
+ }
}
private fun status(): String {
if (!Feed.running) return Feed.error
- val model = Model.byId(store.model).label
+ val model = if (store.cloud) "Cloud" else "Whisper ${Model.byId(store.model).label}"
val engine = when {
- Feed.engine == LocalEngine.STATE_LOADING -> "loading…"
- Feed.engine == LocalEngine.STATE_READY && Feed.speed > 0f -> "ready, %.1f s for each second of speech".format(Feed.speed)
- Feed.engine == LocalEngine.STATE_READY -> "ready"
+ Feed.engine == Engine.STATE_LOADING -> "loading…"
+ Feed.engine == Engine.STATE_READY && Feed.speed > 0f -> "ready, %.1f s for each second of speech".format(Feed.speed)
+ Feed.engine == Engine.STATE_READY -> "ready"
else -> Feed.engine
}
val late = if (Feed.late > 0) ", ${Feed.late} pieces dropped" else ""
@@ -282,7 +365,7 @@ class MainActivity : Activity(), Feed.Listener {
val bars = Feed.bars()
val level = "▮".repeat(bars) + "▯".repeat(10 - bars)
val error = if (Feed.error.isEmpty()) "" else "\n${Feed.error}"
- return "Whisper $model: $engine$late\nShare link: $relay\nSubRead Overlay: $overlay\nSound: $level$error"
+ return "$model: $engine$late\nShare link: $relay\nSubRead Overlay: $overlay\nSound: $level$error"
}
private fun showLink() {
@@ -358,6 +441,8 @@ class MainActivity : Activity(), Feed.Listener {
const val REQUEST_PROJECTION = 2
const val SCREEN_ID = 1
const val MIC_ID = 2
+ const val PHONE_ID = 3
+ const val CLOUD_ID = 4
const val MODEL_ID = 10
}
}
diff --git a/app/src/main/kotlin/com/honjimaku/subrep/Pcm.kt b/app/src/main/kotlin/com/honjimaku/subrep/Pcm.kt
index 7908da9..859905a 100644
--- a/app/src/main/kotlin/com/honjimaku/subrep/Pcm.kt
+++ b/app/src/main/kotlin/com/honjimaku/subrep/Pcm.kt
@@ -30,4 +30,15 @@ object Pcm {
}
return n
}
+
+ /** [samples] from -1 to 1 as 16-bit little-endian PCM: the reverse of [toFloat]. */
+ fun toBytes(samples: FloatArray): ByteArray {
+ val out = ByteArray(samples.size * 2)
+ for ((n, value) in samples.withIndex()) {
+ val sample = Math.round(value * 32768f).coerceIn(-32768, 32767)
+ out[2 * n] = sample.toByte()
+ out[2 * n + 1] = (sample shr 8).toByte()
+ }
+ return out
+ }
}
diff --git a/app/src/main/kotlin/com/honjimaku/subrep/Recognizer.kt b/app/src/main/kotlin/com/honjimaku/subrep/Recognizer.kt
new file mode 100644
index 0000000..cb3c1fd
--- /dev/null
+++ b/app/src/main/kotlin/com/honjimaku/subrep/Recognizer.kt
@@ -0,0 +1,25 @@
+package com.honjimaku.subrep
+
+/** Turns one piece of speech into text: Whisper on the phone, or [CloudRecognizer]. */
+interface Recognizer : AutoCloseable {
+
+ /** The language found in the last piece, "ja", "ru", ..., or "" when not known. */
+ val detectedLanguage: String
+
+ /** The text of [samples]: 16 kHz mono, from -1 to 1. [language] is a code, or "auto". */
+ fun transcribe(samples: FloatArray, language: String): String
+}
+
+/** Recognition on subread.space. Each piece costs its length in seconds from the hours of the account. */
+class CloudRecognizer(private val cloud: Cloud, private val onSecondsLeft: (Long) -> Unit) : Recognizer {
+
+ override val detectedLanguage: String = ""
+
+ override fun transcribe(samples: FloatArray, language: String): String {
+ val answer = cloud.transcribe(Pcm.toBytes(samples), language)
+ onSecondsLeft(answer.secondsLeft)
+ return answer.text
+ }
+
+ override fun close() = Unit
+}
diff --git a/app/src/main/kotlin/com/honjimaku/subrep/Store.kt b/app/src/main/kotlin/com/honjimaku/subrep/Store.kt
index db70927..860bed8 100644
--- a/app/src/main/kotlin/com/honjimaku/subrep/Store.kt
+++ b/app/src/main/kotlin/com/honjimaku/subrep/Store.kt
@@ -30,6 +30,16 @@ class Store(context: Context) {
get() = prefs.getString("title", "").orEmpty()
set(value) = prefs.edit().putString("title", value.trim().take(80)).apply()
+ /** True: the cloud makes the captions, with hours from subread.space. False: the phone. */
+ var cloud: Boolean
+ get() = prefs.getBoolean("cloud", false)
+ set(value) = prefs.edit().putBoolean("cloud", value).apply()
+
+ /** The device cookie of subread.space: the account that holds the cloud hours. */
+ var device: String
+ get() = prefs.getString("device", "").orEmpty()
+ set(value) = prefs.edit().putString("device", value).apply()
+
var overlay: Boolean
get() = prefs.getBoolean("overlay", true)
set(value) = prefs.edit().putBoolean("overlay", value).apply()
diff --git a/app/src/main/kotlin/com/honjimaku/subrep/Whisper.kt b/app/src/main/kotlin/com/honjimaku/subrep/Whisper.kt
index 00dadfe..b57afed 100644
--- a/app/src/main/kotlin/com/honjimaku/subrep/Whisper.kt
+++ b/app/src/main/kotlin/com/honjimaku/subrep/Whisper.kt
@@ -16,16 +16,16 @@ internal object WhisperLib {
class WhisperUnavailable(message: String) : Exception(message)
/** A loaded speech model. Not thread-safe: one transcription at a time. */
-class Whisper private constructor(private var ptr: Long) : AutoCloseable {
+class Whisper private constructor(private var ptr: Long) : Recognizer {
/** The language that Whisper found in the last piece: "ja", "ru", ... */
- val detectedLanguage: String get() = WhisperLib.detectedLanguage(ptr)
+ override val detectedLanguage: String get() = WhisperLib.detectedLanguage(ptr)
/**
* The text of one piece of sound: [samples] at 16 kHz mono, from -1 to 1. [language] is a
* code such as "ja", or "auto".
*/
- fun transcribe(samples: FloatArray, language: String): String {
+ override fun transcribe(samples: FloatArray, language: String): String {
val rc = WhisperLib.transcribe(ptr, samples, threads, language, audioContext(samples.size))
if (rc != 0) throw WhisperUnavailable("The speech model failed on this sound (code $rc).")
return (0 until WhisperLib.segmentCount(ptr)).joinToString(" ") {
diff --git a/app/src/main/res/values/strings.xml b/app/src/main/res/values/strings.xml
index 19baf38..e123b41 100644
--- a/app/src/main/res/values/strings.xml
+++ b/app/src/main/res/values/strings.xml
@@ -1,10 +1,24 @@
<?xml version="1.0" encoding="utf-8"?>
<resources>
<string name="app_name">Subrep</string>
- <string name="about">Live captions of the sound of this device, made on this device by Whisper. The captions go to your share link, where the viewers read along with Yomitan, and to SubRead Overlay over the player. No sound leaves the phone.</string>
+ <string name="about">Live captions of the sound of this device, made on this device by Whisper, or in the cloud. The captions go to your share link, where the viewers read along with Yomitan, and to SubRead Overlay over the player.</string>
- <string name="step_model">1. Speech model</string>
- <string name="step_model_why">One download, kept on the phone.</string>
+ <string name="step_where">1. Where the captions are made</string>
+ <string name="where_phone">On this phone: free, and no sound leaves the phone</string>
+ <string name="where_cloud">In the cloud: a larger model, better captions, less battery. It costs hours.</string>
+ <string name="cloud_privacy">In the cloud, each piece of sound goes to subread.space and to Groq, which make the text. Neither keeps the sound or the text.</string>
+ <string name="cloud_left">Cloud hours left: %1$s. Account: %2$s</string>
+ <string name="cloud_offline">Cloud hours: no connection to subread.space.</string>
+ <string name="cloud_closed">Cloud captions are not open yet on subread.space.</string>
+ <plurals name="cloud_buy">
+ <item quantity="one">Buy %1$d hour for %2$s</item>
+ <item quantity="other">Buy %1$d hours for %2$s</item>
+ </plurals>
+ <string name="cloud_buy_failed">The payment page did not open: %1$s</string>
+ <string name="need_hours">Buy cloud hours first, or choose this phone.</string>
+
+ <string name="step_model">2. Speech model on the phone</string>
+ <string name="step_model_why">One download, kept on the phone. The cloud does not need it.</string>
<string name="model_ready">Downloaded.</string>
<string name="model_missing">Not downloaded yet.</string>
<string name="model_downloading">Downloading, %1$d %%.</string>
@@ -12,19 +26,19 @@
<string name="model_failed">The download failed: %1$s</string>
<string name="download">Download</string>
<string name="need_model">Download the model first.</string>
- <string name="step_lang">2. Language</string>
+ <string name="step_lang">3. Language</string>
<string name="step_lang_why">ja, en, ru, fi, es, pt, or auto. A set language is faster and more exact than auto.</string>
- <string name="step_source">3. Sound</string>
+ <string name="step_source">4. Sound</string>
<string name="source_screen">The apps on this device (Android 10 or later)</string>
<string name="source_mic">The microphone</string>
- <string name="step_share">4. Share</string>
+ <string name="step_share">5. Share</string>
<string name="share_on">Share the captions by link</string>
<string name="share_title_hint">Name of the stream, for the viewers</string>
<string name="copy">Copy</string>
<string name="share">Share</string>
<string name="new_link">New link</string>
<string name="copied">Link copied.</string>
- <string name="step_overlay">5. SubRead Overlay</string>
+ <string name="step_overlay">6. SubRead Overlay</string>
<string name="overlay_on">Show the captions over the player, in SubRead Overlay</string>
<string name="overlay_missing">SubRead Overlay is not installed. With it, the captions are on top of the player, and a tap on a word opens the dictionary.</string>
<string name="install_overlay">Install SubRead Overlay</string>
@@ -38,5 +52,5 @@
<string name="notification_channel">Live captions</string>
<string name="notification_title">Subrep: live captions</string>
- <string name="notification_text">Whisper listens to the sound of this device.</string>
+ <string name="notification_text">Subrep listens to the sound of this device.</string>
</resources>
diff --git a/app/src/test/kotlin/com/honjimaku/subrep/CloudTest.kt b/app/src/test/kotlin/com/honjimaku/subrep/CloudTest.kt
new file mode 100644
index 0000000..7ca0f5b
--- /dev/null
+++ b/app/src/test/kotlin/com/honjimaku/subrep/CloudTest.kt
@@ -0,0 +1,199 @@
+package com.honjimaku.subrep
+
+import com.sun.net.httpserver.HttpExchange
+import com.sun.net.httpserver.HttpServer
+import io.kotest.property.Arb
+import io.kotest.property.arbitrary.float
+import io.kotest.property.arbitrary.list
+import io.kotest.property.checkAll
+import kotlinx.coroutines.runBlocking
+import okhttp3.OkHttpClient
+import org.junit.After
+import org.junit.Assert.assertArrayEquals
+import org.junit.Assert.assertEquals
+import org.junit.Assert.assertNull
+import org.junit.Assert.assertTrue
+import org.junit.Assert.fail
+import org.junit.Before
+import org.junit.Test
+import java.net.InetSocketAddress
+import java.util.concurrent.CopyOnWriteArrayList
+import kotlin.math.abs
+import kotlin.random.Random
+
+/** The cloud client against a local server that answers as subread.space does. */
+class CloudTest {
+
+ private class Seen(val method: String, val query: String?, val cookie: String?, val body: ByteArray)
+
+ private lateinit var server: HttpServer
+ private val seen = CopyOnWriteArrayList<Seen>()
+ private var answer: (HttpExchange) -> Pair<Int, String> = { 200 to "{}" }
+ private var device = ""
+
+ private fun cloud() = Cloud(OkHttpClient(), { device }, { device = it }, "http://127.0.0.1:${server.address.port}")
+
+ @Before
+ fun start() {
+ server = HttpServer.create(InetSocketAddress("127.0.0.1", 0), 0)
+ server.createContext("/") { exchange ->
+ val body = exchange.requestBody.readBytes()
+ seen += Seen(exchange.requestMethod, exchange.requestURI.rawQuery, exchange.requestHeaders.getFirst("Cookie"), body)
+ if (exchange.requestHeaders.getFirst("Cookie") == null) {
+ exchange.responseHeaders.add("Set-Cookie", "subplz_device=dev123; HttpOnly; Path=/; SameSite=lax")
+ }
+ val (code, text) = answer(exchange)
+ val bytes = text.toByteArray()
+ exchange.sendResponseHeaders(code, bytes.size.toLong())
+ exchange.responseBody.use { it.write(bytes) }
+ }
+ server.start()
+ }
+
+ @After
+ fun stop() = server.stop(0)
+
+ @Test
+ fun theFirstAnswerMakesTheAccountAndEachLaterCallSendsIt() {
+ answer = { 200 to STATE }
+ val cloud = cloud()
+ val state = cloud.state()
+ assertEquals("dev123", device)
+ assertNull(seen[0].cookie)
+ cloud.state()
+ assertEquals("subplz_device=dev123", seen[1].cookie)
+ assertEquals(Cloud.State("acct_1", 7200, true, listOf(Cloud.Pack("captions20", "20 hours of cloud captions", 20, "$4.99"))), state)
+ }
+
+ @Test
+ fun aPieceGoesAsPcmWithItsLanguage() {
+ device = "dev123"
+ answer = { 200 to """{"text": "こんにちは", "seconds": 3, "seconds_left": 57}""" }
+ val pcm = byteArrayOf(1, 2, 3, 4)
+ val result = cloud().transcribe(pcm, "ja")
+ assertEquals(Cloud.Answer("こんにちは", 57), result)
+ assertEquals("POST", seen[0].method)
+ assertEquals("lang=ja", seen[0].query)
+ assertArrayEquals(pcm, seen[0].body)
+ }
+
+ @Test
+ fun noHoursLeftIsOutOfHours() {
+ device = "dev123"
+ answer = { 402 to """{"detail": "No cloud caption hours left."}""" }
+ try {
+ cloud().transcribe(ByteArray(2), "ja")
+ fail("no exception")
+ } catch (_: Cloud.OutOfHours) {
+ }
+ }
+
+ @Test
+ fun aServerErrorLosesOnlyThePiece() {
+ device = "dev123"
+ answer = { 502 to """{"detail": "speech service answered 500"}""" }
+ try {
+ cloud().transcribe(ByteArray(2), "ja")
+ fail("no exception")
+ } catch (e: Cloud.PieceFailed) {
+ assertEquals("Cloud: speech service answered 500.", e.message)
+ }
+ }
+
+ @Test
+ fun noServerLosesOnlyThePiece() {
+ val cloud = cloud()
+ server.stop(0)
+ try {
+ cloud.transcribe(ByteArray(2), "ja")
+ fail("no exception")
+ } catch (_: Cloud.PieceFailed) {
+ }
+ }
+
+ @Test
+ fun checkoutGivesThePaymentPage() {
+ answer = { 200 to """{"url": "https://checkout.stripe.com/c/pay/cs_1"}""" }
+ assertEquals("https://checkout.stripe.com/c/pay/cs_1", cloud().checkout("captions20"))
+ assertEquals("""{"pack_id":"captions20"}""", String(seen[0].body))
+ }
+
+ @Test
+ fun theDeviceComesOnlyFromItsOwnCookie() {
+ assertEquals("abc", Cloud.deviceFrom(listOf("other=1; Path=/", "subplz_device=abc; HttpOnly")))
+ assertNull(Cloud.deviceFrom(listOf("subplz_device=; Max-Age=0")))
+ assertNull(Cloud.deviceFrom(emptyList()))
+ }
+
+ @Test
+ fun hoursReadAsHoursAndMinutes() {
+ assertEquals("0 min", Cloud.hours(59))
+ assertEquals("59 min", Cloud.hours(3599))
+ assertEquals("20 h 0 min", Cloud.hours(72000))
+ assertEquals("1 h 1 min", Cloud.hours(3660))
+ }
+
+ /** What goes to the cloud comes back as the same sound, within one step of 16-bit PCM. */
+ @Test
+ fun pcmOfFloatsReadsBackTheSameSound(): Unit = runBlocking {
+ checkAll(Arb.list(Arb.float(-1f, 1f), 0..400)) { list ->
+ val samples = list.toFloatArray()
+ val bytes = Pcm.toBytes(samples)
+ val back = FloatArray(samples.size)
+ assertEquals(samples.size, Pcm.toFloat(bytes, bytes.size, back))
+ for (i in samples.indices) assertTrue("${samples[i]} -> ${back[i]}", abs(samples[i] - back[i]) <= 1f / 32768f)
+ }
+ }
+
+ @Test
+ fun loudSoundIsClippedNotWrapped() {
+ val back = FloatArray(2)
+ Pcm.toFloat(Pcm.toBytes(floatArrayOf(1.5f, -1.5f)), 4, back)
+ assertEquals(32767f / 32768f, back[0])
+ assertEquals(-1f, back[1])
+ }
+
+ /** The engine stops asking the cloud once the hours are gone. */
+ @Test
+ fun theEngineStopsWhenTheHoursAreGone() {
+ val calls = CopyOnWriteArrayList<Int>()
+ val states = CopyOnWriteArrayList<String>()
+ val captions = CopyOnWriteArrayList<String>()
+ val recognizer = object : Recognizer {
+ override val detectedLanguage = ""
+ override fun transcribe(samples: FloatArray, language: String): String {
+ calls += samples.size
+ if (calls.size > 1) throw Cloud.OutOfHours()
+ return "一つ目"
+ }
+ override fun close() = Unit
+ }
+ val engine = Engine({ recognizer }, "ja", { captions += it.text }, {}, { states += it })
+ engine.start()
+ val random = Random(7)
+ repeat(3) { feed(engine, speech(1_500, random) + FloatArray(16_000)) }
+ engine.stop()
+ assertEquals(listOf("一つ目"), captions)
+ assertEquals(2, calls.size)
+ assertTrue(states.toString(), states.last().startsWith(Engine.STATE_ERROR + "No cloud hours left"))
+ }
+
+ private fun speech(ms: Int, random: Random) = FloatArray(16 * ms) { random.nextFloat() * 0.6f - 0.3f }
+
+ /** Feeds [sound] in frames of 100 ms, and waits until the worker took each piece. */
+ private fun feed(engine: Engine, sound: FloatArray) {
+ val bytes = Pcm.toBytes(sound)
+ var at = 0
+ while (at < bytes.size) {
+ val n = minOf(3_200, bytes.size - at)
+ engine.feed(bytes.copyOfRange(at, at + n), n)
+ at += n
+ }
+ Thread.sleep(300)
+ }
+
+ private companion object {
+ const val STATE = """{"account_id": "acct_1", "seconds_left": 7200, "available": true,
+ "packs": [{"id": "captions20", "name": "20 hours of cloud captions", "hours": 20, "price_display": "$4.99"}]}"""
+ }
+}