commit 3f9281b01a14288df49c5bde6a25ee694c3e150c truex <truex@equwal.com> 2026-09-20 22:27:51 -0700 Add voice typing: speak, and the words go where the cursor is (0.0.4-alpha) New action Typing > Voice typing. The app has no speech engine and no network access. It calls the Android SpeechRecognizer interface, so a speech recognition app on the device does the listening. TextInsert puts the words into the field that has input focus through the accessibility service, with a clipboard paste as the fallback. It never writes into a password field. Android gives the microphone to the app in front only, and an accessibility service does not count. The listening runs in DictationActivity, which is see-through and cannot take focus or touches. The Viwoods voice prompt cannot be used. It uploads audio to the Viwoods cloud and has no interface for other apps. Debug builds carry FakeRecognitionService and InsertReceiver, so the chain can be tested with no voice. Tested on the reader: "ab" became "ab hello from the test bench". New permission: RECORD_AUDIO, requested at run time from the Voice typing screen. Listing, privacy policy and declarations updated.
CHANGELOG.md | 22 +++ README.md | 28 ++++ app/src/debug/AndroidManifest.xml | 11 +- .../assistkey/debug/FakeRecognitionService.kt | 34 +++++ .../dev/equwal/assistkey/debug/InsertReceiver.kt | 20 +++ app/src/main/AndroidManifest.xml | 21 +++ .../java/dev/equwal/assistkey/model/ActionSpec.kt | 5 +- .../dev/equwal/assistkey/route/ActionRouter.kt | 4 +- .../equwal/assistkey/ui/ActionPickerActivity.kt | 4 + .../java/dev/equwal/assistkey/ui/MainActivity.kt | 5 + .../java/dev/equwal/assistkey/ui/VoiceActivity.kt | 93 ++++++++++++ .../java/dev/equwal/assistkey/voice/Dictation.kt | 106 ++++++++++++++ .../equwal/assistkey/voice/DictationActivity.kt | 157 +++++++++++++++++++++ .../java/dev/equwal/assistkey/voice/TextInsert.kt | 59 ++++++++ gradle.properties | 4 +- play/DECLARATIONS.md | 9 +- play/LISTING.md | 3 + play/PRIVACY.md | 8 ++ 18 files changed, 585 insertions(+), 8 deletions(-)
diff --git a/CHANGELOG.md b/CHANGELOG.md index bda1a09..afd4431 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,27 @@ # Changelog +## 0.0.4-alpha - 2026-09-20 + +Version code 4. + +- **Voice typing.** A new action. Press the key, speak, and the words go where + the cursor is. The app has no speech engine: it uses the Android + SpeechRecognizer interface, so any speech recognition app on the device does + the listening. The accessibility service puts the text into the focused + field. Password fields are never written to. +- **Fix: Power always wakes and unlocks.** While the app manages the Power + button, a press on a sleeping or locked device is never a gesture. The app + also sends the wake and the keyguard dismissal through the shell, so waking + does not depend on the firmware. Regression test: `tools/e2e/power-wake.sh`. + +Known limits: + +- Voice typing needs a speech recognition app that has its own microphone + permission. The Viwoods voice prompt cannot be used: it sends audio to the + Viwoods cloud and has no interface for other apps. +- The listening screen pauses the app behind it for a moment. An app that + clears its text field on resume loses what was typed before. + ## 0.0.3-alpha - 2026-09-20 Version code 3. diff --git a/README.md b/README.md index b0d608a..5f796ae 100644 --- a/README.md +++ b/README.md @@ -66,6 +66,33 @@ every official route including `cmd display set-brightness`. Writing sticks until the system slider is moved. The node belongs to `system`, so it needs Shizuku running as root or a shell that may `su` (userdebug firmware). +## Voice typing + +`voice/`. The action *Typing > Voice typing* turns a key into a dictation key. +The app has no speech engine and no network access. It calls the Android +`SpeechRecognizer` interface, so a speech recognition app on the device does +the listening: the system one, an offline Whisper app, or any other app that +offers a `RecognitionService`. `TextInsert` then puts the words into the field +that has input focus, through the accessibility service: `ACTION_SET_TEXT` at +the selection, with a clipboard paste as the fallback. It never writes into a +password field. + +Two facts cost time: + +- Android gives the microphone to the app in front only, and an accessibility + service does not count (`RECORD_AUDIO` app-op mode is `foreground`). The + listening therefore runs in `DictationActivity`, which is see-through and + cannot take focus or touches. +- The speech recognition app needs its own microphone permission too. If it has + none, the framework answers `ERROR_INSUFFICIENT_PERMISSIONS` and blames the + caller. + +The Viwoods voice prompt is not reusable. It uploads audio to the Viwoods cloud +(`/api/v1/openAi/speechToTextGemini`) and offers no interface to other apps. + +Debug builds carry `FakeRecognitionService`, which hears a fixed sentence, so +the whole chain can be tested on a bench with no voice. + ## Home screen and recent apps `home/`. The home screen is a clock, a few chosen apps and a search line; the @@ -342,6 +369,7 @@ device/ device profiles display/ the extra-dim light home/ the home screen and the recent-apps list shell/ shell access through Shizuku, and the managed Power button +voice/ voice typing: recognizer choice, the listening screen, text insert license/ licence tiers and Google Play Billing native/ firmware settings: power gestures, and the read-only key hooks route/ turning an action spec into behaviour diff --git a/app/src/debug/AndroidManifest.xml b/app/src/debug/AndroidManifest.xml index d990071..694bfcd 100644 --- a/app/src/debug/AndroidManifest.xml +++ b/app/src/debug/AndroidManifest.xml @@ -6,5 +6,14 @@ release build. --> <manifest xmlns:android="http://schemas.android.com/apk/res/android"> - <application android:name=".debug.ShotApp" /> + <application android:name=".debug.ShotApp"> + <!-- Test hook: adb shell am broadcast -n <pkg>/dev.equwal.assistkey.debug.InsertReceiver -e text "..." --> + <receiver android:name=".debug.InsertReceiver" android:exported="true" /> + <!-- A recognizer that hears a fixed sentence, so voice typing can be tested with no voice. --> + <service android:name=".debug.FakeRecognitionService" android:exported="true" android:label="Test bench recognizer"> + <intent-filter> + <action android:name="android.speech.RecognitionService" /> + </intent-filter> + </service> + </application> </manifest> diff --git a/app/src/debug/java/dev/equwal/assistkey/debug/FakeRecognitionService.kt b/app/src/debug/java/dev/equwal/assistkey/debug/FakeRecognitionService.kt new file mode 100644 index 0000000..330ebe6 --- /dev/null +++ b/app/src/debug/java/dev/equwal/assistkey/debug/FakeRecognitionService.kt @@ -0,0 +1,34 @@ +package dev.equwal.assistkey.debug + +import android.content.Intent +import android.os.Bundle +import android.os.Handler +import android.os.Looper +import android.speech.RecognitionService +import android.speech.SpeechRecognizer + +/** Debug builds only. Answers every request with the same sentence after one second. */ +class FakeRecognitionService : RecognitionService() { + + private val main = Handler(Looper.getMainLooper()) + + override fun onStartListening(intent: Intent, callback: Callback) { + callback.readyForSpeech(Bundle()) + callback.beginningOfSpeech() + main.postDelayed({ + callback.endOfSpeech() + callback.results( + Bundle().apply { + putStringArrayList(SpeechRecognizer.RESULTS_RECOGNITION, arrayListOf(SENTENCE)) + } + ) + }, 1000L) + } + + override fun onStopListening(callback: Callback) = Unit + override fun onCancel(callback: Callback) = main.removeCallbacksAndMessages(null) + + companion object { + const val SENTENCE = "hello from the test bench" + } +} diff --git a/app/src/debug/java/dev/equwal/assistkey/debug/InsertReceiver.kt b/app/src/debug/java/dev/equwal/assistkey/debug/InsertReceiver.kt new file mode 100644 index 0000000..ba0f7d9 --- /dev/null +++ b/app/src/debug/java/dev/equwal/assistkey/debug/InsertReceiver.kt @@ -0,0 +1,20 @@ +package dev.equwal.assistkey.debug + +import android.content.BroadcastReceiver +import android.content.Context +import android.content.Intent +import android.util.Log +import dev.equwal.assistkey.route.ServiceHolder +import dev.equwal.assistkey.voice.TextInsert + +/** + * Debug builds only. Lets a test put text at the cursor with no speech, because + * a test bench cannot talk. + */ +class InsertReceiver : BroadcastReceiver() { + override fun onReceive(c: Context, i: Intent) { + val svc = ServiceHolder.service + val result = if (svc == null) "NO_SERVICE" else TextInsert.insert(svc, i.getStringExtra("text").orEmpty()).name + Log.i("AssistKey", "insert test: " + result) + } +} diff --git a/app/src/main/AndroidManifest.xml b/app/src/main/AndroidManifest.xml index 2a09af1..1972538 100644 --- a/app/src/main/AndroidManifest.xml +++ b/app/src/main/AndroidManifest.xml @@ -18,6 +18,13 @@ <uses-permission android:name="android.permission.PACKAGE_USAGE_STATS" tools:ignore="ProtectedPermissions" /> + <!-- + Voice typing only. The speech recognition app does the recording; Android + still requires the app that asks for it to hold the permission. Requested + at run time, from the Voice typing screen, and never otherwise. + --> + <uses-permission android:name="android.permission.RECORD_AUDIO" /> + <!-- Required of any app that wants to be offered the wallet role. --> <uses-permission android:name="android.permission.QUICK_ACCESS_WALLET" /> @@ -34,6 +41,10 @@ <intent> <action android:name="android.media.action.STILL_IMAGE_CAMERA" /> </intent> + <!-- Speech recognition apps, for voice typing. --> + <intent> + <action android:name="android.speech.RecognitionService" /> + </intent> <package android:name="com.viwoods.launcher" /> <package android:name="com.viwoods.viwoodsai" /> <package android:name="com.viwoods.settings" /> @@ -70,6 +81,16 @@ <activity android:name=".ui.NavigationActivity" android:exported="false" /> <activity android:name=".ui.ShellActivity" android:exported="false" /> <activity android:name=".ui.DisplayActivity" android:exported="false" /> + <activity android:name=".ui.VoiceActivity" android:exported="false" /> + <!-- See-through and unfocusable: on top only so the microphone is allowed. --> + <activity + android:name=".voice.DictationActivity" + android:exported="false" + android:excludeFromRecents="true" + android:noHistory="true" + android:launchMode="singleInstance" + android:taskAffinity="" + android:theme="@style/Theme.Invisible" /> <activity android:name=".home.HomeSettingsActivity" android:exported="false" /> <activity android:name=".home.RecentsActivity" diff --git a/app/src/main/java/dev/equwal/assistkey/model/ActionSpec.kt b/app/src/main/java/dev/equwal/assistkey/model/ActionSpec.kt index fdc4faa..a407199 100644 --- a/app/src/main/java/dev/equwal/assistkey/model/ActionSpec.kt +++ b/app/src/main/java/dev/equwal/assistkey/model/ActionSpec.kt @@ -39,7 +39,10 @@ enum class ActionKind { SWIPE, /** payload = forward|backward; scrolls the focused scrollable node */ - SCROLL; + SCROLL, + + /** Speak, and the words go where the cursor is. The payload is not used. */ + VOICE; companion object { fun fromName(n: String): ActionKind = diff --git a/app/src/main/java/dev/equwal/assistkey/route/ActionRouter.kt b/app/src/main/java/dev/equwal/assistkey/route/ActionRouter.kt index c36618c..7a37cef 100644 --- a/app/src/main/java/dev/equwal/assistkey/route/ActionRouter.kt +++ b/app/src/main/java/dev/equwal/assistkey/route/ActionRouter.kt @@ -32,6 +32,8 @@ object ActionRouter { ActionKind.BROADCAST -> broadcast(ctx, spec.payload) ActionKind.SWIPE -> swipe(spec.payload) ActionKind.SCROLL -> scroll(spec.payload) + ActionKind.VOICE -> ServiceHolder.service + ?.let { dev.equwal.assistkey.voice.Dictation.toggle(it) } ?: false } } catch (e: Exception) { Log.w(TAG, "action " + spec.kind + "/" + spec.payload + " failed", e) @@ -184,7 +186,7 @@ object ActionRouter { /** True when this action cannot run without the accessibility service. */ fun requiresAccessibility(spec: ActionSpec): Boolean = when (spec.kind) { - ActionKind.GLOBAL, ActionKind.SWIPE, ActionKind.SCROLL -> true + ActionKind.GLOBAL, ActionKind.SWIPE, ActionKind.SCROLL, ActionKind.VOICE -> true else -> false } } diff --git a/app/src/main/java/dev/equwal/assistkey/ui/ActionPickerActivity.kt b/app/src/main/java/dev/equwal/assistkey/ui/ActionPickerActivity.kt index d2e1785..e3c200f 100644 --- a/app/src/main/java/dev/equwal/assistkey/ui/ActionPickerActivity.kt +++ b/app/src/main/java/dev/equwal/assistkey/ui/ActionPickerActivity.kt @@ -41,6 +41,10 @@ class ActionPickerActivity : Activity() { basics(col) viwoods(col) navigation(col) + col.header("Typing") + col.row("Voice typing", "Speak, and the words go where the cursor is") { + choose(ActionSpec(ActionKind.VOICE, "", "Voice typing")) + } own(col) reading(col) soundAndMedia(col) diff --git a/app/src/main/java/dev/equwal/assistkey/ui/MainActivity.kt b/app/src/main/java/dev/equwal/assistkey/ui/MainActivity.kt index c463a3b..39dc78c 100644 --- a/app/src/main/java/dev/equwal/assistkey/ui/MainActivity.kt +++ b/app/src/main/java/dev/equwal/assistkey/ui/MainActivity.kt @@ -224,6 +224,11 @@ class MainActivity : Activity() { // ---- everything else --------------------------------------------------- private fun extras(col: LinearLayout) { + col.header("Typing") + col.row("Voice typing", "Speak into any text field, from a key") { + startActivity(Intent(this, VoiceActivity::class.java)) + } + col.header("Display and home") col.row("Extra-dim light", "Below the lowest the system slider allows") { startActivity(Intent(this, DisplayActivity::class.java)) diff --git a/app/src/main/java/dev/equwal/assistkey/ui/VoiceActivity.kt b/app/src/main/java/dev/equwal/assistkey/ui/VoiceActivity.kt new file mode 100644 index 0000000..20781f7 --- /dev/null +++ b/app/src/main/java/dev/equwal/assistkey/ui/VoiceActivity.kt @@ -0,0 +1,93 @@ +package dev.equwal.assistkey.ui + +import android.Manifest +import android.app.Activity +import android.content.Intent +import android.net.Uri +import android.provider.Settings +import dev.equwal.assistkey.channel.Channel +import dev.equwal.assistkey.channel.Channels +import dev.equwal.assistkey.voice.Dictation +import dev.equwal.assistkey.ui.Ui.button +import dev.equwal.assistkey.ui.Ui.header +import dev.equwal.assistkey.ui.Ui.note +import dev.equwal.assistkey.ui.Ui.row +import dev.equwal.assistkey.ui.Ui.title + +/** Setup for the Voice typing action: microphone, speech app, language. */ +class VoiceActivity : Activity() { + + override fun onResume() { + super.onResume() + build() + } + + override fun onRequestPermissionsResult(code: Int, permissions: Array<out String>, results: IntArray) { + super.onRequestPermissionsResult(code, permissions, results) + build() + } + + private fun build() { + val col = Ui.page(this) + col.title("Voice typing") + col.note( + "Bind the Voice typing action to a key. Press the key, speak, and the " + + "words go where the cursor is. Press the key again to stop early." + ) + col.note( + "AssistKey has no speech engine and no internet access. A speech " + + "recognition app on this device does the listening, and AssistKey " + + "only receives the text." + ) + + col.header("1. Microphone") + if (Dictation.hasMicrophone(this)) { + col.row("Allowed", null, enabled = false) + } else { + col.button("Allow the microphone") { + requestPermissions(arrayOf(Manifest.permission.RECORD_AUDIO), 1) + } + col.note("If nothing happens, allow it in App info > Permissions.") + col.button("Open App info") { + runCatching { + startActivity( + Intent(Settings.ACTION_APPLICATION_DETAILS_SETTINGS, Uri.parse("package:" + packageName)) + ) + } + } + } + + col.header("2. Speech recognition app") + val engines = Dictation.engines(this) + if (engines.isEmpty()) { + col.note( + "This device has none. Install a voice input app that offers speech " + + "recognition to other apps. For recognition on the device with a " + + "Whisper model, look for FUTO Voice Input or Whisper by woheller69." + ) + } else { + val current = Dictation.engine(this) + engines.forEach { e -> + val mark = if (e.component == current) "* " else " " + col.row(mark + e.title, e.component.packageName) { + Dictation.setEngine(this, e.component) + build() + } + } + } + + col.header("3. Language") + val lang = Dictation.language(this) + col.row(if (lang.isEmpty()) "The speech app decides" else lang, "A tag such as en-US or de-DE") { + Ui.textInput(this, "Language tag", "en-US", lang) { Dictation.setLanguage(this, it); build() } + } + if (lang.isNotEmpty()) col.button("Clear the language") { Dictation.setLanguage(this, ""); build() } + + col.header("4. Key") + col.note("Open a key on the main screen, choose a gesture, then Typing > Voice typing.") + if (!Channels.isSatisfied(this, Channel.ACCESSIBILITY)) { + col.note("Key remapping is off. Voice typing needs it to reach the text field.") + } + col.note("Voice typing never writes into password fields.") + } +} diff --git a/app/src/main/java/dev/equwal/assistkey/voice/Dictation.kt b/app/src/main/java/dev/equwal/assistkey/voice/Dictation.kt new file mode 100644 index 0000000..5568d72 --- /dev/null +++ b/app/src/main/java/dev/equwal/assistkey/voice/Dictation.kt @@ -0,0 +1,106 @@ +package dev.equwal.assistkey.voice + +import android.Manifest +import android.accessibilityservice.AccessibilityService +import android.content.ComponentName +import android.content.Context +import android.content.Intent +import android.content.pm.PackageManager +import android.speech.RecognitionService +import android.speech.SpeechRecognizer +import android.widget.Toast + +/** + * Voice typing: speak, and the words go where the cursor is. + * + * This app has no speech engine and no network access. It uses the Android + * SpeechRecognizer interface, so the recognition is done by a speech app the + * user already has - the system one, or an offline Whisper app, or any other + * app that offers a RecognitionService. The words come back as text, and + * [TextInsert] puts them into the focused field. + * + * This object holds the settings and starts [DictationActivity], which listens. + */ +object Dictation { + + private const val TAG = "AssistKey" + private const val PREFS = "assistkey_voice" + private const val K_ENGINE = "engine" + private const val K_LANGUAGE = "language" + + // ---- configuration ------------------------------------------------------------------- + + data class Engine(val title: String, val component: ComponentName) + + fun engines(c: Context): List<Engine> = + c.packageManager.queryIntentServices(Intent(RecognitionService.SERVICE_INTERFACE), 0).map { + Engine( + it.loadLabel(c.packageManager).toString(), + ComponentName(it.serviceInfo.packageName, it.serviceInfo.name) + ) + } + + /** The engine the user chose, else the only one, else the system default (null). */ + fun engine(c: Context): ComponentName? { + val all = engines(c) + val saved = c.getSharedPreferences(PREFS, Context.MODE_PRIVATE).getString(K_ENGINE, null) + return all.firstOrNull { it.component.flattenToString() == saved }?.component + ?: all.singleOrNull()?.component + } + + fun setEngine(c: Context, component: ComponentName?) { + c.getSharedPreferences(PREFS, Context.MODE_PRIVATE).edit() + .putString(K_ENGINE, component?.flattenToString()).apply() + } + + /** A language tag such as en-US. Empty means the engine decides. */ + fun language(c: Context): String = + c.getSharedPreferences(PREFS, Context.MODE_PRIVATE).getString(K_LANGUAGE, "").orEmpty() + + fun setLanguage(c: Context, tag: String) { + c.getSharedPreferences(PREFS, Context.MODE_PRIVATE).edit().putString(K_LANGUAGE, tag.trim()).apply() + } + + fun hasMicrophone(c: Context): Boolean = + c.checkSelfPermission(Manifest.permission.RECORD_AUDIO) == PackageManager.PERMISSION_GRANTED + + fun available(c: Context): Boolean = engines(c).isNotEmpty() || SpeechRecognizer.isRecognitionAvailable(c) + + // ---- the action ------------------------------------------------------------------------ + + /** Set by [DictationActivity] while it listens; stops it and uses what was heard. */ + @Volatile var stop: (() -> Unit)? = null + + val listening: Boolean get() = stop != null + + /** + * Starts listening. A second call while listening stops early. + * + * Android gives the microphone to the app in front only, and an accessibility + * service does not count. So the listening runs in [DictationActivity], which + * is see-through and cannot take focus: this app is in front, and the text + * field behind it keeps the cursor. + */ + fun toggle(svc: AccessibilityService): Boolean { + stop?.let { + it() + return true + } + if (!hasMicrophone(svc)) return fail(svc, "Voice typing needs the microphone. Open AssistKey > Voice typing.") + if (!available(svc)) return fail(svc, "No speech recognition app on this device. Open AssistKey > Voice typing.") + return runCatching { + svc.startActivity( + Intent(svc, DictationActivity::class.java) + .addFlags(Intent.FLAG_ACTIVITY_NEW_TASK or Intent.FLAG_ACTIVITY_NO_ANIMATION) + ) + true + }.getOrDefault(false) + } + + fun toast(c: Context, message: String) = Toast.makeText(c, message, Toast.LENGTH_LONG).show() + + private fun fail(c: Context, message: String): Boolean { + toast(c, message) + return false + } +} diff --git a/app/src/main/java/dev/equwal/assistkey/voice/DictationActivity.kt b/app/src/main/java/dev/equwal/assistkey/voice/DictationActivity.kt new file mode 100644 index 0000000..b2aa443 --- /dev/null +++ b/app/src/main/java/dev/equwal/assistkey/voice/DictationActivity.kt @@ -0,0 +1,157 @@ +package dev.equwal.assistkey.voice + +import android.app.Activity +import android.content.Intent +import android.graphics.Color +import android.os.Bundle +import android.os.Handler +import android.os.Looper +import android.speech.RecognitionListener +import android.speech.RecognizerIntent +import android.speech.SpeechRecognizer +import android.util.Log +import android.util.TypedValue +import android.view.Gravity +import android.view.WindowManager +import android.widget.FrameLayout +import android.widget.TextView +import dev.equwal.assistkey.route.ServiceHolder + +/** + * Listens, and hands the words to [TextInsert]. + * + * The activity exists so that this app is in front while the microphone is in + * use; Android allows nothing else. It is see-through, and its window cannot + * take focus or touches, so the text field behind it keeps the cursor and the + * keyboard. The only thing it draws is a one-line label. + */ +class DictationActivity : Activity() { + + private companion object { + /** Time for the window behind to be the front window again. */ + const val INSERT_DELAY_MS = 300L + } + + private var recognizer: SpeechRecognizer? = null + private lateinit var label: TextView + + override fun onCreate(savedInstanceState: Bundle?) { + super.onCreate(savedInstanceState) + window.addFlags( + WindowManager.LayoutParams.FLAG_NOT_FOCUSABLE or + WindowManager.LayoutParams.FLAG_NOT_TOUCHABLE or + WindowManager.LayoutParams.FLAG_NOT_TOUCH_MODAL + ) + label = TextView(this).apply { + text = "Listening..." + setTextColor(Color.WHITE) + setBackgroundColor(Color.BLACK) + setTextSize(TypedValue.COMPLEX_UNIT_SP, 18f) + val p = (16 * resources.displayMetrics.density).toInt() + setPadding(p, p * 3, p, p) + } + setContentView( + FrameLayout(this).apply { + addView( + label, + FrameLayout.LayoutParams( + FrameLayout.LayoutParams.MATCH_PARENT, + FrameLayout.LayoutParams.WRAP_CONTENT, + Gravity.TOP + ) + ) + } + ) + } + + override fun onResume() { + super.onResume() + if (recognizer == null) listen() + } + + override fun onDestroy() { + close() + super.onDestroy() + } + + private fun listen() { + val r = runCatching { + Dictation.engine(this)?.let { SpeechRecognizer.createSpeechRecognizer(this, it) } + ?: SpeechRecognizer.createSpeechRecognizer(this) + }.getOrNull() + if (r == null) { + Dictation.toast(this, "The speech recognition app did not start.") + return done(null) + } + recognizer = r + Dictation.stop = { r.stopListening() } + r.setRecognitionListener(listener) + r.startListening( + Intent(RecognizerIntent.ACTION_RECOGNIZE_SPEECH).apply { + putExtra(RecognizerIntent.EXTRA_LANGUAGE_MODEL, RecognizerIntent.LANGUAGE_MODEL_FREE_FORM) + putExtra(RecognizerIntent.EXTRA_PARTIAL_RESULTS, true) + putExtra(RecognizerIntent.EXTRA_CALLING_PACKAGE, packageName) + Dictation.language(this@DictationActivity).takeIf { it.isNotEmpty() } + ?.let { putExtra(RecognizerIntent.EXTRA_LANGUAGE, it) } + } + ) + } + + private val listener = object : RecognitionListener { + override fun onResults(results: Bundle?) = + done(results?.getStringArrayList(SpeechRecognizer.RESULTS_RECOGNITION)?.firstOrNull()) + + override fun onError(error: Int) { + Log.i("AssistKey", "speech recognition error " + error) + Dictation.toast( + this@DictationActivity, + when (error) { + SpeechRecognizer.ERROR_NO_MATCH, SpeechRecognizer.ERROR_SPEECH_TIMEOUT -> "Heard nothing" + SpeechRecognizer.ERROR_INSUFFICIENT_PERMISSIONS -> "The microphone permission is missing" + SpeechRecognizer.ERROR_NETWORK, SpeechRecognizer.ERROR_NETWORK_TIMEOUT -> + "The speech recognition app could not reach its server" + else -> "Speech recognition failed (" + error + ")" + } + ) + done(null) + } + + override fun onPartialResults(partial: Bundle?) { + partial?.getStringArrayList(SpeechRecognizer.RESULTS_RECOGNITION)?.firstOrNull() + ?.takeIf { it.isNotBlank() }?.let { label.text = it } + } + + override fun onEndOfSpeech() { label.text = "Working..." } + override fun onReadyForSpeech(params: Bundle?) = Unit + override fun onBeginningOfSpeech() = Unit + override fun onRmsChanged(rmsdB: Float) = Unit + override fun onBufferReceived(buffer: ByteArray?) = Unit + override fun onEvent(eventType: Int, params: Bundle?) = Unit + } + + private fun close() { + Dictation.stop = null + runCatching { recognizer?.destroy() } + recognizer = null + } + + /** Leaves first, then writes: the field must be the front window again. */ + private fun done(text: String?) { + Log.i("AssistKey", "voice typing: " + if (text.isNullOrBlank()) "no text" else text.length.toString() + " characters") + close() + finish() + @Suppress("DEPRECATION") + overridePendingTransition(0, 0) + if (text.isNullOrBlank()) return + val app = applicationContext + Handler(Looper.getMainLooper()).postDelayed({ + val svc = ServiceHolder.service ?: return@postDelayed + when (TextInsert.insert(svc, text)) { + TextInsert.Result.DONE -> Unit + TextInsert.Result.NO_FIELD -> Dictation.toast(app, "No text field has the cursor") + TextInsert.Result.PASSWORD -> Dictation.toast(app, "Voice typing does not write into password fields") + TextInsert.Result.REFUSED -> Dictation.toast(app, "That field did not accept the text") + } + }, INSERT_DELAY_MS) + } +} diff --git a/app/src/main/java/dev/equwal/assistkey/voice/TextInsert.kt b/app/src/main/java/dev/equwal/assistkey/voice/TextInsert.kt new file mode 100644 index 0000000..c11009e --- /dev/null +++ b/app/src/main/java/dev/equwal/assistkey/voice/TextInsert.kt @@ -0,0 +1,59 @@ +package dev.equwal.assistkey.voice + +import android.accessibilityservice.AccessibilityService +import android.content.ClipData +import android.content.ClipboardManager +import android.os.Bundle +import android.view.accessibility.AccessibilityNodeInfo + +/** + * Puts text where the cursor is, in the app in front. + * + * The accessibility service does this, not a keyboard. It finds the field that + * has input focus and replaces the selection with the new text. A field that + * refuses a direct edit gets the text through the clipboard and a paste. + * + * Password fields are never written to. + */ +object TextInsert { + + enum class Result { DONE, NO_FIELD, PASSWORD, REFUSED } + + fun insert(svc: AccessibilityService, text: String): Result { + val node = svc.findFocus(AccessibilityNodeInfo.FOCUS_INPUT) + ?: svc.rootInActiveWindow?.findFocus(AccessibilityNodeInfo.FOCUS_INPUT) + ?: return Result.NO_FIELD + if (node.isPassword) return Result.PASSWORD + if (!node.isEditable) return Result.NO_FIELD + + // A field that shows its hint reports the hint as its text. + val old = if (node.isShowingHintText) "" else node.text?.toString().orEmpty() + val start = node.textSelectionStart.takeIf { it in 0..old.length } ?: old.length + val end = node.textSelectionEnd.takeIf { it in start..old.length } ?: start + + // A space between the old text and the new words, when there is none. + val lead = if (start > 0 && !old[start - 1].isWhitespace()) " " else "" + val piece = lead + text + val merged = old.substring(0, start) + piece + old.substring(end) + + val set = Bundle().apply { + putCharSequence(AccessibilityNodeInfo.ACTION_ARGUMENT_SET_TEXT_CHARSEQUENCE, merged) + } + if (node.performAction(AccessibilityNodeInfo.ACTION_SET_TEXT, set)) { + val caret = start + piece.length + node.performAction( + AccessibilityNodeInfo.ACTION_SET_SELECTION, + Bundle().apply { + putInt(AccessibilityNodeInfo.ACTION_ARGUMENT_SELECTION_START_INT, caret) + putInt(AccessibilityNodeInfo.ACTION_ARGUMENT_SELECTION_END_INT, caret) + } + ) + return Result.DONE + } + + // Web pages and some custom fields accept a paste but not a direct edit. + val clipboard = svc.getSystemService(ClipboardManager::class.java) ?: return Result.REFUSED + clipboard.setPrimaryClip(ClipData.newPlainText("AssistKey voice typing", piece)) + return if (node.performAction(AccessibilityNodeInfo.ACTION_PASTE)) Result.DONE else Result.REFUSED + } +} diff --git a/gradle.properties b/gradle.properties index c59f674..985c935 100644 --- a/gradle.properties +++ b/gradle.properties @@ -6,8 +6,8 @@ android.nonTransitiveRClass=true kotlin.code.style=official # Release identity. versionCode must rise with every upload to Play. -assistkey.versionName=0.0.3-alpha -assistkey.versionCode=3 +assistkey.versionName=0.0.4-alpha +assistkey.versionCode=4 # The free beta ends on this date (UTC) at the latest. It is the backstop for # installs that cannot reach Google Play; the real switch is the diff --git a/play/DECLARATIONS.md b/play/DECLARATIONS.md index 995573e..7a5bbd6 100644 --- a/play/DECLARATIONS.md +++ b/play/DECLARATIONS.md @@ -1,7 +1,7 @@ # App content declarations Answers for Play Console > Policy and programs > App content. Each is true of -the build tagged `v0.0.3-alpha`; re-check them if the app changes. +the build tagged `v0.0.4-alpha`; re-check them if the app changes. ## Privacy policy @@ -63,13 +63,15 @@ disabilities)?** No. The manifest does not set `isAccessibilityTool`. ``` AssistKey is a hardware key remapper for the Viwoods AiPaper Reader, an e-ink reading device. Remapping hardware keys is the app's only function, and the AccessibilityService API is the only public Android API that can do it. -The service is used for two things: +The service is used for three things: 1. Key event filtering (flagRequestFilterKeyEvents / onKeyEvent). The service receives presses of the device's AI key and volume keys, recognises the gesture the user configured - single or multiple taps, press-and-hold, or a combination of keys - and consumes the press so that the user's chosen action runs instead of the default one. 2. Performing the action the user bound to that gesture: performGlobalAction (Back, Home, Recents, notifications, quick settings, lock screen, screenshot), dispatchGesture (a swipe, used to turn pages in reading apps that accept only touch input), and ACTION_SCROLL_FORWARD / ACTION_SCROLL_BACKWARD on the scrollable node of the active window. -Window content is retrieved only in case 2, only to locate a scrollable node, and only at the moment the user presses a key bound to the Scroll action. No window content, text, or key event is stored, logged or transmitted. The app does not request the INTERNET permission and cannot transmit anything. +3. Voice typing, if the user binds that action: the service finds the text field that has input focus and inserts the recognised words at the cursor (ACTION_SET_TEXT, or ACTION_PASTE as the fallback). It never writes into password fields. + +Window content is retrieved only in cases 2 and 3: to locate a scrollable node at the moment the user presses a key bound to the Scroll action, and to read the one focused text field at the moment recognised words are inserted into it. No window content, text, or key event is stored, logged or transmitted. The app does not request the INTERNET permission and cannot transmit anything. Before the user is sent to the accessibility settings, the app shows a prominent in-app disclosure describing exactly this use, and proceeds only if the user taps Agree. ``` @@ -85,6 +87,7 @@ pointed at the reader - the reader cannot capture its own screen. |---|---| | `BIND_ACCESSIBILITY_SERVICE` | Above. | | `WRITE_SECURE_SETTINGS` | Cannot be granted to a Play install; it does nothing unless the owner grants it over adb. It then lets the app switch the firmware's power-button behaviour (short press, hold duration). Declared so that the grant is possible at all. | +| `RECORD_AUDIO` | Voice typing only. Requested at run time from the Voice typing screen. The speech recognition app the user chose records the audio; Android requires the calling app to hold the permission too. AssistKey receives text only, and stores and sends nothing. | | `PACKAGE_USAGE_STATS` | Special access the user grants from a settings screen. Used only by the recent-apps list to order apps by last use, on the device; nothing is stored or sent. Not needed when shell access is on. | | `moe.shizuku.manager.permission.API_V23` | Lets the user connect AssistKey to Shizuku, a separate app they install and start themselves, for features that need the shell user. | | `QUICK_ACCESS_WALLET` | Required of any app offered as the wallet app. AssistKey serves an empty card list; it exists so a press of the wallet shortcut reaches the user's chosen action. | diff --git a/play/LISTING.md b/play/LISTING.md index e0b6e78..d5779ad 100644 --- a/play/LISTING.md +++ b/play/LISTING.md @@ -47,6 +47,9 @@ ACTIONS • Send an intent or a broadcast, for automation apps • Do nothing - disable a key you keep pressing by accident +VOICE TYPING +Bind a key to Voice typing: press, speak, and the words appear where the cursor is - in any app, with the keyboard you already have. AssistKey has no speech engine and no internet access; a speech recognition app on your device does the listening. + MORE FOR E-INK • Extra-dim light: go below the lowest brightness the system allows (needs shell access with root) • A plain text home screen: a clock, a few apps, type to find the rest diff --git a/play/PRIVACY.md b/play/PRIVACY.md index 7dda86d..5c6ac4c 100644 --- a/play/PRIVACY.md +++ b/play/PRIVACY.md @@ -26,6 +26,14 @@ can scroll it. It does not read, record or transmit what the window contains. channels are on are saved in the app's private storage on your device. They leave the device only through Android's own backup, if you have it on. +**Voice typing.** If you bind the Voice typing action, AssistKey asks for the +microphone permission. A speech recognition app on your device, which you +choose, does the listening; how it handles audio is covered by that app's own +policy. AssistKey receives only the recognised text, and puts it into the text +field that has the cursor. To do that it reads the current text of that one +field at that moment. It keeps and sends none of it, and never writes into +password fields. + **Recent apps.** If you use the recent-apps list without shell access, it reads the usage log Android keeps, with the usage access you grant, to put apps in order of last use. This happens on the device each time the list opens;