Compare commits
4 commits
edd8ccaa05
...
2f9ada646c
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
2f9ada646c | ||
|
|
1645be8a97 | ||
|
|
c4884a3748 | ||
|
|
9e1f30de1f |
12 changed files with 246 additions and 9 deletions
28
CLAUDE.md
28
CLAUDE.md
|
|
@ -26,9 +26,10 @@ Make the **native Android app** (`kaizen-app`) a valid daily-driver that talks t
|
||||||
- **Sentinels** (`lib/chat/constants.ts`): `\x00` USAGE, `\x01` REASONING, `\x02` SOURCES, `\x03` TOOL, `\x04` QUERY. App currently strips all sentinels and renders only visible content.
|
- **Sentinels** (`lib/chat/constants.ts`): `\x00` USAGE, `\x01` REASONING, `\x02` SOURCES, `\x03` TOOL, `\x04` QUERY. App currently strips all sentinels and renders only visible content.
|
||||||
- **Models:** `GET {baseUrl}/api/v1/models` → `{ models: [{ id, name?, provider? }] }`.
|
- **Models:** `GET {baseUrl}/api/v1/models` → `{ models: [{ id, name?, provider? }] }`.
|
||||||
- **Conversations:** `GET /api/v1/conversations` → `{ conversations, hasMore }`. `POST /api/v1/conversations` → create. `GET /api/v1/conversations/[id]` → `{ messages }`. `POST /api/v1/conversations/[id]/messages` → persist. `POST /api/v1/conversations/[id]/title` → auto-title.
|
- **Conversations:** `GET /api/v1/conversations` → `{ conversations, hasMore }`. `POST /api/v1/conversations` → create. `GET /api/v1/conversations/[id]` → `{ messages }`. `POST /api/v1/conversations/[id]/messages` → persist. `POST /api/v1/conversations/[id]/title` → auto-title.
|
||||||
- **Identity:** `GET /api/v1/me` → `{ id, email, name, role, defaultModel }`.
|
- **Identity:** `GET /api/v1/me` → `{ id, email, name, role, defaultModel, ttsVoice }`. `PATCH /api/v1/me` with `{ ttsVoice }` to update preferences.
|
||||||
- **Capability handshake:** `GET /api/v1/meta` → `{ app, apiVersion, features }` (unauth).
|
- **Capability handshake:** `GET /api/v1/meta` → `{ app, apiVersion, features }` (unauth).
|
||||||
- **Upload:** `POST /api/v1/upload` — multipart file → `{ url, kind, name, mimeType, size, fileId }`.
|
- **Upload:** `POST /api/v1/upload` — multipart file → `{ url, kind, name, mimeType, size, fileId }`.
|
||||||
|
- **Audio gen:** `POST /api/v1/audio-gen` — `{ prompt, kind: "speech"|"music", voice? }` → `{ url, cost }`. Gemini-TTS (speech) or Lyria (music) via Vertex. Audio persisted in Object Storage.
|
||||||
|
|
||||||
## App architecture
|
## App architecture
|
||||||
|
|
||||||
|
|
@ -195,13 +196,16 @@ All backend work for the app is done. `pnpm test:run` green (774+), `tsc` + `lin
|
||||||
- **`api_tokens` table** + SHA-256 hash storage (never raw). Migration `0024`.
|
- **`api_tokens` table** + SHA-256 hash storage (never raw). Migration `0024`.
|
||||||
- **Bearer auth path** (`getTokenSession` / `requireActiveUserOrToken` in `lib/auth/guard.ts`).
|
- **Bearer auth path** (`getTokenSession` / `requireActiveUserOrToken` in `lib/auth/guard.ts`).
|
||||||
- **Token mint endpoint** `POST /api/v1/auth/token` — unauthenticated, Argon2id verify, anti-enumeration, rate-limited.
|
- **Token mint endpoint** `POST /api/v1/auth/token` — unauthenticated, Argon2id verify, anti-enumeration, rate-limited.
|
||||||
- **`GET /api/v1/me`** — identity + `defaultModel` for native clients.
|
- **`GET /api/v1/me`** — identity + `defaultModel` + `ttsVoice` for native clients.
|
||||||
|
- **`PATCH /api/v1/me`** — update user preferences (`ttsVoice`, etc.). Bearer-authed.
|
||||||
- **`GET /api/v1/models`** — Bearer-authed, returns filtered model catalog.
|
- **`GET /api/v1/models`** — Bearer-authed, returns filtered model catalog.
|
||||||
- **`/api/v1/conversations`** — thin re-exports of canonical handlers, Bearer-authed.
|
- **`/api/v1/conversations`** — thin re-exports of canonical handlers, Bearer-authed.
|
||||||
- **`POST /api/v1/auth/unlock`** — Bearer-authed, returns unlock token in response body (not cookie) for native clients. Rate-limited 10/5min per user.
|
- **`POST /api/v1/auth/unlock`** — Bearer-authed, returns unlock token in response body (not cookie) for native clients. Rate-limited 10/5min per user.
|
||||||
- **`GET /api/v1/conversations/[id]`** — accepts `X-Unlock-Token` header alongside the existing `kaizen_unlock` cookie for locked conversations.
|
- **`GET /api/v1/conversations/[id]`** — accepts `X-Unlock-Token` header alongside the existing `kaizen_unlock` cookie for locked conversations.
|
||||||
- **`GET /api/v1/meta`** — capability handshake.
|
- **`GET /api/v1/meta`** — capability handshake.
|
||||||
- **App-devices card** (`components/settings/app-tokens-card.tsx`) — see/revoke signed-in devices on `/settings/security`.
|
- **App-devices card** (`components/settings/app-tokens-card.tsx`) — see/revoke signed-in devices on `/settings/security`.
|
||||||
|
- **`POST /api/v1/audio-gen`** — Gemini-TTS speech generation. Bearer-authed (re-export of `/api/audio-gen`). `{ prompt, kind: "speech", voice? }` → `{ url, cost }`.
|
||||||
|
- **`tts_voice` column** on `users` table (migration 0030). Server-synced voice preference, read via `/api/v1/me`, written via `PATCH /api/v1/me`.
|
||||||
- **API versioning** — `/api/v1/` for outward contract, additive-only within v1. Web-internal endpoints stay unversioned.
|
- **API versioning** — `/api/v1/` for outward contract, additive-only within v1. Web-internal endpoints stay unversioned.
|
||||||
|
|
||||||
### App — shipped features (on-device verified on S25 Ultra)
|
### App — shipped features (on-device verified on S25 Ultra)
|
||||||
|
|
@ -260,6 +264,10 @@ All committed on `main`. Compile-verified on Mac (`./gradlew :app:compileDebugKo
|
||||||
| `fd0de5c` | **Markdown upgrade** — tables (`\| col \| col \|`), ~~strikethrough~~, task lists (`- [x]`), clickable links (ACTION_VIEW), block comments (`/* */`), HTML comments, template strings, C/C++/Ruby/PHP/Dockerfile/TOML highlighting. All regex pre-compiled |
|
| `fd0de5c` | **Markdown upgrade** — tables (`\| col \| col \|`), ~~strikethrough~~, task lists (`- [x]`), clickable links (ACTION_VIEW), block comments (`/* */`), HTML comments, template strings, C/C++/Ruby/PHP/Dockerfile/TOML highlighting. All regex pre-compiled |
|
||||||
| `bf94414` | **CallButton** — styled to match SendButton: 38dp circle, subtle accent gradient (16-28% alpha), AudioLines icon in accent color, press-scale animation |
|
| `bf94414` | **CallButton** — styled to match SendButton: 38dp circle, subtle accent gradient (16-28% alpha), AudioLines icon in accent color, press-scale animation |
|
||||||
| `141d4de` | **Stop button + auto-logout + locale fix + login shadow migration** — red square stop button during streaming (cancels stream, preserves partial content), auto-logout on HTTP 401, EmptyHero date respects system locale, LoginScreen shadows migrated to kaizenShadow |
|
| `141d4de` | **Stop button + auto-logout + locale fix + login shadow migration** — red square stop button during streaming (cancels stream, preserves partial content), auto-logout on HTTP 401, EmptyHero date respects system locale, LoginScreen shadows migrated to kaizenShadow |
|
||||||
|
| `edd8cca` | **Bottom padding fix** — LazyColumn bottom contentPadding 152→220dp, fixes messages overlapping mode pills after ChatInput two-row redesign |
|
||||||
|
| `9e1f30d` | **Voice recording + TTS read-aloud** — Mic button wired to `MediaRecorder` (AAC/M4A), tap-to-record, auto-upload as audio attachment. Volume button on assistant messages → `POST /api/v1/audio-gen` → Gemini-TTS → `MediaPlayer` playback. `RECORD_AUDIO` permission |
|
||||||
|
| `c4884a3` | **TTS audio cache in Room** — `ttsUrl` + `ttsVoice` on `MessageEntity` (Room v3), skip API call on replay with same voice. Voice change invalidates cache |
|
||||||
|
| `1645be8` | **Server-synced TTS voice** — reads `ttsVoice` from `GET /api/v1/me`, uses it for all TTS calls. Voice set in web settings applies in app automatically |
|
||||||
|
|
||||||
**Earlier UI/feel work (Phase 0 prototype → feel-first):**
|
**Earlier UI/feel work (Phase 0 prototype → feel-first):**
|
||||||
- Liquid glass styling, floating panels, glassmorphic sidebar
|
- Liquid glass styling, floating panels, glassmorphic sidebar
|
||||||
|
|
@ -298,6 +306,13 @@ All committed on `main`. Compile-verified on Mac (`./gradlew :app:compileDebugKo
|
||||||
- **Sidebar flicker on refresh** — `replaceAll()` did `DELETE ALL` + `INSERT`, causing Room Flow to emit empty list briefly. Fix: `upsert` + `deleteExcept` (no flash).
|
- **Sidebar flicker on refresh** — `replaceAll()` did `DELETE ALL` + `INSERT`, causing Room Flow to emit empty list briefly. Fix: `upsert` + `deleteExcept` (no flash).
|
||||||
- **Gallery/file picker single-select only** — `GetContent()` allowed only one item. Fix: gallery → `PickMultipleVisualMedia(ImageAndVideo)`, file → `OpenMultipleDocuments(["*/*"])`.
|
- **Gallery/file picker single-select only** — `GetContent()` allowed only one item. Fix: gallery → `PickMultipleVisualMedia(ImageAndVideo)`, file → `OpenMultipleDocuments(["*/*"])`.
|
||||||
|
|
||||||
|
### Bugs fixed (2026-06-23 — voice + UX session)
|
||||||
|
|
||||||
|
- **Messages overlapping mode pills** — LazyColumn bottom `contentPadding` was 152dp but the two-row ChatInput redesign made the dock ~210dp tall. Fix: increased to 220dp.
|
||||||
|
- **EmptyHero date hardcoded German** — `Locale.GERMAN` → `Locale.getDefault()`, now respects system/app language.
|
||||||
|
- **LoginScreen used `Modifier.shadow`** — violated design system pillar 4 (multi-layer shadows). Fix: migrated to `kaizenShadow` (level1 for fields, level2 for button).
|
||||||
|
- **Migration 0030 crash** — `pnpm db:generate` bundled the entire org schema (0026-0029) + `tts_voice` into one migration, causing duplicate CREATE TABLE / ADD COLUMN failures. Fix: reduced 0030 to only `ALTER TABLE users ADD COLUMN IF NOT EXISTS tts_voice text`.
|
||||||
|
|
||||||
## What's NOT done yet (app, priority order)
|
## What's NOT done yet (app, priority order)
|
||||||
|
|
||||||
### 1. Remaining visual/UX gaps (daily-drive blockers)
|
### 1. Remaining visual/UX gaps (daily-drive blockers)
|
||||||
|
|
@ -341,10 +356,13 @@ These are what separate "prototype" from "daily-driver":
|
||||||
### 5. Voice & Call Mode
|
### 5. Voice & Call Mode
|
||||||
|
|
||||||
**v1 — Pipeline (Weg A, uses existing backend):**
|
**v1 — Pipeline (Weg A, uses existing backend):**
|
||||||
- [ ] **Voice recording + send** — Android `MediaRecorder`, upload via `/api/v1/upload`, send as audio attachment via `/api/v1/chat` (Vertex understands audio natively)
|
- [x] **Voice recording + send** — Android `MediaRecorder` (AAC/M4A, 44.1kHz/128kbps), tap-to-record mic button (red stop-square while recording), auto-upload via `/api/v1/upload`, sent as audio attachment via `/api/v1/chat`. Vertex understands audio natively; OpenRouter passes it as `input_audio`. Persisted in DB as normal attachment
|
||||||
- [ ] **TTS playback** — POST response text to `/api/v1/audio-gen` (needs Bearer auth, currently web-session-only via `requireActiveUser`), play returned audio URL via `MediaPlayer`/`ExoPlayer`
|
- [x] **TTS read-aloud** — Volume button on every assistant message. `POST /api/v1/audio-gen { prompt, kind: "speech", voice }` → Gemini-TTS → `MediaPlayer` streams the audio URL. Tap again to stop. Accent-colored icon during playback
|
||||||
|
- [x] **TTS cache** — `ttsUrl` + `ttsVoice` fields on `MessageEntity` (Room v3). First play generates + caches; subsequent plays with same voice skip the API call. Voice change auto-invalidates
|
||||||
|
- [x] **Server-synced voice** — `tts_voice` column on `users` table (migration 0030). Read from `GET /api/v1/me`, written via `PATCH /api/v1/me`. Web's `setCallVoice()` syncs to server. App reads on login/resume. Default "Kore"
|
||||||
|
- [x] **Backend: Bearer-auth audio-gen** — `/api/audio-gen` switched to `requireActiveUserOrToken()`, v1 re-export at `/api/v1/audio-gen`
|
||||||
- [ ] **Call UI** — fullscreen overlay, KaizenOrb with amplitude-reactive animation, push-to-talk mic button, status machine (idle → recording → uploading → thinking → speaking), voice picker (30 Gemini-TTS voices)
|
- [ ] **Call UI** — fullscreen overlay, KaizenOrb with amplitude-reactive animation, push-to-talk mic button, status machine (idle → recording → uploading → thinking → speaking), voice picker (30 Gemini-TTS voices)
|
||||||
- [ ] **Backend: Bearer-auth audio-gen** — switch `/api/audio-gen` from `requireActiveUser()` to `requireActiveUserOrToken()`, add v1 re-export at `/api/v1/audio-gen`
|
- [ ] **In-app voice picker** — settings UI to choose from 30 Gemini-TTS voices (synced to server via `PATCH /api/v1/me`)
|
||||||
|
|
||||||
The web's call mode (`use-call-mode.ts`, `call-view.tsx`) is the reference implementation. Same pipeline: Record → Upload → Chat (with audio attachment) → Text stream → TTS → Playback. No WebSocket needed.
|
The web's call mode (`use-call-mode.ts`, `call-view.tsx`) is the reference implementation. Same pipeline: Record → Upload → Chat (with audio attachment) → Text stream → TTS → Playback. No WebSocket needed.
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -8,6 +8,9 @@
|
||||||
<!-- Required to talk to the Kaizen backend (login + chat streaming) -->
|
<!-- Required to talk to the Kaizen backend (login + chat streaming) -->
|
||||||
<uses-permission android:name="android.permission.INTERNET" />
|
<uses-permission android:name="android.permission.INTERNET" />
|
||||||
|
|
||||||
|
<!-- Required for voice recording (mic button in chat) -->
|
||||||
|
<uses-permission android:name="android.permission.RECORD_AUDIO" />
|
||||||
|
|
||||||
<application
|
<application
|
||||||
android:allowBackup="true"
|
android:allowBackup="true"
|
||||||
android:dataExtractionRules="@xml/data_extraction_rules"
|
android:dataExtractionRules="@xml/data_extraction_rules"
|
||||||
|
|
|
||||||
|
|
@ -331,6 +331,8 @@ fun MessageRow(
|
||||||
onCopy: (String) -> Unit = {},
|
onCopy: (String) -> Unit = {},
|
||||||
onDelete: ((String) -> Unit)? = null,
|
onDelete: ((String) -> Unit)? = null,
|
||||||
onRegenerate: ((String) -> Unit)? = null,
|
onRegenerate: ((String) -> Unit)? = null,
|
||||||
|
onReadAloud: ((String) -> Unit)? = null,
|
||||||
|
isPlayingTts: Boolean = false,
|
||||||
) {
|
) {
|
||||||
val cs = MaterialTheme.colorScheme
|
val cs = MaterialTheme.colorScheme
|
||||||
val isDark = isSystemInDarkTheme()
|
val isDark = isSystemInDarkTheme()
|
||||||
|
|
@ -461,6 +463,14 @@ fun MessageRow(
|
||||||
copied = false
|
copied = false
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
onReadAloud?.let { readAloud ->
|
||||||
|
MessageActionButton(
|
||||||
|
icon = KaizenIcons.Volume2,
|
||||||
|
tint = if (isPlayingTts) accent.primary else cs.onSurfaceVariant.copy(alpha = 0.35f),
|
||||||
|
) {
|
||||||
|
haptics.tick(); readAloud(message.content)
|
||||||
|
}
|
||||||
|
}
|
||||||
onRegenerate?.let { regen ->
|
onRegenerate?.let { regen ->
|
||||||
MessageActionButton(KaizenIcons.RefreshCw, cs.onSurfaceVariant.copy(alpha = 0.35f)) {
|
MessageActionButton(KaizenIcons.RefreshCw, cs.onSurfaceVariant.copy(alpha = 0.35f)) {
|
||||||
haptics.tick(); regen(message.wireId)
|
haptics.tick(); regen(message.wireId)
|
||||||
|
|
@ -527,6 +537,8 @@ fun ChatInput(
|
||||||
enabled: Boolean,
|
enabled: Boolean,
|
||||||
isStreaming: Boolean = false,
|
isStreaming: Boolean = false,
|
||||||
onStop: () -> Unit = {},
|
onStop: () -> Unit = {},
|
||||||
|
isRecording: Boolean = false,
|
||||||
|
onMicClick: () -> Unit = {},
|
||||||
pendingFiles: List<PendingFile> = emptyList(),
|
pendingFiles: List<PendingFile> = emptyList(),
|
||||||
onAddClick: () -> Unit = {},
|
onAddClick: () -> Unit = {},
|
||||||
onRemoveFile: (String) -> Unit = {},
|
onRemoveFile: (String) -> Unit = {},
|
||||||
|
|
@ -584,11 +596,17 @@ fun ChatInput(
|
||||||
}
|
}
|
||||||
Spacer(Modifier.width(8.dp))
|
Spacer(Modifier.width(8.dp))
|
||||||
Box(
|
Box(
|
||||||
Modifier.size(38.dp).clip(KaizenShapes.circle).background(iconBg),
|
Modifier.size(38.dp).clip(KaizenShapes.circle)
|
||||||
|
.background(if (isRecording) Color(0xFFEF4444).copy(alpha = if (isDark) 0.85f else 0.80f) else iconBg)
|
||||||
|
.clickable(onClick = onMicClick),
|
||||||
contentAlignment = Alignment.Center,
|
contentAlignment = Alignment.Center,
|
||||||
) {
|
) {
|
||||||
|
if (isRecording) {
|
||||||
|
Icon(KaizenIcons.Square, stringResource(R.string.chat_recording), tint = Color.White, modifier = Modifier.size(14.dp))
|
||||||
|
} else {
|
||||||
Icon(KaizenIcons.Mic, stringResource(R.string.chat_attachment), tint = cs.onSurfaceVariant, modifier = Modifier.size(19.dp))
|
Icon(KaizenIcons.Mic, stringResource(R.string.chat_attachment), tint = cs.onSurfaceVariant, modifier = Modifier.size(19.dp))
|
||||||
}
|
}
|
||||||
|
}
|
||||||
|
|
||||||
Spacer(Modifier.weight(1f))
|
Spacer(Modifier.weight(1f))
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -82,10 +82,17 @@ import androidx.activity.result.contract.ActivityResultContracts
|
||||||
import androidx.compose.ui.platform.LocalContext
|
import androidx.compose.ui.platform.LocalContext
|
||||||
import androidx.compose.ui.res.stringResource
|
import androidx.compose.ui.res.stringResource
|
||||||
import androidx.lifecycle.compose.LifecycleResumeEffect
|
import androidx.lifecycle.compose.LifecycleResumeEffect
|
||||||
|
import android.Manifest
|
||||||
|
import android.content.pm.PackageManager
|
||||||
|
import android.media.MediaPlayer
|
||||||
|
import android.media.MediaRecorder
|
||||||
|
import android.widget.Toast
|
||||||
|
import androidx.core.content.ContextCompat
|
||||||
import kotlinx.coroutines.Job
|
import kotlinx.coroutines.Job
|
||||||
import kotlinx.coroutines.async
|
import kotlinx.coroutines.async
|
||||||
import kotlinx.coroutines.launch
|
import kotlinx.coroutines.launch
|
||||||
import java.io.ByteArrayOutputStream
|
import java.io.ByteArrayOutputStream
|
||||||
|
import java.io.File
|
||||||
|
|
||||||
// Screen enumeration for modular state-driven navigation (Zero Technical Debt)
|
// Screen enumeration for modular state-driven navigation (Zero Technical Debt)
|
||||||
enum class AppScreen { Chat, Settings }
|
enum class AppScreen { Chat, Settings }
|
||||||
|
|
@ -111,6 +118,16 @@ fun ChatScreen(
|
||||||
var chatModel by remember { mutableStateOf<String?>(null) }
|
var chatModel by remember { mutableStateOf<String?>(null) }
|
||||||
var streamJob by remember { mutableStateOf<Job?>(null) }
|
var streamJob by remember { mutableStateOf<Job?>(null) }
|
||||||
|
|
||||||
|
// Audio recording state
|
||||||
|
var isRecording by remember { mutableStateOf(false) }
|
||||||
|
val recorderRef = remember { mutableStateOf<MediaRecorder?>(null) }
|
||||||
|
val audioFileRef = remember { mutableStateOf<File?>(null) }
|
||||||
|
|
||||||
|
// TTS playback state
|
||||||
|
var playingTtsForId by remember { mutableStateOf<Long?>(null) }
|
||||||
|
val ttsPlayerRef = remember { mutableStateOf<MediaPlayer?>(null) }
|
||||||
|
var ttsVoice by remember { mutableStateOf("Kore") }
|
||||||
|
|
||||||
// Navigation State
|
// Navigation State
|
||||||
var currentScreen by remember { mutableStateOf(AppScreen.Chat) }
|
var currentScreen by remember { mutableStateOf(AppScreen.Chat) }
|
||||||
|
|
||||||
|
|
@ -191,6 +208,78 @@ fun ChatScreen(
|
||||||
uploadPickedFile("photo_${System.currentTimeMillis()}.jpg", "image/jpeg", stream.toByteArray())
|
uploadPickedFile("photo_${System.currentTimeMillis()}.jpg", "image/jpeg", stream.toByteArray())
|
||||||
}
|
}
|
||||||
|
|
||||||
|
val micPermission = rememberLauncherForActivityResult(ActivityResultContracts.RequestPermission()) { granted ->
|
||||||
|
if (granted) startRecording(context, recorderRef, audioFileRef) { isRecording = true }
|
||||||
|
else Toast.makeText(context, context.getString(R.string.chat_mic_permission), Toast.LENGTH_SHORT).show()
|
||||||
|
}
|
||||||
|
|
||||||
|
fun toggleRecording() {
|
||||||
|
if (isRecording) {
|
||||||
|
stopRecording(recorderRef, audioFileRef) { name, mime, bytes ->
|
||||||
|
isRecording = false
|
||||||
|
uploadPickedFile(name, mime, bytes)
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
if (ContextCompat.checkSelfPermission(context, Manifest.permission.RECORD_AUDIO) == PackageManager.PERMISSION_GRANTED) {
|
||||||
|
startRecording(context, recorderRef, audioFileRef) { isRecording = true }
|
||||||
|
} else {
|
||||||
|
micPermission.launch(Manifest.permission.RECORD_AUDIO)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fun playTtsUrl(url: String, messageId: Long) {
|
||||||
|
try {
|
||||||
|
val player = MediaPlayer().apply {
|
||||||
|
setDataSource(url)
|
||||||
|
prepareAsync()
|
||||||
|
setOnPreparedListener { start() }
|
||||||
|
setOnCompletionListener {
|
||||||
|
release()
|
||||||
|
ttsPlayerRef.value = null
|
||||||
|
playingTtsForId = null
|
||||||
|
}
|
||||||
|
setOnErrorListener { _, _, _ ->
|
||||||
|
release()
|
||||||
|
ttsPlayerRef.value = null
|
||||||
|
playingTtsForId = null
|
||||||
|
true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
ttsPlayerRef.value = player
|
||||||
|
} catch (_: Exception) {
|
||||||
|
playingTtsForId = null
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fun readAloud(content: String, messageId: Long, wireId: String) {
|
||||||
|
val cfg = session.config ?: return
|
||||||
|
if (playingTtsForId == messageId) {
|
||||||
|
ttsPlayerRef.value?.release()
|
||||||
|
ttsPlayerRef.value = null
|
||||||
|
playingTtsForId = null
|
||||||
|
return
|
||||||
|
}
|
||||||
|
ttsPlayerRef.value?.release()
|
||||||
|
playingTtsForId = messageId
|
||||||
|
val voice = ttsVoice
|
||||||
|
scope.launch {
|
||||||
|
val cached = chat.messageDao.getTtsUrl(wireId, voice)
|
||||||
|
if (cached != null) {
|
||||||
|
playTtsUrl(cached, messageId)
|
||||||
|
return@launch
|
||||||
|
}
|
||||||
|
val url = KaizenApi.generateSpeech(cfg.baseUrl, cfg.token, content, voice)
|
||||||
|
if (url == null) {
|
||||||
|
playingTtsForId = null
|
||||||
|
Toast.makeText(context, context.getString(R.string.chat_tts_error), Toast.LENGTH_SHORT).show()
|
||||||
|
return@launch
|
||||||
|
}
|
||||||
|
chat.messageDao.updateTts(wireId, url, voice)
|
||||||
|
playTtsUrl(url, messageId)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
LaunchedEffect(session.config?.baseUrl, session.config?.token) {
|
LaunchedEffect(session.config?.baseUrl, session.config?.token) {
|
||||||
val cfg = session.config ?: return@LaunchedEffect
|
val cfg = session.config ?: return@LaunchedEffect
|
||||||
loadError = null
|
loadError = null
|
||||||
|
|
@ -209,6 +298,7 @@ fun ChatScreen(
|
||||||
KaizenApi.fetchMe(cfg.baseUrl, cfg.token)?.let { me ->
|
KaizenApi.fetchMe(cfg.baseUrl, cfg.token)?.let { me ->
|
||||||
me.name?.let { settingsViewModel.updateUserName(it) }
|
me.name?.let { settingsViewModel.updateUserName(it) }
|
||||||
me.email?.let { settingsViewModel.updateUserEmail(it) }
|
me.email?.let { settingsViewModel.updateUserEmail(it) }
|
||||||
|
me.ttsVoice?.let { ttsVoice = it }
|
||||||
}
|
}
|
||||||
chat.conversationRepo.refresh(cfg.baseUrl, cfg.token)
|
chat.conversationRepo.refresh(cfg.baseUrl, cfg.token)
|
||||||
if (errors.isNotEmpty()) {
|
if (errors.isNotEmpty()) {
|
||||||
|
|
@ -659,6 +749,10 @@ fun ChatScreen(
|
||||||
send(userContent)
|
send(userContent)
|
||||||
}
|
}
|
||||||
} else null,
|
} else null,
|
||||||
|
onReadAloud = if (message.role == Role.Assistant && !message.streaming && message.content.isNotBlank()) { content ->
|
||||||
|
readAloud(content, message.id, message.wireId)
|
||||||
|
} else null,
|
||||||
|
isPlayingTts = playingTtsForId == message.id,
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
@ -858,6 +952,8 @@ fun ChatScreen(
|
||||||
enabled = !isStreaming,
|
enabled = !isStreaming,
|
||||||
isStreaming = isStreaming,
|
isStreaming = isStreaming,
|
||||||
onStop = { streamJob?.cancel() },
|
onStop = { streamJob?.cancel() },
|
||||||
|
isRecording = isRecording,
|
||||||
|
onMicClick = { toggleRecording() },
|
||||||
pendingFiles = pendingFiles,
|
pendingFiles = pendingFiles,
|
||||||
onAddClick = { showAttachMenu = !showAttachMenu },
|
onAddClick = { showAttachMenu = !showAttachMenu },
|
||||||
onRemoveFile = { id -> pendingFiles.removeAll { it.id == id } },
|
onRemoveFile = { id -> pendingFiles.removeAll { it.id == id } },
|
||||||
|
|
@ -950,3 +1046,53 @@ private fun chatErrorText(e: Throwable, ctx: android.content.Context): String =
|
||||||
e is ChatHttpException -> ctx.getString(R.string.error_server, e.code)
|
e is ChatHttpException -> ctx.getString(R.string.error_server, e.code)
|
||||||
else -> ctx.getString(R.string.error_no_connection)
|
else -> ctx.getString(R.string.error_no_connection)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@Suppress("DEPRECATION")
|
||||||
|
private fun startRecording(
|
||||||
|
context: android.content.Context,
|
||||||
|
recorderRef: androidx.compose.runtime.MutableState<MediaRecorder?>,
|
||||||
|
audioFileRef: androidx.compose.runtime.MutableState<File?>,
|
||||||
|
onStarted: () -> Unit,
|
||||||
|
) {
|
||||||
|
try {
|
||||||
|
val file = File(context.cacheDir, "voice_${System.currentTimeMillis()}.m4a")
|
||||||
|
val recorder = MediaRecorder(context).apply {
|
||||||
|
setAudioSource(MediaRecorder.AudioSource.MIC)
|
||||||
|
setOutputFormat(MediaRecorder.OutputFormat.MPEG_4)
|
||||||
|
setAudioEncoder(MediaRecorder.AudioEncoder.AAC)
|
||||||
|
setAudioSamplingRate(44100)
|
||||||
|
setAudioEncodingBitRate(128000)
|
||||||
|
setOutputFile(file.absolutePath)
|
||||||
|
prepare()
|
||||||
|
start()
|
||||||
|
}
|
||||||
|
recorderRef.value = recorder
|
||||||
|
audioFileRef.value = file
|
||||||
|
onStarted()
|
||||||
|
} catch (_: Exception) {
|
||||||
|
recorderRef.value = null
|
||||||
|
audioFileRef.value = null
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private fun stopRecording(
|
||||||
|
recorderRef: androidx.compose.runtime.MutableState<MediaRecorder?>,
|
||||||
|
audioFileRef: androidx.compose.runtime.MutableState<File?>,
|
||||||
|
onRecorded: (String, String, ByteArray) -> Unit,
|
||||||
|
) {
|
||||||
|
val recorder = recorderRef.value ?: return
|
||||||
|
val file = audioFileRef.value ?: return
|
||||||
|
try {
|
||||||
|
recorder.stop()
|
||||||
|
recorder.release()
|
||||||
|
} catch (_: Exception) {
|
||||||
|
recorder.release()
|
||||||
|
}
|
||||||
|
recorderRef.value = null
|
||||||
|
audioFileRef.value = null
|
||||||
|
if (file.exists() && file.length() > 0) {
|
||||||
|
val bytes = file.readBytes()
|
||||||
|
file.delete()
|
||||||
|
onRecorded("voice_${System.currentTimeMillis()}.m4a", "audio/mp4", bytes)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
|
||||||
|
|
@ -8,5 +8,6 @@ import dev.kaizen.app.db.MessageRepository
|
||||||
class ChatViewModel(context: Context) {
|
class ChatViewModel(context: Context) {
|
||||||
private val db = KaizenDatabase.get(context)
|
private val db = KaizenDatabase.get(context)
|
||||||
val conversationRepo = ConversationRepository(db.conversationDao())
|
val conversationRepo = ConversationRepository(db.conversationDao())
|
||||||
val messageRepo = MessageRepository(db.messageDao())
|
val messageDao = db.messageDao()
|
||||||
|
val messageRepo = MessageRepository(messageDao)
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -36,4 +36,6 @@ data class MessageEntity(
|
||||||
val sortOrder: Int,
|
val sortOrder: Int,
|
||||||
val reasoning: String? = null,
|
val reasoning: String? = null,
|
||||||
val sources: List<SearchSource> = emptyList(),
|
val sources: List<SearchSource> = emptyList(),
|
||||||
|
val ttsUrl: String? = null,
|
||||||
|
val ttsVoice: String? = null,
|
||||||
)
|
)
|
||||||
|
|
|
||||||
|
|
@ -8,7 +8,7 @@ import androidx.room.TypeConverters
|
||||||
|
|
||||||
@Database(
|
@Database(
|
||||||
entities = [ConversationEntity::class, MessageEntity::class],
|
entities = [ConversationEntity::class, MessageEntity::class],
|
||||||
version = 2,
|
version = 3,
|
||||||
exportSchema = false,
|
exportSchema = false,
|
||||||
)
|
)
|
||||||
@TypeConverters(Converters::class)
|
@TypeConverters(Converters::class)
|
||||||
|
|
|
||||||
|
|
@ -21,4 +21,10 @@ interface MessageDao {
|
||||||
|
|
||||||
@Query("SELECT DISTINCT conversationId FROM messages")
|
@Query("SELECT DISTINCT conversationId FROM messages")
|
||||||
suspend fun cachedConversationIds(): List<String>
|
suspend fun cachedConversationIds(): List<String>
|
||||||
|
|
||||||
|
@Query("UPDATE messages SET ttsUrl = :url, ttsVoice = :voice WHERE id = :messageId")
|
||||||
|
suspend fun updateTts(messageId: String, url: String, voice: String)
|
||||||
|
|
||||||
|
@Query("SELECT ttsUrl FROM messages WHERE id = :messageId AND ttsVoice = :voice")
|
||||||
|
suspend fun getTtsUrl(messageId: String, voice: String): String?
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -27,6 +27,7 @@ import java.util.concurrent.TimeUnit
|
||||||
val name: String? = null,
|
val name: String? = null,
|
||||||
val email: String? = null,
|
val email: String? = null,
|
||||||
val defaultModel: String? = null,
|
val defaultModel: String? = null,
|
||||||
|
val ttsVoice: String? = null,
|
||||||
)
|
)
|
||||||
@Serializable data class WireMessage(val role: String, val content: String)
|
@Serializable data class WireMessage(val role: String, val content: String)
|
||||||
@Serializable private data class ChatRequest(
|
@Serializable private data class ChatRequest(
|
||||||
|
|
@ -572,6 +573,35 @@ object KaizenApi {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@Serializable private data class SpeechRequest(val prompt: String, val kind: String = "speech", val voice: String? = null)
|
||||||
|
@Serializable private data class SpeechResponse(val url: String = "", val cost: Double? = null)
|
||||||
|
|
||||||
|
/** POST /api/v1/audio-gen — generate speech from text via Gemini-TTS. Returns the audio URL or null. */
|
||||||
|
suspend fun generateSpeech(baseUrl: String, token: String, text: String, voice: String? = null): String? =
|
||||||
|
withContext(Dispatchers.IO) {
|
||||||
|
val body = json.encodeToString(SpeechRequest(
|
||||||
|
prompt = text.take(2000),
|
||||||
|
voice = voice,
|
||||||
|
)).toRequestBody(jsonMedia)
|
||||||
|
val req = Request.Builder()
|
||||||
|
.url("$baseUrl/api/v1/audio-gen")
|
||||||
|
.post(body)
|
||||||
|
.header("Authorization", "Bearer $token")
|
||||||
|
.build()
|
||||||
|
try {
|
||||||
|
client.newCall(req).execute().use { resp ->
|
||||||
|
if (!resp.isSuccessful) {
|
||||||
|
Log.w(TAG, "generateSpeech: HTTP ${resp.code}")
|
||||||
|
return@use null
|
||||||
|
}
|
||||||
|
json.decodeFromString<SpeechResponse>(resp.body!!.string()).url.takeIf { it.isNotBlank() }
|
||||||
|
}
|
||||||
|
} catch (e: Exception) {
|
||||||
|
Log.w(TAG, "generateSpeech: ${e.javaClass.simpleName}: ${e.message}")
|
||||||
|
null
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
suspend fun prewarm(baseUrl: String) = withContext(Dispatchers.IO) {
|
suspend fun prewarm(baseUrl: String) = withContext(Dispatchers.IO) {
|
||||||
try {
|
try {
|
||||||
val req = Request.Builder().url("$baseUrl/api/v1/meta").build()
|
val req = Request.Builder().url("$baseUrl/api/v1/meta").build()
|
||||||
|
|
|
||||||
|
|
@ -91,6 +91,9 @@ object KaizenIcons {
|
||||||
val ArrowDown: ImageVector get() = Lucide.ArrowDown
|
val ArrowDown: ImageVector get() = Lucide.ArrowDown
|
||||||
val ArrowUp: ImageVector get() = Lucide.ArrowUp
|
val ArrowUp: ImageVector get() = Lucide.ArrowUp
|
||||||
|
|
||||||
|
// ── Audio ──────────────────────────────────────────────────────────────
|
||||||
|
val Volume2: ImageVector get() = Lucide.Volume2
|
||||||
|
|
||||||
// ── Custom (brand overrides) ────────────────────────────────────────────
|
// ── Custom (brand overrides) ────────────────────────────────────────────
|
||||||
// Add hand-drawn SVGs here as ImageVector literals when ready.
|
// Add hand-drawn SVGs here as ImageVector literals when ready.
|
||||||
// Example:
|
// Example:
|
||||||
|
|
|
||||||
|
|
@ -19,6 +19,11 @@
|
||||||
<string name="chat_error">Error</string>
|
<string name="chat_error">Error</string>
|
||||||
<string name="chat_open_menu">Open menu</string>
|
<string name="chat_open_menu">Open menu</string>
|
||||||
<string name="chat_stop">Stop response</string>
|
<string name="chat_stop">Stop response</string>
|
||||||
|
<string name="chat_recording">Recording …</string>
|
||||||
|
<string name="chat_mic_permission">Microphone access required</string>
|
||||||
|
<string name="chat_read_aloud">Read aloud</string>
|
||||||
|
<string name="chat_tts_error">Read aloud failed</string>
|
||||||
|
<string name="chat_tts_unavailable">Read aloud not available (Vertex required)</string>
|
||||||
<string name="attach_camera">Take Photo</string>
|
<string name="attach_camera">Take Photo</string>
|
||||||
<string name="attach_gallery">Photo & Video</string>
|
<string name="attach_gallery">Photo & Video</string>
|
||||||
<string name="attach_file">Choose File</string>
|
<string name="attach_file">Choose File</string>
|
||||||
|
|
|
||||||
|
|
@ -21,6 +21,11 @@
|
||||||
<string name="chat_error">Fehler</string>
|
<string name="chat_error">Fehler</string>
|
||||||
<string name="chat_open_menu">Menü öffnen</string>
|
<string name="chat_open_menu">Menü öffnen</string>
|
||||||
<string name="chat_stop">Antwort stoppen</string>
|
<string name="chat_stop">Antwort stoppen</string>
|
||||||
|
<string name="chat_recording">Aufnahme läuft …</string>
|
||||||
|
<string name="chat_mic_permission">Mikrofonzugriff wird benötigt</string>
|
||||||
|
<string name="chat_read_aloud">Vorlesen</string>
|
||||||
|
<string name="chat_tts_error">Vorlesen fehlgeschlagen</string>
|
||||||
|
<string name="chat_tts_unavailable">Vorlesen nicht verfügbar (Vertex benötigt)</string>
|
||||||
<string name="attach_camera">Foto aufnehmen</string>
|
<string name="attach_camera">Foto aufnehmen</string>
|
||||||
<string name="attach_gallery">Foto & Video</string>
|
<string name="attach_gallery">Foto & Video</string>
|
||||||
<string name="attach_file">Datei auswählen</string>
|
<string name="attach_file">Datei auswählen</string>
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue