krypto/kaizen-app
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs: comprehensive CLAUDE.md update for 2026-06-23 session

Browse source 
- Document voice recording, TTS read-aloud, TTS cache, server-synced voice
- Add /api/v1/audio-gen and PATCH /api/v1/me to API contract
- Mark 5 Voice & Call Mode items as done (recording, TTS, cache, sync, backend)
- Add bugs-fixed section for 06-23 (padding, locale, shadow, migration)
- Update backend section with audio-gen, tts_voice, me PATCH
This commit is contained in:
Bruno Deanoz 2026-06-23 00:47:01 +02:00
parent 1645be8a97
commit 2f9ada646c
1 changed files with 23 additions and 5 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

28
CLAUDE.md
View file

@ -26,9 +26,10 @@ Make the **native Android app** (`kaizen-app`) a valid daily-driver that talks t
- **Sentinels** (`lib/chat/constants.ts`): `\x00` USAGE, `\x01` REASONING, `\x02` SOURCES, `\x03` TOOL, `\x04` QUERY. App currently strips all sentinels and renders only visible content. - **Sentinels** (`lib/chat/constants.ts`): `\x00` USAGE, `\x01` REASONING, `\x02` SOURCES, `\x03` TOOL, `\x04` QUERY. App currently strips all sentinels and renders only visible content.
- **Models:** `GET {baseUrl}/api/v1/models` → `{ models: [{ id, name?, provider? }] }`. - **Models:** `GET {baseUrl}/api/v1/models` → `{ models: [{ id, name?, provider? }] }`.
- **Conversations:** `GET /api/v1/conversations` → `{ conversations, hasMore }`. `POST /api/v1/conversations` → create. `GET /api/v1/conversations/[id]` → `{ messages }`. `POST /api/v1/conversations/[id]/messages` → persist. `POST /api/v1/conversations/[id]/title` → auto-title. - **Conversations:** `GET /api/v1/conversations` → `{ conversations, hasMore }`. `POST /api/v1/conversations` → create. `GET /api/v1/conversations/[id]` → `{ messages }`. `POST /api/v1/conversations/[id]/messages` → persist. `POST /api/v1/conversations/[id]/title` → auto-title.
- **Identity:** `GET /api/v1/me` → `{ id, email, name, role, defaultModel }`. - **Identity:** `GET /api/v1/me` → `{ id, email, name, role, defaultModel, ttsVoice }`. `PATCH /api/v1/me` with `{ ttsVoice }` to update preferences.
- **Capability handshake:** `GET /api/v1/meta` → `{ app, apiVersion, features }` (unauth). - **Capability handshake:** `GET /api/v1/meta` → `{ app, apiVersion, features }` (unauth).
- **Upload:** `POST /api/v1/upload` — multipart file → `{ url, kind, name, mimeType, size, fileId }`. - **Upload:** `POST /api/v1/upload` — multipart file → `{ url, kind, name, mimeType, size, fileId }`.
- **Audio gen:** `POST /api/v1/audio-gen` — `{ prompt, kind: "speech"|"music", voice? }` → `{ url, cost }`. Gemini-TTS (speech) or Lyria (music) via Vertex. Audio persisted in Object Storage.
## App architecture ## App architecture
@ -195,13 +196,16 @@ All backend work for the app is done. `pnpm test:run` green (774+), `tsc` + `lin
- **`api_tokens` table** + SHA-256 hash storage (never raw). Migration `0024`. - **`api_tokens` table** + SHA-256 hash storage (never raw). Migration `0024`.
- **Bearer auth path** (`getTokenSession` / `requireActiveUserOrToken` in `lib/auth/guard.ts`). - **Bearer auth path** (`getTokenSession` / `requireActiveUserOrToken` in `lib/auth/guard.ts`).
- **Token mint endpoint** `POST /api/v1/auth/token` — unauthenticated, Argon2id verify, anti-enumeration, rate-limited. - **Token mint endpoint** `POST /api/v1/auth/token` — unauthenticated, Argon2id verify, anti-enumeration, rate-limited.
- **`GET /api/v1/me`** — identity + `defaultModel` for native clients. - **`GET /api/v1/me`** — identity + `defaultModel` + `ttsVoice` for native clients.
- **`PATCH /api/v1/me`** — update user preferences (`ttsVoice`, etc.). Bearer-authed.
- **`GET /api/v1/models`** — Bearer-authed, returns filtered model catalog. - **`GET /api/v1/models`** — Bearer-authed, returns filtered model catalog.
- **`/api/v1/conversations`** — thin re-exports of canonical handlers, Bearer-authed. - **`/api/v1/conversations`** — thin re-exports of canonical handlers, Bearer-authed.
- **`POST /api/v1/auth/unlock`** — Bearer-authed, returns unlock token in response body (not cookie) for native clients. Rate-limited 10/5min per user. - **`POST /api/v1/auth/unlock`** — Bearer-authed, returns unlock token in response body (not cookie) for native clients. Rate-limited 10/5min per user.
- **`GET /api/v1/conversations/[id]`** — accepts `X-Unlock-Token` header alongside the existing `kaizen_unlock` cookie for locked conversations. - **`GET /api/v1/conversations/[id]`** — accepts `X-Unlock-Token` header alongside the existing `kaizen_unlock` cookie for locked conversations.
- **`GET /api/v1/meta`** — capability handshake. - **`GET /api/v1/meta`** — capability handshake.
- **App-devices card** (`components/settings/app-tokens-card.tsx`) — see/revoke signed-in devices on `/settings/security`. - **App-devices card** (`components/settings/app-tokens-card.tsx`) — see/revoke signed-in devices on `/settings/security`.
- **`POST /api/v1/audio-gen`** — Gemini-TTS speech generation. Bearer-authed (re-export of `/api/audio-gen`). `{ prompt, kind: "speech", voice? }` → `{ url, cost }`.
- **`tts_voice` column** on `users` table (migration 0030). Server-synced voice preference, read via `/api/v1/me`, written via `PATCH /api/v1/me`.
- **API versioning** — `/api/v1/` for outward contract, additive-only within v1. Web-internal endpoints stay unversioned. - **API versioning** — `/api/v1/` for outward contract, additive-only within v1. Web-internal endpoints stay unversioned.
### App — shipped features (on-device verified on S25 Ultra) ### App — shipped features (on-device verified on S25 Ultra)
@ -260,6 +264,10 @@ All committed on `main`. Compile-verified on Mac (`./gradlew :app:compileDebugKo
| `fd0de5c` | **Markdown upgrade** — tables (`\| col \| col \|`), ~~strikethrough~~, task lists (`- [x]`), clickable links (ACTION_VIEW), block comments (`/* */`), HTML comments, template strings, C/C++/Ruby/PHP/Dockerfile/TOML highlighting. All regex pre-compiled | | `fd0de5c` | **Markdown upgrade** — tables (`\| col \| col \|`), ~~strikethrough~~, task lists (`- [x]`), clickable links (ACTION_VIEW), block comments (`/* */`), HTML comments, template strings, C/C++/Ruby/PHP/Dockerfile/TOML highlighting. All regex pre-compiled |
| `bf94414` | **CallButton** — styled to match SendButton: 38dp circle, subtle accent gradient (16-28% alpha), AudioLines icon in accent color, press-scale animation | | `bf94414` | **CallButton** — styled to match SendButton: 38dp circle, subtle accent gradient (16-28% alpha), AudioLines icon in accent color, press-scale animation |
| `141d4de` | **Stop button + auto-logout + locale fix + login shadow migration** — red square stop button during streaming (cancels stream, preserves partial content), auto-logout on HTTP 401, EmptyHero date respects system locale, LoginScreen shadows migrated to kaizenShadow | | `141d4de` | **Stop button + auto-logout + locale fix + login shadow migration** — red square stop button during streaming (cancels stream, preserves partial content), auto-logout on HTTP 401, EmptyHero date respects system locale, LoginScreen shadows migrated to kaizenShadow |
| `edd8cca` | **Bottom padding fix** — LazyColumn bottom contentPadding 152→220dp, fixes messages overlapping mode pills after ChatInput two-row redesign |
| `9e1f30d` | **Voice recording + TTS read-aloud** — Mic button wired to `MediaRecorder` (AAC/M4A), tap-to-record, auto-upload as audio attachment. Volume button on assistant messages → `POST /api/v1/audio-gen` → Gemini-TTS → `MediaPlayer` playback. `RECORD_AUDIO` permission |
| `c4884a3` | **TTS audio cache in Room** — `ttsUrl` + `ttsVoice` on `MessageEntity` (Room v3), skip API call on replay with same voice. Voice change invalidates cache |
| `1645be8` | **Server-synced TTS voice** — reads `ttsVoice` from `GET /api/v1/me`, uses it for all TTS calls. Voice set in web settings applies in app automatically |
**Earlier UI/feel work (Phase 0 prototype → feel-first):** **Earlier UI/feel work (Phase 0 prototype → feel-first):**
- Liquid glass styling, floating panels, glassmorphic sidebar - Liquid glass styling, floating panels, glassmorphic sidebar
@ -298,6 +306,13 @@ All committed on `main`. Compile-verified on Mac (`./gradlew :app:compileDebugKo
- **Sidebar flicker on refresh** — `replaceAll()` did `DELETE ALL` + `INSERT`, causing Room Flow to emit empty list briefly. Fix: `upsert` + `deleteExcept` (no flash). - **Sidebar flicker on refresh** — `replaceAll()` did `DELETE ALL` + `INSERT`, causing Room Flow to emit empty list briefly. Fix: `upsert` + `deleteExcept` (no flash).
- **Gallery/file picker single-select only** — `GetContent()` allowed only one item. Fix: gallery → `PickMultipleVisualMedia(ImageAndVideo)`, file → `OpenMultipleDocuments(["*/*"])`. - **Gallery/file picker single-select only** — `GetContent()` allowed only one item. Fix: gallery → `PickMultipleVisualMedia(ImageAndVideo)`, file → `OpenMultipleDocuments(["*/*"])`.
### Bugs fixed (2026-06-23 — voice + UX session)
- **Messages overlapping mode pills** — LazyColumn bottom `contentPadding` was 152dp but the two-row ChatInput redesign made the dock ~210dp tall. Fix: increased to 220dp.
- **EmptyHero date hardcoded German** — `Locale.GERMAN` → `Locale.getDefault()`, now respects system/app language.
- **LoginScreen used `Modifier.shadow`** — violated design system pillar 4 (multi-layer shadows). Fix: migrated to `kaizenShadow` (level1 for fields, level2 for button).
- **Migration 0030 crash** — `pnpm db:generate` bundled the entire org schema (0026-0029) + `tts_voice` into one migration, causing duplicate CREATE TABLE / ADD COLUMN failures. Fix: reduced 0030 to only `ALTER TABLE users ADD COLUMN IF NOT EXISTS tts_voice text`.
## What's NOT done yet (app, priority order) ## What's NOT done yet (app, priority order)
### 1. Remaining visual/UX gaps (daily-drive blockers) ### 1. Remaining visual/UX gaps (daily-drive blockers)
@ -341,10 +356,13 @@ These are what separate "prototype" from "daily-driver":
### 5. Voice & Call Mode ### 5. Voice & Call Mode
**v1 — Pipeline (Weg A, uses existing backend):** **v1 — Pipeline (Weg A, uses existing backend):**
- [ ] **Voice recording + send** — Android `MediaRecorder`, upload via `/api/v1/upload`, send as audio attachment via `/api/v1/chat` (Vertex understands audio natively) - [x] **Voice recording + send** — Android `MediaRecorder` (AAC/M4A, 44.1kHz/128kbps), tap-to-record mic button (red stop-square while recording), auto-upload via `/api/v1/upload`, sent as audio attachment via `/api/v1/chat`. Vertex understands audio natively; OpenRouter passes it as `input_audio`. Persisted in DB as normal attachment
- [ ] **TTS playback** — POST response text to `/api/v1/audio-gen` (needs Bearer auth, currently web-session-only via `requireActiveUser`), play returned audio URL via `MediaPlayer`/`ExoPlayer` - [x] **TTS read-aloud** — Volume button on every assistant message. `POST /api/v1/audio-gen { prompt, kind: "speech", voice }` → Gemini-TTS → `MediaPlayer` streams the audio URL. Tap again to stop. Accent-colored icon during playback
- [x] **TTS cache** — `ttsUrl` + `ttsVoice` fields on `MessageEntity` (Room v3). First play generates + caches; subsequent plays with same voice skip the API call. Voice change auto-invalidates
- [x] **Server-synced voice** — `tts_voice` column on `users` table (migration 0030). Read from `GET /api/v1/me`, written via `PATCH /api/v1/me`. Web's `setCallVoice()` syncs to server. App reads on login/resume. Default "Kore"
- [x] **Backend: Bearer-auth audio-gen** — `/api/audio-gen` switched to `requireActiveUserOrToken()`, v1 re-export at `/api/v1/audio-gen`
- [ ] **Call UI** — fullscreen overlay, KaizenOrb with amplitude-reactive animation, push-to-talk mic button, status machine (idle → recording → uploading → thinking → speaking), voice picker (30 Gemini-TTS voices) - [ ] **Call UI** — fullscreen overlay, KaizenOrb with amplitude-reactive animation, push-to-talk mic button, status machine (idle → recording → uploading → thinking → speaking), voice picker (30 Gemini-TTS voices)
- [ ] **Backend: Bearer-auth audio-gen** — switch `/api/audio-gen` from `requireActiveUser()` to `requireActiveUserOrToken()`, add v1 re-export at `/api/v1/audio-gen` - [ ] **In-app voice picker** — settings UI to choose from 30 Gemini-TTS voices (synced to server via `PATCH /api/v1/me`)
The web's call mode (`use-call-mode.ts`, `call-view.tsx`) is the reference implementation. Same pipeline: Record → Upload → Chat (with audio attachment) → Text stream → TTS → Playback. No WebSocket needed. The web's call mode (`use-call-mode.ts`, `call-view.tsx`) is the reference implementation. Same pipeline: Record → Upload → Chat (with audio attachment) → Text stream → TTS → Playback. No WebSocket needed.