diff --git a/CLAUDE.md b/CLAUDE.md index f1d2110..13c5cce 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -26,9 +26,10 @@ Make the **native Android app** (`kaizen-app`) a valid daily-driver that talks t - **Sentinels** (`lib/chat/constants.ts`): `\x00` USAGE, `\x01` REASONING, `\x02` SOURCES, `\x03` TOOL, `\x04` QUERY. App currently strips all sentinels and renders only visible content. - **Models:** `GET {baseUrl}/api/v1/models` → `{ models: [{ id, name?, provider? }] }`. - **Conversations:** `GET /api/v1/conversations` → `{ conversations, hasMore }`. `POST /api/v1/conversations` → create. `GET /api/v1/conversations/[id]` → `{ messages }`. `POST /api/v1/conversations/[id]/messages` → persist. `POST /api/v1/conversations/[id]/title` → auto-title. -- **Identity:** `GET /api/v1/me` → `{ id, email, name, role, defaultModel }`. +- **Identity:** `GET /api/v1/me` → `{ id, email, name, role, defaultModel, ttsVoice }`. `PATCH /api/v1/me` with `{ ttsVoice }` to update preferences. - **Capability handshake:** `GET /api/v1/meta` → `{ app, apiVersion, features }` (unauth). - **Upload:** `POST /api/v1/upload` — multipart file → `{ url, kind, name, mimeType, size, fileId }`. +- **Audio gen:** `POST /api/v1/audio-gen` — `{ prompt, kind: "speech"|"music", voice? }` → `{ url, cost }`. Gemini-TTS (speech) or Lyria (music) via Vertex. Audio persisted in Object Storage. ## App architecture @@ -195,13 +196,16 @@ All backend work for the app is done. `pnpm test:run` green (774+), `tsc` + `lin - **`api_tokens` table** + SHA-256 hash storage (never raw). Migration `0024`. - **Bearer auth path** (`getTokenSession` / `requireActiveUserOrToken` in `lib/auth/guard.ts`). - **Token mint endpoint** `POST /api/v1/auth/token` — unauthenticated, Argon2id verify, anti-enumeration, rate-limited. -- **`GET /api/v1/me`** — identity + `defaultModel` for native clients. +- **`GET /api/v1/me`** — identity + `defaultModel` + `ttsVoice` for native clients. +- **`PATCH /api/v1/me`** — update user preferences (`ttsVoice`, etc.). Bearer-authed. - **`GET /api/v1/models`** — Bearer-authed, returns filtered model catalog. - **`/api/v1/conversations`** — thin re-exports of canonical handlers, Bearer-authed. - **`POST /api/v1/auth/unlock`** — Bearer-authed, returns unlock token in response body (not cookie) for native clients. Rate-limited 10/5min per user. - **`GET /api/v1/conversations/[id]`** — accepts `X-Unlock-Token` header alongside the existing `kaizen_unlock` cookie for locked conversations. - **`GET /api/v1/meta`** — capability handshake. - **App-devices card** (`components/settings/app-tokens-card.tsx`) — see/revoke signed-in devices on `/settings/security`. +- **`POST /api/v1/audio-gen`** — Gemini-TTS speech generation. Bearer-authed (re-export of `/api/audio-gen`). `{ prompt, kind: "speech", voice? }` → `{ url, cost }`. +- **`tts_voice` column** on `users` table (migration 0030). Server-synced voice preference, read via `/api/v1/me`, written via `PATCH /api/v1/me`. - **API versioning** — `/api/v1/` for outward contract, additive-only within v1. Web-internal endpoints stay unversioned. ### App — shipped features (on-device verified on S25 Ultra) @@ -260,6 +264,10 @@ All committed on `main`. Compile-verified on Mac (`./gradlew :app:compileDebugKo | `fd0de5c` | **Markdown upgrade** — tables (`\| col \| col \|`), ~~strikethrough~~, task lists (`- [x]`), clickable links (ACTION_VIEW), block comments (`/* */`), HTML comments, template strings, C/C++/Ruby/PHP/Dockerfile/TOML highlighting. All regex pre-compiled | | `bf94414` | **CallButton** — styled to match SendButton: 38dp circle, subtle accent gradient (16-28% alpha), AudioLines icon in accent color, press-scale animation | | `141d4de` | **Stop button + auto-logout + locale fix + login shadow migration** — red square stop button during streaming (cancels stream, preserves partial content), auto-logout on HTTP 401, EmptyHero date respects system locale, LoginScreen shadows migrated to kaizenShadow | +| `edd8cca` | **Bottom padding fix** — LazyColumn bottom contentPadding 152→220dp, fixes messages overlapping mode pills after ChatInput two-row redesign | +| `9e1f30d` | **Voice recording + TTS read-aloud** — Mic button wired to `MediaRecorder` (AAC/M4A), tap-to-record, auto-upload as audio attachment. Volume button on assistant messages → `POST /api/v1/audio-gen` → Gemini-TTS → `MediaPlayer` playback. `RECORD_AUDIO` permission | +| `c4884a3` | **TTS audio cache in Room** — `ttsUrl` + `ttsVoice` on `MessageEntity` (Room v3), skip API call on replay with same voice. Voice change invalidates cache | +| `1645be8` | **Server-synced TTS voice** — reads `ttsVoice` from `GET /api/v1/me`, uses it for all TTS calls. Voice set in web settings applies in app automatically | **Earlier UI/feel work (Phase 0 prototype → feel-first):** - Liquid glass styling, floating panels, glassmorphic sidebar @@ -298,6 +306,13 @@ All committed on `main`. Compile-verified on Mac (`./gradlew :app:compileDebugKo - **Sidebar flicker on refresh** — `replaceAll()` did `DELETE ALL` + `INSERT`, causing Room Flow to emit empty list briefly. Fix: `upsert` + `deleteExcept` (no flash). - **Gallery/file picker single-select only** — `GetContent()` allowed only one item. Fix: gallery → `PickMultipleVisualMedia(ImageAndVideo)`, file → `OpenMultipleDocuments(["*/*"])`. +### Bugs fixed (2026-06-23 — voice + UX session) + +- **Messages overlapping mode pills** — LazyColumn bottom `contentPadding` was 152dp but the two-row ChatInput redesign made the dock ~210dp tall. Fix: increased to 220dp. +- **EmptyHero date hardcoded German** — `Locale.GERMAN` → `Locale.getDefault()`, now respects system/app language. +- **LoginScreen used `Modifier.shadow`** — violated design system pillar 4 (multi-layer shadows). Fix: migrated to `kaizenShadow` (level1 for fields, level2 for button). +- **Migration 0030 crash** — `pnpm db:generate` bundled the entire org schema (0026-0029) + `tts_voice` into one migration, causing duplicate CREATE TABLE / ADD COLUMN failures. Fix: reduced 0030 to only `ALTER TABLE users ADD COLUMN IF NOT EXISTS tts_voice text`. + ## What's NOT done yet (app, priority order) ### 1. Remaining visual/UX gaps (daily-drive blockers) @@ -341,10 +356,13 @@ These are what separate "prototype" from "daily-driver": ### 5. Voice & Call Mode **v1 — Pipeline (Weg A, uses existing backend):** -- [ ] **Voice recording + send** — Android `MediaRecorder`, upload via `/api/v1/upload`, send as audio attachment via `/api/v1/chat` (Vertex understands audio natively) -- [ ] **TTS playback** — POST response text to `/api/v1/audio-gen` (needs Bearer auth, currently web-session-only via `requireActiveUser`), play returned audio URL via `MediaPlayer`/`ExoPlayer` +- [x] **Voice recording + send** — Android `MediaRecorder` (AAC/M4A, 44.1kHz/128kbps), tap-to-record mic button (red stop-square while recording), auto-upload via `/api/v1/upload`, sent as audio attachment via `/api/v1/chat`. Vertex understands audio natively; OpenRouter passes it as `input_audio`. Persisted in DB as normal attachment +- [x] **TTS read-aloud** — Volume button on every assistant message. `POST /api/v1/audio-gen { prompt, kind: "speech", voice }` → Gemini-TTS → `MediaPlayer` streams the audio URL. Tap again to stop. Accent-colored icon during playback +- [x] **TTS cache** — `ttsUrl` + `ttsVoice` fields on `MessageEntity` (Room v3). First play generates + caches; subsequent plays with same voice skip the API call. Voice change auto-invalidates +- [x] **Server-synced voice** — `tts_voice` column on `users` table (migration 0030). Read from `GET /api/v1/me`, written via `PATCH /api/v1/me`. Web's `setCallVoice()` syncs to server. App reads on login/resume. Default "Kore" +- [x] **Backend: Bearer-auth audio-gen** — `/api/audio-gen` switched to `requireActiveUserOrToken()`, v1 re-export at `/api/v1/audio-gen` - [ ] **Call UI** — fullscreen overlay, KaizenOrb with amplitude-reactive animation, push-to-talk mic button, status machine (idle → recording → uploading → thinking → speaking), voice picker (30 Gemini-TTS voices) -- [ ] **Backend: Bearer-auth audio-gen** — switch `/api/audio-gen` from `requireActiveUser()` to `requireActiveUserOrToken()`, add v1 re-export at `/api/v1/audio-gen` +- [ ] **In-app voice picker** — settings UI to choose from 30 Gemini-TTS voices (synced to server via `PATCH /api/v1/me`) The web's call mode (`use-call-mode.ts`, `call-view.tsx`) is the reference implementation. Same pipeline: Record → Upload → Chat (with audio attachment) → Text stream → TTS → Playback. No WebSocket needed.