docs: add Call Mode roadmap (Weg A pipeline + Weg B Live API), image/media gen section

- Mark auto-logout on 401 as done
- Remove WebSocket text-streaming TODO (not needed)
- Add Voice & Call Mode section with v1 (pipeline) and v2 (Gemini Live API via Vertex/GCP)
- Add Image & media generation section
- Document stop button, locale fix, login shadow migration in shipped features
This commit is contained in:
Bruno Deanoz 2026-06-23 00:10:15 +02:00
parent 141d4de227
commit 3c6dab33a0

View file

@ -259,6 +259,7 @@ All committed on `main`. Compile-verified on Mac (`./gradlew :app:compileDebugKo
| `70aa5c5` | **ChatInput button layout** — Mic (voice) + Plus left, Call/Send toggle right. AudioLines icon for call. SendButton: no border, translucent gradient, 38dp | | `70aa5c5` | **ChatInput button layout** — Mic (voice) + Plus left, Call/Send toggle right. AudioLines icon for call. SendButton: no border, translucent gradient, 38dp |
| `fd0de5c` | **Markdown upgrade** — tables (`\| col \| col \|`), ~~strikethrough~~, task lists (`- [x]`), clickable links (ACTION_VIEW), block comments (`/* */`), HTML comments, template strings, C/C++/Ruby/PHP/Dockerfile/TOML highlighting. All regex pre-compiled | | `fd0de5c` | **Markdown upgrade** — tables (`\| col \| col \|`), ~~strikethrough~~, task lists (`- [x]`), clickable links (ACTION_VIEW), block comments (`/* */`), HTML comments, template strings, C/C++/Ruby/PHP/Dockerfile/TOML highlighting. All regex pre-compiled |
| `bf94414` | **CallButton** — styled to match SendButton: 38dp circle, subtle accent gradient (16-28% alpha), AudioLines icon in accent color, press-scale animation | | `bf94414` | **CallButton** — styled to match SendButton: 38dp circle, subtle accent gradient (16-28% alpha), AudioLines icon in accent color, press-scale animation |
| `141d4de` | **Stop button + auto-logout + locale fix + login shadow migration** — red square stop button during streaming (cancels stream, preserves partial content), auto-logout on HTTP 401, EmptyHero date respects system locale, LoginScreen shadows migrated to kaizenShadow |
**Earlier UI/feel work (Phase 0 prototype → feel-first):** **Earlier UI/feel work (Phase 0 prototype → feel-first):**
- Liquid glass styling, floating panels, glassmorphic sidebar - Liquid glass styling, floating panels, glassmorphic sidebar
@ -319,7 +320,7 @@ These are what separate "prototype" from "daily-driver":
### 3. Security hardening ### 3. Security hardening
- [ ] **Auto-logout on 401** — if the token is rejected, route to login screen instead of showing error banner - [x] **Auto-logout on 401** — if the token is rejected, route to login screen instead of showing error banner
- [ ] **Server version check** — call `/api/v1/meta` after login to detect server skew and warn cleanly - [ ] **Server version check** — call `/api/v1/meta` after login to detect server skew and warn cleanly
- [x] **Biometric lock** (`BiometricPrompt` + KeyStore/CryptoObject) — hardware-enforced, password fallback dialog when unavailable, biometric toggle in Settings, all unlock paths gated - [x] **Biometric lock** (`BiometricPrompt` + KeyStore/CryptoObject) — hardware-enforced, password fallback dialog when unavailable, biometric toggle in Settings, all unlock paths gated
- [x] **Security settings** — biometric toggle, signed-in devices with remote revoke, password change - [x] **Security settings** — biometric toggle, signed-in devices with remote revoke, password change
@ -334,17 +335,35 @@ These are what separate "prototype" from "daily-driver":
- [x] **Web search sources**`SourcesBlock` (collapsible, clickable links via ACTION_VIEW), parses `\x02`/`\x04` sentinels. `webSearch: true` in ChatRequest triggers agentive search - [x] **Web search sources**`SourcesBlock` (collapsible, clickable links via ACTION_VIEW), parses `\x02`/`\x04` sentinels. `webSearch: true` in ChatRequest triggers agentive search
- [x] **Tool call display**`ToolEventsBlock`, parses `\x03` sentinel, status icons per step - [x] **Tool call display**`ToolEventsBlock`, parses `\x03` sentinel, status icons per step
- [ ] **Native passkey login** (WebAuthn/FIDO2 on Android) - [ ] **Native passkey login** (WebAuthn/FIDO2 on Android)
- [ ] **WebSocket streaming** (replace chunked HTTP for lower latency on mobile)
- [ ] **Usage/cost display** - [ ] **Usage/cost display**
- [ ] **User context / KI-Profil** - [ ] **User context / KI-Profil**
### 5. App distribution ### 5. Voice & Call Mode
**v1 — Pipeline (Weg A, uses existing backend):**
- [ ] **Voice recording + send** — Android `MediaRecorder`, upload via `/api/v1/upload`, send as audio attachment via `/api/v1/chat` (Vertex understands audio natively)
- [ ] **TTS playback** — POST response text to `/api/v1/audio-gen` (needs Bearer auth, currently web-session-only via `requireActiveUser`), play returned audio URL via `MediaPlayer`/`ExoPlayer`
- [ ] **Call UI** — fullscreen overlay, KaizenOrb with amplitude-reactive animation, push-to-talk mic button, status machine (idle → recording → uploading → thinking → speaking), voice picker (30 Gemini-TTS voices)
- [ ] **Backend: Bearer-auth audio-gen** — switch `/api/audio-gen` from `requireActiveUser()` to `requireActiveUserOrToken()`, add v1 re-export at `/api/v1/audio-gen`
The web's call mode (`use-call-mode.ts`, `call-view.tsx`) is the reference implementation. Same pipeline: Record → Upload → Chat (with audio attachment) → Text stream → TTS → Playback. No WebSocket needed.
**v2 — Gemini Live API (Weg B, target architecture):**
- [ ] **Bidirectional WebSocket audio streaming** via Vertex AI Gemini Live API (`BidiGenerateContent`). Available on Vertex AI / enterprise Gemini (GCP), not just the consumer API. True real-time conversation: sub-second latency, server-side VAD, user can interrupt the model mid-response. Requires either proxying through the backend (WebSocket relay) or direct app→Vertex connection with credential delegation. History sync and cost tracking need a separate solution since the conversation bypasses the normal chat persistence pipeline.
### 6. Image & media generation
- [ ] **Image generation** — send prompt to `/api/image-gen` (needs Bearer auth + v1 re-export), display returned image URL inline. Web uses `detectMode("image")` + dedicated `imageModelId` picker; app can start simpler (Bild mode pill → image-gen endpoint)
- [ ] **Video generation**`/api/video-gen` (SSE polling, max 4.5 min), display result inline
- [ ] **Audio generation** — music (Lyria) + speech (Gemini-TTS) via `/api/audio-gen`, inline `<audio>` player
### 7. App distribution
- [ ] **Push notifications** (FCM) - [ ] **Push notifications** (FCM)
- [ ] **Play Store / F-Droid listing** - [ ] **Play Store / F-Droid listing**
- [ ] **Auto-update mechanism** - [ ] **Auto-update mechanism**
### 6. Offline-cache enhancements (future iterations, see `OFFLINE_CACHE_ADD.md` §11) ### 8. Offline-cache enhancements (future iterations, see `OFFLINE_CACHE_ADD.md` §11)
- [ ] **Stale-check enforcement** — use `cachedAt` to trigger background refresh when older than N minutes - [ ] **Stale-check enforcement** — use `cachedAt` to trigger background refresh when older than N minutes
- [ ] **Offline-queue for sends** — queue messages locally, send when network returns (needs conflict resolution) - [ ] **Offline-queue for sends** — queue messages locally, send when network returns (needs conflict resolution)