docs: add Call Mode roadmap (Weg A pipeline + Weg B Live API), image/media gen section
- Mark auto-logout on 401 as done - Remove WebSocket text-streaming TODO (not needed) - Add Voice & Call Mode section with v1 (pipeline) and v2 (Gemini Live API via Vertex/GCP) - Add Image & media generation section - Document stop button, locale fix, login shadow migration in shipped features
This commit is contained in:
parent
141d4de227
commit
3c6dab33a0
1 changed files with 23 additions and 4 deletions
27
CLAUDE.md
27
CLAUDE.md
|
|
@ -259,6 +259,7 @@ All committed on `main`. Compile-verified on Mac (`./gradlew :app:compileDebugKo
|
|||
| `70aa5c5` | **ChatInput button layout** — Mic (voice) + Plus left, Call/Send toggle right. AudioLines icon for call. SendButton: no border, translucent gradient, 38dp |
|
||||
| `fd0de5c` | **Markdown upgrade** — tables (`\| col \| col \|`), ~~strikethrough~~, task lists (`- [x]`), clickable links (ACTION_VIEW), block comments (`/* */`), HTML comments, template strings, C/C++/Ruby/PHP/Dockerfile/TOML highlighting. All regex pre-compiled |
|
||||
| `bf94414` | **CallButton** — styled to match SendButton: 38dp circle, subtle accent gradient (16-28% alpha), AudioLines icon in accent color, press-scale animation |
|
||||
| `141d4de` | **Stop button + auto-logout + locale fix + login shadow migration** — red square stop button during streaming (cancels stream, preserves partial content), auto-logout on HTTP 401, EmptyHero date respects system locale, LoginScreen shadows migrated to kaizenShadow |
|
||||
|
||||
**Earlier UI/feel work (Phase 0 prototype → feel-first):**
|
||||
- Liquid glass styling, floating panels, glassmorphic sidebar
|
||||
|
|
@ -319,7 +320,7 @@ These are what separate "prototype" from "daily-driver":
|
|||
|
||||
### 3. Security hardening
|
||||
|
||||
- [ ] **Auto-logout on 401** — if the token is rejected, route to login screen instead of showing error banner
|
||||
- [x] **Auto-logout on 401** — if the token is rejected, route to login screen instead of showing error banner
|
||||
- [ ] **Server version check** — call `/api/v1/meta` after login to detect server skew and warn cleanly
|
||||
- [x] **Biometric lock** (`BiometricPrompt` + KeyStore/CryptoObject) — hardware-enforced, password fallback dialog when unavailable, biometric toggle in Settings, all unlock paths gated
|
||||
- [x] **Security settings** — biometric toggle, signed-in devices with remote revoke, password change
|
||||
|
|
@ -334,17 +335,35 @@ These are what separate "prototype" from "daily-driver":
|
|||
- [x] **Web search sources** — `SourcesBlock` (collapsible, clickable links via ACTION_VIEW), parses `\x02`/`\x04` sentinels. `webSearch: true` in ChatRequest triggers agentive search
|
||||
- [x] **Tool call display** — `ToolEventsBlock`, parses `\x03` sentinel, status icons per step
|
||||
- [ ] **Native passkey login** (WebAuthn/FIDO2 on Android)
|
||||
- [ ] **WebSocket streaming** (replace chunked HTTP for lower latency on mobile)
|
||||
- [ ] **Usage/cost display**
|
||||
- [ ] **User context / KI-Profil**
|
||||
|
||||
### 5. App distribution
|
||||
### 5. Voice & Call Mode
|
||||
|
||||
**v1 — Pipeline (Weg A, uses existing backend):**
|
||||
- [ ] **Voice recording + send** — Android `MediaRecorder`, upload via `/api/v1/upload`, send as audio attachment via `/api/v1/chat` (Vertex understands audio natively)
|
||||
- [ ] **TTS playback** — POST response text to `/api/v1/audio-gen` (needs Bearer auth, currently web-session-only via `requireActiveUser`), play returned audio URL via `MediaPlayer`/`ExoPlayer`
|
||||
- [ ] **Call UI** — fullscreen overlay, KaizenOrb with amplitude-reactive animation, push-to-talk mic button, status machine (idle → recording → uploading → thinking → speaking), voice picker (30 Gemini-TTS voices)
|
||||
- [ ] **Backend: Bearer-auth audio-gen** — switch `/api/audio-gen` from `requireActiveUser()` to `requireActiveUserOrToken()`, add v1 re-export at `/api/v1/audio-gen`
|
||||
|
||||
The web's call mode (`use-call-mode.ts`, `call-view.tsx`) is the reference implementation. Same pipeline: Record → Upload → Chat (with audio attachment) → Text stream → TTS → Playback. No WebSocket needed.
|
||||
|
||||
**v2 — Gemini Live API (Weg B, target architecture):**
|
||||
- [ ] **Bidirectional WebSocket audio streaming** via Vertex AI Gemini Live API (`BidiGenerateContent`). Available on Vertex AI / enterprise Gemini (GCP), not just the consumer API. True real-time conversation: sub-second latency, server-side VAD, user can interrupt the model mid-response. Requires either proxying through the backend (WebSocket relay) or direct app→Vertex connection with credential delegation. History sync and cost tracking need a separate solution since the conversation bypasses the normal chat persistence pipeline.
|
||||
|
||||
### 6. Image & media generation
|
||||
|
||||
- [ ] **Image generation** — send prompt to `/api/image-gen` (needs Bearer auth + v1 re-export), display returned image URL inline. Web uses `detectMode("image")` + dedicated `imageModelId` picker; app can start simpler (Bild mode pill → image-gen endpoint)
|
||||
- [ ] **Video generation** — `/api/video-gen` (SSE polling, max 4.5 min), display result inline
|
||||
- [ ] **Audio generation** — music (Lyria) + speech (Gemini-TTS) via `/api/audio-gen`, inline `<audio>` player
|
||||
|
||||
### 7. App distribution
|
||||
|
||||
- [ ] **Push notifications** (FCM)
|
||||
- [ ] **Play Store / F-Droid listing**
|
||||
- [ ] **Auto-update mechanism**
|
||||
|
||||
### 6. Offline-cache enhancements (future iterations, see `OFFLINE_CACHE_ADD.md` §11)
|
||||
### 8. Offline-cache enhancements (future iterations, see `OFFLINE_CACHE_ADD.md` §11)
|
||||
|
||||
- [ ] **Stale-check enforcement** — use `cachedAt` to trigger background refresh when older than N minutes
|
||||
- [ ] **Offline-queue for sends** — queue messages locally, send when network returns (needs conflict resolution)
|
||||
|
|
|
|||
Loading…
Reference in a new issue