Compare commits

...

7 commits

Author SHA1 Message Date
Bruno Deanoz
da3eded81e Remove temporary bug screenshots + ignore bugs/ dir 2026-06-24 03:27:53 +02:00
Bruno Deanoz
ff66b80966 Wire Live Call end-to-end: turn persistence, model check, permission
Turn persistence: onTurnEnd handler creates conversation if needed,
adds user + assistant messages to UI list, caches in Room. Server
already persists via WS-Server — app only needs local state sync.

Model check: CallButton only triggers Live Call when a vertex:*live*
model is in the catalog (liveAvailable). Falls back to PTT otherwise.

Permission: RECORD_AUDIO runtime request via ActivityResultContracts
before starting the call. Denied → toast, no crash.

Title generation: endLiveCall triggers generateTitle + sidebar refresh,
same pattern as text chat.
2026-06-24 03:25:32 +02:00
Bruno Deanoz
31278f2440 Add Live Call UI overlay + wiring in ChatScreen
LiveCallOverlay — fullscreen composable: KaizenOrb (amplitude-reactive,
streaming/recording variants), duration timer, status text (connecting/
listening/speaking), animated subtitle (transcript/response), end call
button (red gradient circle with PhoneOff icon), interrupt on orb tap.

ChatScreen wiring: CallButton → startLiveCall() → binds LiveCallService
→ Foreground Service with AudioPipeline + LiveClient. Overlay renders
above chat content with fade animation. endLiveCall() unbinds + stops.

ChatInput: onCallClick callback propagated from CallButton to ChatScreen.

i18n: live_connecting/listening/ready/speaking/end/reconnecting in de+en.
KaizenIcons: added PhoneOff (Lucide).
2026-06-24 03:14:08 +02:00
Bruno Deanoz
65ea3a5656 Add Live Call transport layer (P10 foundation)
LiveProtocol — type-safe Kaizen WS protocol serialization/parsing,
mirrors lib/live/protocol.ts 1:1 (setup/resume/audio/text/interrupt/end
+ all 12 server event types).

LiveClient — OkHttp WebSocket client with auto-reconnect (exponential
backoff, max 5 attempts), resume protocol (sessionId + lastSeqId),
backpressure handling, terminal error detection. 15s ping interval
keeps mobile NAT alive.

AudioPipeline — bidirectional audio: AudioRecord (16kHz PCM16 mono,
VOICE_COMMUNICATION for hardware AEC/noise suppression) → 100ms chunks
→ Base64. AudioTrack (24kHz PCM16 mono, LOW_LATENCY, MODE_STREAM) ←
server chunks. Mic-ducking during KI speech (RMS threshold gating).

LiveCallService — Foreground Service (MICROPHONE type), survives
rotation/backgrounding. Notification with duration timer + end button.
Status state machine (IDLE→CONNECTING→LISTENING→SPEAKING). Orchestrates
LiveClient + AudioPipeline, emits onTurnEnd for tree integration.
2026-06-24 03:06:48 +02:00
Bruno Deanoz
25fbb17d18 fix: token badge text wrapping on long model names
Added maxLines=1 and softWrap=false to prevent "0 / 1M" from breaking
into two lines when the top bar is compressed by a long model name.
2026-06-24 03:02:33 +02:00
Bruno Deanoz
bb53b24655 fix: TTS read-aloud downloads audio via OkHttp before playing
MediaPlayer.setDataSource(url) with HTTPS URLs can silently fail due to
certificate issues, redirect handling, or listener race conditions (listeners
were set after prepareAsync). Fix: download the WAV file via OkHttp (same
client as all other network calls) to a temp file, set listeners before
prepareAsync, play from local path. Temp file cleaned up on completion/error.
2026-06-24 01:33:23 +02:00
Bruno Deanoz
287cdb4c6b feat: branch navigation for edit/regenerate instead of destructive delete
Edit creates a sibling branch (same parentId) instead of deleting messages.
Regenerate creates a new assistant branch. BranchNavigator UI shows chevron
arrows with "1/3" counter on messages with siblings. Tree state (activeRootId,
activeChild) synced to server via PATCH. New ChatTree data structure mirrors
the web frontend's tree.ts. StoredMessage.parentId parsed from API response.
2026-06-24 01:29:02 +02:00
16 changed files with 1607 additions and 88 deletions

1
.gitignore vendored
View file

@ -13,3 +13,4 @@
.externalNativeBuild
.cxx
local.properties
bugs/

View file

@ -280,7 +280,7 @@ All committed on `main`. Compile-verified on Mac (`./gradlew :app:compileDebugKo
| `fb6bc98` | **Web search provider picker** — for Vertex models: Google vs Enterprise Web Search dropdown pill, `webSearchProvider` sent in chat request |
| `1899f2c` | **Voice message display** — audio-only user messages render as compact "🎤 Sprachnachricht" pill instead of ugly filename chip |
| `44ff82a` | **Voice message ordering fix** — caches user + assistant messages in Room with correct `sortOrder` after server save (was missing, caused wrong order on reload) |
| `8f72afd` | **Edit and resend user messages** — SquarePen icon on user messages, removes message + all after it, puts text back in input field |
| `8f72afd` | **Edit and resend user messages** — SquarePen icon on user messages, creates branch (old messages preserved), text back in input field |
| `b2df7e9` | **Modern message action icons** — ClipboardCopy (copy), SquarePen (edit), RotateCcw (regenerate), 36dp touch target, 20dp icons |
| `421b29e` | **ReasoningBlock + SourceCards redesign** — accent-tinted background + gradient borders, KaizenShapes.md (16dp radius), domain instead of full URL in source cards |
| `75ca8c0` | **Visual refresh** — body text W400→W300 (airier), headings scaled up (H1 26sp, H2 22sp, H3 18sp), `animateItem()` on messages (350ms fade-in, spring placement), blobs 20% larger + better distribution, higher alpha on OLED, light-mode factor 0.65→0.78, distinct `primaryDeep` per theme |
@ -292,6 +292,7 @@ All committed on `main`. Compile-verified on Mac (`./gradlew :app:compileDebugKo
| `1c9491e` | **Text selection**`SelectionContainer` on user + assistant messages. Long-press to select, copy, share |
| `1c9491e` | **Markdown spacing** — heading spacing 18→24/20dp, list items 8dp gap + muted bullets, blockquote 12dp inset |
| `430d5b6` | **Flat sidebar user row** — removed heavy card (background, border, shadow), replaced with flat row + thin separator |
| — | **Branch navigation (tree support)**`ChatTree` data structure mirrors web frontend's tree. Edit creates a branch (sibling node with same parentId) instead of deleting messages. Regenerate creates a new assistant branch. `BranchNavigator` UI (chevron arrows + "1/3" counter) on messages with siblings. `activeRootId`/`activeChild` synced to server via PATCH. `StoredMessage.parentId` parsed from API. `patchConversation` handles JSON object values for `activeChild` |
**Earlier UI/feel work (Phase 0 prototype → feel-first):**
- Liquid glass styling, floating panels, glassmorphic sidebar

View file

@ -11,6 +11,10 @@
<!-- Required for voice recording (mic button in chat) -->
<uses-permission android:name="android.permission.RECORD_AUDIO" />
<!-- Required for Live Call foreground service (mic access while backgrounded) -->
<uses-permission android:name="android.permission.FOREGROUND_SERVICE" />
<uses-permission android:name="android.permission.FOREGROUND_SERVICE_MICROPHONE" />
<application
android:allowBackup="true"
android:dataExtractionRules="@xml/data_extraction_rules"
@ -32,6 +36,11 @@
<category android:name="android.intent.category.LAUNCHER" />
</intent-filter>
</activity>
<service
android:name=".live.LiveCallService"
android:foregroundServiceType="microphone"
android:exported="false" />
</application>
</manifest>

View file

@ -437,6 +437,51 @@ fun TokenBadge(used: Int, contextLength: Int, modifier: Modifier = Modifier) {
color = cs.onSurfaceVariant,
fontSize = 12.sp,
fontWeight = FontWeight.Medium,
maxLines = 1,
softWrap = false,
)
}
}
}
@Composable
fun BranchNavigator(
branchInfo: BranchInfo,
onNavigate: (Int) -> Unit,
modifier: Modifier = Modifier,
) {
val cs = MaterialTheme.colorScheme
Row(
modifier = modifier,
verticalAlignment = Alignment.CenterVertically,
horizontalArrangement = Arrangement.spacedBy(2.dp),
) {
Box(
Modifier.size(28.dp).clip(KaizenShapes.circle)
.clickable(enabled = branchInfo.index > 0) { onNavigate(-1) },
contentAlignment = Alignment.Center,
) {
Icon(
KaizenIcons.ChevronLeft, null,
tint = cs.onSurfaceVariant.copy(alpha = if (branchInfo.index > 0) 0.65f else 0.20f),
modifier = Modifier.size(16.dp),
)
}
Text(
"${branchInfo.index + 1}/${branchInfo.total}",
color = cs.onSurfaceVariant.copy(alpha = 0.55f),
fontSize = 12.sp,
fontWeight = FontWeight.Medium,
)
Box(
Modifier.size(28.dp).clip(KaizenShapes.circle)
.clickable(enabled = branchInfo.index < branchInfo.total - 1) { onNavigate(1) },
contentAlignment = Alignment.Center,
) {
Icon(
KaizenIcons.ChevronRight, null,
tint = cs.onSurfaceVariant.copy(alpha = if (branchInfo.index < branchInfo.total - 1) 0.65f else 0.20f),
modifier = Modifier.size(16.dp),
)
}
}
@ -451,6 +496,8 @@ fun MessageRow(
onEdit: ((String, String) -> Unit)? = null,
onReadAloud: ((String) -> Unit)? = null,
isPlayingTts: Boolean = false,
branchInfo: BranchInfo? = null,
onNavigateBranch: ((String, Int) -> Unit)? = null,
) {
val cs = MaterialTheme.colorScheme
val isDark = isSystemInDarkTheme()
@ -535,7 +582,12 @@ fun MessageRow(
Row(
Modifier.padding(top = 6.dp, end = 4.dp),
horizontalArrangement = Arrangement.spacedBy(2.dp),
verticalAlignment = Alignment.CenterVertically,
) {
if (branchInfo != null && onNavigateBranch != null) {
BranchNavigator(branchInfo, onNavigate = { dir -> onNavigateBranch(message.wireId, dir) })
Spacer(Modifier.width(4.dp))
}
MessageActionButton(
icon = if (copied) KaizenIcons.Check else KaizenIcons.ClipboardCopy,
tint = if (copied) KaizenSemanticColors.success else cs.onSurfaceVariant.copy(alpha = 0.55f),
@ -597,7 +649,12 @@ fun MessageRow(
Row(
Modifier.padding(start = 38.dp, top = 6.dp),
horizontalArrangement = Arrangement.spacedBy(2.dp),
verticalAlignment = Alignment.CenterVertically,
) {
if (branchInfo != null && onNavigateBranch != null) {
BranchNavigator(branchInfo, onNavigate = { dir -> onNavigateBranch(message.wireId, dir) })
Spacer(Modifier.width(4.dp))
}
MessageActionButton(
icon = if (copied) KaizenIcons.Check else KaizenIcons.ClipboardCopy,
tint = if (copied) KaizenSemanticColors.success else cs.onSurfaceVariant.copy(alpha = 0.55f),
@ -734,6 +791,7 @@ fun ChatInput(
isRecording: Boolean = false,
recordingAmplitude: Float = 0f,
onMicClick: () -> Unit = {},
onCallClick: () -> Unit = {},
pendingFiles: List<PendingFile> = emptyList(),
onAddClick: () -> Unit = {},
onRemoveFile: (String) -> Unit = {},
@ -865,7 +923,7 @@ fun ChatInput(
} else if (hasContent) {
SendButton(enabled = canSend, onClick = onSend)
} else {
CallButton(onClick = { /* TODO: call feature */ })
CallButton(onClick = onCallClick)
}
}
}

View file

@ -42,6 +42,7 @@ data class Message(
val tools: List<ToolEvent> = emptyList(),
val query: String = "",
val sources: List<SearchSource> = emptyList(),
val branchInfo: BranchInfo? = null,
)
data class Suggestion(val labelRes: Int, val promptRes: Int, val icon: ImageVector)

View file

@ -92,11 +92,17 @@ import androidx.core.content.ContextCompat
import androidx.compose.animation.core.Spring
import androidx.compose.animation.core.spring
import androidx.compose.animation.core.tween
import android.content.ComponentName
import android.content.ServiceConnection
import android.os.IBinder
import kotlinx.coroutines.Job
import kotlinx.coroutines.async
import kotlinx.coroutines.launch
import java.io.ByteArrayOutputStream
import java.io.File
import dev.kaizen.app.live.CallStatus
import dev.kaizen.app.live.LiveCallOverlay
import dev.kaizen.app.live.LiveCallService
// Screen enumeration for modular state-driven navigation (Zero Technical Debt)
enum class AppScreen { Chat, Settings }
@ -123,12 +129,47 @@ fun ChatScreen(
var chatModel by remember { mutableStateOf<String?>(null) }
var streamJob by remember { mutableStateOf<Job?>(null) }
// Tree state for branch navigation
var chatTree by remember { mutableStateOf(ChatTree.EMPTY) }
val storedMessagesMap = remember { mutableMapOf<String, dev.kaizen.app.net.StoredMessage>() }
var editParentId by remember { mutableStateOf<String?>(null) }
fun rebuildVisibleMessages() {
val path = getVisiblePath(chatTree)
messages.clear()
for (entry in path) {
val sm = storedMessagesMap[entry.messageId] ?: continue
messages.add(
Message(
id = nextId++,
role = if (sm.role == "assistant") Role.Assistant else Role.User,
content = sm.content,
wireId = sm.id,
attachments = sm.attachments,
reasoning = sm.reasoning ?: "",
sources = sm.sources ?: emptyList(),
branchInfo = entry.branchInfo,
),
)
}
}
// Audio recording state
var isRecording by remember { mutableStateOf(false) }
val recorderRef = remember { mutableStateOf<MediaRecorder?>(null) }
val audioFileRef = remember { mutableStateOf<File?>(null) }
var recordingAmplitude by remember { mutableStateOf(0f) }
// ─── Live Call state (uses context/ttsVoice/currentConversationId declared below) ───
var liveService by remember { mutableStateOf<LiveCallService?>(null) }
var liveCallActive by remember { mutableStateOf(false) }
val liveStatus by liveService?.status?.collectAsState() ?: remember { mutableStateOf(CallStatus.IDLE) }
val liveSubtitle by liveService?.subtitle?.collectAsState() ?: remember { mutableStateOf("") }
val liveAmplitude by liveService?.amplitude?.collectAsState() ?: remember { mutableStateOf(0f) }
val liveUserSpeaking by liveService?.userSpeaking?.collectAsState() ?: remember { mutableStateOf(false) }
var liveStartedAt by remember { mutableStateOf(0L) }
var liveElapsedMs by remember { mutableStateOf(0L) }
LaunchedEffect(isRecording) {
if (!isRecording) { recordingAmplitude = 0f; return@LaunchedEffect }
while (isRecording) {
@ -176,6 +217,144 @@ fun ChatScreen(
val pendingFiles = remember { mutableStateListOf<PendingFile>() }
val context = LocalContext.current
// ─── Live Call service binding + control ──────────────────────────────
val liveServiceConnection = remember {
object : ServiceConnection {
override fun onServiceConnected(name: ComponentName?, binder: IBinder?) {
liveService = (binder as? LiveCallService.LocalBinder)?.service
}
override fun onServiceDisconnected(name: ComponentName?) {
liveService = null
}
}
}
LaunchedEffect(liveCallActive) {
if (liveCallActive) {
liveStartedAt = System.currentTimeMillis()
while (liveCallActive) {
liveElapsedMs = System.currentTimeMillis() - liveStartedAt
kotlinx.coroutines.delay(500)
}
}
}
val liveAvailable = models.any { it.id.startsWith("vertex:") && it.id.contains("live") }
fun startLiveCallInner() {
val cfg = session.config ?: return
liveCallActive = true
val intent = LiveCallService.intent(context)
context.startForegroundService(intent)
context.bindService(intent, liveServiceConnection, 0)
scope.launch {
var attempts = 0
while (liveService == null && attempts < 20) {
kotlinx.coroutines.delay(50)
attempts++
}
val svc = liveService ?: return@launch
// Create conversation if needed (same pattern as text chat)
val cid = conversationId ?: run {
val newId = KaizenApi.createConversation(cfg.baseUrl, cfg.token)
if (newId != null) {
conversationId = newId
chat.conversationRepo.insertOptimistic(newId, "Live Call")
}
newId
}
svc.onTurnEnd = { turn ->
scope.launch {
val convId = conversationId ?: return@launch
// Add messages to local UI
if (turn.userText.isNotEmpty()) {
val userMsg = Message(nextId++, Role.User, turn.userText, wireId = turn.userMessageId ?: java.util.UUID.randomUUID().toString())
messages.add(userMsg)
}
if (turn.assistantText.isNotEmpty()) {
val aMsg = Message(nextId++, Role.Assistant, turn.assistantText, wireId = turn.assistantMessageId ?: java.util.UUID.randomUUID().toString())
messages.add(aMsg)
chatModel = cfg.model
usedTokens = turn.usage.inputTokens + turn.usage.outputTokens
}
// Cache in Room (server already persisted via WS-Server)
val msgCount = messages.size
val entitiesToCache = mutableListOf<dev.kaizen.app.db.MessageEntity>()
if (turn.userMessageId != null && turn.userText.isNotEmpty()) {
entitiesToCache.add(dev.kaizen.app.db.MessageEntity(
id = turn.userMessageId, conversationId = convId,
role = "user", content = turn.userText,
attachments = emptyList(), sortOrder = msgCount - 2,
))
}
if (turn.assistantMessageId != null && turn.assistantText.isNotEmpty()) {
entitiesToCache.add(dev.kaizen.app.db.MessageEntity(
id = turn.assistantMessageId, conversationId = convId,
role = "assistant", content = turn.assistantText,
attachments = emptyList(), sortOrder = msgCount - 1,
))
}
if (entitiesToCache.isNotEmpty()) {
chat.messageRepo.cacheMessages(convId, entitiesToCache)
}
}
}
svc.onError = { code, message ->
scope.launch {
val errorText = message ?: code
Toast.makeText(context, errorText, Toast.LENGTH_SHORT).show()
}
}
svc.startCall(
baseUrl = cfg.baseUrl,
token = cfg.token,
model = cfg.model,
voice = ttsVoice,
conversationId = cid,
)
}
}
val liveCallPermission = rememberLauncherForActivityResult(ActivityResultContracts.RequestPermission()) { granted ->
if (granted) startLiveCallInner()
else Toast.makeText(context, context.getString(R.string.chat_mic_permission), Toast.LENGTH_SHORT).show()
}
fun startLiveCall() {
if (ContextCompat.checkSelfPermission(context, Manifest.permission.RECORD_AUDIO) != PackageManager.PERMISSION_GRANTED) {
liveCallPermission.launch(Manifest.permission.RECORD_AUDIO)
return
}
startLiveCallInner()
}
fun endLiveCall() {
liveService?.onTurnEnd = null
liveService?.onError = null
liveService?.endCall()
liveCallActive = false
try { context.unbindService(liveServiceConnection) } catch (_: Exception) {}
liveService = null
// Generate title for new conversations
val cfg = session.config ?: return
val cid = conversationId ?: return
scope.launch {
val title = KaizenApi.generateTitle(cfg.baseUrl, cfg.token, cid)
if (title != null) chat.conversationRepo.updateTitle(cid, title)
chat.conversationRepo.refresh(cfg.baseUrl, cfg.token)
}
}
var showAttachMenu by remember { mutableStateOf(false) }
fun uploadPickedFile(name: String, mimeType: String, bytes: ByteArray) {
@ -309,6 +488,17 @@ fun ChatScreen(
sources = finalSources?.map { dev.kaizen.app.net.SearchSource(it.title, it.url, it.snippet) } ?: emptyList(),
),
))
storedMessagesMap[userMsg.wireId] = dev.kaizen.app.net.StoredMessage(
id = userMsg.wireId, role = "user", content = "",
parentId = userParentId, attachments = listOf(att),
)
storedMessagesMap[assistantWireId] = dev.kaizen.app.net.StoredMessage(
id = assistantWireId, role = "assistant", content = finalContent,
parentId = userMsg.wireId, model = cfg.model,
reasoning = finalReasoning,
sources = finalSources?.map { dev.kaizen.app.net.SearchSource(it.title, it.url, it.snippet) },
)
chatTree = buildChatTree(storedMessagesMap.values.toList(), chatTree.activeRootId, chatTree.activeChild)
}
if (convIdDeferred != null) {
chat.conversationRepo.insertOptimistic(cid, "(Voice)")
@ -334,28 +524,44 @@ fun ChatScreen(
}
fun playTtsUrl(url: String, messageId: Long) {
scope.launch {
try {
val tmpFile = kotlinx.coroutines.withContext(kotlinx.coroutines.Dispatchers.IO) {
val req = okhttp3.Request.Builder().url(url).build()
KaizenApi.imageClient.newCall(req).execute().use { resp ->
if (!resp.isSuccessful) return@withContext null
val bytes = resp.body!!.bytes()
java.io.File.createTempFile("tts_", ".wav", context.cacheDir).also { it.writeBytes(bytes) }
}
}
if (tmpFile == null) {
playingTtsForId = null
return@launch
}
val player = MediaPlayer().apply {
setDataSource(url)
prepareAsync()
setOnPreparedListener { start() }
setOnCompletionListener {
release()
ttsPlayerRef.value = null
playingTtsForId = null
tmpFile.delete()
}
setOnErrorListener { _, _, _ ->
release()
ttsPlayerRef.value = null
playingTtsForId = null
tmpFile.delete()
true
}
setDataSource(tmpFile.absolutePath)
prepareAsync()
}
ttsPlayerRef.value = player
} catch (_: Exception) {
playingTtsForId = null
}
}
}
fun readAloud(content: String, messageId: Long, wireId: String) {
val cfg = session.config ?: return
@ -443,6 +649,8 @@ fun ChatScreen(
viewingLockedChat = false
usedTokens = 0
chatModel = null
chatTree = ChatTree.EMPTY
storedMessagesMap.clear()
}
fun unlockAndOpen(summary: ConversationSummary, password: String? = null) {
@ -462,22 +670,12 @@ fun ChatScreen(
passwordUnlockTarget = null
passwordUnlockError = null
passwordUnlockForToggle = false
val stored = KaizenApi.fetchLockedMessages(cfg.baseUrl, cfg.token, summary.id, unlockToken)
if (stored.isNotEmpty()) {
messages.clear()
stored.forEach { m ->
messages.add(
Message(
id = nextId++,
role = if (m.role == "assistant") Role.Assistant else Role.User,
content = m.content,
wireId = m.id,
attachments = m.attachments,
reasoning = m.reasoning ?: "",
sources = m.sources ?: emptyList(),
),
)
}
val data = KaizenApi.fetchLockedConversationData(cfg.baseUrl, cfg.token, summary.id, unlockToken)
if (data.messages.isNotEmpty()) {
storedMessagesMap.clear()
data.messages.forEach { storedMessagesMap[it.id] = it }
chatTree = buildChatTree(data.messages, data.activeRootId, data.activeChild)
rebuildVisibleMessages()
conversationId = summary.id
viewingLockedChat = true
}
@ -546,46 +744,36 @@ fun ChatScreen(
)
}
conversationId = summary.id
val stored = KaizenApi.fetchMessages(cfg.baseUrl, cfg.token, summary.id)
if (stored.isNotEmpty()) {
val entities = stored.mapIndexed { i, m -> m.toEntity(summary.id, i) }
val data = KaizenApi.fetchConversationData(cfg.baseUrl, cfg.token, summary.id)
if (data.messages.isNotEmpty()) {
val entities = data.messages.mapIndexed { i, m -> m.toEntity(summary.id, i) }
chat.messageRepo.cacheMessages(summary.id, entities)
val lastAssistant = stored.lastOrNull { it.role == "assistant" }
val lastAssistant = data.messages.lastOrNull { it.role == "assistant" }
lastAssistant?.model?.let { chatModel = it }
usedTokens = stored.sumOf { (it.inputTokens ?: 0) + (it.outputTokens ?: 0) }
if (stored.size != cachedList.size || stored.zip(cachedList).any { (s, c) -> s.id != c.id || s.content != c.content }) {
messages.clear()
stored.forEach { m ->
messages.add(
Message(
id = nextId++,
role = if (m.role == "assistant") Role.Assistant else Role.User,
content = m.content,
wireId = m.id,
attachments = m.attachments,
reasoning = m.reasoning ?: "",
sources = m.sources ?: emptyList(),
),
)
}
}
usedTokens = data.messages.sumOf { (it.inputTokens ?: 0) + (it.outputTokens ?: 0) }
storedMessagesMap.clear()
data.messages.forEach { storedMessagesMap[it.id] = it }
chatTree = buildChatTree(data.messages, data.activeRootId, data.activeChild)
rebuildVisibleMessages()
}
}
}
fun send(text: String) {
fun send(text: String, branchParentId: String? = null, branchAttachments: List<Attachment> = emptyList()) {
val trimmed = text.trim()
val uploadedAtts = pendingFiles.mapNotNull { it.uploaded }
val uploadedAtts = if (branchAttachments.isNotEmpty()) branchAttachments else pendingFiles.mapNotNull { it.uploaded }
if (trimmed.isEmpty() && uploadedAtts.isEmpty()) return
if (isStreaming) return
val cfg = session.config ?: return
haptics.click()
val userParentId = messages.lastOrNull()?.wireId
val userParentId = editParentId ?: branchParentId ?: messages.lastOrNull()?.wireId
editParentId = null
val userMsg = Message(nextId++, Role.User, trimmed, attachments = uploadedAtts)
messages.add(userMsg)
input = ""
pendingFiles.clear()
if (branchAttachments.isEmpty()) pendingFiles.clear()
val assistantId = nextId++
val assistantWireId = java.util.UUID.randomUUID().toString()
messages.add(Message(assistantId, Role.Assistant, "", streaming = true, thinking = true, wireId = assistantWireId))
@ -681,6 +869,24 @@ fun ChatScreen(
sources = finalSources ?: emptyList(),
),
))
// Update tree with the new nodes
storedMessagesMap[userMsg.wireId] = dev.kaizen.app.net.StoredMessage(
id = userMsg.wireId, role = "user", content = trimmed,
parentId = userParentId, attachments = uploadedAtts,
)
storedMessagesMap[assistantWireId] = dev.kaizen.app.net.StoredMessage(
id = assistantWireId, role = "assistant", content = finalContent,
parentId = userMsg.wireId, model = cfg.model,
reasoning = finalReasoning,
sources = finalSources?.map { dev.kaizen.app.net.SearchSource(it.title, it.url, it.snippet) },
)
chatTree = buildChatTree(
storedMessagesMap.values.toList(),
chatTree.activeRootId,
chatTree.activeChild,
)
if (wasNewConversation) {
val title = KaizenApi.generateTitle(cfg.baseUrl, cfg.token, convId)
if (title != null) {
@ -888,24 +1094,22 @@ fun ChatScreen(
onRegenerate = if (message.role == Role.Assistant && !isStreaming) { _ ->
val lastUserIdx = messages.indexOfLast { it.role == Role.User && messages.indexOf(it) < messages.indexOf(message) }
if (lastUserIdx >= 0) {
val userContent = messages[lastUserIdx].content
messages.removeAt(messages.indexOf(message))
send(userContent)
val userMsg = messages[lastUserIdx]
val parentId = chatTree.nodes[userMsg.wireId]?.parentId
val msgIdx = messages.indexOf(message)
if (msgIdx >= 0) {
messages.subList(msgIdx, messages.size).clear()
}
send(userMsg.content, branchParentId = parentId, branchAttachments = userMsg.attachments)
}
} else null,
onEdit = if (message.role == Role.User && !isStreaming) { wireId, content ->
val node = chatTree.nodes[wireId]
val parentId = node?.parentId
val idx = messages.indexOfFirst { it.wireId == wireId }
if (idx >= 0) {
val cfg = session.config ?: return@MessageRow
val toRemove = messages.subList(idx, messages.size).map { it.wireId }
messages.removeAll { it.wireId in toRemove }
if (conversationId != null) {
scope.launch {
toRemove.forEach { id ->
KaizenApi.deleteMessage(cfg.baseUrl, cfg.token, conversationId!!, id)
}
}
}
messages.subList(idx, messages.size).clear()
editParentId = parentId
input = content
}
} else null,
@ -913,6 +1117,24 @@ fun ChatScreen(
readAloud(content, message.id, message.wireId)
} else null,
isPlayingTts = playingTtsForId == message.id,
branchInfo = message.branchInfo,
onNavigateBranch = if (!isStreaming) { wireId, direction ->
val newTree = treeNavigate(chatTree, wireId, direction)
chatTree = newTree
rebuildVisibleMessages()
if (conversationId != null) {
val cfg = session.config ?: return@MessageRow
scope.launch {
KaizenApi.patchConversation(
cfg.baseUrl, cfg.token, conversationId!!,
mapOf(
"activeRootId" to newTree.activeRootId,
"activeChild" to newTree.activeChild,
),
)
}
}
} else null,
)
}
}
@ -1137,6 +1359,7 @@ fun ChatScreen(
isRecording = isRecording,
recordingAmplitude = recordingAmplitude,
onMicClick = { toggleRecording() },
onCallClick = if (liveAvailable) {{ startLiveCall() }} else {{ /* PTT TODO */ }},
pendingFiles = pendingFiles,
onAddClick = { showAttachMenu = !showAttachMenu },
onRemoveFile = { id -> pendingFiles.removeAll { it.id == id } },
@ -1234,6 +1457,23 @@ fun ChatScreen(
}
}
}
// ─── Live Call Overlay ────────────────────────────────
androidx.compose.animation.AnimatedVisibility(
visible = liveCallActive,
enter = androidx.compose.animation.fadeIn(tween(250)),
exit = androidx.compose.animation.fadeOut(tween(200)),
) {
LiveCallOverlay(
status = liveStatus,
subtitle = liveSubtitle,
amplitude = liveAmplitude,
userSpeaking = liveUserSpeaking,
elapsedMs = liveElapsedMs,
onEndCall = { endLiveCall() },
onInterrupt = { liveService?.interrupt() },
)
}
}
}
}

View file

@ -0,0 +1,106 @@
package dev.kaizen.app.chat
import dev.kaizen.app.net.StoredMessage
data class ChatNode(
val id: String,
val role: String,
val parentId: String?,
val childIds: List<String> = emptyList(),
)
data class BranchInfo(val index: Int, val total: Int)
data class VisibleEntry(
val messageId: String,
val branchInfo: BranchInfo?,
)
data class ChatTree(
val nodes: Map<String, ChatNode>,
val rootIds: List<String>,
val activeRootId: String?,
val activeChild: Map<String, String>,
) {
companion object {
val EMPTY = ChatTree(emptyMap(), emptyList(), null, emptyMap())
}
}
fun buildChatTree(
messages: List<StoredMessage>,
savedActiveRootId: String?,
savedActiveChild: Map<String, String>,
): ChatTree {
val nodes = mutableMapOf<String, ChatNode>()
val rootIds = mutableListOf<String>()
for (msg in messages) {
nodes[msg.id] = ChatNode(
id = msg.id,
role = msg.role,
parentId = msg.parentId,
)
}
for (msg in messages) {
val pid = msg.parentId
if (pid != null && nodes.containsKey(pid)) {
val parent = nodes[pid]!!
nodes[pid] = parent.copy(childIds = parent.childIds + msg.id)
} else if (pid == null) {
rootIds.add(msg.id)
}
}
val activeRootId = if (savedActiveRootId != null && nodes.containsKey(savedActiveRootId))
savedActiveRootId
else rootIds.lastOrNull()
return ChatTree(nodes, rootIds, activeRootId, savedActiveChild)
}
fun getVisiblePath(tree: ChatTree): List<VisibleEntry> {
if (tree.rootIds.isEmpty()) return emptyList()
val activeRootId = tree.activeRootId ?: tree.rootIds.last()
val result = mutableListOf<VisibleEntry>()
var currentId: String? = activeRootId
while (currentId != null) {
val node = tree.nodes[currentId] ?: break
val siblings = if (node.parentId == null) {
tree.rootIds
} else {
tree.nodes[node.parentId]?.childIds ?: listOf(node.id)
}
val branchInfo = if (siblings.size > 1) {
BranchInfo(index = siblings.indexOf(node.id), total = siblings.size)
} else null
result.add(VisibleEntry(node.id, branchInfo))
if (node.childIds.isEmpty()) break
currentId = tree.activeChild[node.id] ?: node.childIds.last()
}
return result
}
fun treeNavigate(tree: ChatTree, nodeId: String, direction: Int): ChatTree {
val node = tree.nodes[nodeId] ?: return tree
val siblings = if (node.parentId == null) {
tree.rootIds
} else {
tree.nodes[node.parentId]?.childIds ?: return tree
}
val idx = siblings.indexOf(nodeId)
val newIdx = idx + direction
if (newIdx < 0 || newIdx >= siblings.size) return tree
val newActiveId = siblings[newIdx]
return if (node.parentId == null) {
tree.copy(activeRootId = newActiveId)
} else {
tree.copy(activeChild = tree.activeChild + (node.parentId to newActiveId))
}
}

View file

@ -0,0 +1,210 @@
package dev.kaizen.app.live
import android.Manifest
import android.content.Context
import android.content.pm.PackageManager
import android.media.AudioAttributes
import android.media.AudioFormat
import android.media.AudioRecord
import android.media.AudioTrack
import android.media.MediaRecorder
import android.util.Base64
import android.util.Log
import androidx.core.content.ContextCompat
import kotlinx.coroutines.*
import java.nio.ByteBuffer
import java.nio.ByteOrder
import kotlin.math.abs
import kotlin.math.sqrt
/**
* Bidirectional audio pipeline for Live Call.
*
* Capture: AudioRecord (16kHz PCM16 mono, VOICE_COMMUNICATION for hardware AEC)
* 100ms chunks Base64 [onChunk] callback.
*
* Playback: AudioTrack (24kHz PCM16 mono, LOW_LATENCY, MODE_STREAM)
* PCM16 chunks from server.
*
* Mic-ducking: gain reduced to 10% while the KI speaks (prevents echo feedback
* loop on devices with weak AEC). Not a digital gain we skip sending chunks
* whose RMS is below the ducking threshold.
*/
class AudioPipeline(private val context: Context) {
companion object {
private const val TAG = "AudioPipeline"
private const val CAPTURE_SAMPLE_RATE = 16_000
private const val PLAYBACK_SAMPLE_RATE = 24_000
private const val CHANNEL_IN = AudioFormat.CHANNEL_IN_MONO
private const val CHANNEL_OUT = AudioFormat.CHANNEL_OUT_MONO
private const val ENCODING = AudioFormat.ENCODING_PCM_16BIT
private const val CHUNK_MS = 100
private const val CAPTURE_CHUNK_SAMPLES = CAPTURE_SAMPLE_RATE * CHUNK_MS / 1000 // 1600
private const val CAPTURE_CHUNK_BYTES = CAPTURE_CHUNK_SAMPLES * 2 // 3200
private const val DUCKING_GAIN = 0.10f
}
var onChunk: ((base64Pcm: String) -> Unit)? = null
var onAmplitude: ((Float) -> Unit)? = null
private var recorder: AudioRecord? = null
private var track: AudioTrack? = null
private var captureJob: Job? = null
@Volatile private var ducking = false
@Volatile private var capturing = false
// ─── Capture ──────────────────────────────────────────────────────────
fun startCapture(scope: CoroutineScope): Boolean {
if (ContextCompat.checkSelfPermission(context, Manifest.permission.RECORD_AUDIO) != PackageManager.PERMISSION_GRANTED) {
Log.w(TAG, "RECORD_AUDIO permission not granted")
return false
}
val minBuf = AudioRecord.getMinBufferSize(CAPTURE_SAMPLE_RATE, CHANNEL_IN, ENCODING)
if (minBuf == AudioRecord.ERROR || minBuf == AudioRecord.ERROR_BAD_VALUE) {
Log.e(TAG, "Invalid min buffer size: $minBuf")
return false
}
val rec = AudioRecord(
MediaRecorder.AudioSource.VOICE_COMMUNICATION,
CAPTURE_SAMPLE_RATE,
CHANNEL_IN,
ENCODING,
maxOf(minBuf * 4, CAPTURE_CHUNK_BYTES * 4),
)
if (rec.state != AudioRecord.STATE_INITIALIZED) {
Log.e(TAG, "AudioRecord init failed")
rec.release()
return false
}
recorder = rec
rec.startRecording()
capturing = true
captureJob = scope.launch(Dispatchers.IO) {
val buffer = ShortArray(CAPTURE_CHUNK_SAMPLES)
val byteBuffer = ByteBuffer.allocate(CAPTURE_CHUNK_BYTES).order(ByteOrder.LITTLE_ENDIAN)
while (isActive && capturing) {
val read = rec.read(buffer, 0, CAPTURE_CHUNK_SAMPLES)
if (read <= 0) continue
val rms = computeRms(buffer, read)
onAmplitude?.invoke(rms)
if (ducking && rms < DUCKING_GAIN) continue
byteBuffer.clear()
for (i in 0 until read) byteBuffer.putShort(buffer[i])
val base64 = Base64.encodeToString(byteBuffer.array(), 0, read * 2, Base64.NO_WRAP)
onChunk?.invoke(base64)
}
}
return true
}
fun stopCapture() {
capturing = false
captureJob?.cancel()
captureJob = null
try {
recorder?.stop()
} catch (_: IllegalStateException) {}
recorder?.release()
recorder = null
}
// ─── Playback ─────────────────────────────────────────────────────────
fun initPlayback(): Boolean {
val minBuf = AudioTrack.getMinBufferSize(PLAYBACK_SAMPLE_RATE, CHANNEL_OUT, ENCODING)
if (minBuf == AudioTrack.ERROR || minBuf == AudioTrack.ERROR_BAD_VALUE) {
Log.e(TAG, "Invalid playback buffer size: $minBuf")
return false
}
val t = AudioTrack.Builder()
.setAudioAttributes(
AudioAttributes.Builder()
.setUsage(AudioAttributes.USAGE_ASSISTANT)
.setContentType(AudioAttributes.CONTENT_TYPE_SPEECH)
.build()
)
.setAudioFormat(
AudioFormat.Builder()
.setSampleRate(PLAYBACK_SAMPLE_RATE)
.setEncoding(ENCODING)
.setChannelMask(CHANNEL_OUT)
.build()
)
.setBufferSizeInBytes(maxOf(minBuf * 4, 24_000 * 2))
.setPerformanceMode(AudioTrack.PERFORMANCE_MODE_LOW_LATENCY)
.setTransferMode(AudioTrack.MODE_STREAM)
.build()
if (t.state != AudioTrack.STATE_INITIALIZED) {
Log.e(TAG, "AudioTrack init failed")
t.release()
return false
}
track = t
t.play()
return true
}
fun playChunk(base64Pcm: String) {
val bytes = Base64.decode(base64Pcm, Base64.NO_WRAP)
track?.write(bytes, 0, bytes.size)
}
fun flushPlayback() {
track?.pause()
track?.flush()
track?.play()
}
fun stopPlayback() {
try {
track?.pause()
track?.flush()
} catch (_: IllegalStateException) {}
track?.release()
track = null
}
// ─── Ducking ──────────────────────────────────────────────────────────
fun setDucking(enabled: Boolean) {
ducking = enabled
}
// ─── Cleanup ──────────────────────────────────────────────────────────
fun release() {
stopCapture()
stopPlayback()
}
// ─── Helpers ──────────────────────────────────────────────────────────
private fun computeRms(buffer: ShortArray, count: Int): Float {
var sum = 0.0
for (i in 0 until count) {
val sample = buffer[i].toDouble() / Short.MAX_VALUE
sum += sample * sample
}
return sqrt(sum / count).toFloat()
}
}

View file

@ -0,0 +1,181 @@
package dev.kaizen.app.live
import androidx.compose.animation.*
import androidx.compose.animation.core.*
import androidx.compose.foundation.background
import androidx.compose.foundation.clickable
import androidx.compose.foundation.interaction.MutableInteractionSource
import androidx.compose.foundation.interaction.collectIsPressedAsState
import androidx.compose.foundation.isSystemInDarkTheme
import androidx.compose.foundation.layout.*
import androidx.compose.material3.Icon
import androidx.compose.material3.MaterialTheme
import androidx.compose.material3.Text
import androidx.compose.runtime.*
import androidx.compose.ui.Alignment
import androidx.compose.ui.Modifier
import androidx.compose.ui.draw.alpha
import androidx.compose.ui.draw.clip
import androidx.compose.ui.draw.scale
import androidx.compose.ui.graphics.Brush
import androidx.compose.ui.graphics.Color
import androidx.compose.ui.res.stringResource
import androidx.compose.ui.text.font.FontWeight
import androidx.compose.ui.text.style.TextAlign
import androidx.compose.ui.text.style.TextOverflow
import androidx.compose.ui.unit.dp
import androidx.compose.ui.unit.sp
import dev.kaizen.app.R
import dev.kaizen.app.chat.KaizenOrb
import dev.kaizen.app.ui.effect.GlassSurface
import dev.kaizen.app.ui.effect.KaizenShadows
import dev.kaizen.app.ui.icon.KaizenIcons
import dev.kaizen.app.ui.shape.KaizenShapes
import dev.kaizen.app.ui.theme.LocalKaizenAccent
@Composable
fun LiveCallOverlay(
status: CallStatus,
subtitle: String,
amplitude: Float,
userSpeaking: Boolean,
elapsedMs: Long,
onEndCall: () -> Unit,
onInterrupt: () -> Unit,
modifier: Modifier = Modifier,
) {
val isDark = isSystemInDarkTheme()
val accent = LocalKaizenAccent.current
val cs = MaterialTheme.colorScheme
Box(
modifier
.fillMaxSize()
.background(
if (isDark) Color.Black.copy(alpha = 0.85f)
else Color.White.copy(alpha = 0.88f)
)
.statusBarsPadding()
.navigationBarsPadding(),
contentAlignment = Alignment.Center,
) {
Column(
horizontalAlignment = Alignment.CenterHorizontally,
verticalArrangement = Arrangement.spacedBy(24.dp),
modifier = Modifier.fillMaxWidth().padding(horizontal = 32.dp),
) {
// ─── Duration ─────────────────────────────────────────
val elapsed = elapsedMs / 1000
val min = elapsed / 60
val sec = elapsed % 60
Text(
String.format("%d:%02d", min, sec),
color = cs.onSurface.copy(alpha = 0.5f),
fontSize = 14.sp,
fontWeight = FontWeight.W400,
letterSpacing = 1.sp,
)
// ─── Orb ──────────────────────────────────────────────
val isActive = status == CallStatus.LISTENING || status == CallStatus.SPEAKING
Box(
Modifier
.size(180.dp)
.then(
if (status == CallStatus.SPEAKING) {
Modifier.clickable(
interactionSource = remember { MutableInteractionSource() },
indication = null,
onClick = onInterrupt,
)
} else Modifier
),
contentAlignment = Alignment.Center,
) {
KaizenOrb(
size = 160.dp,
streaming = status == CallStatus.SPEAKING,
recording = status == CallStatus.LISTENING && userSpeaking,
amplitude = if (isActive) amplitude else null,
)
}
// ─── Status text ──────────────────────────────────────
val statusText = when (status) {
CallStatus.IDLE -> ""
CallStatus.CONNECTING -> stringResource(R.string.live_connecting)
CallStatus.LISTENING -> if (userSpeaking) stringResource(R.string.live_listening)
else stringResource(R.string.live_ready)
CallStatus.SPEAKING -> stringResource(R.string.live_speaking)
}
Text(
statusText,
color = when (status) {
CallStatus.SPEAKING -> accent.primary
CallStatus.LISTENING -> if (userSpeaking) accent.primary.copy(alpha = 0.8f)
else cs.onSurface.copy(alpha = 0.4f)
else -> cs.onSurface.copy(alpha = 0.4f)
},
fontSize = 13.sp,
fontWeight = FontWeight.W400,
letterSpacing = 0.5.sp,
)
// ─── Subtitle (transcript / KI response) ─────────────
AnimatedVisibility(
visible = subtitle.isNotEmpty(),
enter = fadeIn(tween(200)) + expandVertically(),
exit = fadeOut(tween(150)) + shrinkVertically(),
) {
Text(
subtitle,
color = cs.onSurface.copy(alpha = 0.75f),
fontSize = 15.sp,
fontWeight = FontWeight.W300,
textAlign = TextAlign.Center,
maxLines = 4,
overflow = TextOverflow.Ellipsis,
lineHeight = 22.sp,
modifier = Modifier.fillMaxWidth(),
)
}
Spacer(Modifier.height(32.dp))
// ─── End Call button ──────────────────────────────────
EndCallButton(onClick = onEndCall)
}
}
}
@Composable
private fun EndCallButton(onClick: () -> Unit) {
val interaction = remember { MutableInteractionSource() }
val pressed by interaction.collectIsPressedAsState()
val pressScale by animateFloatAsState(if (pressed) 0.9f else 1f, label = "end")
Box(
Modifier
.size(64.dp)
.scale(pressScale)
.clip(KaizenShapes.circle)
.background(
Brush.verticalGradient(
listOf(
Color(0xFFEF4444),
Color(0xFFDC2626),
)
)
)
.clickable(interactionSource = interaction, indication = null, onClick = onClick),
contentAlignment = Alignment.Center,
) {
Icon(
KaizenIcons.PhoneOff,
contentDescription = stringResource(R.string.live_end),
tint = Color.White,
modifier = Modifier.size(28.dp),
)
}
}

View file

@ -0,0 +1,304 @@
package dev.kaizen.app.live
import android.app.Notification
import android.app.NotificationChannel
import android.app.NotificationManager
import android.app.PendingIntent
import android.app.Service
import android.content.Context
import android.content.Intent
import android.content.pm.ServiceInfo
import android.os.Binder
import android.os.IBinder
import android.util.Log
import dev.kaizen.app.MainActivity
import dev.kaizen.app.R
import kotlinx.coroutines.*
import kotlinx.coroutines.flow.MutableStateFlow
import kotlinx.coroutines.flow.StateFlow
/**
* Foreground Service for Live Call keeps mic + WS alive when the app is
* backgrounded or the screen rotates. Shows a persistent notification with
* call duration and an end button.
*
* Lifecycle: started by ChatScreen binds [startCall] audio flows
* [endCall] or notification "Beenden" cleanup stopSelf.
*/
class LiveCallService : Service() {
companion object {
private const val TAG = "LiveCallService"
private const val CHANNEL_ID = "kaizen_live_call"
private const val NOTIFICATION_ID = 9001
private const val ACTION_END = "dev.kaizen.app.LIVE_CALL_END"
fun intent(context: Context): Intent = Intent(context, LiveCallService::class.java)
}
// ─── Public state ─────────────────────────────────────────────────────
private val _status = MutableStateFlow(CallStatus.IDLE)
val status: StateFlow<CallStatus> = _status
private val _subtitle = MutableStateFlow("")
val subtitle: StateFlow<String> = _subtitle
private val _amplitude = MutableStateFlow(0f)
val amplitude: StateFlow<Float> = _amplitude
private val _userSpeaking = MutableStateFlow(false)
val userSpeaking: StateFlow<Boolean> = _userSpeaking
var onTurnEnd: ((ServerEvent.TurnEnd) -> Unit)? = null
var onError: ((String, String?) -> Unit)? = null
// ─── Internal ─────────────────────────────────────────────────────────
private val scope = CoroutineScope(SupervisorJob() + Dispatchers.Main.immediate)
private var client: LiveClient? = null
private var audio: AudioPipeline? = null
private var durationJob: Job? = null
private var startTimeMs = 0L
// ─── Binder ───────────────────────────────────────────────────────────
inner class LocalBinder : Binder() {
val service: LiveCallService get() = this@LiveCallService
}
private val binder = LocalBinder()
override fun onBind(intent: Intent?): IBinder = binder
// ─── Lifecycle ────────────────────────────────────────────────────────
override fun onCreate() {
super.onCreate()
createNotificationChannel()
}
override fun onStartCommand(intent: Intent?, flags: Int, startId: Int): Int {
if (intent?.action == ACTION_END) {
endCall()
return START_NOT_STICKY
}
return START_NOT_STICKY
}
override fun onDestroy() {
endCall()
scope.cancel()
super.onDestroy()
}
// ─── Start / End ──────────────────────────────────────────────────────
fun startCall(baseUrl: String, token: String, model: String, voice: String, conversationId: String?) {
if (_status.value != CallStatus.IDLE) return
_status.value = CallStatus.CONNECTING
startForeground(NOTIFICATION_ID, buildNotification("Verbinde…"), ServiceInfo.FOREGROUND_SERVICE_TYPE_MICROPHONE)
val pipeline = AudioPipeline(this)
audio = pipeline
val liveClient = LiveClient(baseUrl, token, scope)
client = liveClient
pipeline.onAmplitude = { amp -> _amplitude.value = amp }
scope.launch {
liveClient.events.collect { event -> handleEvent(event, pipeline) }
}
pipeline.onChunk = { base64 -> liveClient.sendAudio(base64) }
liveClient.connect(model, voice, conversationId)
}
fun endCall() {
if (_status.value == CallStatus.IDLE) return
client?.end()
audio?.release()
durationJob?.cancel()
client?.dispose()
client = null
audio = null
durationJob = null
_status.value = CallStatus.IDLE
_subtitle.value = ""
_amplitude.value = 0f
_userSpeaking.value = false
stopForeground(STOP_FOREGROUND_REMOVE)
stopSelf()
}
fun sendText(text: String) {
client?.sendText(text)
}
fun interrupt() {
client?.interrupt()
}
// ─── Event handling ───────────────────────────────────────────────────
private fun handleEvent(event: ServerEvent, pipeline: AudioPipeline) {
when (event) {
is ServerEvent.Ready -> {
_status.value = CallStatus.LISTENING
if (!pipeline.initPlayback()) {
Log.e(TAG, "Playback init failed")
}
if (!pipeline.startCapture(scope)) {
Log.e(TAG, "Capture init failed")
endCall()
return
}
startTimeMs = System.currentTimeMillis()
startDurationTimer()
updateNotification("Anruf aktiv")
}
is ServerEvent.Resumed -> {
_status.value = CallStatus.LISTENING
updateNotification("Anruf fortgesetzt")
}
is ServerEvent.Audio -> {
if (_status.value != CallStatus.SPEAKING) {
_status.value = CallStatus.SPEAKING
pipeline.setDucking(true)
}
pipeline.playChunk(event.data)
}
is ServerEvent.Text -> {
_subtitle.value += event.delta
}
is ServerEvent.Transcript -> {
if (event.final) {
_subtitle.value = event.text
}
}
is ServerEvent.Vad -> {
_userSpeaking.value = event.speaking
if (event.speaking) {
_status.value = CallStatus.LISTENING
_subtitle.value = ""
pipeline.setDucking(false)
}
}
is ServerEvent.Interrupted -> {
pipeline.flushPlayback()
pipeline.setDucking(false)
_status.value = CallStatus.LISTENING
}
is ServerEvent.TurnEnd -> {
pipeline.setDucking(false)
_status.value = CallStatus.LISTENING
onTurnEnd?.invoke(event)
}
is ServerEvent.ToolCall -> {
if (event.status == "running") {
_subtitle.value = "${event.name}"
}
}
is ServerEvent.Backpressure -> {
// LiveClient handles throttling internally
}
is ServerEvent.GoAway -> {
if (event.reason == "timeout") {
endCall()
}
}
is ServerEvent.Error -> {
onError?.invoke(event.code, event.message)
if (event.code in setOf("auth_failed", "restricted_model", "restricted_feature", "budget_exceeded", "concurrent_limit")) {
endCall()
}
}
}
}
// ─── Duration timer ───────────────────────────────────────────────────
private fun startDurationTimer() {
durationJob?.cancel()
durationJob = scope.launch {
while (isActive) {
delay(1000)
val elapsed = (System.currentTimeMillis() - startTimeMs) / 1000
val min = elapsed / 60
val sec = elapsed % 60
updateNotification(String.format("Anruf · %d:%02d", min, sec))
}
}
}
// ─── Notification ─────────────────────────────────────────────────────
private fun createNotificationChannel() {
val channel = NotificationChannel(
CHANNEL_ID,
getString(R.string.app_name),
NotificationManager.IMPORTANCE_LOW,
).apply {
description = "Live Call"
setShowBadge(false)
}
getSystemService(NotificationManager::class.java)?.createNotificationChannel(channel)
}
private fun buildNotification(text: String): Notification {
val openIntent = PendingIntent.getActivity(
this, 0,
Intent(this, MainActivity::class.java).apply {
flags = Intent.FLAG_ACTIVITY_SINGLE_TOP
},
PendingIntent.FLAG_IMMUTABLE,
)
val endIntent = PendingIntent.getService(
this, 1,
Intent(this, LiveCallService::class.java).apply { action = ACTION_END },
PendingIntent.FLAG_IMMUTABLE,
)
return Notification.Builder(this, CHANNEL_ID)
.setSmallIcon(R.drawable.ic_launcher_foreground)
.setContentTitle(getString(R.string.app_name))
.setContentText(text)
.setContentIntent(openIntent)
.addAction(Notification.Action.Builder(null, "Beenden", endIntent).build())
.setOngoing(true)
.build()
}
private fun updateNotification(text: String) {
getSystemService(NotificationManager::class.java)
?.notify(NOTIFICATION_ID, buildNotification(text))
}
}
enum class CallStatus {
IDLE,
CONNECTING,
LISTENING,
SPEAKING,
}

View file

@ -0,0 +1,211 @@
package dev.kaizen.app.live
import android.util.Log
import kotlinx.coroutines.*
import kotlinx.coroutines.flow.MutableSharedFlow
import kotlinx.coroutines.flow.SharedFlow
import okhttp3.OkHttpClient
import okhttp3.Request
import okhttp3.Response
import okhttp3.WebSocket
import okhttp3.WebSocketListener
import java.util.concurrent.TimeUnit
import java.util.concurrent.atomic.AtomicInteger
import java.util.concurrent.atomic.AtomicReference
/**
* WebSocket client for the Kaizen Live Call protocol.
*
* Connects to `wss://{baseUrl}/ws/live` with Bearer auth, speaks the
* provider-agnostic JSON protocol defined in `lib/live/protocol.ts`.
* Handles auto-reconnect with exponential backoff + resume.
*
* Thread-safe: OkHttp dispatches callbacks on its own threads,
* all state mutations go through atomic refs or the coroutine scope.
*/
class LiveClient(
private val baseUrl: String,
private val token: String,
private val scope: CoroutineScope,
) {
companion object {
private const val TAG = "LiveClient"
private const val INITIAL_BACKOFF_MS = 500L
private const val MAX_BACKOFF_MS = 8_000L
private const val MAX_RECONNECT_ATTEMPTS = 5
}
// ─── Public event stream ──────────────────────────────────────────────
private val _events = MutableSharedFlow<ServerEvent>(extraBufferCapacity = 64)
val events: SharedFlow<ServerEvent> = _events
// ─── State ────────────────────────────────────────────────────────────
private val ws = AtomicReference<WebSocket?>(null)
private val sessionId = AtomicReference<String?>(null)
private val lastSeqId = AtomicInteger(0)
private val reconnectAttempts = AtomicInteger(0)
private var chunkIntervalMs = 100L
@Volatile private var active = false
@Volatile private var pendingSetup: SetupParams? = null
private data class SetupParams(
val model: String,
val voice: String,
val conversationId: String?,
)
private val client = OkHttpClient.Builder()
.readTimeout(0, TimeUnit.MILLISECONDS)
.pingInterval(15, TimeUnit.SECONDS)
.connectTimeout(10, TimeUnit.SECONDS)
.build()
// ─── Connect ──────────────────────────────────────────────────────────
fun connect(model: String, voice: String, conversationId: String?) {
active = true
pendingSetup = SetupParams(model, voice, conversationId)
reconnectAttempts.set(0)
openSocket()
}
private fun openSocket() {
if (!active) return
val wsUrl = baseUrl
.replace("https://", "wss://")
.replace("http://", "ws://") + "/ws/live"
val request = Request.Builder()
.url(wsUrl)
.header("Authorization", "Bearer $token")
.build()
val socket = client.newWebSocket(request, Listener())
ws.set(socket)
}
// ─── Send ─────────────────────────────────────────────────────────────
fun sendAudio(base64Pcm: String) {
ws.get()?.send(encodeAudio(base64Pcm))
}
fun sendText(text: String) {
ws.get()?.send(encodeText(text))
}
fun interrupt() {
ws.get()?.send(encodeInterrupt())
}
fun end() {
active = false
ws.get()?.send(encodeEnd())
ws.getAndSet(null)?.close(1000, "end")
}
fun dispose() {
active = false
ws.getAndSet(null)?.cancel()
client.dispatcher.executorService.shutdown()
}
// ─── Reconnect ────────────────────────────────────────────────────────
private fun scheduleReconnect() {
if (!active) return
val attempt = reconnectAttempts.incrementAndGet()
if (attempt > MAX_RECONNECT_ATTEMPTS) {
Log.w(TAG, "Max reconnect attempts reached")
scope.launch { _events.emit(ServerEvent.Error("reconnect_failed", "Max reconnect attempts")) }
active = false
return
}
val backoff = (INITIAL_BACKOFF_MS * (1 shl (attempt - 1))).coerceAtMost(MAX_BACKOFF_MS)
Log.d(TAG, "Reconnecting in ${backoff}ms (attempt $attempt)")
scope.launch {
delay(backoff)
if (active) openSocket()
}
}
// ─── WebSocket Listener ───────────────────────────────────────────────
private inner class Listener : WebSocketListener() {
override fun onOpen(webSocket: WebSocket, response: Response) {
Log.d(TAG, "WebSocket opened")
reconnectAttempts.set(0)
val sid = sessionId.get()
if (sid != null) {
webSocket.send(encodeResume(sid, lastSeqId.get()))
} else {
val setup = pendingSetup ?: return
webSocket.send(encodeSetup(setup.model, setup.voice, setup.conversationId))
}
}
override fun onMessage(webSocket: WebSocket, text: String) {
val event = parseServerEvent(text) ?: return
when (event) {
is ServerEvent.Ready -> {
sessionId.set(event.sessionId)
}
is ServerEvent.Resumed -> {
sessionId.set(event.sessionId)
}
is ServerEvent.Audio -> {
lastSeqId.set(maxOf(lastSeqId.get(), event.seq))
}
is ServerEvent.Text -> {
lastSeqId.set(maxOf(lastSeqId.get(), event.seq))
}
is ServerEvent.Backpressure -> {
chunkIntervalMs = if (event.throttleMs > 0) event.throttleMs.toLong() else 100L
}
is ServerEvent.GoAway -> {
if (event.reason == "shutdown") {
scope.launch {
delay(500)
if (active) openSocket()
}
}
}
is ServerEvent.Error -> {
val terminal = event.code in setOf(
"auth_failed", "restricted_model", "restricted_feature",
"budget_exceeded", "concurrent_limit",
)
if (terminal) active = false
}
else -> {}
}
scope.launch { _events.emit(event) }
}
override fun onClosing(webSocket: WebSocket, code: Int, reason: String) {
webSocket.close(code, reason)
}
override fun onClosed(webSocket: WebSocket, code: Int, reason: String) {
Log.d(TAG, "WebSocket closed: $code $reason")
ws.compareAndSet(webSocket, null)
if (active && code != 1000) scheduleReconnect()
}
override fun onFailure(webSocket: WebSocket, t: Throwable, response: Response?) {
Log.w(TAG, "WebSocket failure: ${t.message}")
ws.compareAndSet(webSocket, null)
if (active) scheduleReconnect()
}
}
}

View file

@ -0,0 +1,156 @@
package dev.kaizen.app.live
import kotlinx.serialization.Serializable
import kotlinx.serialization.json.Json
import kotlinx.serialization.json.JsonObject
import kotlinx.serialization.json.boolean
import kotlinx.serialization.json.booleanOrNull
import kotlinx.serialization.json.double
import kotlinx.serialization.json.doubleOrNull
import kotlinx.serialization.json.int
import kotlinx.serialization.json.intOrNull
import kotlinx.serialization.json.jsonObject
import kotlinx.serialization.json.jsonPrimitive
import kotlinx.serialization.json.long
import kotlinx.serialization.json.longOrNull
private val json = Json { ignoreUnknownKeys = true; encodeDefaults = false }
// ─── Client → Server ──────────────────────────────────────────────────────
@Serializable
data class SetupMessage(
val type: String = "setup",
val model: String,
val voice: String,
val conversationId: String? = null,
val tools: Boolean? = null,
)
@Serializable
data class ResumeMessage(
val type: String = "resume",
val sessionId: String,
val lastSeqId: Int? = null,
)
@Serializable
data class AudioMessage(val type: String = "audio", val data: String)
@Serializable
data class TextMessage(val type: String = "text", val text: String)
@Serializable
data class SimpleMessage(val type: String)
fun encodeSetup(model: String, voice: String, conversationId: String?, tools: Boolean? = true): String =
json.encodeToString(SetupMessage.serializer(), SetupMessage(model = model, voice = voice, conversationId = conversationId, tools = tools))
fun encodeResume(sessionId: String, lastSeqId: Int?): String =
json.encodeToString(ResumeMessage.serializer(), ResumeMessage(sessionId = sessionId, lastSeqId = lastSeqId))
fun encodeAudio(base64Pcm: String): String =
json.encodeToString(AudioMessage.serializer(), AudioMessage(data = base64Pcm))
fun encodeText(text: String): String =
json.encodeToString(TextMessage.serializer(), TextMessage(text = text))
fun encodeInterrupt(): String = """{"type":"interrupt"}"""
fun encodeEnd(): String = """{"type":"end"}"""
// ─── Server → Client ──────────────────────────────────────────────────────
data class LiveUsage(
val inputTokens: Int = 0,
val outputTokens: Int = 0,
val cost: Double? = null,
val durationMs: Long = 0,
)
sealed interface ServerEvent {
data class Ready(val sessionId: String) : ServerEvent
data class Resumed(val sessionId: String) : ServerEvent
data class Audio(val data: String, val seq: Int) : ServerEvent
data class Text(val delta: String, val seq: Int) : ServerEvent
data class Transcript(val text: String, val final: Boolean) : ServerEvent
data class Vad(val speaking: Boolean) : ServerEvent
data object Interrupted : ServerEvent
data class TurnEnd(
val usage: LiveUsage,
val userText: String,
val assistantText: String,
val userMessageId: String? = null,
val assistantMessageId: String? = null,
) : ServerEvent
data class ToolCall(val name: String, val status: String) : ServerEvent
data class Backpressure(val throttleMs: Int) : ServerEvent
data class GoAway(val remainingMs: Int, val reason: String) : ServerEvent
data class Error(val code: String, val message: String?) : ServerEvent
}
fun parseServerEvent(raw: String): ServerEvent? {
val obj: JsonObject
try {
obj = json.parseToJsonElement(raw).jsonObject
} catch (_: Exception) {
return null
}
val type = obj["type"]?.jsonPrimitive?.content ?: return null
return when (type) {
"ready" -> ServerEvent.Ready(
sessionId = obj["sessionId"]?.jsonPrimitive?.content ?: return null,
)
"resumed" -> ServerEvent.Resumed(
sessionId = obj["sessionId"]?.jsonPrimitive?.content ?: return null,
)
"audio" -> ServerEvent.Audio(
data = obj["data"]?.jsonPrimitive?.content ?: return null,
seq = obj["seq"]?.jsonPrimitive?.intOrNull ?: 0,
)
"text" -> ServerEvent.Text(
delta = obj["delta"]?.jsonPrimitive?.content ?: return null,
seq = obj["seq"]?.jsonPrimitive?.intOrNull ?: 0,
)
"transcript" -> ServerEvent.Transcript(
text = obj["text"]?.jsonPrimitive?.content ?: "",
final = obj["final"]?.jsonPrimitive?.booleanOrNull ?: false,
)
"vad" -> ServerEvent.Vad(
speaking = obj["speaking"]?.jsonPrimitive?.booleanOrNull ?: false,
)
"interrupted" -> ServerEvent.Interrupted
"turn_end" -> {
val u = obj["usage"]?.jsonObject
ServerEvent.TurnEnd(
usage = LiveUsage(
inputTokens = u?.get("inputTokens")?.jsonPrimitive?.intOrNull ?: 0,
outputTokens = u?.get("outputTokens")?.jsonPrimitive?.intOrNull ?: 0,
cost = u?.get("cost")?.jsonPrimitive?.doubleOrNull,
durationMs = u?.get("durationMs")?.jsonPrimitive?.longOrNull ?: 0,
),
userText = obj["userText"]?.jsonPrimitive?.content ?: "",
assistantText = obj["assistantText"]?.jsonPrimitive?.content ?: "",
userMessageId = obj["userMessageId"]?.jsonPrimitive?.content,
assistantMessageId = obj["assistantMessageId"]?.jsonPrimitive?.content,
)
}
"tool_call" -> ServerEvent.ToolCall(
name = obj["name"]?.jsonPrimitive?.content ?: "",
status = obj["status"]?.jsonPrimitive?.content ?: "",
)
"backpressure" -> ServerEvent.Backpressure(
throttleMs = obj["throttleMs"]?.jsonPrimitive?.intOrNull ?: 0,
)
"go_away" -> ServerEvent.GoAway(
remainingMs = obj["remainingMs"]?.jsonPrimitive?.intOrNull ?: 0,
reason = obj["reason"]?.jsonPrimitive?.content ?: "",
)
"error" -> ServerEvent.Error(
code = obj["code"]?.jsonPrimitive?.content ?: "unknown",
message = obj["message"]?.jsonPrimitive?.content,
)
else -> null
}
}

View file

@ -99,6 +99,7 @@ data class StoredMessage(
val id: String,
val role: String,
val content: String = "",
val parentId: String? = null,
val attachments: List<Attachment> = emptyList(),
val reasoning: String? = null,
val sources: List<SearchSource>? = null,
@ -107,7 +108,17 @@ data class StoredMessage(
val outputTokens: Int? = null,
)
@Serializable private data class ConversationDetail(val messages: List<StoredMessage> = emptyList())
@Serializable private data class ConversationDetail(
val messages: List<StoredMessage> = emptyList(),
val activeRootId: String? = null,
val activeChild: Map<String, String> = emptyMap(),
)
data class ConversationData(
val messages: List<StoredMessage> = emptyList(),
val activeRootId: String? = null,
val activeChild: Map<String, String> = emptyMap(),
)
/** A message to persist. `parentId` chains the linear history; `kind` is always "chat" for the app. */
@Serializable
@ -309,6 +320,9 @@ object KaizenApi {
/** GET /api/v1/conversations/[id] — the stored messages, in order (empty on failure/locked). */
suspend fun fetchMessages(baseUrl: String, token: String, id: String): List<StoredMessage> =
fetchConversationData(baseUrl, token, id).messages
suspend fun fetchConversationData(baseUrl: String, token: String, id: String): ConversationData =
withContext(Dispatchers.IO) {
val req = Request.Builder()
.url("$baseUrl/api/v1/conversations/$id")
@ -318,13 +332,14 @@ object KaizenApi {
client.newCall(req).execute().use { resp ->
if (!resp.isSuccessful) {
Log.w(TAG, "fetchMessages($id): HTTP ${resp.code}")
return@use emptyList()
return@use ConversationData()
}
json.decodeFromString<ConversationDetail>(resp.body!!.string()).messages
val detail = json.decodeFromString<ConversationDetail>(resp.body!!.string())
ConversationData(detail.messages, detail.activeRootId, detail.activeChild)
}
} catch (e: Exception) {
Log.w(TAG, "fetchMessages($id): ${e.javaClass.simpleName}: ${e.message}")
emptyList()
ConversationData()
}
}
@ -348,17 +363,20 @@ object KaizenApi {
}
}
/** PATCH /api/v1/conversations/[id] — update title, locked, pinned. */
/** PATCH /api/v1/conversations/[id] — update title, locked, pinned, activeRootId, activeChild. */
suspend fun patchConversation(baseUrl: String, token: String, id: String, body: Map<String, Any?>): Boolean =
withContext(Dispatchers.IO) {
val payload = json.encodeToString(kotlinx.serialization.json.JsonObject(
body.mapValues { (_, v) ->
when (v) {
fun toJsonElement(v: Any?): kotlinx.serialization.json.JsonElement = when (v) {
is String -> kotlinx.serialization.json.JsonPrimitive(v)
is Boolean -> kotlinx.serialization.json.JsonPrimitive(v)
is Map<*, *> -> kotlinx.serialization.json.JsonObject(
v.entries.associate { (k, v2) -> k.toString() to toJsonElement(v2) }
)
null -> kotlinx.serialization.json.JsonNull
else -> kotlinx.serialization.json.JsonNull
}
}
val payload = json.encodeToString(kotlinx.serialization.json.JsonObject(
body.mapValues { (_, v) -> toJsonElement(v) }
)).toRequestBody(jsonMedia)
val req = Request.Builder()
.url("$baseUrl/api/v1/conversations/$id")
@ -414,6 +432,9 @@ object KaizenApi {
/** GET /api/v1/conversations/[id] with unlock token — fetch messages of a locked conversation. */
suspend fun fetchLockedMessages(baseUrl: String, token: String, id: String, unlockToken: String): List<StoredMessage> =
fetchLockedConversationData(baseUrl, token, id, unlockToken).messages
suspend fun fetchLockedConversationData(baseUrl: String, token: String, id: String, unlockToken: String): ConversationData =
withContext(Dispatchers.IO) {
val req = Request.Builder()
.url("$baseUrl/api/v1/conversations/$id")
@ -424,13 +445,14 @@ object KaizenApi {
client.newCall(req).execute().use { resp ->
if (!resp.isSuccessful) {
Log.w(TAG, "fetchLockedMessages($id): HTTP ${resp.code}")
return@use emptyList()
return@use ConversationData()
}
json.decodeFromString<ConversationDetail>(resp.body!!.string()).messages
val detail = json.decodeFromString<ConversationDetail>(resp.body!!.string())
ConversationData(detail.messages, detail.activeRootId, detail.activeChild)
}
} catch (e: Exception) {
Log.w(TAG, "fetchLockedMessages($id): ${e.javaClass.simpleName}: ${e.message}")
emptyList()
ConversationData()
}
}

View file

@ -22,6 +22,8 @@ object KaizenIcons {
// ── Navigation ──────────────────────────────────────────────────────────
val ArrowLeft: ImageVector get() = Lucide.ArrowLeft
val ChevronDown: ImageVector get() = Lucide.ChevronDown
val ChevronLeft: ImageVector get() = Lucide.ChevronLeft
val ChevronRight: ImageVector get() = Lucide.ChevronRight
val ChevronUp: ImageVector get() = Lucide.ChevronUp
val Menu: ImageVector get() = KaizenCustomIcons.Menu
val MoreVertical: ImageVector get() = Lucide.EllipsisVertical
@ -95,8 +97,9 @@ object KaizenIcons {
val ArrowDown: ImageVector get() = Lucide.ArrowDown
val ArrowUp: ImageVector get() = Lucide.ArrowUp
// ── Audio ──────────────────────────────────────────────────────────────
// ── Audio / Call ────────────────────────────────────────────────────────
val Volume2: ImageVector get() = Lucide.Volume2
val PhoneOff: ImageVector get() = Lucide.PhoneOff
// ── Custom (brand overrides) ────────────────────────────────────────────
// Add hand-drawn SVGs here as ImageVector literals when ready.

View file

@ -180,4 +180,12 @@
<string name="settings_language_system">System</string>
<string name="settings_language_de">Deutsch</string>
<string name="settings_language_en">English</string>
<!-- Live Call -->
<string name="live_connecting">Connecting…</string>
<string name="live_listening">Listening…</string>
<string name="live_ready">Just start talking</string>
<string name="live_speaking">Responding…</string>
<string name="live_end">End</string>
<string name="live_reconnecting">Reconnecting…</string>
</resources>

View file

@ -182,4 +182,12 @@
<string name="settings_language_system">System</string>
<string name="settings_language_de">Deutsch</string>
<string name="settings_language_en">English</string>
<!-- Live Call -->
<string name="live_connecting">Verbinde…</string>
<string name="live_listening">Hört zu…</string>
<string name="live_ready">Sprich einfach los</string>
<string name="live_speaking">Antwortet…</string>
<string name="live_end">Beenden</string>
<string name="live_reconnecting">Verbindung wird wiederhergestellt…</string>
</resources>