Voice-activated assistant integration for hands-free development.

Voice SDK

The Voice SDK (@codmir/voice-daemon) enables voice-activated interactions with codmir. Say "codmir" to activate, then speak your command.

Overview

The voice system consists of:

Wake Word Listener - Detects "codmir" activation phrase
Voice Daemon - WebSocket server coordinating voice pipeline
Ball Overlay - Visual feedback UI (PyQt5)
TTS Worker - Text-to-speech responses

Architecture

┌─────────────────┐     ┌─────────────────┐
│  Wake Word      │────▶│  Voice Daemon   │
│  Listener       │     │  (port 9410/11) │
└─────────────────┘     └────────┬────────┘
                                 │
                    ┌────────────┼────────────┐
                    ▼            ▼            ▼
             ┌──────────┐ ┌──────────┐ ┌──────────┐
             │  Ball    │ │  AI      │ │  TTS     │
             │  Overlay │ │  Agent   │ │  Worker  │
             └──────────┘ └──────────┘ └──────────┘

Installation

Prerequisites

Ubuntu 22.04+ or compatible Linux
Python 3.10+ with PyQt5
Node.js 20+

Setup

# Navigate to voice daemon
cd workers/voice-daemon

# Install Node dependencies
pnpm install

# Install Python dependencies for overlay
pip install PyQt5

# Build TypeScript
pnpm build

Usage

Start Services

# Using the CLI helper
codmir-activate start

# Or manually
systemctl --user start codmir-voice-daemon
systemctl --user start codmir-voice-overlay

Commands

codmir-activate start   # Start voice services
codmir-activate stop    # Stop services
codmir-activate status  # Check service status
codmir-activate test    # Run overlay in demo mode

WebSocket API

The voice daemon exposes a WebSocket server on port 9410.

Connect

const ws = new WebSocket('ws://127.0.0.1:9410');

ws.onopen = () => {
  console.log('Connected to voice daemon');
};

ws.onmessage = (event) => {
  const data = JSON.parse(event.data);
  console.log('Event:', data.type, data.payload);
};

Events

// Wake word detected
{ type: 'wake', payload: { timestamp: number } }

// Listening started
{ type: 'listening', payload: {} }

// Transcription result
{ type: 'transcript', payload: { text: string, final: boolean } }

// AI response
{ type: 'response', payload: { text: string } }

// Speaking started
{ type: 'speaking', payload: { text: string } }

// Idle (ready for next command)
{ type: 'idle', payload: {} }

// Error
{ type: 'error', payload: { message: string } }

Send Commands

// Simulate wake word (for testing)
ws.send(JSON.stringify({ type: 'simulate_wake' }));

// Send text command (skip voice)
ws.send(JSON.stringify({ type: 'command', text: 'Create a new ticket' }));

// Cancel current operation
ws.send(JSON.stringify({ type: 'cancel' }));

HTTP Webhook

Wake word detection posts to http://127.0.0.1:9411/webhook/wakeword:

curl -X POST http://127.0.0.1:9411/webhook/wakeword

Ball Overlay States

The floating ball UI indicates assistant state:

State	Appearance
`idle`	Subtle pulse, gray/blue
`listening`	Active pulse, bright blue
`thinking`	Spinning animation
`speaking`	Ripple effect
`error`	Red flash

Configuration

Environment variables:

# Voice daemon
CODMIR_VOICE_WS_PORT=9410
CODMIR_VOICE_HTTP_PORT=9411
CODMIR_AI_ENDPOINT=http://localhost:3000/api/voice

# Overlay
CODMIR_OVERLAY_POSITION=bottom-right
CODMIR_OVERLAY_SIZE=64

Systemd Services

Voice Daemon Service

# ~/.config/systemd/user/codmir-voice-daemon.service
[Unit]
Description=codmir Voice Daemon
After=network.target

[Service]
Type=simple
WorkingDirectory=/path/to/workers/voice-daemon
ExecStart=/usr/bin/node dist/index.js
Restart=on-failure

[Install]
WantedBy=default.target

Overlay Service

# ~/.config/systemd/user/codmir-voice-overlay.service
[Unit]
Description=codmir Voice Overlay
After=codmir-voice-daemon.service

[Service]
Type=simple
WorkingDirectory=/path/to/workers/voice-daemon/overlay
ExecStart=/usr/bin/python3 ball.py
Restart=on-failure
Environment=DISPLAY=:0

[Install]
WantedBy=default.target

Integration with Desktop

The voice assistant can control:

Terminal - Run commands
Browser - Open URLs
File Manager - Navigate files
codmir App - Create tickets, run builds

// Example: Voice command handling
daemon.on('command', async (text) => {
  if (text.includes('create ticket')) {
    await createTicket(text);
  }
  if (text.includes('run tests')) {
    await runTests();
  }
});

Troubleshooting

Daemon Won't Start

# Check logs
journalctl --user -u codmir-voice-daemon -f

# Check port availability
lsof -i :9410

Overlay Not Showing

# Check X11 display
echo $DISPLAY

# Run overlay manually
cd workers/voice-daemon/overlay
python3 ball.py

Wake Word Not Detecting

Ensure the wake word listener is running and configured for "codmir" activation phrase.