Voice Assistant

A local voice-control surface for a self-hosted homelab. Spoken commands are captured by an M5Stack Atom Echo S3R, transcribed by a local Whisper instance, and routed through Home Assistant to a FastAPI orchestrator that dispatches work to Gemini Flash (for intent) and a Claude CLI subprocess (for live homelab queries).

Atom Echo S3R ──Wyoming──► Home Assistant ──► Orchestrator (Gemini Flash)
                                │                    │
                                │              ┌─────┴──────┐
                          Whisper (STT)    homelab_query   desk_leds
                           Piper (TTS)    (claude CLI)    speaker_power

Hardware

Item	Notes
M5Stack Atom Echo S3R	ESP32-S3, ES8311 codec, 8 MB PSRAM
Always-on Linux host	Runs HA, Whisper, Piper, orchestrator

Components

ESPHome firmware (atom-echo-s3r.yaml)

Flashes the Atom Echo S3R with the Wyoming voice assistant protocol. Press the front button once to start listening; VAD detects end-of-speech so you don't need to hold.

esphome run atom-echo-s3r.yaml

Create secrets.yaml (gitignored) with:

wifi_ssid: "..."
wifi_password: "..."
api_encryption_key: "..."   # generate with esphome generate-key
ota_password: "..."
fallback_password: "..."

Orchestrator (orchestrator/)

FastAPI service that exposes an OpenAI-compatible /v1/chat/completions endpoint. Home Assistant's custom conversation agent calls it; it runs a Gemini Flash tool-call loop and dispatches work to the tools below.

Copy .env.example to .env and fill in your keys:

cp orchestrator/.env.example orchestrator/.env

Tools

Tool	What it does
homelab_query	Runs claude --print --dangerously-skip-permissions as a subprocess with homelab context injected. If a response would exceed ~25 s, replies immediately and pushes the result via Pushbullet.
desk_leds	Controls a WS2812B LED strip via GET http://desk-leds.home/set. Supports animations, brightness, speed, glitter overlays, and audio-reactive mode.
speaker_power	Turns the Sony receiver / desk speakers on/off via Home Assistant's Matter integration (the Kasa plug's TPAP firmware isn't python-kasa-drivable). Uses HA_TOKEN + SPEAKER_SWITCH_ENTITY.
web_search	Answers current/real-world questions via a separate search-grounded Gemini call.
play_music	Plays a Navidrome album/playlist in AJ's browser tab. Resolves the request via the Subsonic API (navidrome.py) and drives the web player through the browser extension over a localhost WebSocket.
control_playback	Pause / resume / next / prev for the music playing in the browser.
set_alarm / cancel_alarm / skip_next_alarm / alarm_status	Runtime morning alarm owned by the orchestrator (alarm.py). At the set time it turns the speakers on, wakes and unlocks the screen, and plays the album through the browser. Reschedule by voice ("wake me at 6:30", "skip tomorrow") — no rebuild.
set_bedtime / cancel_bedtime / skip_next_bedtime / bedtime_status	Nightly bedtime wind-down (bedtime.py): pauses the music, turns speakers + LEDs off, and locks/blanks the screen — without suspending (the box stays up so HA and the alarm keep running). Shares the scheduler with the alarm.
list_tasks	Returns recent homelab query tasks from a local SQLite registry so you can ask "what's running?"

Editing the alarm & bedtime without voice

The alarm and bedtime keep their live schedule in JSON state files you can edit by hand instead of using voice. Edit, then systemctl restart voice-orchestrator to load it. (Voice commands rewrite these files, so anything you say afterwards wins.)

• ~/.local/share/voice-assistant/alarm.json — { "enabled", "time", "album", "skip_next" }
• ~/.local/share/voice-assistant/bedtime.json — { "enabled", "time", "skip_next" }

time is 24-hour HH:MM. Both files are created on first orchestrator start. The defaults used to seed a fresh file (or if you delete one) live in code — orchestrator/alarm.py (06:00, the flamingos album) and orchestrator/bedtime.py (23:00); changing those needs a voice-orchestrator restart too.

Task registry

Every homelab_query call is persisted to ~/.local/share/voice-assistant/tasks.db (SQLite). Statuses: pending → running → done/failed.

Browser extension (browser-extension/)

Firefox extension that lets the orchestrator drive the Navidrome (later Jellyfin) web player in AJ's own tab, so playback is seamless with his normal browsing rather than a headless player. It's a persistent WebSocket client to the orchestrator; see browser-extension/README.md and project-memory/browser-playback-spec.md.

Ownership boundary: orchestrator-driven behaviour lives here as Python/JS, never as a script in nixos-config. nixos-config is touched only for a new system package or a systemd unit. See project-memory/browser-playback-spec.md.

Home Assistant custom component (ha-custom-components/orchestrator_conversation/)

Packaged as a NixOS derivation (modules/home-assistant.nix). Proxies HA's conversation pipeline to the orchestrator and passes conversation_id so follow-up button presses have context.

NixOS setup

The nixos-config repo declares all services:

• modules/home-assistant.nix — HA + Wyoming STT/TTS + the custom component
• modules/orchestrator.nix — systemd service for the FastAPI orchestrator

STT: wyoming-faster-whisper (model small-int8, CPU, port 10300)
TTS: wyoming-piper (voice en_US-lessac-medium, port 10200)

Running the orchestrator locally

cd orchestrator
pip install fastapi uvicorn httpx python-dotenv
uvicorn main:app --host 127.0.0.1 --port 8001

Conversation history

In-memory per conversation_id, capped at 10 turn-pairs. Expires on service restart. Persistent JSONL storage is planned.

Troubleshooting

Device not reachable / no logs from today

Symptom: HA logs show Can't connect to ESPHome API for atom-echo-s3r @ <ip>: Connect call failed, button does nothing, no orchestrator traffic.

Diagnosis:

# Check if device is on the network
ping atom-echo-s3r.local

# Watch HA for connection errors
journalctl -u home-assistant -f | grep -i "atom-echo\|esphome"

# Connect to serial to see boot output
esphome logs atom-echo-s3r.yaml --device /dev/ttyACM0

You may need sg dialout first if your shell session predates your dialout group membership.

Crash loop (serial port opens then immediately closes): the firmware is corrupted, most likely from a power cut during an NVS flash write. Fix by reflashing:

esphome run atom-echo-s3r.yaml --device /dev/ttyACM0

Prevention: power the Atom Echo from a dedicated 5V wall adapter, not a computer USB port. Computer USB is subject to power management and can cut power unexpectedly mid-write.

stt-no-text-recognized in ESPHome logs

Whisper received audio but couldn't transcribe it. Usually means the mic picked up silence or background noise rather than speech. Speak closer to the device and try again. If it persists, check that wyoming-faster-whisper is running: systemctl status wyoming-faster-whisper-en-us.