H HospAir

Operations Runbook

Day-to-day studio use, streams, deploys, troubleshooting, and backups.

🛠 Operations Runbook

Day-to-day care and feeding of a HospAir station: written for volunteers.

Everything here assumes the stack from the main README is installed. For first-time production setup see NGINX-PROXY.md and CLOUDFLARE-DNS.md.

Jump to: Host requirements · Quick start · Daily studio use · Streams & encoders · Deploying updates · Troubleshooting · Backups · Maintenance

💻 Host requirements

This stack needs at least 4 GB RAM, or 2 GB RAM + 2 GB swap. Without swap, docker compose build can push load above 80 and trigger an out-of-memory reboot.

Add swap once on the host:

sudo fallocate -l 2G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab

On small servers, build images one at a time and free disk after every build , old layers can fill a 38 GB disk and make the next build OOM:

COMPOSE_PARALLEL_LIMIT=1 docker compose build web && ./scripts/docker-post-build-cleanup.sh
COMPOSE_PARALLEL_LIMIT=1 docker compose build worker && ./scripts/docker-post-build-cleanup.sh
COMPOSE_PARALLEL_LIMIT=1 docker compose build liquidsoap && ./scripts/docker-post-build-cleanup.sh

Or npm run docker:build (builds everything, then prunes cache automatically). The cleanup script removes build cache and unused images only, it never touches volumes, so the database and audio library are safe.

🚀 Quick start (local dev)

cp .env.example .env
COMPOSE_PARALLEL_LIMIT=1 docker compose up -d --build

Open http://localhost:3000 and sign in:

Browser live (WebRTC bridge) is optional and off by default:

docker compose --profile live up -d webrtc-bridge

For production use the nginx overlay so only nginx binds host ports (80, 443, 8000, 8001, 3333, 8005), see NGINX-PROXY.md:

docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d --build

🎙 Daily studio use

  1. Sign in and open Studio
  2. Search the library and add tracks to the queue, or let the scheduled clocks build the hour
  3. Drag audio files into the Media Browser upload area (or Library), the worker ingests them; add READY tracks to the log
  4. Fire jingles/adverts on top of the current song: Media Wall (click a cart) or Library → Jingles/Adverts → On air (or double-click). Automation keeps playing underneath, ducked
  5. Pause is instant; remote-DJ takeover crossfades in/out (no longer waits for the track to end)
  6. Use Emergency Stop only when needed

Presenter headphones: YOU is instant foldback of your own voice; MONITOR is program audio only (~200 ms delay, no mic echo). Music ducking on air follows your voice automatically, you never press a duck button.

📡 Streams & encoders

Listener URLs

Icecast admin (listener stats) is on the same hosts/ports under /admin/ (user admin, password from .env ICECAST_*_ADMIN_PASS, default hackme).

BUTT / external encoder (remote DJs)

Each presenter who broadcasts remotely gets their own harbor login, so you know who's on air and shows can fade the right DJ in automatically.

  1. Admin → Users → enable Remote broadcast for the presenter. A personal harbor username + password is generated and shown there (rotate or disable any time).
  2. The DJ points BUTT/Mixxx at:
Setting Value
Server your server IP / hostname (connect.YOURHOST in production, DNS-only record)
Port 8005 (nginx TCP proxy → Liquidsoap Harbor)
Mount /live
Icecast user their personal remote login (Admin → Users)
Password their personal remote password (Admin → Users)

Leave “Use legacy Icecast protocol” unchecked: the username is only sent on the modern Icecast source protocol (legacy/Shoutcast drops it, so per-user auth can't work).

The station-wide Harbor password (Admin → Broadcast) still works as a shared break-glass login if the web app is unreachable, but it's anonymous, it won't trigger the automatic show fade-in.

Going on air with a show

Make a show remote-aware in Admin → Scheduling → Shows → (show) → Remote DJs: toggle Allow remote DJs and pick which presenters may take it. Then:

  • A permitted DJ can connect early: they stay silent until their show's hour begins.
  • When the show starts (and they're connected) they fade on air automatically; if they drop or the show ends, playout fades back to the generated log.
  • If no permitted DJ is connected, the show's automation log just plays as normal.
  • The operator can always force a manual takeover with REMOTE DJ in Studio, the scheduled evaluator never overrides a manual session or an emergency lock.

Connected DJs appear as pulsing avatars in the studio header (red = on air, amber = waiting for their show); the DJ themselves sees a countdown on their dashboard.

The browser live mic uses the WebRTC bridge (WEBRTC_BRIDGE_PUBLIC_URL, port 3333 / live.YOURHOST), not port 8005.

🚢 Deploying updates

On deploy, the migrate service runs prisma migrate deploy before web/worker start. Whenever the schema changed, rebuild migrate first or /api/station will 500:

docker compose build migrate && docker compose run --rm migrate
docker compose up -d

P3009: failed migration

If migrate exits with P3009 and logs like relation "…" already exists, the SQL partially applied on a previous attempt and Prisma refuses to continue.

When the schema objects already exist in Postgres, mark the migration applied:

# repeat per migration name from the error
docker compose run --rm migrate sh -c \
 "cd packages/db && npx prisma migrate resolve --applied 20260601200000_show_prep_polish"

# confirm: should print "No pending migrations to apply."
docker compose run --rm migrate sh -c "npm run migrate -w @hospair/db"
docker compose up -d

If objects are genuinely missing (migration failed mid-way), use prisma migrate resolve --rolled-back <name> instead, then run migrate deploy again so Prisma re-applies the SQL, only when you are sure the DB is incomplete.

🩺 Troubleshooting

Icecast shows no mounts / sources

Liquidsoap must connect to /public and /wards:

docker compose logs liquidsoap | grep -i mount

You should see Connecting mount /public and Connection setup was successful. If you see literal names like ICECAST_PUBLIC_MOUNT, a script change reversed getenv arguments (Liquidsoap uses getenv(default, name)). Fix and docker compose restart liquidsoap.

No audio on stream

  1. Admin → Health: all services OK?
  2. Library has READY tracks?
  3. At least one row in the log/queue?
  4. docker compose restart liquidsoap

Browser live not working

  1. Allow microphone access in the browser
  2. curl http://localhost:3333/health
  3. docker compose restart webrtc-bridge

ARM MIC: ducking but no voice (or voice very late)

Ducking follows the same Harbor path as your voice on air, so duck-with-no-speech means the audio path broke after the VAD:

  • Rebuild after fixes: docker compose up -d --build liquidsoap webrtc-bridge web
  • On MONITOR you never hear yourself, that's intentional; use YOU for foldback
  • First link works, second phrase after ~30 s silence doesn't (YOU fine, air not): Liquidsoap dropped the Harbor feed (input.harbor default 30 s timeout), rebuild liquidsoap + webrtc-bridge, hard-refresh Studio
  • First phrase works, second immediately doesn't: rebuild webrtc-bridge (an old build paused the WebSocket while Harbor was busy and never resumed)

Upload stuck on PROCESSING

  1. docker compose logs worker
  2. Confirm ffmpeg is available inside the worker container
  3. Reprocess from the library API, or npm run library:unstick-processing

Queue out of sync (desync banner)

The engine pulls its queue one row at a time from the control server, so the old "stale pushed buffer" problem class is gone, at most one prefetched row can be stale after a log edit, and routine reconciliation corrects the cursor automatically. A desync banner means the log row marked on-air and the audio Liquidsoap reports disagree. Typical causes:

  • A skip/resync raced metadata that was still settling (clears itself in seconds)
  • A delayed webhook reported an older row
  • Audio is playing a row that was removed from the log

First response:

  1. If a cart is playing over a song, wait, cart overlays intentionally freeze the log row
  2. If the banner clears within a few seconds, it was transitional; do nothing
  3. If it stays, press Resync once, it hard-aligns audio to the current log head
  4. Don't hammer Skip to fix desync; each skip really does advance a row
  5. Background reconcile is paused while Studio is open, so manual edits aren't overwritten

If Resync doesn't recover:

docker compose logs --tail=100 web worker liquidsoap
docker compose restart liquidsoap

Then confirm in Studio: Now/Next matches the highlighted row, the audio matches it, and the next queued row is the intended one.

Top of hour (terminal fade)

Governed by Strict top-of-hour lock (Admin → Playout/Timing), it runs in full automation whether the studio is open or not:

  • FADE: during :59, upcoming rows are trimmed and a crossfade skip is scheduled so the next hour starts exactly at :00
  • CUT: hard handoff at :00 (flush + skip)
  • OFF: the on-air track may overrun :00 up to a grace window (STRICT_OFF_OVERRUN_GRACE_SEC, default 90 s; 0 = let it run), then fades
  • Tune the arm window (default 20 s) and tolerance (default 2 s) in Admin → Timing

The remaining-time input for this scheduling comes from the engine's wall-clock playhead (track start events + cue lengths), no polling of Liquidsoap internals.

Dead-air protection (standby fallback)

A standby source engages automatically when the automation + mic bus falls silent unexpectedly for longer than DEAD_AIR_SECONDS (default 12 s). Intentional silence (Emergency Stop, pause, remote encoder takeover) does not trigger it. Detection is event-driven: standby tracks are tagged, and the track event flips the state.

  • Standby content lives in FALLBACK_AUDIO_PATH (default /data/audio/fallback). The worker auto-seeds it every ~10 min with symlinks to up to 24 READY MUSIC tracks, so it's never empty once the library has content. You can drop your own .mp3/.ogg files in; they reload without a restart
  • While standby is on air: Studio shows a red DEAD-AIR PROTECTION ACTIVE banner, a critical dead_air alert is raised, and the worker is kicked to refill the log
  • State is exposed via the dead_air.status telnet command, GET /dead-air on the engine's harbor control port, and fallbackActive on GET /control/telemetry
# Is standby on air right now?
docker compose exec liquidsoap bash -c 'echo "dead_air.status" | nc -q1 127.0.0.1 1234'
# What's seeded as standby?
docker compose exec liquidsoap ls -l /data/audio/fallback

Engine health (pull queue)

The playout engine logs every pull and track change. A healthy engine shows a steady stream of these; silence for longer than the longest track is suspicious:

docker logs hospair-liquidsoap-1 --since 10m | grep -E "pull_queue|track-push"

To check what listeners actually hear (volume of the live stream):

docker compose exec worker sh -c \
 'ffmpeg -hide_banner -t 4 -i http://icecast-public:8000/public -af volumedetect -f null - 2>&1 | grep mean_volume'

Music sits around −10 to −20 dB; sustained readings below −40 dB mean silence or hum. Never poll Liquidsoap source state (remaining/is_ready) from outside as a health check: see the thread-safety section in liquidsoap-roadmap.md.

💾 Backups

Database

docker compose exec postgres pg_dump -U hospair hospair > hospair-backup.sql
# restore
cat hospair-backup.sql | docker compose exec -T postgres psql -U hospair hospair

Audio library

Where the audio lives depends on the storage mode (Admin → Storage):

  • Local mode (default): everything is on disk under AUDIO_STAGING_PATH (default /data/audio), back up that directory (it's a Docker volume):
docker run --rm -v hospair_audio_staging:/data -v "$PWD":/backup alpine \
tar czf /backup/hospair-audio.tgz -C /data .
  • S3 mode: the external bucket is the durable home, back it up with your provider's tooling (e.g. aws s3 sync/rclone). /data/audio-cache is only a disposable local cache and does not need backing up; it refills on demand.

Switch modes and migrate existing audio between disk and S3 from Admin → Storage (manual migrate local→S3 and pull S3→local actions). There is no MinIO service to run any more.

🔧 Maintenance

Rotate the Harbor password

  1. Update HARBOR_PASSWORD in .env
  2. Update the station record via Admin (or seed)
  3. docker compose up -d --build liquidsoap webrtc-bridge web

Stop / start / logs

docker compose down && docker compose up -d

docker compose logs -f web
docker compose logs -f worker
docker compose logs -f liquidsoap
docker compose logs -f webrtc-bridge