ganymede — operator's manual

Read this offline

This page is a single self-contained file — no external CSS, fonts or scripts. Save it anywhere and it still works with the network down, which is exactly when you'll want it.

Run this on your own machine now, while everything works. It makes ~/homelab-offline/ and pulls down this manual plus the credentials, so a dead network leaves you with both the instructions and the logins.

mkdir -p ~/homelab-offline && chmod 700 ~/homelab-offline && \
scp jcasey@192.168.1.10:/opt/homelab/compose/dashboard/html/manual.html ~/homelab-offline/ && \
scp -r jcasey@192.168.1.10:/opt/homelab/secrets/plain ~/homelab-offline/secrets && \
chmod -R go-rwx ~/homelab-offline/secrets && ls -R ~/homelab-offline

The IP is used rather than ganymede on purpose — if DNS is the thing that broke, the hostname won't resolve. Then open ~/homelab-offline/manual.html in any browser. It prints cleanly too, for the truly bad days.

What is where

One Proxmox host, two containers, everything else in Docker inside them.

Services

Everything marked lan is reachable on your LAN or over Tailscale only. The dashboard's links detect how you reached it and switch between local addresses and public hostnames automatically.

Credentials

They live under /opt/homelab/secrets/, version-controlled and sops-encrypted. Per-app API keys stay inside each app's own config directory under /appdata/.

Network & DNS — the part that bites

This is the most surprising piece of the whole setup, so it's worth understanding before you change anything.

Why AdGuard runs DHCP

j5casey.win resolves publicly to your WAN address. The AT&T gateway cannot NAT-hairpin — a device inside the house that tries to reach your own public IP hands the packet to the gateway, and the gateway refuses to turn it around. Verified: TCP to the WAN IP times out from every LAN host while working fine from the internet.

The fix is split-horizon DNS. On the LAN, j5casey.win resolves to 192.168.1.11 and Caddy serves it directly with a valid certificate. For that, every device has to use AdGuard as its resolver — and the gateway has no DNS override field anywhere in its UI. Replacing its DHCP server was the only remaining way to hand devices a resolver.

The rules

Check who is serving DHCP:

nmap --script broadcast-dhcp-discover -e vmbr0
# every "Server Identifier" must be 192.168.1.11

Upstream resolver

AdGuard uses Quad9 over DNS-over-HTTPS, with plain 9.9.9.9 and 1.1.1.1 as fallback. The fallback exists because Quad9's DoH endpoint collapsed once and took the whole house's DNS with it — with no fallback configured, every lookup failed while j5casey.win kept resolving (rewrites are answered locally, no upstream needed).

Browsers that bypass AdGuard

Firefox enables DNS-over-HTTPS by default in the US, sending lookups straight to Cloudflare and skipping AdGuard entirely — so jellyfin.j5casey.win resolves to the public IP and fails on the LAN. The tell is that the failing name never appears in AdGuard's query log while other names from the same device do.

AdGuard answers Firefox's canary domain with NXDOMAIN, which switches DoH off network-wide:

dig use-application-dns.net @192.168.1.11   # must return NXDOMAIN

A browser restart is required — the canary is only checked at startup. If someone has set DoH to "Max Protection" it ignores the canary and must be turned off manually.

How a download actually happens

1. You (or Jellyseerr) add a title to Sonarr or Radarr
2. It asks Prowlarr, which queries NZBgeek (usenet) and IPTorrents (torrents)
3. Usenet gets first refusal. Torrents are considered 15 minutes later
4. Usenet grabs go to SABnzbd, torrents to qBittorrent
5. On completion the *arr imports it into /data/media/... as a hardlink — the file exists in both places but occupies disk once
6. Jellyfin picks it up; Bazarr hunts subtitles

Quality profiles, in plain terms

Every "quality" is two things bolted together: a source and a resolution. WEBDL-1080p means "pulled from a streaming service, 1080p". The source is what actually decides how good it looks:

Which profile a show gets depends on what can exist, not preference:

• WEB-1080p — current streaming shows (House of the Dragon, Strange New Worlds). There is no Blu-ray yet; WEB-DL is the best available
• HD 720p/1080p — finished shows with real Blu-ray remasters (TNG, Game of Thrones, Avatar TLA)
• Classic TV (any quality) — never remastered, exists only as 480p DVD (DS9, Voyager, MacGyver, Grandmother's shows). An HD-only profile would search forever and find nothing

Grandmother's library

She gets a curated subset, not the whole library — a separate Jellyfin library pointed at /data/grandmother/, built from a manifest.

/opt/homelab/grandmother-library.txt     # one path per line, relative to /data/media/

The tree is rebuilt as hardlinks, so her entire library costs about 2 MB of actual disk. Adding a show to her view duplicates nothing.

grandmother-list                      # what she can see
grandmother-add "andy griffith"       # fuzzy match; lists candidates if ambiguous
grandmother-remove "the crown"
grandmother-sync                      # rebuild the hardlink tree (runs nightly and on add)

Music — two separate libraries

Navidrome runs two libraries because the collections barely overlap and merging them would bury a 200-artist collection under a 600-artist one:

Both accounts can see both libraries and switch between them. Favourites, play counts and playlists are per-account automatically. Navidrome exposes each library to phone clients as a Subsonic "music folder", so Symfonium can browse them separately too.

retag-youtube.py --root /tank/data/media/music-karington/YouTube \
                 --uncovered /opt/homelab/spotify-uncovered-karington.json

Subtitles

Bazarr handles these, with English set as the default profile for every series and film — anything added from now on gets subtitles hunted automatically.

Many files already carry subtitles inside them (House of the Dragon has 36 tracks). Those need nothing; just pick them in Jellyfin's player. Bazarr only fetches where none exist.

Add a show or film

Easiest path is Jellyseerr (requests.j5casey.win) — it hands off to Sonarr/Radarr automatically. Otherwise add it directly in Sonarr or Radarr and pick the profile using the table above.

For a large backfill, search season by season rather than episode by episode — one query per season instead of 26, which keeps you well clear of indexer rate limits.

Add something to Grandmother's library

grandmother-add "love worth finding"

If the title isn't in TVDB or TMDB at all — as with the Adrian Rogers sermons — Jellyfin has nothing to match against and shows it bare. Local metadata fixes that: put poster.jpg, fanart.jpg and a tvshow.nfo in the series folder, with <lockdata>true</lockdata> so a later refresh doesn't wipe it.

Push something to the front of the queue

SABnzbd supports per-item priority. Setting an item to Force jumps it ahead of everything, including things queued later.

# find the items, then set priority 2 (Force) on each nzo_id
curl "http://192.168.1.11:8080/api?mode=queue&output=json&apikey=$KEY"
curl "http://192.168.1.11:8080/api?mode=queue&name=priority&value=$NZO&value2=2&apikey=$KEY"

This only reorders the queue — nothing is cancelled or re-searched.

Reclaim disk

Most torrents cost nothing to keep seeding — Sonarr hardlinks them into the library, so the seed and the library file are the same bytes. Deleting them frees zero and only costs ratio.

The exception is RAR'd scene releases: the library copy had to be extracted, so it's genuinely separate data. Those are the only reclaimable ones, and a weekly job handles them:

retire-seeded-rars.py --dry-run    # what's eligible and why
retire-seeded-rars.py --retire     # do it (cron runs this Mondays)

It deletes a torrent only when it is RAR-backed, has seeded at least 30 days, and Sonarr confirms the season is complete in the library.

Nobody has internet

Symptom: devices connect but nothing loads, or "sign in required"

1. Is DNS answering at all?

   dig @192.168.1.11 j5casey.win     # should return 192.168.1.11
   dig @192.168.1.11 example.com     # should return a public IP

2. If the first works and the second doesn't, the upstream is broken, not AdGuard. It should fall back to plain DNS automatically; check docker logs adguard for repeated upstream errors.
3. If neither works, is AdGuard even running?

   pct exec 101 -- docker ps | grep adguard
   pct exec 101 -- ss -tulnp | grep -E ':(53|67) '

4. Two DHCP servers? nmap --script broadcast-dhcp-discover -e vmbr0

dig @192.168.1.11 connectivitycheck.gstatic.com   # must NOT be 192.168.1.254
curl -u USER:PASS -X POST http://192.168.1.11:8083/control/cache_clear

One device can't reach j5casey.win

Symptom: the rest of the house is fine, but one machine can't load the internal hostnames

Almost always a stale DHCP lease. The device is still holding an address handed out by the AT&T gateway from before AdGuard took over DHCP, so it has never been told about the internal resolver. It browses the internet perfectly well, which is what makes this confusing — only the split-horizon names fail.

1. Confirm from ganymede. A device that isn't using AdGuard leaves no trace in either place:

   # is there a lease for its MAC?
   curl -s -u USER:PASS http://192.168.1.11:8083/control/dhcp/status | python3 -m json.tool
   
   # has it ever asked AdGuard anything?
   curl -s -u USER:PASS "http://192.168.1.11:8083/control/querylog?limit=500" | grep -c '192.168.1.NNN'

   No lease and zero queries together is the signature.
2. On the device itself, read what it actually believes:

   cat /etc/resolv.conf
   nmcli dev show | grep -E 'GENERAL.CONNECTION|IP4.ADDRESS|IP4.DNS'

   nameserver 192.168.1.254 or search attlocal.net means an AT&T lease. AdGuard hands out neither, so either string is proof on its own.
3. Force a clean lease. Deleting the cached one is the part that matters:

   sudo sh -c 'rm -f /var/lib/NetworkManager/*wlan0*.lease'
   sudo nmcli con down CONN && sudo nmcli con up CONN

   Use the connection name from nmcli con show --active, and the right interface (wlan0 for wifi, eth0/enp* for wired).
4. Verify — cat /etc/resolv.conf should now say 192.168.1.11, and dig j5casey.win +short should return the LAN address rather than the public one.

Something won't play in Jellyfin

Symptom: "Unable to find a valid media source to play"

Almost always codecs. YouTube-sourced files increasingly arrive as AV1 + Opus, which most clients can't decode. Check:

pct exec 101 -- docker exec jellyfin /usr/lib/jellyfin-ffmpeg/ffprobe -v error \
  -show_entries stream=codec_type,codec_name -of default=noprint_wrappers=1 "/data/path/to.mkv"

If it says av1 / opus, re-pull it preferring H.264 and AAC. Anything destined for Grandmother especially should be H.264 — it direct-plays everywhere and never troubles the iGPU.

Symptom: playback fails instantly with a bare "fatal error"

Hardware transcoding can't open the GPU. The give-away is ffmpeg exiting within milliseconds:

pct exec 101 -- docker exec jellyfin sh -c 'ls -t /config/log/*.log | head -1 | xargs grep -i "Device creation failed"'

Exit code 234 is -22 (EINVAL) from VA-API. Confirm the Jellyfin process has the render group — not just the container:

pct exec 101 -- docker exec jellyfin sh -c 'grep Groups /proc/$(pgrep -f bin/jellyfin | head -1)/status'
# must include the gid that owns /dev/dri/renderD128

Symptom: wrong artwork or synopsis

Jellyfin matched the wrong title — common where a remake shares the name (Avatar: The Last Airbender 2005 vs 2024). Put the year in the folder name, then re-identify the series in Jellyfin.

Episodes look missing but probably aren't

Symptom: a completed show shows e.g. 174/176

Those two are almost always specials — Season 0 retrospectives, syndication re-cuts, behind-the-scenes clips. TVDB counts them; Sonarr shows them in the total but leaves them unmonitored so they're never chased. I Love Lucy has 651 of them against 193 real episodes.

To see real gaps, count only season > 0 and monitored+aired episodes. Genuinely missing episodes are usually usenet retention on old shows — re-search and let torrents take them.

Downloads finish but never appear

Symptom: torrent at 100% and seeding, library empty

Scene RAR release. Sonarr will say completed / importBlocked, reason Sample:

extract-torrent-rars.py --scan          # what needs unpacking
extract-torrent-rars.py --extract-all   # unpack, leaving the seed untouched

Then import the extracted tree. The torrent's own files are never touched, so seeding continues.

chown -R 1000:1000 /tank/data/torrents/extracted

Symptom: downloads complete but sit in /data/usenet/complete forever

Import never ran. DownloadedEpisodesScan does not recurse into release folders that contain a nested folder of the same name, so the obvious fix silently does nothing:

import-orphans.py --dry-run   # what's importable
import-orphans.py --import

This once recovered 309 episodes / 462 GB that were already on disk.

Symptom: everything says "downloadClientUnavailable"

The *arr apps can't authenticate to qBittorrent. Because they all reach it through the same Docker bridge gateway, one stale password locks out everything, including your browser. Test the download clients in each app; qBittorrent's ban threshold is raised so this shouldn't recur.

Symptom: nothing grabs after an outage

Prowlarr benches indexers for hours after failures, and Sonarr, Radarr and Lidarr each keep their own separate failure state. Clearing Prowlarr alone is not enough — test the indexers in all four.

After a power cut

Both containers have onboot: 1 and come up automatically — the whole stack was live 28 seconds after boot last time. What takes minutes is the host itself: POST, ZFS import, Proxmox init. If you check from a phone during that window it looks dead when it isn't.

1. Check from a machine that's already online: dig @192.168.1.11 j5casey.win
2. Devices with an existing lease keep working — leases are 7 days and survive reboots. Only devices asking for a fresh lease are affected
3. If you fell back to AT&T DHCP in the meantime: turn it off again, then flush AdGuard's cache (see above)
4. Long-running jobs (yt-dlp fetches, imports) do not survive — restart them. Most are resumable and pick up where they stopped

Scripts

All in /opt/homelab/scripts/. Every one takes --dry-run or --help where it could do damage.

Scheduled jobs

Logs: /var/log/adguard-watchdog.log, /var/log/retire-seeded-rars.log, /var/log/fetch-lwf.log, /var/log/cloudflare-ddns.log.

Disk layout

/tank/data/                 -> /data inside CT 101
  media/tv/                 TV, one folder per show
  media/movies/             films
  media/music/              Jack's music
  media/music-karington/    Karington's music
  grandmother/tv|movies/    hardlinks to her curated subset
  usenet/complete/          SABnzbd finished downloads
  torrents/complete/        qBittorrent data (keep — this is what seeds)
  torrents/extracted/       unpacked scene RARs, staging only
  imports/                  Spotify exports and other one-off inputs

/tank/appdata/              -> /appdata  (every app's config)
/rpool/scratch/             -> /scratch  (SAB unpack scratch, on NVMe)
/opt/homelab/               git repo: compose files, scripts, docs, secrets

Gotchas worth remembering

• Hardlinks mean deleting a torrent usually frees nothing. Check the link count before assuming you'll reclaim space.
• Specials inflate every episode count. A show at 174/176 is almost certainly complete.
• Sonarr reports success it didn't have. A ManualImport can say "completed — imported 4 files" while every file failed. Check the log, not the command result.
• qBittorrent 5.x names its cookie QBT_SID_<port>, not SID. Scripts matching SID exactly read a successful login as a failure.
• YouTube 403s are usually throttling, not blocking. Wider pacing and retries fix them. A genuine block shows as "This video is not available" on every client except tv, which admits "DRM protected".
• Never forward RCON (25575). Minecraft's game port is fine; RCON is remote command execution.
• IPTorrents only. Public trackers (The Pirate Bay, YTS) are disabled in Prowlarr on purpose, and DHT/PeX/LSD are off in qBittorrent because private trackers require it.