GPU Server Setup & LXC Passthrough

✅ RESOLVED (2026-08-03): the RTX A4000 outage (GSP boot timeout, ~15:38–15:45) was recovered without a reboot via a targeted PCI function-level reset (FLR) — echo 1 > /sys/bus/pci/devices/0000:65:00.0/reset. Full story in Known Pitfalls #6.

Hardware

Node: workhorse (gpu01) — 192.168.0.250
CPU: Threadripper, 128GB RAM
Driver: 580.126.18 (installed via .run file, DKMS enabled)
CUDA Version: 13.0 (verified 2026-08-03)

All 3 GPUs enumerated and healthy as of 2026-08-03 15:45. Note: nvidia-smi order (5060 Ti, 5060 Ti, A4000) differs from CUDA/PyTorch order inside CT 111 — PyTorch sees cuda:0 = A4000, cuda:1/2 = 5060 Ti (verified with the ComfyUI venv). CUDA_VISIBLE_DEVICES=0 in the comfyui-gpu0 unit therefore maps to the A4000, despite nvidia-smi listing it last.

Proxmox LXC GPU Passthrough

LXC containers can share host GPUs via device passthrough — no PCIe passthrough needed. The host kernel module handles the GPU; the container only needs the userspace libraries.

Correct LXC Config Pattern

Working pattern (CT 110 + CT 111 on workhorse). All GPU nodes are bind-mounted with the optional flag:

# Allow NVIDIA device nodes (major numbers from host — verify with ls -la /dev/nvidia*)
lxc.cgroup2.devices.allow: c 195:* rwm    # /dev/nvidia[0-9] + /dev/nvidiactl
lxc.cgroup2.devices.allow: c 511:* rwm    # /dev/nvidia-uvm, /dev/nvidia-uvm-tools
lxc.cgroup2.devices.allow: c 236:* rwm    # /dev/nvidia-caps/nvidia-cap[1-2]

# Mount all available GPUs
lxc.mount.entry: /dev/nvidia0 dev/nvidia0 none bind,optional,create=file
lxc.mount.entry: /dev/nvidia1 dev/nvidia1 none bind,optional,create=file
lxc.mount.entry: /dev/nvidia2 dev/nvidia2 none bind,optional,create=file
lxc.mount.entry: /dev/nvidiactl dev/nvidiactl none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-uvm dev/nvidia-uvm none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-uvm-tools dev/nvidia-uvm-tools none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-modeset dev/nvidia-modeset none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-caps/nvidia-cap1 dev/nvidia-caps/nvidia-cap1 none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-caps/nvidia-cap2 dev/nvidia-caps/nvidia-cap2 none bind,optional,create=file

optional flag — If a device doesn’t exist on the host the mount silently skips instead of failing the container start. This matters: /dev/nvidia-modeset is absent on workhorse (confirmed 2026-08-03) yet both CT configs mount it — the flag carries it.

CT 110 additionally mounts /dev/net/tun (VPN use) and the big storage: lxc.mount.entry: /bigboi/ai-models root/kobold/models none bind,create=dir 0 0.

Container Requirements

Inside the LXC:

1. NVIDIA userspace libraries matching the host kernel module version
2. nvidia-smi for verification
3. No kernel modules needed inside — they come from the host

Installing / Updating Userspace Libraries

When the host driver version changes, the container’s userspace libs must match.

Option A — From the host’s .run file (recommended):

# Copy the .run file into the container
pct push <CTID> /root/NVIDIA-Linux-x86_64-<VERSION>.run /tmp/

# Install userspace only (no kernel modules)
pct exec <CTID> -- bash /tmp/NVIDIA-Linux-x86_64-<VERSION>.run --no-kernel-modules --silent

Option B — Copy from host filesystem:

# Copy .so files from host -> container (advanced, use Option A instead)

After install, restart the container: pct stop <CTID> && pct start <CTID>

Verification

# Inside container: check all GPUs visible
nvidia-smi

# Python/ComfyUI: check PyTorch sees them
python -c "import torch; print(torch.cuda.is_available(), torch.cuda.device_count())"

Checking Major Device Numbers

Host device nodes tell you the major number to use in the LXC config:

ls -la /dev/nvidia*
# crw-rw-rw- 1 root root 195,  0 ... /dev/nvidia0
# crw-rw-rw- 1 root root 510,  0 ... /dev/nvidia-uvm
# crw-rw-rw- 1 root root 236,  1 ... /dev/nvidia-caps/nvidia-cap1

Live majors on workhorse (2026-08-03): 195 (nvidia*), 510 (uvm), 236 (caps).

Config drift observed: both CT configs still allow c 511:* for uvm while the live node is major 510 — yet CUDA works fine inside the containers (torch cuda: True, count 2, verified 2026-08-03, and gestalt-ardy actively uses ~8 GiB). Conclusion: the lxc.cgroup2.devices.allow entries are not strictly enforced on this host, so drift is harmless in practice. Keep them in sync anyway when you touch the configs.

Known Pitfalls

1. Driver Version Mismatch

Symptom: "Failed to initialize NVML: Driver/library version mismatch"
Cause: Container userspace libs are older than the host kernel module.
Fix: Reinstall userspace libs from the .run file matching the host version.

2. Wrong Major Device Numbers

Symptom: Device nodes missing inside container, or nvidia-smi can’t find devices.
Fix: Check ls -la /dev/nvidia* on the host. Update lxc.cgroup2.devices.allow lines with the correct major numbers (especially after driver upgrades, as major numbers can change). Note the allow lines can drift without visible breakage on this host (see above) — don’t rely on that.

3. Device Major Numbers Shift on Driver Reinstalls

History: On workhorse, nvidia-uvm has bounced between majors across driver reinstalls: 510 → 511 (June 2026, after the 580.126.18 update) → 510 again (Jul 25 2026 reboot/reinstall, current). nvidia-caps has been 235 → 236 (current). Always check live values rather than copying old configs.

4. Missing GPU Mounts

Symptom: Only some GPUs visible inside container.
Fix: Verify all /dev/nvidia[0-9] devices are mounted. Each GPU has its own device node.

5. SSH via Host (pct exec)

Use the Proxmox node to reach containers when SSH aliases are stale:

ssh workhorse "pct exec <CTID> -- <command>"

6. GPU Present in lspci but Missing from nvidia-smi (A4000 incident)

Symptom (2026-08-03): lspci shows the RTX A4000 at 0000:65:00.0, /dev/nvidia2 exists, but nvidia-smi enumerates only 2 GPUs — in the host AND all containers.
dmesg evidence:

NVRM: gpuHandleSanityCheckRegReadError_GM107: Possible bad register read: addr: 0x110100, regvalue: 0xbadf5620
NVRM: kflcnWaitForHalt_TU102: Timeout waiting for Falcon to halt
NVRM: kgspWaitForGfwBootOk_TU102: failed to wait for GFW boot complete: 0x65 VBIOS version 94.04.63.00.11
NVRM: GPU 0000:65:00.0: RmInitAdapter failed! (0x62:0x65:2015)

Diagnosis: No Xid errors logged; no failure at the Jul 25 boot — the card was healthy until ~15:38 on Aug 3. The failure coincided with pixelrag’s CUDA init on CT 110 (15:37), so a user-space CUDA init may have poked a card that was already in a bad state and the driver’s GSP re-init failed. Telling sign: the PCIe link had fallen back to 2.5 GT/s (Gen1) — lspci -vvv showed LnkSta: Speed 2.5GT/s (downgraded) — and register reads came back garbage (0xbadf5620), i.e. the card was wedged at the link level.

Resolution (what worked, 15:45): PCI remove/rescan (echo 1 > /sys/bus/pci/devices/0000:65:00.0/remove + echo 1 > /sys/bus/pci/rescan) alone did NOT fix it — the card re-enumerated but the link stayed at 2.5 GT/s and GSP still timed out. What fixed it: a targeted sysfs function-level reset on the GPU function only:

echo 1 > /sys/bus/pci/devices/0000:65:00.0/reset   # FLR — no driver reload, no reboot

The driver re-initialized the card and the link retrained to full 16 GT/s x16. Zero disruption to the 5060 Ti’s, running containers, or the nvidia driver module — this is the preferred first-line recovery for a wedged card; keep the reboot-based options (NVreg_EnableGpuFirmware=0, physical reseat) as fallbacks only. The 0xbadf5620 register read was a symptom of the wedged link, not evidence of a hardware fault — no reseat needed.

GPU Inventory

Fleet-Wide (verified 2026-08-03)

Driver version note: workhorse runs 580.126.18, pve runs 580.119.02. When setting up new LXC GPU passthrough on a node, match the container’s userspace libs to the host driver version for that node.

How GPUs Are Assigned (live 2026-08-03, post-recovery)

The A4000 is back online (recovered 15:45 via FLR) and Ollama + ComfyUI are running again. Current reality:

CUDA ordering inside CT 111: PyTorch reports cuda:0 = A4000, cuda:1/2 = 5060 Ti (verified post-recovery) — this differs from nvidia-smi order (A4000 is nvidia-smi index 2). The per-GPU systemd units use CUDA_VISIBLE_DEVICES (0/1/2), which resolves to A4000/5060 Ti/5060 Ti correctly. gestalt-ardy on CT 110 pins CUDA_VISIBLE_DEVICES=0 → 5060 Ti #0.

ComfyUI units fixed 2026-08-03: the three per-GPU units (comfyui-gpu0/1/2.service) had been broken since ~Jul 22 — their ExecStart pointed at /opt/ComfyUI/venv/bin/python but the venv lives at /opt/ComfyUI/.venv/bin/python (status 203/EXEC). Fixed with sed + systemctl daemon-reload; all three started cleanly. The legacy single-GPU comfyui.service (port 8188) conflicts with comfyui-gpu0 and stays disabled — the per-GPU units are the current design.

llama.cpp Server (CT 110)

Service: llama-server (systemd) — currently disabled + inactive
Endpoint: http://192.168.0.17:8080 (OpenAI-compatible API)
Port: 8080 (separate from Ollama on 11434)

Model: Gemma 4 Harmonia 31B

• Size: 30.7B parameters, Q4_K_M quantization (~22 GB on disk) — file verified present 2026-08-03
• VRAM: Split across both 5060 Ti’s (~12.7 + ~13.1 GB) when running
• Performance: ~17.6 tok/s generation, ~7 tok/s prompt eval (last measured)
• Context: 32K (reduced from native 256K to avoid OOM)
• Config: 1 parallel slot, 99 GPU layers, continuous batching

Systemd Service Config (as filed — includes GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 added after last run)

[Unit]
Description=llama.cpp server (Gemma 4 31B)
After=network.target
Wants=network.target

[Service]
Type=simple
User=root
WorkingDirectory=/root
Environment=CUDA_VISIBLE_DEVICES=1,2
Environment=GGML_CUDA_ENABLE_UNIFIED_MEMORY=1
ExecStart=/usr/local/bin/llama-server --model /root/kobold/models/Gemma-4-Harmonia-31B-it-uncensored-heretic-Q4_K_M.gguf --host 0.0.0.0 --port 8080 --n-gpu-layers 99 --threads 16 --cont-batching --no-warmup --ctx-size 32768 --parallel 1 --batch-size 256 --ubatch-size 64
Restart=on-failure
RestartSec=10

Important Notes

• Vision: the mmproj file (...-mmproj-BF16.gguf) is now present on disk, but vision was not working with this llama.cpp build and the server hasn’t run since — still untested.
• Built from source with CUDA 12.0 (skipping Blackwell compute_120a by targeting Ampere/Ada archs)
• OOM guard: Reduced context to 32K and parallel to 1 slot to prevent OOM on the ~22 GB model across two 16 GB cards

Available Models

Other GGUF models at /root/kobold/models/ (verified present 2026-08-03): Gemma3 27B, Cydonia 24B, Cydonia Heresy, Mistral Small 22B, Qwen 32B, GLM4.7 Flash, DeepSeek-R1-Distill-70B Q8, Melody1437-12B, plus more 12–70B models.

Services on CT 110 (live 2026-08-03)

Units exist for all of the above (systemd). Active: ollama, qwen3-tts, gestalt-ardy, pixelrag. Ollama at CT 110 has no CUDA_VISIBLE_DEVICES pin — it sees all 3 GPUs again now that the A4000 is back.

CT Reference

Related

• entities/homelab-fleet — node layout, CT numbering, IPs
• entities/kokoro-tts — TTS service details
• references/pixelrag-visual-rag — PixelRAG on CT 110 (A4000), local API down until index built
• concepts/recent-infrastructure — ComfyUI status, shared infra notes