cuda-dev (shared GPU dev box)

cuda-dev2 (and the rest of the cuda-dev range as it comes online) is the shared, GPU-equipped dev box you use when work doesn’t fit on your laptop: long-running Claude jobs, autonomous agents in bypass-permissions mode, CUDA workloads, big container builds. You don’t get a shell on the host itself — every developer gets their own Incus container, configured to look like a small Ubuntu box with Docker, mosh and the standard toolchain pre-installed. Each container has GPU access (cooperative; one heavy job at a time) and is treated as throwaway: if it gets into a bad state we rebuild it from the golden image and you keep going.

Prerequisites

• A working Company VPN connection. The dev box lives on the office LAN (192.168.2.26), so the WireGuard tunnel needs to be up — see Services → Company VPN. ping 192.168.2.26 should answer once it is.
• A request in to infra. Mention you want a cuda-dev container; the onboarding tool takes it from there. In one go it generates a dedicated SSH key pair for you, creates and starts your container on cuda-dev2, drops the pubkey into it, picks up the container’s LAN IP, and emails you everything you need — a one-time Bitwarden Send link for the private key, a permanent Bitwarden item reference, and a ready-to-paste ~/.ssh/config snippet with the actual IP baked in. Nothing for you to edit by hand and no second hand-off.

Connecting

Containers attach to the office LAN via macvlan on the host’s NIC, so each one gets its own 192.168.2.x DHCP lease from the office router. You SSH straight to the container — no jump host, no ProxyJump, no host shell involved.

Step 1: get your private SSH key onto your laptop

The provisioning email gives you two ways to do this. Pick one.

Option A — Bitwarden Desktop SSH agent (recommended if you already use it for the rest of your infra keys). Open Bitwarden, find the item cuda-dev key / <localpart> in the d-centralize - Infra collection, and make sure the SSH agent toggle is on for that item. Then drop the public half of the key into ~/.ssh/cuda-dev-<localpart>.pub on your laptop — that file is just a selector telling SSH which agent-loaded key to use, not a private key:

# Paste the `ssh-ed25519 AAAA…` line from the Bitwarden item.
$EDITOR ~/.ssh/cuda-dev-<localpart>.pub
chmod 644 ~/.ssh/cuda-dev-<localpart>.pub

The matching private half stays inside Bitwarden Desktop forever; the agent serves it on demand. This is the same pattern as every other <host> SSH key Bitwarden item — see Setup → Git and the Host * / IdentityAgent block near the top of your ~/.ssh/config.

Option B — copy the private key to disk. Click the one-time Bitwarden Send link in your email (good for 7 days, three retrievals), copy the contents, and save them as the file:

$EDITOR ~/.ssh/cuda-dev-<localpart>     # paste the private key
chmod 600 ~/.ssh/cuda-dev-<localpart>

The same key also lives permanently in Bitwarden as cuda-dev key / <localpart> if you miss the Send window — you can re-export the private half from there.

Step 2: drop the SSH config block

The provisioning email arrives with the actual IP filled in; the example below uses a placeholder. The IdentityFile line points at the path you just created in step 1 — for option A it’s the .pub selector, for option B it’s the private key itself. SSH treats both the same way at the call site.

Host dev-<localpart>
  HostName 192.168.2.<your-container-ip>
  User root
  IdentityFile ~/.ssh/cuda-dev-<localpart>      # option B (private key)
  # IdentityFile ~/.ssh/cuda-dev-<localpart>.pub  # option A (BW agent selector)
  IdentitiesOnly yes

Then:

ssh dev-<localpart>

You land as root inside your own container. The first connection prompts you to accept the host key — that’s expected.

mosh for flaky links

The dev profile installs mosh so you can survive hotel Wi-Fi without losing your shell. Same target, no SSH config tricks:

mosh dev-<localpart>

What’s inside

The container is built from the cuda-dev-golden image. The exact list of installed packages is defined in golden-image.yml in the infra repo — open an MR there if you want something else baked in. At the time of writing it includes:

• Ubuntu 26.04 (matches the host).
• Docker + docker compose (nested — the dev profile sets security.nesting=true).
• mosh, tmux, git, vim, htop, build-essential, Python 3 + pipx + python3-venv.
• Claude Code CLI (claude).
• The NVIDIA driver from the host, bind-mounted via nvidia.runtime=true; nvidia-smi works and so do CUDA container images.

docker --version
nvidia-smi
claude --version

Working with the GPU

The host has a single shared consumer GPU (currently GTX 1660 SUPER, 6 GB VRAM). Allocation is cooperative — everyone can see the device, but in practice one heavy job at a time is what fits. If you’re about to kick off a multi-hour CUDA run, drop a note in #infra so others know to hold off.

Live host status, GPU utilisation and per-container metrics are at https://infra.d-centralize.nl/devices/cuda-dev-range/cuda-dev2/#monitoring (also picked up by uptime-kuma at https://status.d-centralize.nl/status/dcentralize). nvidia-smi -l 5 from inside your container is still the quickest way to check who’s on the GPU right now.

Resource limits

The default per-container limits are generous so a sole-occupant developer gets most of the host:

Limit	Value
CPU	8 cores
Memory	24 GB
Root disk	100 GB

If you genuinely need more — large dataset, build pipeline, etc. — ping ops; the limits are profile-level and trivial to bump per container.

The host also has 64 GB swap, so a temporary spike beyond your memory limit will spill there rather than OOM-kill (but you’ll feel it).

Treating the container as throwaway

The container is yours, but it’s not precious:

• Keep important state outside it. Code in git. Datasets on the NAS or in object storage. Build caches you actually want to keep belong on a mounted volume, not in ~/.
• Snapshot before risky experiments. A one-line message to ops gets you a named snapshot (incus snapshot dev-<localpart> pre-experiment); self-serve via an incus remote on the laptop is on the follow-up list and will be documented here once it’s wired in.
• Rebuild on compromise or rot. Same shape — ask ops to wipe and re-create from cuda-dev-golden. You’re back in business in under a minute.

When things go wrong

• Permission denied (publickey) on ssh dev-… — your pubkey isn’t in the container’s ~/.ssh/authorized_keys yet. Confirm with ops that the right key landed.
• ping 192.168.2.26 doesn’t answer — the VPN’s down; see Services → Company VPN.
• GPU not visible inside the container — nvidia-smi failing is almost always either a driver mismatch (host rebooted, container needs a restart) or the GPU device fell off the profile. Ping ops.