How we built a 40-GPU inference cluster in 48 hours for an AI hackathon

Infiano was running an AI hackathon with 57 teams and 240 submissions that had to be evaluated within just one week. They brought us in to build the infrastructure capable of handling this workload using a specific open-source model.

Here’s how we built a 40-GPU inference cluster in just 48 hours.

The setup: renting GPUs on vast.ai

We spun up 10 bare-metal servers on vast.ai — each with 4× NVIDIA RTX 5090 (32GB VRAM). That’s 40 GPUs total, plus a high-memory CPU server (503GB RAM) for orchestration, all provisioned within 48 hours.

Why vast.ai? Speed and cost. For a 6-day project running a custom 20B-parameter model, spot GPU rentals beat any cloud alternative — we kept total infrastructure cost under $3,000.

Each GPU server ran 4 independent vLLM processes (one per GPU), load-balanced by nginx, fronted by Caddy for authentication. A single providers.json file on the orchestration server was the single source of truth for all 31 API endpoints — swap a server, edit one line, done.

What we built on top

The customer’s evaluation pipeline was solid, but running 240 submissions at scale required several engineering additions:

• •Watchdog v3 (GPU-side). A ping-first monitoring process on each server. Fast /ping check first — if the server responds but end-to-end inference times out, that’s load, not failure (WARN only). Three consecutive ping failures — automatic supervisorctl restart. This eliminated false restarts under heavy load.
• •Watchdog v3 (CPU-side). Tracked submission throughput every minute. Zero OK responses for 5 consecutive checks — kill all stuck processes and restart. Also killed any submission running over 60 minutes or consuming over 8GB RAM. Auto-restarted the orchestration process if it disappeared.
• •GPU health monitor. A separate cron job checking all GPU servers every minute via HTTP. Three consecutive failures — server marked DEAD, traffic rerouted automatically. When a server came back, auto-recovery was logged. All config loaded from providers.json.
• •Zabbix monitoring. 15 metrics per GPU streamed to our Zabbix instance: KV cache utilization, queue depth, avg latency, time-to-first-token, tokens/sec, HTTP error counts. Gave us a live dashboard across all 40 GPUs.
• •Skip-task bug fix. The upstream evaluation script had a subtle bug: when a task was skipped (empty LLM response, missing file), it was dropped from the queue but not counted toward progress — causing the process to hang indefinitely waiting for completion. We patched two code paths to append None for skipped tasks, unblocking hundreds of submissions.
• •400-spam protection. One submission was sending max_tokens=-12000, flooding all GPU endpoints with HTTP 400 errors and degrading the entire cluster. We added detection: more than 500 HTTP 400s in the last 200 log lines — kill the offending process automatically.

Incidents we handled

• •SRV9 died overnight — a vast.ai spot instance vanished. The health monitor caught it in under a minute and rerouted traffic. Replaced next morning: one line in config.
• •GPU hardware failure — GPU0 on SRV7 threw a hardware error. Removed from rotation; the remaining 3 GPUs on that server kept working.
• •Cache collision — two parallel submissions writing to the same cache directory. The first one finished and cleaned up, destroying the second’s intermediate state. Fixed by sequencing conflicting jobs.

The result

236 submissions scored across 55 teams in 6 days. Zero infrastructure-caused restarts. Zero HTTP 5xx errors. Time-to-first-token consistently 1.1—1.4s across all 40 GPUs.

Same workload, less than half the cost — and delivered in days, not weeks.