Multi-GPU
A server can have 1 to 8 GPUs, always on the same machine. The GPUs are connected by the machine’s PCIe or NVLink, which is fast enough to work as one big accelerator. GPUs on different machines are not combined: over the internet that would be too slow to be useful.
Serving large models: tensor parallelism
Section titled “Serving large models: tensor parallelism”The vllm and sglang templates start the model with --tensor-parallel-size (--tp in SGLang) equal to the number of GPUs you rented. Every layer of the model is split across all GPUs, so:
- memory adds up: a model that needs 80 GB runs on 2×48 GB or 4×24 GB;
- throughput grows: more memory for the KV cache means more parallel requests and longer contexts.
| Model size | 1 GPU | Combined |
|---|---|---|
| up to 24 GB (e.g. Qwen3 8B, gpt-oss 20B) | RTX 3090 / 4090, A5000, L4 | 2×16 GB |
| up to 40 GB (Qwen3 14B, Phi-4) | A100 40GB, any 48 GB card | 2×24 GB |
| up to 80 GB (Qwen3 32B, gpt-oss 120B) | A100 80GB, H100, RTX PRO 6000 | 2×48 GB, 4×24 GB |
| 160 GB (Llama 3.3 70B) | B200 | 2×80 GB, 2×H200 |
| 320 GB (Qwen3 235B FP8, GLM-4.5 Air) | — | 4×80 GB, 4×H200, 2×B200 |
| 640 GB (Qwen3 Coder 480B FP8) | — | 8×80 GB, 8×H200, 4×B200 |
| 1000 GB (DeepSeek-R1 / V3.1) | — | 8×H200, 8×B200 |
vLLM works best when the number of GPUs divides the model’s attention heads — use 1, 2, 4 or 8 GPUs.
Many small models instead of one big one
Section titled “Many small models instead of one big one”For high traffic on a small model it is often cheaper to run several 1-GPU servers and spread requests between them than one multi-GPU server. You can launch as many servers as your balance allows, in parallel, through the dashboard or the API.
Training on several GPUs
Section titled “Training on several GPUs”Templates marked distributed (pytorch, jupyter, cuda, custom, llamafactory, axolotl, unsloth, tensorflow) see all rented GPUs. Use the usual tools:
# PyTorch DDP / FSDP on all GPUs of the servertorchrun --nproc_per_node=$(nvidia-smi -L | wc -l) train.py
# Axolotlaccelerate launch -m axolotl.cli.train config.yml
# DeepSpeeddeepspeed --num_gpus=4 train.py --deepspeed ds_config.jsonLLaMA-Factory uses all GPUs automatically when you start training from its UI.
Apps that use one GPU
Section titled “Apps that use one GPU”ComfyUI, Ollama, Open WebUI, Text Generation WebUI and the audio templates use one GPU per process. With more GPUs you can run one process per GPU (for example a second ComfyUI with --cuda-device 1 on another port), but for simple cases renting 1 GPU is enough.