Notebook, Virtual Machine, Docker, or Kubernetes: Where Should Your LLM Run?
You have selected a model and a GPU. Now you need somewhere to run it.
Should that be a Jupyter notebook, a virtual machine, a Docker container, or Kubernetes?
For most small teams, the progression is simple:
Notebook for experiments → VM for a simple server → Docker for repeatable deployment → Kubernetes only when scaling actually requires it
You do not need to start with the most sophisticated option.
Pick Based on What You Are Doing
Your situationBest starting optionTesting a model or promptNotebookRunning one private LLM serverVirtual machineBuilding a reproducible APIDockerSharing the same environment across a teamDockerRunning multiple replicasKubernetesManaging several GPU nodesKubernetesExperimenting with fine-tuningNotebook or DockerSmall production LLM APIVM + DockerLarge production platformKubernetesThe biggest mistake is treating these as competing technologies.
They often sit on top of each other. You can run Docker inside a VM, and Kubernetes itself runs containers across multiple machines.
Use a Notebook When You Are Still Experimenting
Jupyter notebooks combine executable code, text, equations, and outputs in one document. That makes them useful for interactive experimentation.
A notebook works well when you are:
- Downloading a model for the first time
- Testing prompts
- Comparing quantization methods
- Checking GPU memory usage
- Running a small evaluation
- Exploring a dataset
- Trying a LoRA fine-tuning script
If you are still deciding what to test, this open-source LLM comparison can help narrow models by use case, size, licensing, and GPU requirements before you build the environment.
You can execute one step, inspect the result, change something, and run it again.
Example
from transformers import pipeline pipe = pipeline( "text-generation", model="your-model-name", device=0 ) pipe("Explain KV cache in simple terms")
For experimentation, this is easier than building an API before you know whether the model works.
Do not treat the notebook as the production server
A notebook becomes awkward when:
- Applications need to call the model
- Several people need simultaneous access
- The process must restart automatically
- You need reliable logging
- The model should run continuously
- Deployments need to be repeatable
Once another application depends on the model, move beyond the notebook.
Use a Virtual Machine When You Need One Persistent Server
A GPU virtual machine is often the simplest next step.
You get a machine where you can install:
- NVIDIA drivers
- CUDA
- Python
- PyTorch
- vLLM
- Model files
- Your API
A simple setup might look like:
Application ↓ LLM API ↓ vLLM ↓ GPU VM
This can be perfectly reasonable for an internal assistant or an early production application.
If your local hardware is no longer enough, a cloud GPU environment lets you choose different GPU classes without purchasing the hardware first.
A VM is enough when:
- You have one model
- One server handles the traffic
- The team understands Linux
- You do not deploy frequently
- Occasional manual maintenance is acceptable
The main problem appears when everyone starts installing packages directly on the server.
One person upgrades Python. Another changes CUDA libraries. A third updates the inference server.
Soon you get:
“It worked yesterday.”
That is where containers become useful.
Use Docker When You Want the Deployment to Be Repeatable
Docker packages your application, dependencies, and configuration into a container.
The host still needs the appropriate GPU drivers and NVIDIA Container Toolkit, but Docker can expose NVIDIA GPUs to containers.
This lets you define the environment once instead of manually rebuilding it on every server.
For example, vLLM provides an official Docker image for running its OpenAI-compatible server.
A simplified command looks like:
docker run --gpus all \ -p 8000:8000 \ vllm/vllm-openai:latest \ --model your-model
Docker becomes useful when:
- More than one developer works on the project
- You deploy to different machines
- You need predictable dependencies
- You want easy rollback between versions
- You need to reproduce the same environment
- The model is exposed as an API
For many small teams, Docker on a single GPU VM is enough for production.
If you are choosing between common inference GPUs at this stage, an L4 vs L40S comparison can help determine whether you need 24GB or 48GB VRAM and how much performance headroom is useful.
You do not automatically need Kubernetes after Docker.
Use Kubernetes When One Server Is No Longer Enough
Kubernetes schedules containers across a cluster of machines.
It also supports GPUs through vendor device plugins, allowing GPU resources to be exposed to workloads running in Pods.
That sounds useful, but it also adds significantly more infrastructure.
You now need to understand concepts such as:
- Pods
- Deployments
- Services
- Nodes
- Device plugins
- Storage
- Scheduling
- Health probes
- Resource requests
Kubernetes makes sense when that additional complexity solves a real problem.
Consider it when:
- You have several GPU nodes
- You need multiple inference replicas
- Different teams share GPU infrastructure
- Applications must automatically recover from failed instances
- You need controlled rolling deployments
- Several models need scheduling
- Traffic requires horizontal scaling
Do not introduce Kubernetes just to run one model
If your architecture is:
One application ↓ One LLM server ↓ One GPU
Kubernetes may solve problems you do not have.
Docker on a VM is usually easier to understand, debug, and maintain.
These Options Can Be Combined
The choice is not always:
Notebook OR VM OR Docker OR Kubernetes
A real development path might look like:
Notebook ↓ Test the model ↓ GPU Virtual Machine ↓ Docker container ↓ Production API
Later, if traffic grows:
Docker image ↓ Kubernetes ↓ Multiple GPU nodes
The same container you tested on one VM can become the building block for a larger deployment.
A Practical Example
Suppose three developers are building an internal RAG assistant.
Week 1
They are evaluating several 7B models.
Use: Jupyter notebook
There is no reason to build production infrastructure yet.
Week 2
They select one model and need the backend application to call it.
Use: GPU VM
The model runs continuously behind an HTTP API.
Week 3
Different developers start changing dependencies.
Use: Docker on the same VM
The environment becomes reproducible.
Three months later
The assistant has hundreds of users, and one GPU can no longer handle peak traffic.
Now consider: Kubernetes or another orchestration approach.
Complexity was added only when the application created a reason for it.
What Should You Start With?
Use this rule:
Still testing the model?
Use a notebook.
Need one persistent LLM server?
Use a GPU VM.
Need repeatable deployments?
Add Docker.
Need to manage several servers or replicas?
Consider Kubernetes.
For most small LLM teams, the sweet spot is:
GPU VM + Docker + an inference server such as vLLM
Before settling on the VM size, compare the complete infrastructure cost rather than only the GPU model. This cloud GPU pricing comparison is useful for checking how GPU type, memory, pricing model, and provider differences affect the final deployment cost.
Move to Kubernetes when one machine or one deployment is genuinely becoming difficult to manage, not because Kubernetes appears on every production AI architecture diagram.
