Yes, a small team can self-host an LLM without a dedicated MLOps engineer.
But only if the deployment stays reasonably simple.
Running one model behind an API on a single GPU is very different from operating autoscaling, multi-GPU inference across several environments.
For a small team, the goal should not be to build a complete MLOps platform. It should be to run the model reliably with the fewest moving parts.
When Self-Hosting Is Realistic
A small team can usually manage self-hosting when:
- One model handles most requests
- One GPU or one GPU server is enough
- Traffic is relatively predictable
- The model does not change every few days
- Basic Linux and Docker skills already exist in the team
- Someone can monitor the service and respond when it fails
- High availability is useful but not mission-critical
A typical setup might be:
Application ↓ LLM API ↓ vLLM or another inference server ↓ GPU
That does not require a dedicated MLOps engineer.
A backend, DevOps, or infrastructure engineer who is comfortable with Linux, Docker, networking, and monitoring can often manage it.
vLLM, for example, can expose a model through an OpenAI-compatible HTTP server, which makes integration with existing applications relatively straightforward.
What Someone Still Has to Manage
Self-hosting does not mean simply starting the model once.
Someone on the team needs to own these areas.
GPU and driver setup
The server needs:
- NVIDIA drivers
- CUDA-compatible libraries
- Container GPU access
- Enough VRAM for the selected model
Docker supports GPU access through the NVIDIA Container Toolkit, but the host still needs the correct GPU driver and runtime configuration.
Model serving
You need an inference server that handles requests efficiently.
Common options include:
- vLLM
- Hugging Face Text Generation Inference
- Ollama for simpler environments
- llama.cpp for some local and quantized deployments
For production-facing APIs, tools such as vLLM are useful because they handle request serving, batching, and model execution rather than forcing you to build everything yourself.
Monitoring
At minimum, monitor:
- GPU memory usage
- GPU utilization
- Request latency
- Queue size
- Time to first token
- Tokens per second
- Failed requests
- Server restarts
vLLM exposes production metrics through a /metrics endpoint, making it possible to integrate model-serving metrics with monitoring systems such as Prometheus.
Security
Do not expose an inference server directly to the internet without protection.
Add:
- Authentication
- HTTPS
- Network restrictions
- Secret management
- Request limits
- Logging
Also protect model files, cloud credentials, Hugging Face tokens, and any prompts stored in logs.
Updates and recovery
Someone needs to know what happens when:
- The model process crashes
- The GPU runs out of memory
- A driver update breaks the environment
- Storage fills up
- A new model version is deployed
- The VM restarts
If nobody owns these situations, self-hosting will eventually become frustrating.
You Probably Do Not Need Kubernetes Yet
This is where small teams often make the setup unnecessarily complicated.
You probably do not need Kubernetes when:
- One GPU VM handles the workload
- One inference server is enough
- Traffic is predictable
- Occasional restarts are acceptable
Docker and a simple process supervisor may be enough.
Kubernetes becomes more useful when you need:
- Several GPU nodes
- Multiple model replicas
- Automated scheduling
- Rolling deployments
- Higher availability
- Complex scaling
- Multiple teams sharing GPU resources
Kubernetes supports GPU scheduling through device plugins, but that introduces another infrastructure layer the team must operate.
Do not introduce Kubernetes simply because the application uses AI.
The Small-Team Stack Can Be Very Simple
A reasonable first production setup could be:
Cloud GPU VM │ ├── Docker │ ├── vLLM │ ├── Your model │ ├── API authentication │ └── Basic monitoring
Your application calls the model server just like it would call a hosted LLM API.
For example, vLLM provides an OpenAI-compatible interface, so many applications can switch the API base URL without redesigning the entire integration.
This is often enough for:
- Internal assistants
- Small SaaS products
- RAG applications
- Development environments
- Low-to-moderate traffic APIs
When Self-Hosting Starts Getting Hard
The operational workload increases quickly when you need several of these at once:
Multiple GPUs
You may need tensor parallelism, distributed communication, and faster GPU interconnects.
Several models
Now you must decide:
- Which model runs on which GPU?
- Can models share memory?
- When should one model be unloaded?
- How should requests be routed?
Variable traffic
If traffic is highly unpredictable, keeping a GPU running wastes money while shutting it down introduces startup delays.
Autoscaling GPU workloads is possible, but it creates more infrastructure and monitoring work.
High availability
If the LLM API cannot go offline, you may need:
- Multiple replicas
- Health checks
- Load balancing
- Failover
- Deployment automation
Several GPU nodes
At that point, orchestration systems such as Kubernetes become more relevant because they can schedule workloads against GPU resources exposed by device plugins.
If your diagram starts looking like this:
Load balancer ↓ Multiple API replicas ↓ Kubernetes ↓ Several GPU nodes ↓ Shared storage + monitoring + autoscaling
you are moving beyond a simple self-hosted deployment.
A dedicated platform or MLOps skillset starts becoming much more valuable.
A Small Team Still Needs an Owner
You may not need an MLOps engineer, but you do need someone who owns the LLM infrastructure.
That person should be able to:
- Restart and troubleshoot the service
- Check GPU memory
- Read application logs
- Update containers
- Manage credentials
- Track infrastructure costs
- Test model upgrades
- Investigate latency problems
This can be an existing backend, DevOps, or infrastructure engineer.
The mistake is assuming nobody needs to own it because the serving software is open source.
When Managed Inference Is the Better Choice
Do not self-host just because you technically can.
Managed inference may be better when:
- Nobody wants to maintain GPU infrastructure
- Traffic changes dramatically
- Downtime is expensive
- Your team is focused entirely on product development
- The model workload is still experimental
- You need several models but do not want to manage scheduling
- Infrastructure debugging would take time away from core development
You pay more for convenience, but engineering time also has a cost.
Compare:
Self-hosting cost = GPU + storage + networking + engineering time
against:
Managed cost = Service charges + usage
The cheapest GPU bill is not always the cheapest overall solution.
Start With One Model on One GPU
A small team can absolutely self-host an LLM without building a large MLOps function.
Keep the first deployment boring:
- One GPU
- One model
- One inference server
- Docker
- Authentication
- Basic monitoring
- One person responsible for it
Add orchestration, replicas, autoscaling, and multi-GPU infrastructure only when a real requirement appears.
If one GPU can reliably serve the application, there is little value in operating infrastructure designed for fifty.
