Self-Hosting Open-Weight LLMs: Production Ollama and vLLM on Linux

1. Why Self-Host Open-Weight Models?

The default reflex for most teams is to call a hosted API, and for good reason: it is zero-ops and the frontier models are genuinely better on hard reasoning tasks. Self-hosting open-weight models like Llama 3, Mistral, or Qwen stops being a hobby project and starts being a business decision under three fairly common conditions: sustained high-volume inference where per-token API costs dominate the bill, data residency or contractual requirements that forbid sending customer data to a third party, and latency-sensitive applications where a round trip to an external API adds unacceptable tail latency compared to inference on hardware you control on the same network as your application.

The economics flip at a predictable point: a single GPU running a 7B to 14B parameter model at reasonable utilization can serve a sustained request volume that would cost several times the GPU's amortized hourly cost on a per-token hosted API, once you are running tens of millions of tokens a day. Below that volume, the fixed cost of GPU capacity (paid whether or not it is busy) usually loses to the pay-per-token model, so self-hosting is a volume and utilization play, not a universal upgrade.

The honest trade-off is operational ownership: you now own GPU driver compatibility, model upgrades, capacity planning, and the failure modes of a stateful service instead of an HTTP client with retries. Teams that self-host successfully treat it as infrastructure with the same rigor as a database, with monitoring, capacity headroom, and a rollback plan, not as a one-off script running on a spare machine.

2. Setting Up Ollama in a Production systemd Service

Ollama is the fastest path from zero to a running local model, wrapping llama.cpp with model management, an HTTP API, and automatic GPU detection, which makes it the right default for moderate-throughput self-hosting and for teams who want a production-ready single-node setup without operating a full inference cluster. Running it as a manually-started process is fine for local development but wrong for production: it needs to survive reboots, restart on crash, and log to a place your monitoring stack can see.

The standard production setup wraps the Ollama server in a systemd unit with Restart=always, a dedicated non-root system user, and explicit environment variables controlling which network interface it binds to, the models directory location (which should be on fast local NVMe storage, not network-mounted, since model files are tens of gigabytes and load-time directly affects cold-start latency), and concurrency limits.

Two environment variables matter disproportionately in production: OLLAMA_NUM_PARALLEL controls how many requests a single loaded model instance can process concurrently (default is conservative; tuning it up trades memory for throughput), and OLLAMA_MAX_LOADED_MODELS controls how many distinct models can be resident in memory simultaneously, which matters if you serve more than one model from the same node and need to avoid GPU memory thrashing from repeated load and unload cycles.