Ollama — LLM on Your Laptop in 5 Minutes

“I want to try an LLM locally but don’t want to set up CUDA, quantization, and compile llama.cpp.” Ollama is the answer: one command to install, one to run a model. It’s Docker for LLMs — it downloads the model, sets up the inference runtime, and exposes an API. In five minutes you have a working local AI without needing to understand GPU memory management or model formats.

Why Local Inference¶

• Privacy: Data never leaves your machine — critical for sensitive documents and code
• Offline: Works without internet — ideal for working on a plane or in secured environments
• Cost: $0 per token — unlimited experimentation without watching the budget
• Latency: No network roundtrip — response depends only on local hardware

For developers, local inference is invaluable when prototyping AI features. You test prompts, tune RAG pipelines, and iterate on outputs without waiting for an API and without costs. The resulting prompt easily transfers to a cloud model for production.

OpenAI-Compatible API¶

Ollama exposes an OpenAI-compatible API on localhost:11434. Redirect your existing code by changing the base URL — no changes to application logic needed. LangChain, LlamaIndex, Continue.dev, and most AI tools integrate Ollama natively. You can develop locally with Mistral and switch to GPT-4 in production by changing a single variable.

Recommended Models¶

• Mistral (7B): Versatile, decent Czech language support, best quality/size ratio for local use
• codellama (7B/13B): Optimized for code generation, completion, and review
• phi-2 (2.7B): Ultra lightweight model from Microsoft, surprisingly capable for its size
• llama3 (8B): Meta’s latest open model with excellent reasoning

With 16 GB RAM you can run 7B models, with 32 GB even 13B. Models are automatically quantized (Q4_0 or Q5_K_M) for optimal quality-to-memory ratio.

Local AI Is a Reality¶

Every developer can run a quality LLM locally. Ollama is a must-have tool in the developer toolbox for prototyping, testing, and offline AI work.