Run LLMs Locally with Ollama: A Beginner’s Guide

You can run LLMs locally on an ordinary laptop, and it's a good way to experiment without building an account system or sending every prompt to a hosted API. It also makes a few hidden details obvious: a model has to be downloaded, loading it takes time, and long conversations consume memory. Ollama handles the serving work, so you can focus on sending a question and reading the answer.

We'll start with a terminal conversation and then write a small Python chat that remembers previous turns. The model is qwen3:4b-instruct, a small model that answers directly. Download the starter folder for local_chat.py and the shared HTTP helper, or read the script first.

Tested with Ollama 0.40.5 on 2026-10-20.

Install Ollama to run LLMs locally

Install Ollama from its official download page. On a Mac, open the app after installation. On Linux, start the installed service, or run ollama serve if no server is running yet. Then run the commands below: pull downloads the named model, list shows what is stored locally, and run opens a terminal conversation. If the server says its port is already in use, Ollama is probably running already.

ollama --version
ollama pull qwen3:4b-instruct
ollama list
ollama run qwen3:4b-instruct

Why the -instruct tag? The plain qwen3:4b tag is a thinking model. In our tests on Ollama 0.40.5, asking it not to think still produced a long reasoning trace inside the reply, ending in a literal </think> tag, and some replies ran out of tokens before the answer. That's interesting when you want to study how reasoning models work, and annoying in a chat window.

The model library lists the available sizes and tags. Our tag is a download of roughly 2.5 GB, but that isn't the memory it needs while running: Ollama also needs room for the conversation context, and your operating system needs memory for everything else. Start with a small model and an ordinary question, then watch how your machine behaves. A slow or failed load is a reason to shrink the model or the context before touching your Python code.

Understand what is running where

The downloaded model is a set of files; Ollama is the process that loads them and answers requests. Your terminal or Python script talks to that process over HTTP, normally at http://127.0.0.1:11434. That address means "this same machine". If you run Python on a remote computer, its 127.0.0.1 isn't your Mac.

With this setup, prompt text goes to your local Ollama process. Installing and pulling models still need the network, and tools you add later could make their own requests. If you point OLLAMA_URL at another machine, your prompts travel there instead. "Local" describes the route your request takes, so keep that route explicit when you share code.

Make one request without a Python library

Ollama's chat API takes a model name and a list of messages. Set stream to false for the first request so the response arrives as one JSON object rather than a sequence of chunks. Each message has a role and content; here the only message comes from the user. Try this before building any interface, so you can tell model-server problems apart from application bugs.

curl http://127.0.0.1:11434/api/chat \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "qwen3:4b-instruct",
    "messages": [{"role": "user", "content": "Explain a Python list in two sentences."}],
    "stream": false,
    "options": {"num_ctx": 4096, "num_predict": 400}
  }'

The response includes the assistant's message plus request statistics such as prompt_eval_count (tokens read) and eval_count (tokens generated). It also has a done_reason: stop means the model finished, and length means it hit the num_predict limit mid-reply. Check that field before trusting an answer. A successful HTTP status doesn't mean the answer is complete.

Call the same endpoint from Python

You don't need an SDK for this. The starter's common.py uses the standard library's urllib.request, sends JSON, and reads the JSON response. It takes the server address from OLLAMA_URL, falling back to the local address, and sets a timeout so a stalled server doesn't leave the script hanging. Calling the HTTP API directly shows you exactly which options are sent.

from common import request_json
 
response = request_json("chat", {
    "model": "qwen3:4b-instruct",
    "messages": [{"role": "user", "content": "What is a Python dictionary?"}],
    "stream": False,
    "options": {"num_ctx": 4096, "num_predict": 400},
})
print(response["message"]["content"])

Save that snippet beside common.py to try a single call, or run local_chat.py for the full terminal conversation. The script reads a question, adds it to the messages, sends the request, and prints the answer. If the request fails, for example because Ollama isn't running, it prints the error and leaves the saved history untouched, so you can simply ask again. It also warns you when a reply stopped at the num_predict limit.

Conversation memory is a list you send again

The model server doesn't remember earlier turns for you. Our script keeps a list containing the system instruction, the user's questions, and the assistant's replies, and sends the whole list every time. That's why a follow-up such as "Show me an example of that" works: the earlier answer is part of the next prompt. Nothing about the model's weights changes, and closing the program discards the conversation.

pending = messages + [{"role": "user", "content": question}]
response = request_json("chat", {
    "model": "qwen3:4b-instruct",
    "messages": pending,
    "stream": False,
    "options": {"num_ctx": 4096, "num_predict": 400},
})
messages = pending + [response["message"]]

Step through a real session below. It was recorded with the starter script's settings, and the token counts are the ones Ollama reported.

A conversation can outgrow the available context, so keeping every message forever isn't a memory strategy. Our script uses a 4,096-token context and provides /reset to clear the history. For longer chats, decide which messages to keep, summarize, or retrieve before each request. Dropping history loses details, and summaries can introduce mistakes, so test the follow-up questions your application actually needs.

Separate output length from context length

The context window is the space available for the whole request: conversation text plus room for the reply. The output limit controls how much the model may generate. In our example, num_ctx is 4,096 and num_predict is 400. Raising the output limit helps when replies are cut off, but it won't bring back an earlier turn that didn't fit. Raising the context increases memory use even if you only expect short answers.

Record these settings when you compare runs, along with the model tag and the ID shown by ollama list. The first request after starting can include model loading, so it often feels slower than the next one. Don't compare a cold request from one model with a warm request from another.

Try a short conversation you can verify

Unzip the starter, then run the chat script. Ask it to remember a made-up project name, ask for that name in a follow-up, then use /reset and ask again. After the reset the name is no longer in the messages, although a language model could still guess one. Use an invented label rather than anything sensitive.

unzip ai-blog-starters.zip
cd ai-blog-starters
python3 local_chat.py
# /reset clears history; /quit ends the program.

You can also try tasks with known answers, such as explaining a short Python expression or summarizing a paragraph you supply. Check the result instead of judging how confident it sounds: a local model can produce fluent errors just like a hosted one. For questions about your own files, you need to send their contents or build retrieval. The PDF chatbot tutorial shows how to attach passages and page references to a request, and the RAG evaluation guide shows how to check that it finds the right ones.

What to build after the terminal version

Once the single request and the history loop work, a browser interface is mostly another way to collect a question and show the response. Keep the Ollama call on your application's backend if the browser can't reach the model server directly. Add streaming when you want answers to appear gradually; your code then has to assemble chunks and handle an interrupted response. The non-streaming version is easier to debug while you learn the request format.

If you want the model to fetch a value rather than invent it, continue with tool calling in Python. If you want to ask about an image, use a model that supports vision and follow the vision-language app article. Both use the same server connection but change what goes into a message and what your application has to validate.