# Private AI model: setup steps for IT

The steps below use [Ollama](https://ollama.com), free software that runs AI models locally, and [Hugging Face](https://huggingface.co), the largest public library of open models. This is the technical companion to the article [How to run your own private AI model, step by step](https://tonytonggg.com/writing/run-your-own-private-ai-model).

### Step 1: Install Ollama

On a Mac or Windows PC, download the installer from [ollama.com/download](https://ollama.com/download). On a Linux server:

```bash
curl -fsSL https://ollama.com/install.sh | sh
```

### Step 2: Download and run a model

Open a terminal and run:

```bash
ollama run gemma3
```

The first time, Ollama downloads the model, which can take a few minutes. Then you can type questions directly. Other popular choices from the [Ollama library](https://ollama.com/library) at the time of writing (September 2026) include `qwen3` and `llama3.1`, and most come in several sizes:

```bash
ollama pull qwen3:8b     # download a specific size
ollama list              # see what's installed
```

**Check the licence before business use.** Each model has its own terms, listed on its library page. Most popular open models allow commercial use, but some come with conditions.

### Step 3: Get specialised models from Hugging Face

Hugging Face hosts hundreds of thousands of models, including versions tuned for specific languages, industries or tasks. Ollama can run many of them directly. Look for models published in the **GGUF** format, then run them by name:

```bash
ollama run hf.co/bartowski/Llama-3.2-1B-Instruct-GGUF:Q4_K_M
```

The last part (`Q4_K_M`) is the level of compression. Smaller numbers mean smaller, faster files at some cost in quality, and Q4 is a sensible default.

### Step 4: Give your team a chat window

Most staff won't use a terminal. [Open WebUI](https://github.com/open-webui/open-webui) gives your private model a ChatGPT-style interface in the browser. If Docker is installed:

```bash
docker run -d -p 127.0.0.1:3000:8080 \
  --add-host=host.docker.internal:host-gateway \
  -v open-webui:/app/backend/data \
  --name open-webui ghcr.io/open-webui/open-webui:main
```

Then open `http://localhost:3000`. The `127.0.0.1` part means the chat window can only be reached from that same machine. Create your admin account straight away, because the first account created becomes the admin. It connects to Ollama on the same machine, and conversations stay on that machine.

To share it with your team on purpose, remove the `127.0.0.1:` so the command reads `-p 3000:8080`, and make sure that port is only reachable over your office network or VPN, never the open internet.

### Step 5: Connect it to your own tools

Ollama runs a local API that your existing software and automations can call:

```bash
curl http://localhost:11434/api/chat -d '{
  "model": "gemma3",
  "messages": [{ "role": "user", "content": "Summarise this enquiry in one line: ..." }],
  "stream": false
}'
```

It also offers an OpenAI-compatible endpoint at `http://localhost:11434/v1`. Many tools built for ChatGPT can point at your private model instead by changing a single address. That makes it easy to prototype with a hosted model on non-sensitive data, then switch the sensitive workflow to your own.

### Security checklist

Running a model locally only keeps data private if the machine itself is secure:

- **Keep Ollama private.** By default it only listens on the machine itself (`localhost`). Don't expose port 11434 to the internet. If colleagues need access, put the machine on a private network such as Tailscale or a VPN.
- **Lock down the server** the same way as any other: firewall on, no password logins, and admin access only over a private network. The bots in [my server logs](https://tonytonggg.com/writing/165000-break-in-attempts) will find an open AI server just as quickly as an open database.
- **Watch what else sends data out.** Plugins, web search features and "cloud sync" options can quietly send content elsewhere. Turn off anything you don't need.
- **Keep an access list.** Know who can use the system and what data it can read.