How to set up a private, self-hosted AI chat for a small team with Ollama or vLLM and Open WebUI

Private means one thing here: the model runs on a machine we control, and no prompt, file or answer goes to a third party. That rules out cloud APIs. It leaves two engines worth running. Ollama is one binary that pulls models by name and is the right start for a handful of people. vLLM is a Python server built for many simultaneous chats; it batches every open conversation into the same pass through the GPU, so a queue of requests does not turn into a queue of people. Both expose an API that Open WebUI, the chat front end we use, can talk to.

The question that decides it is not how many people have accounts but how many type at the same time. Five people who each ask something every few minutes rarely overlap. Twenty people in a workshop do. Our rule from running both: Ollama up to about five concurrent chats, vLLM beyond that, and vLLM only when the server has an NVIDIA GPU. The rest of this guide has an Ollama step and a vLLM step; you do one of them. Press through the tree with your real numbers and it tells you which.

Two things live in GPU memory. The weights, which depend on the model size and the quantisation, and the KV cache, which is the working memory of each open conversation and grows with context length and with the number of people chatting at once. Unsloth's requirements page gives the weight side: an 8B model at 4-bit needs about 6 GB, a 27B about 22 GB, a 70B about 41 GB. Those figures include a little overhead, which is why the calculator multiplies the raw size by 1.1.

The cache side has no single published number because it depends on the model's architecture. We use a rough rule: about 1 GB per 8,000 tokens of context for an 8B model, scaled by model size, per concurrent user. Newer models with grouped-query attention use less, so treat the total as a ceiling, not a target. The calculator also hands you the two Ollama variables from step 3 that match your answer, and names a GPU class: 16 GB is a T4 or an RTX 4080, 24 GB is an RTX 4090 or an L4, 48 GB is an L40S or an RTX 6000 Ada, 80 GB is an A100 or an H100. If the total lands above 80 GB you need vLLM with tensor parallel across two cards, or a smaller model. We have not verified any vendor's users-per-GPU claims; this is the arithmetic we trust instead.

Install Ollama on the server with the one-line script from ollama.com/download, or run it inside the Open WebUI container in step 5. Either way, three variables matter, and all three default to values meant for one person on a laptop.

OLLAMA_HOST defaults to localhost. Set it to 0.0.0.0:11434 so other machines, and the Open WebUI container, can reach it. OLLAMA_NUM_PARALLEL defaults to 1, which means the second person to press Enter waits for the first answer to finish. Set it to your concurrent user count from the calculator. OLLAMA_CONTEXT_LENGTH defaults to 4096 tokens, which is one long document before the model forgets the start of the chat. Set it to 8192 or 16384. Memory scales with parallel times context, which is why the calculator ties them together. OLLAMA_MAX_LOADED_MODELS defaults to 3 per GPU and is fine unless people switch models all day.

On Linux with the installer, the variables go into the systemd unit: run the edit command below, add the lines under [Service], then reload and restart. On macOS set each one with launchctl setenv and restart the Ollama app. In the bundled container, pass them as -e flags on docker run; Ollama inside reads the same names from the container environment. Pull a model, ask it something from a second machine with curl, and only then move on.

vLLM is for the server that is never idle. It keeps the GPU full by batching every open conversation into the same forward pass, so ten people chatting feels like one. It needs an NVIDIA GPU, a recent driver and Python. Install it with pip in a fresh virtual environment and start it with the command below.

Four flags matter. --gpu-memory-utilization 0.92 lets vLLM reserve 92 percent of the card for weights and cache, leaving the rest for the driver. --max-model-len 16384 caps the context; the cache is sized from it, so this is where you trade context for concurrent users. --tensor-parallel-size 1 is one GPU; set it to 2 for a model split across two cards. --api-key is the password every client must send, and Open WebUI will send it for the whole team. The server speaks the OpenAI API at /v1/chat/completions, /v1/embeddings and /v1/models, which is why almost every AI tool can point at it.

Start with a model that fits with room to spare. vLLM is happiest with full-precision or AWQ and GPTQ weights from Hugging Face; the GGUF files Ollama uses are not the main path here. Confirm it is up with the curl line, which lists the loaded model, and leave the terminal open the first day so you see the batching stats scroll past.

Open WebUI is the chat window: accounts, model picker, file upload and chat history, all stored in a volume on your server. Install Docker, then run the command below. The :ollama image bundles Ollama in the same container, with --gpus=all handing it the card and the ollama volume keeping downloaded models. Port 3000 on the host maps to 8080 inside. If you chose vLLM, use the :main image instead, drop --gpus=all, and add -e OPENAI_API_BASE_URL=http://SERVER-IP:8000/v1 so Open WebUI talks to vLLM's API. The key goes in the matching OPENAI_API_KEY variable; we have not re-verified that name against the current environment variable reference, so check it there before you rely on it.

WEBUI_SECRET_KEY signs every login session. The quick-start generates one inline, which is fine the first time and a trap the second: run the same command on redeploy and a new key is minted, and everyone is logged out. Generate it once, keep it in a file, and pass the same value forever.

Open http://SERVER-IP:3000. The first account created becomes the admin. Create it yourself, immediately. Then go to Admin Panel > Settings > General and turn off Enable New Sign Ups, so nobody who finds the address can make an account. From here on people join through SSO in step 7, or you add them by hand.

Single sign-on means the team logs in with the Google Workspace or Microsoft 365 account they already have, and leaving the company means leaving the chat. The first half happens at the identity provider, where you register Open WebUI as an application and get a client ID and a secret.

For Google: open the Google Cloud console, pick or create a project, go to APIs & Services, then Credentials, and create an OAuth client ID of type Web application. Add your public address to the authorised JavaScript origins and the callback to the authorised redirect URIs. The callback for Google is your WEBUI_URL followed by /oauth/google/callback. Copy the client ID and the secret.

For Microsoft: register an application in Entra ID, add a web redirect URI in the same shape, create a client secret, and note the tenant ID; Open WebUI needs all three, and the Microsoft scope must include offline_access or logins expire early. Open WebUI's SSO page lists the Microsoft and generic OIDC callback paths; we only give the Google one here because it is the one we have run.

A rule that saves an afternoon: WEBUI_URL in the next step must match the redirect URI exactly, same scheme and host and no trailing slash, and it must be the address people type. That means a reverse proxy with HTTPS goes in front before you do any of this.

Now the Open WebUI half. It is all environment variables, and the names are the whole game. The ones that exist: GOOGLE_CLIENT_ID and GOOGLE_CLIENT_SECRET; MICROSOFT_CLIENT_ID, MICROSOFT_CLIENT_SECRET and MICROSOFT_CLIENT_TENANT_ID; and for any other OpenID Connect provider OAUTH_CLIENT_ID, OAUTH_CLIENT_SECRET and OPENID_PROVIDER_URL, which points at the provider's /.well-known/openid-configuration document. The ones that do not exist, and that we have seen in forum answers: OPENID_CLIENT_ID and WEBUI_ENABLE_SSO. Set them and nothing happens, with no error.

Three more. ENABLE_OAUTH_SIGNUP=true lets a person who signs in through SSO get an account without an admin creating one first. OAUTH_MERGE_ACCOUNTS_BY_EMAIL=true links an SSO login to an existing account with the same email, which is what you want for the accounts you made by hand, and it is only safe when the provider verifies email addresses, which Google and Microsoft do. WEBUI_URL is the public address; without it the callback is built from the wrong host and the provider rejects it. For roles, ENABLE_OAUTH_ROLE_MANAGEMENT with OAUTH_ADMIN_ROLES lets a group at the provider decide who is an admin.

Fill in the builder, copy the command, stop and remove the old container (the volumes keep everything), and run the new one. For Google, delete the third variable line; the builder cannot hide it. Two open issues worth knowing as of October 2026: users provisioned through SCIM cannot sign in with SSO since v0.12.0 (open-webui #32347), and OIDC token refresh ignores OAUTH_TOKEN_ENDPOINT_AUTH_METHOD (#32506). Plain Google and Microsoft logins without SCIM are not affected.

Private is a property of the whole setup, not of the model. Walk the list. Open WebUI can call outside services if you let it: a cloud API key in the admin settings turns your private chat into a proxy for a cloud model, and web search features send queries to a search provider. Leave them off unless the team has agreed in writing. Keep port 11434 and port 8000 closed to the internet; only Open WebUI needs them, from inside the server or the Docker network. Put the chat behind HTTPS on a reverse proxy, with the firewall letting in 443 and nothing else.

Back up the open-webui volume. It holds every account, chat and uploaded file, and a lost volume is a lost year of team knowledge. The ollama volume is just downloaded models and can be pulled again. Tell the team what is stored and who can read it: an admin can be given access to everyone's chats, which is a feature for support and a surprise for people who did not know. Decide, then say so.

Pin the model. Models are pulled by tag and tags move, so an update can change answers overnight; ollama list shows the digest you are running, and vLLM takes a revision on the Hugging Face name. Finally, test the claim instead of believing it. Block outbound traffic at the firewall, or pull the cable, and ask a question. If the answer still comes, nothing is leaving.

The honest comparison is per month. ChatGPT Team charges per seat; check the current price at openai.com because it changes and we are not going to guess it here. Self-hosting charges you for a GPU, whether you buy a card and pay for power or rent one by the hour, plus the hours a person spends keeping it running, which is the line everyone forgets.

Put your numbers in. The rental rate is whatever your cloud charges for the GPU class the step 2 calculator gave you; a 24 GB card sits in a different price band from an 80 GB one. A card you own is the purchase price spread over three years plus electricity, turned into an hourly figure. The maintenance line is honest at two hours a month once it runs; the first month is far more. The calculator gives you both monthly figures and the number of seats at which they cross.

Our experience: for a team under ten, self-hosting rarely wins on money alone, and the reason to do it is the data, not the invoice. Somewhere past twenty or thirty, with a 24 GB card rented at the cheaper end of the market serving an 8B or 14B model, the arithmetic turns in your favour, and the model is still smaller than the cloud ones. That trade is the whole decision, and it is a fine one to make on purpose.