Research Update

AI Hosting & Model Selection

Which AI models we can actually use, and where we can host them, given the constraint that client data must not sit on a third party's infrastructure.

Prepared by Saam Sani, IT & Innovations · Metro Testing & Engineering · July 2026

This is a research update, nothing more. No proposal, no recommendation, no vendor, no budget ask. We don't know the project yet. The goal is simply to get the reading out of my head and onto paper, so there's something concrete to work from when a real project lands. The likely eventual use case is a RAG setup (an AI that answers questions from our own documents), but most of what follows is about AI hosting and model choice generally.

1Which AIs we can use, the central finding

This turned out to be the most important thing I learned, and it's a hard constraint rather than a preference. AI models split into two categories that are not interchangeable.

Models we can OWN open-weight

The model file is published and free to download. Once we have it, it runs on our hardware, offline, indefinitely. No account, no API key, no subscription, no per-query fee, nothing phoning home.

Examples: Gemma (Google), Qwen (Alibaba), Mistral, Llama (Meta), DeepSeek, GLM.

Models we can only RENT proprietary

Accessed over the internet as a service. The model file is never released, so it cannot be installed on our hardware at any price, under any contract. This covers Claude, GPT and Gemini.

Examples: Claude (Anthropic), GPT (OpenAI), Gemini (Google).

I checked the Claude case specifically because it came up: Anthropic doesn't publish model weights and there's no self-hosting option, all inference runs on their infrastructure or an approved cloud partner's. Same for GPT and Gemini. There are custom arrangements for classified government work, but nothing available to a private company.

The consequence: if the requirement is genuinely that client data stays on infrastructure we control, the model choice is limited to open-weight models. That isn't a compromise we're choosing, it's the only category that can physically run in our building. Worth knowing the quality gap has narrowed considerably: open-weight models now trail the proprietary leaders by a fairly small margin, and the remaining gaps are mostly in long multi-step reasoning and agentic work, not in reading documents and summarising them accurately, which is what our likely use case needs.

2The Builder vs the Engine

Worth separating these, because they get conflated and have completely different privacy implications.

The Builder

The AI coding assistant that helps developers write the code. We'd use Claude Code for this. It sees source code only (Dockerfiles, pipeline code, config), never client data, and it's involved during the build and then done.

Flagged mainly as a policy point: source code would be going to a cloud service. Most organisations are comfortable with that, and it's the same category of exposure as any cloud dev tooling, but you may want it explicitly signed off rather than assumed. It has no bearing on the finished system either way.

The Engine

The AI models that run inside the system and process our data at runtime. These see client data on every query, so these are the ones that must be open-weight and local.

Section 3 covers which ones. This is where the data-residency requirement actually applies.

3Which engine models to consider

First thing that surprised me: it's three models, not one. They work in sequence and each does a different job.

3a. Embedding model

Converts documents into a form searchable by meaning rather than keywords, so a search for "payment terms" finds a document that says "invoicing schedule."

BGE-M3 MIT the common default for self-hosted setups. 568M parameters, 100+ languages, and it does dense, sparse and multi-vector retrieval in a single model, which means hybrid search without a separate keyword index. Small and cheap to run.
Qwen3-Embedding Apache 2.0 0.6B / 4B / 8B, scores higher on benchmarks. The 0.6B version is the best quality-per-VRAM option at around 1.5GB; the 8B if we have GPU headroom.
nomic-embed-text the lightest option, about 0.3GB, if we want a minimal footprint.

3b. Reranker

The embedding model returns roughly 50 rough matches; the reranker re-reads them properly and picks the best 5. Reportedly lifts precision substantially, and it's the component first-time builds most often skip.

BGE-reranker-v2 Apache 2.0 pairs naturally with BGE-M3.
Qwen3-Reranker pairs with Qwen3-Embedding.

3c. Main model (the one that writes the answer)

Options here depend entirely on how much GPU memory we have, so they map to the hosting tiers in Section 4.

GPU memory tier	Maps to	Candidate main models
~16 to 32GB	Single added GPU (A1) or modest workstation	Gemma 4 12B (practical start for ~16GB) · Qwen3.6-27B at 4-bit
~32 to 96GB	Proper workstation (A2)	Qwen3.6-27B comfortably at better precision · larger Gemma 4 variants · Mistral Small
80GB+	Rack server, multi-GPU (A3 / A4)	GLM-5.2 (strongest all-round open-weight, long context) · Kimi K3 (top on coding/agentic) · DeepSeek V4 · MiniMax M3 · Mistral Large 3

The 80GB+ models don't run on a single consumer GPU; they need 80GB-class multi-GPU setups even at 4-bit, plus headroom on top.

Rough sizing rule: a model needs about 2GB of GPU memory per billion parameters at full precision, roughly 1GB at 8-bit, roughly 0.6GB at 4-bit. Then add 20 to 40% on top for working memory, which matters more than usual for us because retrieved document text makes the inputs long.

Important, we would NOT need one of the big ones. The answer comes from our documents, not the model's own knowledge, so a mid-sized model that summarises retrieved text accurately beats a much larger one costing several times more in hardware. Quality is driven mainly by retrieval, whether the right passages got found, not by model size. Reportedly a common and expensive mistake.

3d. Serving software

The models need something to run them.

vLLM the production default. Supports 200+ model architectures and runs on NVIDIA, AMD, Intel, Apple Silicon and CPU, which makes it the safest bet if we might change hardware.
SGLang reportedly better specifically for retrieval-heavy and multi-turn workloads, so worth benchmarking against vLLM for our case.
Ollama easiest to install, good for early testing, not intended for production concurrency.

One caveat on all of the above: this list will be out of date within a couple of months. New open-weight models ship constantly and the leaderboard order changes materially every few weeks, several of the models above didn't exist six months ago. Two implications: don't treat these as final selections, re-check at the point of procurement; and whatever we build, keep the model swappable via configuration rather than hardcoded, so upgrading is a config change rather than a rebuild.

4Where an AI can be hosted

Four broad options. Only the first three keep data on infrastructure we control; the fourth is included for honest comparison.

Option	Data residency	Indicative cost	Notes
A. On-premises	Never leaves our premises	Card only $2k to $10k · workstation $8k to $20k · rack server $50k to $130k · HA $250k to $400k per node	Our hardware, our building, open-weight models. Software and models $0. Needs a physical GPU.
B. Colocation	Same isolation as A	Same hardware plus rack rental (quote)	We own identical kit, but it sits in a professionally managed datacenter. Easier auditor story than an office server room.
C. Our cloud tenant	Our tenant, but vendor is a third party	Permanent monthly GPU bill (quote)	Rented GPU instances in our own AWS / Azure / GCP account running the same open models. Whether this satisfies the requirement is a compliance call.
D. Rented model	Leaves our infrastructure	Lowest by far at low volume, usage-based	Claude / GPT / Gemini via API, Bedrock or Azure. Best quality, fastest to deploy, but does NOT meet our stated requirement. For comparison only.

The on-prem ladder (Option A in detail)

"On-prem" isn't one thing, it spans reusing what we already own through to dedicated new kit. All four count.

A1. Existing server (VM or container). The software can run as a VM or container on a virtualization host we already own (VMware, Hyper-V, Proxmox). No dedicated box required at the software level. The constraint is underneath it, see the GPU note below.
A2. Dedicated workstation. A single powerful desktop-class machine in a locked room. Genuinely on-prem, no rack or server room needed. Indicative range $8k to $20k.
A3. Dedicated rack server. Built for continuous operation, 2 to 4 professional GPUs, redundant power, out-of-band management. Sits in a rack. Indicative range $50k to $130k.
A4. Multi-node / high availability. Two or more nodes so there's no single point of failure, for when the system becomes business-critical. Indicative range $250k to $400k per node.

From the software's point of view all four are identical, Linux plus containers either way. The choice is purely capacity, cost and facilities.

The GPU reality. AI inference needs a physical GPU. Virtualization can slice up CPU and memory in software, but it cannot create a GPU, the maths genuinely requires GPU silicon on the host. Standard business servers almost never have one. The cheapest route is checking whether an existing host can take a card: it needs a free full-length PCIe x16 slot, physical clearance (these cards are long and often double-width), spare power (300 to 600W per card), and airflow rated to cool it. Clear all four and the only purchase might be the card itself. In practice most general-purpose servers fail at least one, since they're specified for CPU and RAM, not for a card that runs like a space heater. Worth checking our estate before assuming we need to buy anything.

Common to every on-prem and colocation option: data never leaves our premises · software and models cost $0 (everything named in Section 3 is free to download) · $0 per query, ever · ongoing cost is 0.5 to 1 FTE to run it plus power and cooling (~$1k/yr at workstation scale, ~$10k/yr at multi-node scale). Auditors reportedly follow "the data never leaves our building" with "who can physically reach the machine, and would you know?", so access control and logging on the room matters as much as the hardware.

Worth being clear-eyed: if our email and files already live in Microsoft 365 or Google Workspace, that data already sits with a third party today. Worth understanding where our actual exposure is, rather than isolating one system while the same data lives elsewhere.

5Licensing, one trap to know

"Free to download" doesn't automatically mean "free to use commercially." The licence on each model matters and they are not consistent.

Safest MIT and Apache 2.0. Both fine for commercial use with no conditions worth worrying about (Apache adds an explicit patent grant, which is why lawyers tend to prefer it). Of the models above, BGE-M3 is MIT; the Qwen and BGE-reranker models are Apache 2.0.
Needs a read Gemma, Llama, Mistral's larger models and several others ship under custom or community licences. Generally permissive for commercial use, but they carry conditions, and terms can differ between releases within the same family. Verify per specific model rather than per family.
The trap CC-BY-NC means non-commercial use only, so unusable for us. Some of the highest-scoring embedding models are licensed this way and sit at the top of the leaderboards, so it's easy to pick one without noticing. Anything SSPL also needs legal review before production.

Sensible rule: if it isn't MIT or Apache 2.0, someone reads the licence before it goes in.

6Security, applies to every option equally

From the reading, this is the real project risk, more than the hosting or model decision. Because we hold multiple clients' data, permissions cannot be handled by instructing the AI. The model doesn't reliably understand authorisation, once one client's content is in front of it, it can surface in an answer to another client. Filtering has to happen at the data layer, before anything reaches the model.

Every document chunk tagged with its owner at ingestion time
Users authenticate through our existing directory (AD / LDAP)
Search filtered before it runs, so unauthorised content is never retrieved
Stronger option: a separate collection per client, making cross-contamination structurally impossible rather than merely filtered against
Full audit logging: every query, every document retrieved, and who asked
Encryption at rest, TLS between components, a sync job that updates permissions when staff change roles or leave, and a deletion process that purges source documents, chunks, embeddings AND logs when a client exercises a deletion right

If multiple applications end up connecting to it, the API layer needs its own controls, each one authenticates and the server enforces permissions itself. A client application should never be trusted to do the filtering, or a compromised client has access to everything.

One hardening note specific to self-hosting: several tools in this space ship with usage telemetry enabled by default. Turning it off and putting the stack behind egress-blocked networking would let us demonstrate to an auditor that nothing leaves, rather than just asserting it. Legal and compliance would be worth involving early rather than retrofitting.

7Cost, one honest point

Self-hosting is not the cheaper option at our likely volume, renting a commercial model would cost less per query. What we'd be buying is data sovereignty: client data physically cannot leave a machine we own and lock. That's a legitimate reason to spend the money, but worth framing it that way rather than as a cost saving, because the numbers won't support a cost-saving argument. And staffing (0.5 to 1 FTE) is likely the largest line item over three years, often exceeding the hardware itself, and it tends to get left out of these estimates entirely.

8Two other findings worth knowing

There are mature open-source platforms that already ship the unglamorous parts: document connectors, permission syncing, hybrid search, reranking, audit logging and an API for other systems to plug into. Onyx, RAGFlow, AnythingLLM, Open WebUI. Onyx (MIT licensed, self-hosted, bring-your-own-model) looked the strongest fit on paper for permission-aware retrieval, though I haven't tested anything. Worth knowing the option exists before anyone scopes a build from scratch.

Document ingestion, not the AI, is reportedly where these projects fail. Messy scanned PDFs, tables and poor text splitting wreck answer quality far more often than model choice does, and it's consistently underestimated.

Where this sits. A research update, and that's all it is. We don't know the project yet, so I'm not proposing anything, recommending a vendor, or asking for budget. Once we have it and a detailed picture of what we actually want (scale, data sources, user numbers, how business-critical it needs to be) choosing between these options becomes much more straightforward, and most of the ranges above will narrow considerably. Sending it now so we're not starting from a blank page. You'll obviously want to run your own evaluation and land on what fits our environment best, happy to talk through any of it, or dig further into whichever part looks most relevant to you.