You want capable models on your own terms, but your data can’t leave the building.
Open-weight models (Llama, Qwen, Kimi, DeepSeek) on infrastructure you control: on-prem, in your VPC, or air-gapped. Evaluated on your data, sized, built and handed to your team. The same discipline covers image and video generation, and personal assistants on your own cloud.
Five components of a private AI deployment
Running your own models is an infrastructure decision as much as an AI one. So the first thing we deliver is the evaluation that tells you whether to make it.
Model selection and evaluation
We benchmark open-weight models on your tasks, not a public leaderboard: quality, latency and cost per request. If a hosted API still wins for your workload, the evaluation is where that shows.
An evaluation harness you can re-run, the scored results, and the reasoning behind the recommendation.
Deployment architecture
Serving runtime, hardware sizing, quantisation trade-offs, and the network boundary it all sits inside. Sized for the load you have, and drawn so your security team can review it before procurement.
An architecture document with a deployment diagram, the sizing work, and the options we rejected with reasons.
Retrieval over your own documents
For when the answers live in your documents rather than the model. Chunking, embeddings, hybrid search and reranking, with retrieval scored separately from generation, so you know which half failed.
The pipeline, the index, and a retrieval evaluation set with scores you can re-run after any change.
Guardrails and monitoring
Input and output policy, rate limits per caller, and an audit log with the retention your compliance people ask for. Metrics for latency, GPU use and cost per request.
Policy configuration, dashboards, and an audit trail that answers "who asked what, and what came back".
Handover
Runbooks for the boring failures, an upgrade path for the next model, and a walkthrough that ends when your engineers can swap a model without calling us.
Documentation, the repository, and your own team operating the system.
Where everything actually sits
One deployment, drawn honestly: what runs inside your network, what talks to what, and what does not exist at all when the install is air-gapped.
Public internet and hosted model APIs. Optional egress, off by default, and absent entirely in an air-gapped install.
Your network boundary · VPC, on-prem, or air-gapped
- Web application
- Flutter client
- Internal tools
Where people meet the system.
- FastAPI
- Auth
- Rate limits
- Input & output policy
- Audit log
One door in. Every request and response passes through it, which is the only reason the audit log can be complete.
- vLLM on a dedicated GPU
- Ollama / llama.cpp for smaller footprints
Model weights on your storage: Llama, Qwen, Kimi or DeepSeek at the size the evaluation justified.
- pgvector or Qdrant
- Hybrid search + rerank
Your documents, indexed inside the boundary. Nothing is copied to a third party to be embedded.
- Metrics
- Logs
- Audit trail
- GPU utilisation
- Cost per request
Everything shipped as Docker images. Compose or Kubernetes is your call; the images are the same either way.
fig. 01 · nothing inside the boundary needs the internet to answer a request. In an air-gapped install the top row does not exist, and model updates arrive the way the rest of your software does.
How the work is structured
Fixed scope, starting with the evaluation, so the decision is made on evidence rather than preference.
Evaluation first
Your real tasks against candidate models, before anyone commits to hardware. If a hosted API is the better answer, that is the deliverable and the engagement can stop there.
Architecture and sizing
The serving stack, the boundary, the sizing, the failure modes. Reviewed with your security people before procurement, not after.
Build and integrate
Serving, retrieval, gateway and policy layer, all in your repository, deployed as they land.
Deploy and handover
Monitoring, audit logging and cost tracking in place before go-live, then a recorded walkthrough with the engineers who will own it.
- The evaluation harness and the model comparison on your data
- Architecture, sizing, and the serving and retrieval stack
- Guardrails, monitoring and the audit trail
- Runbooks, documentation, and a handover session with your engineers
- The hardware, or a cloud account with GPU capacity we can deploy into
- A representative sample of real tasks and documents to evaluate against
- A named technical contact, and network or security approval where it is needed
- Someone who will own the deployment after handover (this is the one we insist on)
The day-by-day version is on the process page.
When a hosted API is honestly the better answer
Private deployment is a constraint being satisfied, not a default. Here is the argument, including the half that costs us the project.
No residency rule, no regulator, no client contract saying otherwise. A hosted API then gives you more capability per pound than a GPU you have to keep busy.
On the hardest reasoning tasks the leading hosted models are still ahead. If your work sits at that edge, your own model is a downgrade you will feel daily.
A GPU costs the same idle as busy; hosted inference bills per token. Below a steady baseline the arithmetic does not favour owning hardware.
Drivers, capacity, upgrades, someone on call. If nobody on your side will own that, a private deployment becomes an orphan.
Patient records, case files, defence work, unreleased financials. If sending it to a third party is not a decision you are allowed to make, the question is settled.
Data residency, sector rules, or a client agreement that names where processing happens. An architecture you can show them beats an assurance you cannot.
At a steady, high request rate the fixed cost of a server stops looking expensive. This is arithmetic, worth doing before either of us assumes an answer.
A hosted model can change underneath you, and your evaluations change with it. A model you host is the version you chose.
Most real deployments sit on both sides: an open model inside the boundary for your data, a hosted one for work that never sees it. We design it that way when it fits.
Tell us what the constraint is.
A regulator, a client contract, a network with no route out. Whichever it is, the answer starts with an evaluation on your own data.