When to run your own model instead of calling an API

The question usually arrives as an identity: a team that wants to run its own models, or one that would rather never see a GPU. It is neither. It is arithmetic and risk, and the honest answer changes per workload inside the same product.

Here is what actually decides it, in the order the decisions tend to bind.

01. Start with the data

The strongest reason to run your own model has nothing to do with models. It is that a regulator, a lawyer or a client's security team has said the data cannot leave a boundary, and a hosted API is, by construction, the data leaving.

Be precise about which boundary, because the arguments differ and a technical audience will test them.

United Kingdom. Under UK GDPR and the Data Protection Act 2018, sending personal data to a processor outside the UK is a restricted transfer, and it is lawful with the right instrument: adequacy, or an IDTA or the UK Addendum to the EU standard contractual clauses, supported by a transfer risk assessment. Note what that implies. A hosted API with a signed data processing agreement, UK or EU regional processing and no training on your data is frequently lawful. "GDPR" on its own is not an argument for self-hosting, and a CTO who has read the regulation will say so. What usually bites instead is narrower: NHS data-handling requirements, FCA expectations on outsourcing and operational resilience for regulated firms, legal privilege, or a clause in your client's own contract that is stricter than anything a regulator wrote.

Gulf. The expectations tend to be more direct. Saudi Arabia's Personal Data Protection Law, administered by SDAIA, restricts transfers of personal data abroad and attaches conditions to them; the NDMO's rules for government and public-sector data, and SAMA's framework for financial institutions, are read by security teams as meaning in-Kingdom. Hyperscalers now operate Saudi regions, which changes what in-Kingdom can mean in practice. But the question asked in the room is usually the physical one: which building is this running in, and who can be compelled to hand it over? An open-weight model on the client's own hardware answers that in a sentence. A hosted API answers it with a diagram and a legal opinion, and sometimes that is fine.

So: if the constraint is written down and specific, self-hosting is the shortest route to satisfying it. If the constraint is "we are worried about privacy", a regional endpoint with the right contract may be a far cheaper answer, and it is worth an hour to check before anyone buys a GPU.

02. The cost arithmetic

The comparison is usually done in a spreadsheet with a token price and a monthly volume, and it is wrong because the two sides have different shapes.

A hosted API is a variable cost: you pay per token, and at three on a Sunday morning you pay nothing. Self-hosting is a capacity cost: you pay for the GPU-hours you have reserved whether or not anything is queued. So the comparison is not price-per-token against price-per-token. It is your token bill against the cost of a machine divided by how busy you can keep it.

HOURS_PER_MONTH = 730

# Fill these in with today's numbers. All four of them move.
API_PRICE_PER_MTOK = ...   # blended input+output, your provider, this week
GPU_COST_PER_HOUR  = ...   # reserved instance, or amortised hardware + power + rack
TOKENS_PER_SEC     = ...   # measured on YOUR model, batch size and prompt shape
UTILISATION        = 0.25  # measured, not hoped for


def hosted_monthly(mtok_per_month: float) -> float:
    return mtok_per_month * API_PRICE_PER_MTOK


def self_hosted_monthly(gpus: int) -> float:
    return gpus * GPU_COST_PER_HOUR * HOURS_PER_MONTH


def capacity_mtok(gpus: int) -> float:
    """Tokens you can actually serve per month, not the spec-sheet number."""
    seconds = HOURS_PER_MONTH * 3600 * UTILISATION
    return gpus * TOKENS_PER_SEC * seconds / 1_000_000

Three things fall out of that which people miss.

• Utilisation dominates everything else. At ten per cent utilisation you are paying ten times the notional price per token. An interactive feature used during office hours in one time zone is the worst case for this. Queued, batchable work is the best case, and it is the one that makes self-hosting look obviously correct.
• Throughput is not a spec-sheet number. Tokens per second depends on the model, the quantisation, the serving stack, the concurrency and the ratio of prompt to completion in your real traffic. A long-context retrieval workload and a classification workload on the same card can differ by an order of magnitude. Measure it on your own prompts before it goes anywhere near a business case.
• The GPU is not the whole bill. Add the second machine you need for redundancy, the environment you evaluate on, storage for weights, egress, and the engineering time in section five, which is usually the largest line and the one left off the sheet.

The honest shape of the answer: below some sustained volume the API wins on cost and it is not close; above it, with work you can batch, self-hosting wins and that is not close either. The crossover is specific to your traffic. The useful exercise is to compute where yours sits and then check whether you are within about three times of it. If you are, cost is not deciding this. Pick on one of the other reasons.

03. Latency

Running your own model does not automatically make anything faster. What it makes is predictable, and it moves the compute closer to the user. Those are different wins and worth separating.

Distance is the part you can bank. A user in Riyadh calling an endpoint in Northern Virginia pays that round trip on every request, and an agent that makes six sequential model calls pays it six times. Moving inference into a Saudi region (your own or a provider's) removes it. This is measurable before you build anything: time the endpoints from where the users actually are.

Tail latency is the other part. On a shared API you inherit someone else's queue, someone else's rate limits and occasional multi-second outliers you cannot explain to a client. On your own serving stack the queue is yours: you choose the batch size, you choose whether request two hundred waits or is rejected, and the ninety-ninth percentile becomes something you engineer rather than observe.

What you do not get free is raw speed. Frontier providers have put a great deal of specialist work into serving. Matching it means doing some of that work yourself: a real inference server rather than a naive loop, continuous batching, a quantisation you have evaluated rather than guessed at, and prefix and KV-cache reuse for the parts of your prompt that never change. Hold two numbers separately while you tune: time to first token, which is what a user feels, and inter-token latency, which decides whether streaming reads as fast. Optimising the wrong one of those is a common way to lose a week.

04. The quality gap

This is where most writing on the subject stops being useful, because it argues about leaderboards.

The honest position: open-weight models (Llama, Qwen, DeepSeek, Mistral, Gemma) are close enough to hosted frontier models to be the right answer on a large class of tasks, and clearly behind on another class. Which class yours is in is not a question a public benchmark answers, because a benchmark is not your data.

Where the gap is small or gone in practice: extraction into a schema, classification and routing, summarisation within a domain you can supply examples from, rewriting, and anything where the output format can be constrained and then validated in code. A well-chosen model in the 8B to 70B range, with a decent prompt, a few examples and grammar-constrained decoding, does these to a standard your users will not distinguish.

Where the gap is still real: long-horizon agentic work with many tool calls, where a small per-step error rate compounds into a visibly worse system; code generation in unfamiliar repositories; dependable use of very long contexts, where "supports 128k" and "is accurate at 128k" are different claims; and robustness when the input is messy, adversarial, or in a language other than English. Arabic performance in particular varies far more between open models than headline evaluations suggest, which matters if half your users are in the Gulf.

So the only recommendation worth giving: build a task-specific evaluation set from your own data before you choose. A hundred real examples with agreed correct answers, scored by the same script against both options, settles the question in an afternoon, and settles it for your task rather than someone else's. It is also the artefact you will want the next time you change weights.

If the gap turns out to matter and the data constraint is real, the answer is not "accept worse". It is usually a narrower task, better retrieval, or a fine-tune on your own examples, all of which are more available to you when the model is yours.

05. What you operate

This is the line that decides most of these projects and is rarely in the business case. Running your own model is not a deployment. It is a service you now operate, and it arrives with:

• GPU capacity to plan, plus the driver, CUDA and kernel lifecycle underneath it
• an inference server (vLLM, TensorRT-LLM, llama.cpp) with its own release cadence and its own breaking changes
• weights to version, store and roll back, and a re-evaluation every time they change, because "same model, newer quantisation" is a behaviour change
• observability: per-request latency, queue depth, GPU memory, tokens in and out, and cost attribution per tenant if you have more than one
• a decided behaviour under load: at three times expected traffic, do you queue, shed, or spill over to a hosted API?
• someone on call who knows what an out-of-memory in a serving process looks like
• and the security work (patching, network policy, access control), which was the point of doing this in the first place

For a team already running Kubernetes with an on-call rota, that is an increment. For a team of four shipping a product, it is a person. Say the number out loud before signing: this is a standing cost in engineering attention, and it does not fall after launch, because model releases keep arriving whether or not you have time for them.

Which leads to an uncomfortable check. If the reason for self-hosting is cost, and the operational burden is a person, recompute. The API bill is often smaller than the salary.

06. The split that usually wins

Most systems end up mixed rather than pure, because the constraint is rarely uniform across a product. The pattern is to route on data class and task difficulty, not on preference: anything touching regulated or personal data goes to the local model inside the boundary; anything that needs the hardest reasoning and can be made safe (by redaction, by restatement, or because it was public to begin with) may go to a hosted frontier model. The routing rule is code. It is auditable, it is testable, and it is the artefact you put in front of a security review.

        request
           │
           ▼
  ┌────────────────────────────┐
  │ classifier + redaction     │   in-boundary,
  │ gateway                    │   no egress
  └─────────────┬──────────────┘
     regulated  │  data?
        ┌───────┴────────┐
    yes │                │ no
        ▼                ▼
  ┌─────────────┐  ┌──────────────────┐
  │ local model │  │ hosted frontier  │
  │ VPC/on-prem │  │ model API        │
  └──────┬──────┘  └────────┬─────────┘
         └────────┬─────────┘
                  ▼
        response + audit record

fig. 01: the classifier and the audit record are the parts that make this defensible. A split you cannot log is a split you cannot defend in a review.

07. A checklist

Run these in order. The first genuine yes usually decides it.

1. Is there a written constraint (regulation, contract, or a security policy someone has actually read) saying this data cannot leave a boundary? If yes, self-host, or find a compliant regional endpoint and check it satisfies the same clause.
2. Is the volume sustained and batchable, and can you state a utilisation figure you have measured? If you cannot name it, cost is not your reason.
3. Is there a latency requirement you can state as a number, from a place you can name? If so, measure the round trip first. The answer may be a region rather than a GPU.
4. Have you scored both options on a hundred of your own examples? If not, the quality question is still open and everything above it is premature.
5. Who operates it on the Tuesday after launch, and do they already do this kind of work?

If the first four are no and the fifth is nobody, call the API. That is advice we give more often than the opposite, and it is usually the cheaper of the two.