The graphics card most private AI deployments are sized around, Nvidia's 96 GB RTX PRO 6000 Blackwell, launched at about $8,565. In autumn 2026, Nvidia's own store lists it at $16,000. The 128 GB DGX Spark, sold as a desktop AI computer, went from $3,999 at launch to $6,950. A shortage of memory chips did that, and it changes the arithmetic for every company thinking about running AI inside its own walls.
We set up private AI deployments for companies, so we have an interest in this answer. Here it is anyway.
Buy your own AI hardware for control, not to save money, and size it by memory bandwidth, not memory size.
Why hosted AI is cheap per word
The concession first. If your only goal is a lower bill, a hosted service will usually beat a box in your server room. Inference providers sell open models like Llama, Qwen and gpt-oss for cents per million tokens because they keep each GPU busy with thousands of customers at once. A GPU answering one person leaves most of its compute idle while it waits on memory. The same GPU answering a hundred people costs almost the same per step.
Even with that advantage, the margins are thin. Fireworks AI, one of the largest of these providers, reportedly passed $1 billion in annualized revenue in 2026 at a gross margin of around 50%, because GPU time sits in its cost of sales. Groq, which built its own inference chip, made its biggest payday by licensing that chip to Nvidia in a deal reported at about $20 billion. Selling AI by the token is a volume business. A single company's private server will never run at that kind of volume, so it will never match those prices per word.
So why do companies buy hardware at all? Because the price per word is not what they are buying.
What you are buying instead
A model on your own hardware means the prompts, documents and answers never leave your building. No vendor sees them, stores them, or trains on them. No vendor retires the model you built a process around, changes how it behaves overnight, or raises the per-seat price at renewal. The system keeps working when the internet does not. And the cost is a fixed asset with a known running bill, not a line that grows with every new hire who gets a seat.
For a clinic, a law firm, an accountant, a manufacturer protecting designs, or any business whose clients ask where their data goes, that control is the product. The trade-off is that you also own the operations: updates, security, backups and monitoring become your job, or the job of whoever you pay to do them.
Memory decides if a model loads. Bandwidth decides how fast it talks.
This is where most hardware decisions go wrong. A model's weights must sit in GPU memory, plus room for the conversation. A 70-billion-parameter model stored at 4-bit precision needs about 39 GB for its weights alone, so it will not load on a 32 GB card, and it loads comfortably on a 96 GB one.
Loading is only half of it. To write each token, the model reads all of its active weights from memory once. That puts a hard ceiling on speed: memory bandwidth divided by the size of those weights. Run the same 70B model through that formula on three machines:
- DGX Spark, 128 GB at 273 GB/s: about 7 tokens per second, at best.
- RTX PRO 6000, 96 GB at about 1,792 GB/s: about 46 tokens per second.
- One H200, 141 GB at 4.8 TB/s: about 122 tokens per second.
The Spark has the most memory of the first two and is the slowest by a factor of six. It loads the model and then answers at the pace of a slow typist, which is fine for overnight document processing and painful for staff waiting on a reply.
Real speeds land below the ceiling. Apple's M3 Ultra, with 819 GB/s, has a ceiling of about 40 tokens per second on 4-bit DeepSeek R1, a mixture-of-experts model that reads about 37 billion of its parameters per token. Early reviews reported between about 16 and 20 tokens per second, under half the ceiling. We plan on roughly half to two-thirds of the ceiling, and test before we promise anything.
Mixture-of-experts models are the exception worth knowing. A model like gpt-oss-120b has 117 billion parameters but reads only about 5 billion of them per token. It needs big memory to load and runs fast once loaded, which suits exactly the large-memory, slower-memory boxes that struggle with dense 70B models.
What it costs to own
Owning has three cost lines. The hardware, spread over its useful life. A one-time setup: choosing the model, installing a production inference server, connecting it to your documents and software, and fine-tuning it on your own work where that pays off. Then the running cost, which is mostly support and maintenance, plus electricity.
Take a hypothetical firm of 25 people, each paying $30 a month for an AI assistant, plus $500 a month in API bills. Call it $1,250 a month. Put a 96 GB workstation in at $20,000, with $12,000 of setup and $500 a month of support. Electricity for a 900-watt machine busy ten hours a day at $0.20 per kWh adds about $55 a month.
Payback takes about 46 months. Over the hardware's three-year life, owning costs about $52,000 and staying on seats costs $45,000. On cost alone, this firm should not buy. It should buy only if control of its data is worth that difference.
Now change two numbers. Sixty seats and $1,500 a month in API calls put current spend at $3,300 a month. The same hardware pays back in under a year, and over three years the gap is about $67,000 in favor of owning. Most of the decision comes from how much you spend now, not which box you pick.
We built a free calculator that runs this with your own numbers, checks whether your chosen model fits, and estimates its speed ceiling.
Where this stops applying
A 70B open model on your own hardware is not the best model available. On hard reasoning, long multi-step work and broad general knowledge, frontier hosted models are still ahead. Private deployments work best on defined, repeated tasks such as answering from your own documents, drafting in your house style, classifying and extracting, and they should be proven on a test set of your real work before anyone retires a subscription.
On-premise also does not make you compliant on its own. It removes one data transfer. Access control, logging, retention and the rest of your obligations still apply inside your walls.
And the prices in this piece have already moved once this year. Get a quote that is days old, not weeks.
Questions to answer before you buy
- Which tasks would the model do, and do you have fifty real examples with correct answers to test it on?
- Does your data have to stay inside your walls, or would you simply prefer it did?
- What do you spend on AI seats and API calls each month today, and how fast is that growing?
- Does your chosen model fit in memory with room to spare, and is its speed ceiling fast enough for people waiting on answers?
- Who will patch, back up and monitor the machine in year two?
If you want help scoping a private deployment, it sits within our AI automation work.