"Self-host AI on-premise so data never leaves the organisation" is an increasingly popular direction. But before buying a GPU, the first question has to be: how much does it cost? This article breaks down the three real costs of running a large language model (LLM) yourself: VRAM (how much GPU memory a model needs), electricity (what a GPU running 24/7 costs at official Vietnam rates), and hardware — then compares them against calling a cloud API for the same model. Every figure is sourced from public data, checked on 18 July 2026.
Quick summary
- VRAM needed: estimate quickly as parameters × bytes per parameter. A 70–73 billion-parameter model needs ~36 GB of VRAM at INT4 quantization, ~145 GB at FP16 (weights only, before overhead).
- GPU electricity: at Vietnam's average retail electricity price of VND 2,204.0655/kWh (excl. VAT, from 10 May 2025 — Ministry of Industry and Trade), one H100 (700W TDP) running 24/7 draws ~504 kWh ≈ VND 1.11 million/month in electricity — GPU alone.
- Hardware is the biggest and most variable cost — the capital to buy GPUs typically dwarfs electricity; it is the variable that decides the break-even point.
- Versus cloud API: for the same Qwen2.5-72B, the reference cloud price is ~$0.475 in / $0.495 out per 1M tokens (Artificial Analysis). At low–moderate volume, renting the API is still cheaper than self-hosting because there is no hardware capital.
- When self-hosting pays off: high, continuous usage — or when you are required to keep data on-site (compliance, data sovereignty) — not as a default "self-hosting is always cheaper".
Key facts (sourced)
- Vietnam average retail electricity price: VND 2,204.0655/kWh (excl. VAT), effective 10 May 2025 — Ministry of Industry and Trade / EVN.
- GPU TDP (official NVIDIA): RTX 4090 = 450W; A100 = 300W (PCIe)/400W (SXM); H100 = up to 700W (SXM).
- Parameter counts (HuggingFace model cards): Qwen2.5-7B = 7.61B; Qwen2.5-32B = 32.5B; Llama 3.3 70B = 70.6B; Qwen2.5-72B = 72.7B.
- Cloud API price for the same model (Qwen2.5-72B, provider median — Artificial Analysis): $0.475 input / $0.495 output per 1M tokens.
- Reference exchange rate: ~VND 26,440/USD (Vietcombank sell, 17 Jul 2026).
How much VRAM does a model need?
A model needs a minimum amount of GPU memory (VRAM) equal to its parameter count times the bytes stored per parameter. At FP16/BF16 precision each parameter takes 2 bytes; quantizing down to INT8 makes it 1 byte, INT4 makes it 0.5 byte. So a 72.7-billion-parameter model (Qwen2.5-72B, per its HuggingFace model card) needs ~145 GB of VRAM at FP16, but only ~36 GB at INT4 — enough to fit one 80 GB GPU. That is why quantization is nearly mandatory for self-hosting.
Important caveat: the numbers in the table below are weights only. In practice you must add roughly 20–30% for KV-cache, activations and runtime overhead — so treat these as a floor, not a recommended configuration. (GB here uses billions of bytes, to compare cleanly against GPU capacities.)
| Open model | Parameters | VRAM FP16 (2 bytes) | VRAM INT4 (0.5 byte) | Suggested GPU (INT4, incl. overhead) |
|---|---|---|---|---|
| Qwen2.5-7B | 7.61B | ~15.2 GB | ~3.8 GB | 1× RTX 4090 (24 GB) — ample |
| Qwen2.5-32B | 32.5B | ~65 GB | ~16.3 GB | 1× RTX 4090 (24 GB), just fits |
| Llama 3.3 70B | 70.6B | ~141 GB | ~35.3 GB | 1× A100/H100 80 GB, or 2× RTX 4090 |
| Qwen2.5-72B | 72.7B | ~145 GB | ~36.4 GB | 1× A100/H100 80 GB, or 2× RTX 4090 |
The takeaway: choose the model size first, then the GPU. A quantized 7–32B model can run on a single 24 GB consumer card; but the 70B-and-up tier already needs an 80 GB data-centre GPU or multiple cards stacked — a large jump in cost.
Electricity: what does a GPU running 24/7 cost?
The monthly electricity of a GPU running continuously is TDP (kW) × 720 hours × electricity price. At Vietnam's average retail price of VND 2,204.0655/kWh (excl. VAT, effective 10 May 2025 per the Ministry of Industry and Trade), an H100 rated at 700W running 24/7 draws ~504 kWh and costs about VND 1.11 million/month — and that is the electricity of the GPU alone.
We use the average price as a transparent, verifiable anchor. In reality, a business's bill also depends on voltage level and time-of-use band (off-peak/normal/peak) under the tariff in Decision 1279/QĐ-BCT, plus VAT — and, importantly, total system electricity (CPU, RAM, fans) plus room cooling is typically 1.5–2× the bare GPU draw. In other words, roughly double the figures below to estimate a real bill.
| GPU | VRAM | TDP (power) | Energy/month (24/7) | Electricity/month* |
|---|---|---|---|---|
| NVIDIA RTX 4090 | 24 GB | 450W | 324 kWh | ~VND 714,000 |
| NVIDIA A100 (PCIe) | 40/80 GB | 300W | 216 kWh | ~VND 476,000 |
| NVIDIA A100 (SXM) | 80 GB | 400W | 288 kWh | ~VND 635,000 |
| NVIDIA H100 (SXM) | 80 GB | 700W | 504 kWh | ~VND 1,111,000 |
*Electricity for the GPU alone, at the average price excl. VAT. A real bill is higher due to CPU/RAM/cooling (typically ×1.5–2), VAT, and voltage/time-of-use tariffs.
The surprise for many: GPU electricity is usually NOT the biggest line. A data-centre GPU costs VND 0.5–1.1 million/month in power — significant, but small next to the capital to buy that same GPU. That is why a TCO analysis cannot skip hardware.
The three components of on-premise TCO
The total cost of ownership (TCO) of self-hosting an LLM has three parts, and the largest is hardware capital — usually far more than electricity and operations combined. The table below ranks the three by magnitude and volatility, so a business knows which variable to lock down before deciding.
| Component | What it includes | Magnitude & volatility |
|---|---|---|
| 1. Hardware capital (CAPEX) | GPUs, servers, networking, storage; depreciated over the lifecycle | Largest, most volatile with the GPU market — decides the break-even point. Out of scope for this article's electricity math. |
| 2. Electricity (OPEX) | GPU power + the rest of the machine + cooling | Medium. GPU alone ~VND 0.5–1.1M/month/GPU; multiply by ~1.5–2 for the full system. |
| 3. Operations (OPEX) | MLOps staff, model updates, monitoring, maintenance, redundancy | Steady but easily forgotten; for small organisations it can be the "most expensive" part due to lack of people. |
Self-hosting versus renting a cloud API
At low-to-moderate usage, renting a cloud API for the same model is usually cheaper than self-hosting, because you avoid hardware capital. Take Qwen2.5-72B itself: the reference cloud price (provider median, per Artificial Analysis) is $0.475 per 1M input tokens and $0.495 per 1M output tokens. Converting at ~VND 26,440/USD (Vietcombank, 17 Jul 2026):
| Monthly volume | Cloud API cost | Self-host electricity (80 GB GPU, excl. CAPEX) |
|---|---|---|
| 50M input + 50M output tokens | $48.5 ≈ VND 1.28M | ~VND 0.64–1.11M (A100/H100) + GPU capital |
| 200M input + 100M output tokens | $144.5 ≈ VND 3.82M | ~VND 0.64–1.11M (A100/H100) + GPU capital |
Read the table correctly: self-host electricity is nearly fixed per running hour, while cloud cost scales with tokens. So cloud is cheaper at low usage; only above a high, continuous volume threshold does self-hosting's electricity + hardware depreciation become cheaper per token. But the deciding figure is GPU capital — which this article deliberately does not price, because it swings sharply with the market; a business must get real quotes and amortise them to compare properly.
What it means for Vietnamese businesses
For most Vietnamese businesses, the sensible choice is not "all self-hosted" or "all cloud", but segmenting by data and volume. This is the framework we recommend, extending our roadmap to build internal AI and our piece on AI data sovereignty:
- Start with a small, quantized model: a 7–32B need runs well on a single 24 GB GPU — the cheapest entry point to try real on-premise before scaling up.
- Sensitive data → self-host; routine tasks → consider the API: if the use case requires keeping data on-site (compliance with Decree 13/2023 on personal data protection, customer contracts), self-hosting is a requirement — the cost is the price of sovereignty, not a way to save money.
- Compute TCO with your own numbers: take your real token volume + real GPU quotes + real electricity bills, then compare cloud vs self-host. Do not decide on the gut feeling that "on-premise must be cheaper".
Self-hosting an LLM on-premise is not cheaper by default — it becomes cheaper when usage is high and continuous, and it is always worth it when data must stay inside the organisation.
Frequently asked questions
How do I quickly estimate the VRAM a model needs?
Take parameters × bytes per parameter: FP16 = 2 bytes, INT8 = 1 byte, INT4 = 0.5 byte. For example a 72.7B model needs ~145 GB at FP16 and ~36 GB at INT4 (weights only); in practice add 20–30% for KV-cache and activations.
What does running a GPU 24/7 cost in Vietnam?
It's TDP(kW) × 720 hours × electricity price. At the average price of VND 2,204.0655/kWh (excl. VAT, from 10 May 2025 per the Ministry of Industry and Trade): RTX 4090 (450W) ~VND 714,000/month, A100 SXM (400W) ~VND 635,000, H100 SXM (700W) ~VND 1.11 million — this is the GPU alone; a real bill is higher due to cooling, VAT and voltage/time-of-use tariffs.
Is self-hosting cheaper than renting a cloud API?
Not by default. At low-to-moderate volume, cloud APIs are usually cheaper because there is no hardware capital. Self-hosting is cheaper per token only when usage is high and continuous enough to amortise the GPU. Self-hosting also has non-monetary value: keeping data on-site.
What is the biggest cost of self-hosting an LLM?
Usually hardware capital (GPUs), far exceeding electricity and operations. GPU prices swing sharply with the market, so get real quotes and amortise them over the lifecycle to compute an accurate TCO.
How should a small business start?
Start with a small open model (7–32B parameters) quantized, running on a single 24 GB GPU — the lowest entry cost. Use it for internal tasks, measure real token volume, then decide whether to scale up or move part of it to cloud based on real data.
Compute an accurate on-premise AI TCO for your case
Namtech helps businesses pick the model size, GPU configuration and realistic TCO (VRAM, electricity, hardware, operations) — plus a hybrid on-premise/cloud plan to optimise both cost and data sovereignty.
Book a free consultationNote: This article is compiled from public sources as of 18 July 2026. VRAM figures are estimates from the params × bytes formula (weights only, before 20–30% overhead); electricity is computed from official NVIDIA TDPs and the official average electricity price, excl. VAT and auxiliary power; cloud API prices and exchange rates may change. Informational only — not technical, investment or legal advice.
- Ministry of Industry and Trade — Retail electricity price adjustment from 10 May 2025 (VND 2,204.0655/kWh, Decision 1279/QĐ-BCT)
- NVIDIA — GeForce RTX 4090 (Total Graphics Power 450W)
- NVIDIA — A100 Tensor Core GPU (Max TDP 300W PCIe / 400W SXM)
- NVIDIA — H100 Tensor Core GPU (Max TDP up to 700W SXM)
- HuggingFace — Qwen2.5-72B-Instruct (72.7B parameters)
- HuggingFace — Qwen2.5-32B-Instruct (32.5B parameters)
- HuggingFace — Qwen2.5-7B-Instruct (7.61B parameters)
- HuggingFace — Llama-3.3-70B-Instruct (70.6B parameters)
- Artificial Analysis — Qwen2.5-72B pricing ($0.475 in / $0.495 out per 1M tokens)
- VTC News — USD/VND exchange rate 17 Jul 2026 (Vietcombank sell ~VND 26,440)