Choosing a vector database for on-prem RAG in 2026: pgvector, Qdrant, Milvus, Weaviate, Chroma compared

When building on-prem RAG — letting AI read and answer over your own documents — the first engineering question is: where do the vectors live? A vector database stores the "embeddings" (numeric representations of your documents) so they can be retrieved when a user asks a question. This piece compares five open-source vector databases on verifiable facts (license, version, popularity), shows how to pick an embedding model for Vietnamese, and estimates the storage your index will need. All figures checked on 20 Jul 2026.

Why does on-prem RAG need a vector database?

RAG (Retrieval-Augmented Generation) needs a vector database to store and quickly find document chunks that are semantically close to a question. The flow: split documents into chunks → run them through an embedding model to turn each chunk into a numeric vector → store them in the vector DB. When a user asks, the question is embedded too, and the vector DB returns the "nearest" chunks for the language model to synthesize an answer. If this is new to you, read RAG: teach your internal AI your documents and the internal AI system architecture first.

For businesses that must keep data on-site, the key is choosing a vector DB that is open source and self-hostable — so internal documents never leave for a third-party service. All five options here meet that bar; they differ in license, maturity and how you operate them.

Five open-source vector databases compared

All five vector databases below are open source and self-hostable, but they differ in license and popularity: Milvus leads with 45,274 GitHub stars, while pgvector is the "lightest" choice because it is just an extension that plugs straight into PostgreSQL. The table below gathers facts verified on 20 Jul 2026 — use it to filter quickly by license (commercial-friendly?) and existing infrastructure.

Read the table correctly: all five licenses (PostgreSQL, Apache-2.0, BSD-3-Clause) are permissive — commercial use and self-hosting are fully allowed. GitHub stars reflect popularity and community, not absolute quality, but they are a good signal of the support and documentation you'll find when something breaks.

Choosing an embedding model for Vietnamese

For Vietnamese documents, pick a multilingual embedding model and balance two decisive specs: vector dimensions (larger = more storage) and context length (longest chunk the model handles at once). For example, BAAI/bge-m3 produces 1,024-dim vectors and accepts up to 8,192 tokens — 16× a 512-token model — so you can chunk more freely. The table below is taken directly from the official config files (config.json) on HuggingFace.

*bge-m3 ships an FP32 checkpoint of 2.27 GB → estimated ~568M parameters; the other three are taken directly from HuggingFace safetensors metadata. RoBERTa-family models (bge-m3, e5) declare a max_position_embeddings that is 2 higher due to an offset — the usable context is 8,192 / 512 as shown.

The lesson: smaller dimensions = faster retrieval and lighter disk; larger dimensions = more semantic nuance retained. For most Vietnamese document-QA tasks, a 768–1,024-dim multilingual model is a good balance; the 384-dim MiniLM fits when you need ultra-light and can trade some accuracy.

Storage: how much does a vector index take?

The raw storage of a vector index equals dimensions × bytes per number × number of vectors. At FP32 (4 bytes/number), one million 1,024-dim vectors is ~4.1 GB; scalar quantization to int8 (1 byte) drops it to ~1.0 GB. This is the number to know before allocating RAM/disk to a vector DB, since many engines keep the index in memory for fast retrieval.

*Raw vector storage only. It excludes the index structure (e.g. the HNSW graph), payload/metadata and replicas — a real bill is higher. int8 quantization cuts storage sharply but trades some retrieval accuracy.

The pragmatic point: for a mid-sized corpus (under a few million chunks), vector storage is rarely the bottleneck — a few GB is trivial. The bottleneck is usually retrieval latency and chunk quality. But when vectors reach tens or hundreds of millions, dimension choice and quantization start to drive real RAM cost.

How to choose for a Vietnamese business

For most Vietnamese businesses starting with on-prem RAG, the sensible path is to go from simple to specialized rather than jump straight into a complex distributed system. Here is the framework we recommend:

• Already on PostgreSQL → use pgvector. You add vector capability to the database you already operate, with no new system to build and maintain — the fastest, cheapest entry to a working RAG.
• Need a dedicated engine at scale → Qdrant or Milvus. When vector counts are large, you need complex metadata filtering or steady low latency, these two most popular dedicated engines offer more index options and scaling.
• Fast prototyping → Chroma. Lightweight and easy to embed, ideal for the experimentation phase before committing infrastructure.
• Pick the embedding first, the DB second. Lock in the embedding model (and therefore the vector dimensions) first, because changing the embedding model forces you to rebuild the entire index. For Vietnamese, a 768–1,024-dim multilingual model is a safe default.

There is no absolute "best" vector database for on-prem RAG — start with pgvector if you already run PostgreSQL, and only move to Qdrant or Milvus when your vector count and latency requirements genuinely exceed it.

Frequently asked questions

Note: This article compiles public sources as of 20 Jul 2026. Version numbers and GitHub star counts are a snapshot at the time of lookup and will change; storage figures are computed from vector dimensions (raw, excluding index structure/metadata); model specs are taken from official config files. For reference only — not technical, investment or legal advice.