RAG for business – vector stores, embeddings and production costs (2026)

RAG (Retrieval-Augmented Generation) is an architectural pattern where the LLM receives context from an external knowledge base FOR EVERY query, instead of being trained on your data once (fine-tuning). Schema: user query → embedding query → retrieval of relevant documents from vector DB → injection into prompt → LLM generates answer with context.

Why it dominates in 2026: (1) fresh data without retraining — a new document in the knowledge base is available immediately; (2) auditability — you know which documents the answer came from (compliance + EU AI Act); (3) cheaper than fine-tuning — typical RAG is EUR 12-125k vs fine-tuning EUR 125-500k; (4) replaceable models — data stays, model can change.

When RAG is NOT the best choice: when you have very specialized domain language (medical, legal, extremely specific finance) that needs to be 'built into' the model — fine-tuning may be better. When you have a very small knowledge base (under 1000 documents) — long-context prompting may suffice without RAG infrastructure.

RAG architecture has 6 layers: (1) Ingestion pipeline — documents enter the system (PDF, Word, web, API); (2) Chunking — documents split into chunks of defined size; (3) Embedding generation — each chunk gets a numeric vector; (4) Vector storage — vectors stored in vector DB with metadata; (5) Query pipeline — user query embedded, retrieval, ranking; (6) LLM generation — context + query → LLM → answer.

Every layer has its own technical choices and common mistakes. Most often organizations learn from their own mistakes after 3-6 months in production. Below we show which choices are critical.

Full 2026 production stack: LangChain or LlamaIndex as framework, Pinecone/Weaviate/pgvector as vector DB, OpenAI text-embedding-3-large or Cohere embed-v4 as embedder, GPT-4o or Claude 3.7 as generator. LangSmith for observability. Everything has stabilized significantly in the last 2 years.

Chunking is dividing long documents into smaller fragments that go into the vector DB as separate entries. Seems trivial — in practice 60% of RAG quality problems come from bad chunking.

Three main strategies: (1) Fixed-size chunking — simplest, splits every 500-1500 tokens. Works badly for structured documents (invoices, contracts). (2) Semantic chunking — splits at meaning boundaries (headings, paragraphs). Better for natural language. (3) Document-structure aware — uses document structure (sections, tables). Required for technical documents.

Practical recommendation: start with recursive text splitter, chunk_size=800-1200 tokens, overlap=100-200 tokens. This works for 70% of scenarios. For structured documents (compliance, finance) — semantic or structure-aware. Invest in evaluation to measure chunking impact on quality.

The hottest decision in RAG architecture. Four options dominate in 2026: Pinecone (managed SaaS), Weaviate (open-source + cloud), pgvector (PostgreSQL extension), Azure AI Search.

Pinecone wins for MVPs and small scale — simplest setup, good API, great developer experience. Price grows fast: for 5M embeddings ~USD 1500-2500/month (pod tier dependent). Above that scale economically inefficient.

Self-hosted Weaviate wins for medium scale — full control, low operational costs (mainly GPU/CPU hosting), full hybrid search out-of-the-box. Requires DevOps capacity. Typical TCO for 50M embeddings: USD 1000-2500/month.

pgvector wins when you already have PostgreSQL — simplest operational story (one DB, one backup, one monitoring). Performance better than expected (works great up to 50M vectors). Doesn't have all Pinecone/Weaviate features but sufficient for most use cases.

Azure AI Search wins for Microsoft ecosystem — native integration with Azure OpenAI, Microsoft Entra ID, SharePoint connectors. Premium pricing, but organizations usually already have Azure subscription.

Pure vector search has a weakness: it doesn't understand exact match. The question 'What does the policy say about e-invoicing' will find chunks about e-invoices (semantically similar) but may miss the document that literally has 'KSeF' in the heading.

Hybrid retrieval combines vector search (semantic) with BM25 (keyword). Each returns top-K, system merges results and reranks. Typical gain: 15-30% improvement in recall@10 for domain-specific queries.

In practice 2026: hybrid retrieval is standard, not an option. Weaviate and Pinecone have hybrid out-of-box. For pgvector you implement separately, but that's ~50 lines of code. Plus Cohere Rerank as final layer (cost ~USD 2 per 1M tokens) lifts quality another 10-15%.

Embeddings are often an underestimated RAG cost. OpenAI text-embedding-3-large costs ~USD 0.13 per 1M tokens. For 1M documents × average 500 tokens each = 500M tokens = ~USD 65 ONE-TIME. Sounds cheap.

But: every embedder model change = full base regeneration. Switching from OpenAI v3 large to Cohere embed-v4 for 50M documents = ~USD 3,250 one-time plus engineering time. Practically you stick with the chosen model 1-2 years.

Second cost: query-time embeddings. Every user query embedded. At 10k queries/day × 30 query tokens = 300k tokens/day = ~USD 12/month. Scales proportionally.

RAG without evaluation is flying blind. Three key dimensions to measure: (1) Retrieval quality — are retrieved chunks relevant; (2) Answer quality — is the generated answer correct; (3) Faithfulness — is the answer grounded in retrieved context (no hallucination).

Practical stack: RAGAS (open-source evaluation framework), LangSmith (production observability), own test set of 100-500 questions with gold standard answers. Cycle: change in the system → re-run evaluation → comparison vs baseline → decision.

Production monitoring: every response logged with (query, retrieved chunks, answer, user feedback). LLM-as-a-judge for automatic faithfulness scoring. Human review of 5-10% of answers for validation. Without these metrics you won't optimize quality.