Real-Time Data Mesh — Streaming Data Architecture for Enterprise

Retrieval-Augmented Generation (RAG) has become the de facto standard for enterprise AI applications that need to work with internal data. But between “works in a demo” and “works in production” is a chasm. How to bridge it?

Why RAG and Why Now¶

Fine-tuning LLM models on company data is expensive, slow, and hard to maintain. RAG offers an elegant alternative: keep the model general and supply relevant context at runtime. In 2026, we have mature embedding models, stable vector databases, and enough production experience to know what works.

Typical enterprise use cases include internal knowledge bases (documentation, wiki, processes), customer support over product documentation, compliance — searching regulations and internal policies, and contract and legal document analysis.

RAG Pipeline Architecture in 2026¶

A modern RAG pipeline has four key phases:

• Ingestion: Processing source documents — parsing, cleaning, chunking
• Indexing: Generating embeddings and storing in a vector database
• Retrieval: Finding relevant chunks based on the query
• Generation: Assembling the prompt with context and generating the answer

Each phase has its pitfalls. Let’s look at them in detail.

Chunking — The Foundation of Success¶

Chunking is the most underestimated part of the RAG pipeline. Bad chunking = bad results, regardless of model quality. These strategies have proven effective:

• Semantic chunking: Instead of fixed length, split text by semantic boundaries — headings, paragraphs, topical units. Requires preprocessing but dramatically improves retrieval quality.
• Overlap with context: 10–20% overlap between chunks ensures information at boundaries isn’t lost. Metadata included — document name, section, date.
• Hierarchical chunking: Two levels — parent chunks (broader context) and child chunks (detail). Retrieval searches at the child level, but the parent chunk goes into the prompt.

Optimal chunk size depends on the use case. For factual Q&A typically 256–512 tokens, for analytical tasks 512–1024 tokens. Always measure on real data.

Embedding Models — Selection and Trade-offs¶

In 2026, we can choose from several embedding model categories:

• OpenAI text-embedding-3-large: Solid performance, simple integration, but data leaves the perimeter
• Cohere embed-v4: Strong multilingual performance, suitable for Czech data
• Open-source (nomic-embed, BGE, E5): Host on-premise, full data control
• Domain-specific models: Fine-tuned embeddings for a specific domain — best performance but requires training investment

For Czech enterprise clients, we recommend a hybrid approach: open-source model on-premise for sensitive data, commercial API for less sensitive use cases.

Retrieval — More Than Just Cosine Similarity¶

Naive RAG relies on vector similarity. In practice, that’s not enough. Modern retrieval combines:

• Hybrid search: Vector search + BM25 (keyword search). Fusion algorithm (RRF) combines results from both.
• Query transformation: Synonym expansion, sub-query decomposition, HyDE (Hypothetical Document Embeddings).
• Reranking: Cross-encoder model reranks top-K results. Slower but significantly more accurate.
• Metadata filtering: Filtering by date, department, document type — reduces noise and speeds up retrieval.

Vector Databases — Technology Choice¶

The market has consolidated in 2026:

• pgvector (PostgreSQL): Great start if you have Postgres. HNSW indexes handle millions of vectors.
• Qdrant: Rust-based, high performance, good filtering. Popular in the EU.
• Weaviate: Built-in vectorization, GraphQL API, multi-tenancy.
• Managed services (Pinecone, Azure AI Search): Easiest operations, but data on provider’s cloud.

For most projects, we recommend pgvector — minimizes complexity and most teams already know Postgres.

Evaluation — Measuring RAG Quality¶

We measure at three levels:

• Retrieval quality: Precision@K, Recall@K, MRR — does the retriever return relevant documents?
• Generation quality: Faithfulness, relevance, completeness
• End-to-end: User satisfaction, answer correctness verified by domain expert

RAGAS automates evaluation via LLM-as-judge. But for production, regular human evaluation on a data sample is essential.

Common Mistakes¶

• Ignoring preprocessing: Garbage in, garbage out.
• Too much context: “Lost in the middle” effect — model ignores mid-context info.
• Missing observability: Log every pipeline step.
• Static pipeline: Data changes; implement incremental indexing.

RAG Is Engineering, Not Magic¶

Our tip: Start simple, measure baseline, iterate. Most improvements come from better chunking and reranking, not swapping the LLM.