Real-time analytics — architektura pro analýzu v reálném čase

26. 11. 2023 1 min read intermediate

Real-time analytics umožňuje analyzovat data v momentě vzniku. Lambda vs Kappa, streaming pipeline a OLAP databáze.

Architecture¶

Lambda vs Kappa¶

Lambda — batch + speed vrstva. Kappa — pouze streaming.

# Kappa stack:
# Kafka → Flink → ClickHouse → Grafana
# 1. Kafka: příjem
# 2. Flink: enrichment, agregace
# 3. ClickHouse: sub-second dotazy
# 4. Grafana: vizualizace

OLAP enginy¶

ClickHouse — nejrychlejší agregace
Apache Druid — time-series
Apache Pinot — user-facing analytics
DuckDB — embedded OLAP

Metriky¶

End-to-end latence — <10s je real-time
Query latence — cíl <1s

Summary¶

Kappa s Kafka + Flink + ClickHouse je dnes preferovaný stack pro real-time analytics.

real-timeanalyticslambdakappa

CORE SYSTEMS tým

Stavíme core systémy a AI agenty, které drží provoz. 15 let zkušeností s enterprise IT.

Všechny články

Další know-how

Apache Flink — Stream Processing for Real-Time Analytics

Kafka for messaging, Flink for processing. How we built a real-time analytics pipeline for an e-commerce client.

dbt + Snowflake — A Modern Data Stack for Analytics

ELT instead of ETL. dbt for SQL transformations with engineering practices, Snowflake as a cloud data warehouse.

Apache Flink — real-time stream processing engine

Apache Flink je framework pro stateful stream processing. Windowing, event time a exactly-once sémantika.