Batch and stream processing are two fundamental approaches. Batch in batches, stream in real time. The right choice depends on latency and complexity.
Batch vs Stream¶
Batch processing¶
- Periodic — hourly, daily
- High throughput
- Simpler logic
- Tools: Spark, dbt, Airflow
Stream processing¶
- Continuous — event by event
- Low latency
- More complex — windowing, state
- Tools: Kafka, Flink
Decision criteria¶
- Latency <1 min → stream
- Latency hours/days → batch
- Complex transformations → batch simpler
- Event-driven actions → stream
Summary¶
Most organizations combine both approaches — batch for analytics, stream for operational use cases.
batchstreamarchitecturelatency