Core Data Platform & Integration

Most data projects fail on architecture, not technology. The team picks Snowflake or Databricks, starts building pipelines, and after 6 months discovers they have no defined source of truth, data quality is a disaster and nobody knows who owns what data.

Our blueprint process: (1) Data audit — we map all sources, flows, transformations, consumers. (2) Domain mapping — who owns what data, who are the consumers, what are the SLAs. (3) Architecture design — Medallion architecture (Bronze → Silver → Gold), technology selection based on requirements. (4) Implementation roadmap — use-case prioritization by business value, MVP pipeline in 4-6 weeks.

Medallion Architecture: Bronze = raw data (as-is from sources, immutable). Silver = cleaned, validated, conformed. Gold = business-ready aggregations, denormalized views for consumers. Each layer has clearly defined responsibilities and quality gates.

Technology selection: We don’t do “Snowflake projects” or “Databricks projects”. We choose technology based on requirements: batch vs. streaming, data volume, latency, budget, existing stack. Sometimes PostgreSQL + dbt is the best solution. Sometimes you need a Spark cluster.

A pipeline that fails silently is worse than no pipeline. We build data pipelines with a production approach: monitoring, alerting, retry logic, dead letter queue, data quality checks on input and output.

ETL vs. ELT: ETL transforms data before storage — suitable for regulated environments where you want to control what gets stored. ELT stores raw data and transforms in the warehouse — more efficient with modern systems (Snowflake, Databricks, BigQuery). We usually choose ELT, but it depends on context.

Orchestration with Airflow: DAG-based orchestration. Dependency management, retry logic, SLA alerting, backfill capability. Taskflow API for cleaner code. Custom operators for specific sources. Monitoring through Grafana.

Transformations with dbt: SQL-first transformations with versioning, testing and documentation. dbt tests for data quality (unique, not_null, accepted_values, custom). dbt docs for automatic documentation and lineage visualization. Incremental models for efficient processing.

Error handling: Every pipeline has: retry with exponential backoff, dead letter queue for failed records, failure alerting, automatic recovery after transient errors. SLA monitoring — pipeline must complete within defined time, otherwise alert.

Batch processing has its place — but when you need to make real-time decisions, it’s not enough. Fraud detection, dynamic pricing, supply chain optimization, IoT telemetry — these are use cases where delay costs money.

Apache Kafka as backbone: Kafka isn’t just a message broker — it’s a distributed commit log, event streaming platform and integration backbone in one. Guaranteed delivery, ordering per partition, replay capability, retention policies.

Stream processing: Kafka Streams for simple transformations (filtering, enrichment, aggregation). Apache Flink for complex stream processing (windowing, complex event processing, ML inference). ksqlDB for SQL-like stream processing.

Kafka Connect: Pre-built connectors for hundreds of sources and targets. CDC (Change Data Capture) from PostgreSQL, MySQL, SQL Server via Debezium. Sink to Elasticsearch, S3, Snowflake. New connector approval in hours, not weeks of custom development.

Production operations: Multi-broker cluster, replication factor 3, ISR monitoring. Schema Registry for schema evolution (Avro/Protobuf). Kafka Lag monitoring — consumer group lag = early warning for processing bottlenecks.

Data without quality is noise. A dashboard nobody trusts is more expensive than no dashboard — people ignore it and make decisions based on intuition. Data quality isn’t nice-to-have, it’s a prerequisite for any data initiative.

Data Quality Framework: We measure 6 dimensions: completeness (missing values), consistency (agreement between sources), accuracy (correctness of values), timeliness (data freshness), uniqueness (duplicates), validity (format and range). Automated checks on input and output of every pipeline.

Data Contracts: Formal agreement between data producer and consumer. Defines schema, quality expectations, SLA, ownership. Breaking change = versioning + notification + migration period. Contracts in code (protobuf, JSON Schema), not in documents.

Data Lineage: We automatically track where data came from, how it was transformed, where it goes. Visualization in data catalog. When source system changes, you know exactly what’s affected. Impact analysis in minutes, not days.

Data Catalog: Central place to find all company data. Description, owner, quality metrics, lineage, examples. Self-service — analyst finds what they need without IT ticket. DataHub, Apache Atlas, or Atlan.

Manual integration (CSV export, FTP upload, email with attachment) is technical debt that accumulates. Every new system means new manual connections. We build an integration layer that connects systems reliably, automatically and with monitoring.

Integration patterns: Synchronous (REST/gRPC) for queries and commands where you need immediate response. Asynchronous (Kafka, RabbitMQ) for events and notifications where eventual consistency suffices. CDC (Change Data Capture) for real-time data replication without changing source system.

API Design: RESTful API with OpenAPI specification. Versioning (URL or header). Rate limiting, authentication (OAuth2/API key), pagination, error handling. API gateway (Kong, Azure APIM) for central management.

Resilience: Retry with exponential backoff, circuit breaker (Polly, Resilience4j), timeout handling, bulkhead isolation, dead letter queue. Every integration has defined SLA and monitoring. One system outage doesn’t stop others.

Typical integrations: SAP/ERP (orders, invoices, inventory), Salesforce/CRM (contacts, opportunities), e-commerce platforms (Shopify, Magento), payment gateways (Stripe, GoPay), delivery services (PPL, DPD, Zásilkovna). We connect most in 1-3 weeks.

If business has to ask IT for every report, you have a problem. Self-service analytics means analysts, product managers and leadership get data themselves — from verified, quality sources. IT builds the platform, business uses it.

Semantic Layer: Unified definition of business metrics. “Revenue” means the same thing in every report, for every team. We implement through dbt metrics, Cube.js, or Power BI semantic model. No more “our revenue differs from yours by 3%”.

Data Catalog: Central place for discovery. Analyst searches “customer churn” → finds defined metric, owner, quality score, query examples. Reducing “who should I ask about this data” from days to minutes.

Dashboards and reports: Power BI for executive reporting and ad-hoc analysis. Grafana for operational dashboards. Embedded analytics for customer portals. Standard templates for typical use cases (sales, operations, finance).

Governance: Who sees what (row-level security), who can change what (role-based access), certified vs. exploratory datasets. Balance between control and freedom — too restrictive = people return to Excel.