home / case studies / Southeast Asian payments processor (composite)
Data Streaming

Real-Time Transaction Event Backbone for a Payments Processor

We replaced nightly batch file transfers between payment, ledger and reporting systems with a Kafka-based event backbone. Downstream teams moved from next-day data to second-level freshness without changing their core systems all at once.

Southeast Asian payments processor (composite) · Data Streaming

Real-Time Transaction Event Backbone for a Payments Processor

Challenge

Delivering exactly-once semantics across legacy databases that lacked clean primary keys, and keeping PCI scope tight while still sharing data broadly. We tokenised card data at the CDC edge and enforced field-level schema rules before any event reached shared topics.

Approach

We designed a multi-cluster Apache Kafka deployment with Schema Registry (Avro) and topic naming and retention conventions per domain. Change Data Capture (Debezium) streamed events from the existing Oracle and PostgreSQL databases, so the source applications needed almost no changes. Apache Flink jobs handled enrichment (merchant, currency, risk tier) and wrote curated streams to a lakehouse (Iceberg on object storage) and to an operational analytics store. We added an outbox pattern for new services, consumer-lag dashboards in Grafana, and a self-service topic provisioning workflow tied to Git.

Outcome

Reporting freshness moved from T+1 to near real time (seconds to low minutes) Illustratively ~40–60% fewer point-to-point integrations as teams subscribed to shared topics Reconciliation breaks surfaced the same day rather than the next morning A governed schema contract reduced breaking changes between teams A reusable pattern for onboarding new payment rails

T+1Reporting freshness moved from to near real time (seconds to low minutes)
~40–60%Illustratively fewer point-to-point integrations as teams subscribed to shared topics
✓Reconciliation breaks surfaced the same day rather than the next morning

Problem solved

Settlement, reconciliation and merchant reporting depended on files generated overnight. Errors showed up a day late, and every new consumer needed another point-to-point integration.

What we built

We designed a multi-cluster Apache Kafka deployment with Schema Registry (Avro) and topic naming and retention conventions per domain. Change Data Capture (Debezium) streamed events from the existing Oracle and PostgreSQL databases, so the source applications needed almost no changes. Apache Flink jobs handled enrichment (merchant, currency, risk tier) and wrote curated streams to a lakehouse (Iceberg on object storage) and to an operational analytics store. We added an outbox pattern for new services, consumer-lag dashboards in Grafana, and a self-service topic provisioning workflow tied to Git.

Benefits

Reporting freshness moved from T+1 to near real time (seconds to low minutes) Illustratively ~40–60% fewer point-to-point integrations as teams subscribed to shared topics Reconciliation breaks surfaced the same day rather than the next morning A governed schema contract reduced breaking changes between teams A reusable pattern for onboarding new payment rails

Apache KafkaDebeziumApache FlinkIcebergGrafanaOraclePostgreSQL

Talk to us