Tom Kaszuba
Feb 16, 2024

There are several approaches to resolve eventual consistency problems. The easiest is to use a global kTable which consumes the lag fully before starting up the topology. You can use an inner join which will block until both messages arrive. As is mentioned you can use a suppression mechanism, a time window is one approach. I personally prefer to use a session window. You could also create your own lag detection mechanism, I've done it but it is not trivial.

Tom Kaszuba
Tom Kaszuba

Written by Tom Kaszuba

Java, Scala and .Net Consultant with over 20 years experience in the Financial Industry, specializing in Big Data and Integration... and especially Kafka.

No responses yet