Tom Kaszuba
Mar 29, 2024

--

First off, I've worked with infinite data retention on trading and insurance so definitely not simple use cases and definitely not theoretical. Second, data didn't sit around and was routinely reprocessed to rebuild state during releases if the topology or schema changed. Third, the main reason why you would want to have ALL your data in Kafka is that if Kafka is your source of truth why copy the data around and complicate your architecture if you don't need to? As for streaming, Kafka is so much more. Granted it is not as good at reading as a traditional database, no indexes, no ad hoc or push down queries etc... but for simple queries getting the data from a local state store is faster and simpler than from a networked external db and many times this is enough.

--

--

Tom Kaszuba
Tom Kaszuba

Written by Tom Kaszuba

Java, Scala and .Net Consultant with over 20 years experience in the Financial Industry, specializing in Big Data and Integration... and especially Kafka.

Responses (1)