Now throw in eventual consistency into the mix and you get reconciliation reporting which is the only real thing that can verify the validity of your domain data. But for this you need skilled business analysts that really understand the domain. You can't really automate this step, no matter what the consultants tell you.

KIP-82 Introduced headers to Kafka and they’ve been a hit ever since. Headers allow you to pass metadata information such as logging context, error information or data lineage of a message without having to dirty your payload. At least in theory. …

This was the course that got me into my FP journey about 8 years ago. As you said, it is very difficult but if you can trudge through it, it really pays off in day to day work.

This is similar to how we handle protobuf, avro and json. Each service has their our own metadata store that contains that services registered schemas. In a predeploy job we check those schemas against confluents schema registry for breaking changes. If there are changes the service doesn't get deployed.

If you are heavy into stateful transformations in Kafka Streams you’ve most likely hit a case where you wanted to slow or suppress the emitted events and forward only the last pertinent aggregated event, for example, to reason about the end of a batched load. In Kafka 2.1, the suppression…

"This is why Kafka cannot be regarded as a database; not without twisting the basic definition". While I agree with this statement, saying that kafka is not a database because it doesn't have all the bells and whistles of a traditional database is dangerous. It feeds corpo politicians who don't believe kafka can replace a database which is false. So even if it's not 100% true (yet) I still thinks it makes more sense to call Kafka a database to make it more accessible.

The Kafka streams api uses local state stores backed up by a change log topic to maintain state. Since Kafka works on the level of bytes the state stores also require that this level of granularity be used. The translation between the domain language and the Kafka world is the…

If you have spent any significant time with Avro (or Protbuf) and are using the Confluent Schema Registry you probably have encountered a breaking schema change characterized by the following mysterious exception. { “error_code”: 409, “message”: “Schema being registered is incompatible with an earlier schema” } What happens is the…

Kafka is most likely not the first platform you reach for when thinking of processing batched data. Most likely you’ve heard of Kafka being used to process millions of continuous real time events such as twitter feeds or IoT feeds not running end of day batches from old mainframes. Does…