Tom Kaszuba – Medium

Pinned

How to sink a Kafka roll out

Has management started to flirt with the idea of doing event sourcing and rolling out a Kafka cluster? Have you read up on the technology and you want nothing to do with it? What is the best way to nip this rollout in the bud? Is arguing whether Kafka is…

Kafka

6 min read

Kafka

6 min read

Dec 25, 2022

The curious case of missing headers in Kafka Streams

KIP-82 Introduced headers to Kafka and they’ve been a hit ever since. Headers allow you to pass metadata information such as logging context, error information or data lineage of a message without having to dirty your payload. At least in theory. …

6 min read

The curious case of missing headers in Kafka Streams

6 min read

Dec 16, 2022

Review of Functional Programming Principles in Scala Coursera Course

Daryan Hanshew

This was the course that got me into my FP journey about 8 years ago.

This was the course that got me into my FP journey about 8 years ago. As you said, it is very difficult but if you can trudge through it, it really pays off in day to day work.

1 min read

This was the course that got me into my FP journey about 8 years ago. As you said, it is very difficult but if you can trudge through it, it really pays off in day to day work.

Aug 4, 2022

The Journey of Adopting gRPC and Protobuf

Ron Aharoni

This is similar to how we handle protobuf, avro and json.

This is similar to how we handle protobuf, avro and json. Each service has their our own metadata store that contains that services registered schemas. In a predeploy job we check those schemas against confluents schema registry for breaking changes. If there are changes the service doesn't get deployed.

1 min read

Oct 10, 2021

Custom Suppression in Kafka Streams

If you are heavy into stateful transformations in Kafka Streams you’ve most likely hit a case where you wanted to slow or suppress the emitted events and forward only the last pertinent aggregated event, for example, to reason about the end of a batched load. In Kafka 2.1, the suppression…

Kafka

4 min read

Kafka

4 min read

Sep 2, 2021

You can replace Kafka with a database

528

Emil Koutanov

"This is why Kafka cannot be regarded as a database; not without twisting the basic definition".

"This is why Kafka cannot be regarded as a database; not without twisting the basic definition". While I agree with this statement, saying that kafka is not a database because it doesn't have all the bells and whistles of a traditional database is dangerous. It feeds corpo politicians who don't believe kafka can replace a database which is false. So even if it's not 100% true (yet) I still thinks it makes more sense to call Kafka a database to make it more accessible.

1 min read

Apr 22, 2021

Speed up your Kafka Deserialization with a Caching Serde

The Kafka streams api uses local state stores backed up by a change log topic to maintain state. Since Kafka works on the level of bytes the state stores also require that this level of granularity be used. The translation between the domain language and the Kafka world is the…

Kafka

3 min read

Kafka

3 min read

Dec 25, 2020

Avro Schema Evolution Strategies on Kafka

If you have spent any significant time with Avro (or Protbuf) and are using the Confluent Schema Registry you probably have encountered a breaking schema change characterized by the following mysterious exception. { “error_code”: 409, “message”: “Schema being registered is incompatible with an earlier schema” } What happens is the…

Avro

7 min read

Avro

7 min read

Published in The Startup

·Oct 2, 2020

ETL Batch Processing With Kafka?

Kafka is most likely not the first platform you reach for when thinking of processing batched data. Most likely you’ve heard of Kafka being used to process millions of continuous real time events such as twitter feeds or IoT feeds not running end of day batches from old mainframes. Does…

Kafka

6 min read

Kafka

6 min read