Hard vs Soft deletes in Kafka

There are several ways on how to model a delete event in Kafka with each having their own advantages and disadvantages. While using explicit delete events seem like the no brainer correct approach every time, it is not the preferred way to do this in Kafka and could lead to serious problems down the line. Employing a Kafka expert is a good way to identify potential operational issues before committing to an approach that has previously worked flawlessly in other distributed systems.

Tombstones in Kafka are events with a null value. If tombstone events are encountered in a compacted topic they tell the Kafka broker that the existing event should be permanently deleted. The tombstone itself is then permanently removed after the delete.retention.ms has passed. Since the tombstone is permanently deleted it is therefore impossible to tell if the record was deleted or if it ever existed. Explicit delete events on the other hand do not use the built in Kafka deletion mechanism but just mark the event as deleted using an event such as the following and expect the consumer to deal with this event explicitly.

{
  ...
  "isDeleted": true
}

Delete events are therefore a “soft” delete, while using tombstones is a “hard” delete. There is of course nothing wrong with the “soft” approach and in fact many times it is something that is flat out required in a full audit scenario or dealing with delete events explicitly but it does introduce some operational problems.

The main one being, old irrelevant data is never cleaned up, it just hangs around for years. All those old deleted products, users, transactions, schemas are there for the lifetime of the topic since a compacted topic by default has infinite retention, unless a “compact,delete” policy is specifically set. During a replay scenario, the consumers will need to deal with perhaps 5 year old events that people have long forgotten about. Even in a simple schema migration scenario, these old events will need to be accounted for. For topics with domain objects that have very short lifespans it causes the amount of data to explode. I’ve seen topics go from 10 mil to 25k with proper deletion strategies. All that irrelevant data also needs to be replicated to other broker nodes, which impacts the broker performance. Leaving the broker aspects aside having less relevant data over a ton of irrelevant data also has its advantages in Kafka streams clients. State stores (kTables) from topics with less data are rebuilt much faster and hence make services stand up quicker.* Having smaller relevant datasets for lookups is also more advantageous since a more performant in memory store could be used over RocksDB. Tombstones/null values are handled more efficiently during a left/outer join since you do not take the serde hit. Therefore doing a join that gets filtered out anyways will incur performance penalties.

In conclusion, soft deletes with explicit delete events are great but they should not be the default if the scenario doesn’t specifically require them. Using tombstones helps topics stay lean and should be first evaluated before using explicit delete events.

*The Confluent Schema Registry is a good example of this as it uses soft deletes as a default. If the same schema registry is used to bring up and down multiple environments, for ex. testing, all those old schemas are hanging around in the backup topic even though they are not visible through the REST endpoints. If the amount of schemas gets big enough the startup times of the schema registry will suffer. Waiting 15 minutes for the schema registry to restart is not a nice calling card for a highly available environment.