I've seen Kafka used a single source of truth with data over 5 years. It works very well as long as you are attentive with what can happen if you replay that data. Your wishlist is on point. Regarding the first point, I've implemented this feature with a governance metadata store. Generally governance and read capabilities (indexing, filtering etc...) are the achilles heel of Kafka. https://tkaszuba.medium.com/how-to-sink-a-kafka-roll-out-6dc127c555c4