Designing a Kafka Consumer That Survives Restarts

My weather platform received continuous telemetry from multiple monitoring stations. A basic consumer could read and store those events, but restarting it introduced a harder question: how could processing resume without silently losing uncommitted observations?

I separated message retrieval from confirmed processing and tracked progress using checkpoints. This made recovery possible, but it also introduced the possibility that a message could be handled more than once if the consumer stopped after storage but before checkpoint confirmation.

I treated the system as an at-least-once pipeline and designed the storage path with repeated delivery in mind. Checkpointing and restart testing allowed the consumer to resume from a known position instead of assuming that uninterrupted execution was normal.

For a production deployment, I would formalize idempotency keys, add a dead-letter path, test partition rebalancing, and define monitoring around consumer lag, duplicate processing, and failed writes.