The requirements describe a real-time streaming use case where multiple processing applications must read the same stream in parallel, each must be able to resume processing after interruptions without data loss, and the system should support adding new processors with minimal duplication. Amazon Kinesis Data Streams (KDS) is purpose-built for this pattern.
With Kinesis Data Streams, producers write records to a stream that is split into shards. Multiple consumer applications can independently read from the stream using separate consumer groups (for example, using Kinesis Client Library). Each consumer maintains its own checkpoint (sequence number position), which allows it to resume from the last processed record after a restart or failure. This directly meets the “resume without losing data” requirement.
KDS also supports a configurable retention period (commonly 24 hours by default, extendable), enabling replay and recovery without forcing producers to re-send data or duplicating messages for each processor. Adding new consumers is straightforward: a new processor can begin reading the existing stream (from latest or from a trim horizon), which minimizes duplication compared to fan-out duplication patterns.
Why the others don’t fit:
SQS is a queue, not a stream. Multiple consumers typically compete for messages rather than all consuming the same messages independently, and duplication is required if multiple apps must process the same data.
Kinesis Data Firehose is primarily for delivery to destinations (S3, Redshift, OpenSearch) and not designed for multiple real-time parallel consumers with replay semantics.
EventBridge is event routing with limited replay/resume semantics compared to KDS and is not intended for high-throughput stream processing with per-consumer checkpoints.
Therefore, Amazon Kinesis Data Streams satisfies all requirements.
