Step 1: Understanding the Problem
Processing in Batches: The application must process records in groups.
Sequential Processing: Each record in the batch must be processed before writing to Aurora.
Solution Goals: Use services that support ordered, batched processing and integrate with Aurora.
Step 2: Solution Analysis
Option A:
Amazon Kinesis Data Streams supports ordered data processing.
AWS Lambda can process batches of records via event source mapping with MaximumBatchingWindowInSeconds for timing control.
Configuring the batching window ensures efficient processing and compliance with the workflow.
Correct Option.
Option B:
Amazon SQS is not designed for streaming; it provides reliable, unordered message delivery.
Setting MaximumBatchingWindowInSeconds to 0 disables batching, which is contrary to the requirement.
Not suitable.
Option C:
Amazon MSK provides Kafka-based streaming but requires custom EC2-based processing.
This increases system complexity and operational overhead.
Not ideal for serverless requirements.
Option D:
DynamoDB Streams is event-driven but lacks strong native integration for batch ordering.
Using ECS adds unnecessary complexity.
Not suitable.
Step 3: Implementation Steps for Option A
Set up Kinesis Data Stream:
Configure shards based on the expected throughput.
Configure Lambda with Event Source Mapping:
Enable Kinesis as the event source for Lambda.
Set MaximumBatchingWindowInSeconds to 300 to accumulate data for processing.
Example:
{
"EventSourceArn": "arn:aws:kinesis:region:account-id:stream/stream-name",
"BatchSize": 100,
"MaximumBatchingWindowInSeconds": 300
}
Write Processed Data to Aurora:
Use AWS RDS Data API for efficient database operations from Lambda.
AWS Developer References:
Amazon Kinesis Data Streams Developer Guide
AWS Lambda Event Source Mapping
Batch Processing with Lambda
