Changelog

v0.1.1 (24/09/2024)

✨NEW

• Add a composite transform (WriteToDynamoDB) that writes records to a DynamoDB table with help of the batch_writer of the boto3 package.

  • The batch writer will automatically handle buffering and sending items in batches. In addition, it will also automatically handle any unprocessed items and resend them as needed.

• Provide an option that handles duplicate records

  • dedup_pkeys - List of keys to be used for deduplicating items in buffer.

• Create a dedicated pipeline option (DynamoDBOptions) that reads AWS related values (e.g. aws_access_key_id) from pipeline arguments.

• Implement a metric object that records the total counts.

• Add unit and integration testing cases. The moto and localstack-utils are used for unit and integration testing respectively.

• Integrate with GitHub Actions by adding workflows for testing, documentation and release management.