Changelog

v0.1.0 (20/08/2024)

✨NEW

  • Add a composite transform (WriteToSqs) that sends messages to a SQS queue in batch, using the send_message_batch method of the boto3 package.

  • Provide options that handle failed records.

    • max_trials - The maximum number of trials when there is one or more failed records.

    • append_error - Whether to append error details to failed records.

  • Return failed elements by a tagged output, which allows users to determine how to handle them subsequently.

  • Create a dedicated pipeline option (SqsOptions) that reads AWS related values (e.g. aws_access_key_id) from pipeline arguments.

  • Implement metric objects that record the total, succeeded and failed elements counts.

  • Add unit and integration testing cases. The moto and localstack-utils are used for unit and integration testing respectively. Also, a custom test client is created for testing retry behavior, which is not supported by the moto package.

  • Integrate with GitHub Actions by adding workflows for testing, documentation and release management.