We are happy to present the first release of the Apache Beam Python I/O connector for Amazon SQS.

✨NEW

  • Add a composite transform (WriteToSqs) that sends messages to a SQS queue in batch, using the send_message_batch method of the boto3 package.
  • Provide options that handle failed records.
    • max_trials - The maximum number of trials when there is one or more failed records.
    • append_error - Whether to append error details to failed records.
  • Return failed elements by a tagged output, which allows users to determine how to handle them subsequently.
  • Create a dedicated pipeline option (SqsOptions) that reads AWS related values (e.g. aws_access_key_id) from pipeline arguments.
  • Implement metric objects that record the total, succeeded and failed elements counts.
  • Add unit and integration testing cases. The moto and localstack-utils are used for unit and integration testing respectively. Also, a custom test client is created for testing retry behavior, which is not supported by the moto package.
  • Integrate with GitHub Actions by adding workflows for testing, documentation and release management.

See this post for more examples.