We are happy to present the first release of the Apache Beam Python I/O connector for Amazon DynamoDB.

✨NEW

  • Add a composite transform (WriteToDynamoDB) that writes records to a DynamoDB table with help of the batch_writer of the boto3 package.
    • The batch writer will automatically handle buffering and sending items in batches. In addition, it will also automatically handle any unprocessed items and resend them as needed.
  • Provide an option that handles duplicate records
    • dedup_pkeys - List of keys to be used for deduplicating items in buffer.
  • Create a dedicated pipeline option (DynamoDBOptions) that reads AWS related values (e.g. aws_access_key_id) from pipeline arguments.
  • Implement a metric object that records the total counts.
  • Add unit and integration testing cases. The moto and localstack-utils are used for unit and integration testing respectively.
  • Integrate with GitHub Actions by adding workflows for testing, documentation and release management.

See Introduction to DynamoDB PyIO Sink Connector for more examples.