We are happy to present the Apache Beam Python I/O connector for Amazon Data Firehose 0.2.1 release.

♻️UPDATE

• Return failed elements by a tagged output, which allows users to determine how to handle them subsequently.
• Provide options that handle failed records.
  • max_trials - The maximum number of trials when there is one or more failed records.
  • append_error - Whether to append error details to failed records.

We are happy to present the first release of the Apache Beam Python I/O connector for Amazon SQS.

✨NEW

• Add a composite transform (WriteToSqs) that sends messages to a SQS queue in batch, using the send_message_batch method of the boto3 package.

We are happy to present the first release of the Apache Beam Python I/O connector for Amazon Data Firehose.

✨NEW

• Add a composite transform (WriteToFirehose) that puts records into a Firehose delivery stream in batch, using the put_record_batch method of the boto3 package.

Apache Beam is an open source unified programming model to define and execute data processing pipelines, including ETL, batch and stream processing. We consider it has a huge potential to improve traditional development patterns in both transactional and analytical processing of data. Specifically it can be applied to event-driven applications, data pipelines and streaming analytics.

Employing dataflow programming, Beam supports a range of I/O connectors, but we find some gaps in the existing connectors especially in relation to the Python SDK. It fueled us to start the Apache Beam Python I/O Connectors project.