Apache Beam is an open source unified programming model to define and execute data processing pipelines, including ETL, batch and stream processing. We consider it has a huge potential to improve traditional development patterns in both transactional and analytical processing of data. Specifically it can be applied to event-driven applications, data pipelines and streaming analytics.

Employing dataflow programming, Beam supports a range of I/O connectors, but we find some gaps in the existing connectors especially in relation to the Python SDK. It fueled us to start the Apache Beam Python I/O Connectors project.

We are happy to present the Apache Beam Python I/O connector for Amazon Data Firehose 0.2.1 release.

♻️UPDATE

• Return failed elements by a tagged output, which allows users to determine how to handle them subsequently.
• Provide options that handle failed records.
  • max_trials - The maximum number of trials when there is one or more failed records.
  • append_error - Whether to append error details to failed records.

We are happy to present the first release of the Apache Beam Python I/O connector for Amazon SQS.

✨NEW

• Add a composite transform (WriteToSqs) that sends messages to a SQS queue in batch, using the send_message_batch method of the boto3 package.

Amazon Simple Queue Service (Amazon SQS) offers a secure, durable, and available hosted queue that lets you integrate and decouple distributed software systems and components. The Apache Beam Python I/O connector for Amazon SQS (sqs_pyio) aims to integrate with the queue service by supporting source and sink connectors. Currently, the sink connector is available.

We are happy to present the first release of the Apache Beam Python I/O connector for Amazon Data Firehose.

✨NEW

• Add a composite transform (WriteToFirehose) that puts records into a Firehose delivery stream in batch, using the put_record_batch method of the boto3 package.

Amazon Data Firehose is a fully managed service for delivering real-time streaming data to destinations such as Amazon Simple Storage Service (Amazon S3), Amazon Redshift, Amazon OpenSearch Service and Amazon OpenSearch Serverless. The Apache Beam Python I/O connector for Amazon Data Firehose (firehose_pyio) provides a data sink feature that facilitates integration with those services.