Changelog

v0.2.1 (22/08/2024)

♻️UPDATE

Return failed elements by a tagged output, which allows users to determine how to handle them subsequently.
Provide options that handle failed records.
- max_trials - The maximum number of trials when there is one or more failed records.
- append_error - Whether to append error details to failed records.

✨NEW

Add a composite transform (WriteToFirehose) that puts records into a Firehose delivery stream in batch, using the put_record_batch method of the boto3 package.
Provide options that control individual records.
- jsonify - A flag that indicates whether to convert a record into JSON. Note that a record should be of bytes, bytearray or file-like object, and, if it is not of a supported type (e.g. integer), we can convert it into a Json string by specifying this flag to True.
- multiline - A flag that indicates whether to add a new line character (\n) to each record. It is useful to save records into a CSV or Jsonline file.
Create a dedicated pipeline option (FirehoseOptions) that reads AWS related values (e.g. aws_access_key_id) from pipeline arguments.
Implement metric objects that record the total, succeeded and failed elements counts.
Add unit and integration testing cases. The moto and localstack-utils packages are used for unit and integration testing respectively. Also, a custom test client is created for testing retry of failed elements, which is not supported by the moto package.
Integrate with GitHub Actions by adding workflows for testing, documentation and release management.