Many thanks to Snowplow community member Dani Solà from Simply Business for his contribution to this release!
After the release of r61 Pygmy Parrot, we became aware that the updated file handling code for access logs generated by the Clojure Collector did not work with certain legacy Elastic Beanstalk environments, thus:
tomcat7_rotated- works fine
tomcat8_rotated- works fine
tomcat7- EmrEtlRunner does not move logs to Staging bucket
This is an easy issue to diagnose: if it affects you, then following an upgrade to r61 Pygmy Parrot, your Snowplow pipeline will copy no Clojure Collector access logs to Staging, and thus generate no enriched events.
This issue (#1480) is resolved in this release: EmrEtlRunner now supports all Clojure Collector access log filename formats again.
The EmrEtlRunner now has support for adding one or more of your own custom bootstrap actions (#1405). This is particularly useful if you are running your own Hadoop job steps as part of your scheduled jobflow on EMR. Many thanks to Dani Solà for contributing this feature.
You simply set your custom bootstrap actions in your EmrEtlRunner’s
config.yml as an array:
We have made a variety of improvements “under the hood” to EmrEtlRunner:
include Contractsinto classes and modules following best practice (#1438)
:archive:property has been added into the
time_diffas a dependency because it was no longer used (#1352)
You need to update EmrEtlRunner to the latest code (0.13.0) on GitHub:
You must also update your EmrEtlRunner’s configuration file, or else you will get a Contract failure on start. See the next section for details.
Whether or not you use the new bootstrap option, you must update your EmrEtlRunner’s
config.yml file to include an entry for it:
:emr: section of your EmrEtlRunner’s
config.yml file, add in a
:bootstrap: property like so:
For a complete example, see our sample
For more details on this release, please check out the r62 Tropical Parula Release Notes on GitHub.
If you have any questions or run into any problems, please raise an issue or get in touch with us through the usual channels.