Snowplow S3 Loader 0.6.0 released

14 September 2017  •  Enes Aldemir
We are pleased to release version 0.6.0 of Snowplow S3 Loader, formerly known as Kinesis S3, our project dedicated to storing data, including Snowplow raw and enriched event streams, to Amazon S3. This post will cover: NSQ Support Support for “AT_TIMESTAMP” as initial position Upgrading Contributing 1. NSQ Support This release introduces NSQ as an event source - it is for this reason that we have renamed the project from Kinesis S3. Adding NSQ support...

Elasticsearch Loader 0.10.0 released

12 September 2017  •  Enes Aldemir
We are thrilled to announce version 0.10.0 of the Snowplow Elasticsearch Loader, our application for writing Snowplow enriched events and more to Elasticsearch. In this post, we will cover: NSQ support Support for writing raw JSONs Support for “AT_TIMESTAMP” as initial position Configuration changes Contributing 1. NSQ Support With this release, we are adding support for NSQ as an event source: the loader can now sink Snowplow enriched events from an NSQ topic to Elasticsearch....

Snowplow 92 Maiden Castle released

11 September 2017  •  Ben Fradet
We are pleased to announce the release of Snowplow 92 Maiden Castle. This release is a direct follow-up of Snowplow 91 Stonehenge, incorporating various improvements from seeing R90 and R91 operate in the wild. In particular, this release fixes some important gotchas in EmrEtlRunner’s --skip behavior, as well as a bug in the handling of run locks. If you’d like to know more about R92 Maiden Castle, named after the Iron Age hill fort in...

RDB Loader 0.13.0 released

06 September 2017  •  Anton Parkhomenko
We are thrilled to announce version 0.13.0 of Relational Database Loader, our Snowplow component that lets you load your data into relational databases such as Redshift and PostgreSQL. This release marks the migration of our RDB Loader and RDB Shredder apps from part of the snowplow/snowplow “mono-repo” into an independent project with its own release cadence. In this post, we will cover: Dedicated repository Single folder load Dry run Other changes Upgrading Contributing 1. Dedicated...

Snowplow Mini 0.3.0 released

30 August 2017  •  Enes Aldemir
We are pleased to announce the 0.3.0 release of Snowplow Mini, our accessible “Snowplow in a box” distribution. Snowplow Mini is the complete Snowplow real-time pipeline running on a single instance, available for easy deployment as a pre-built AMI. Use it to: Set up an inexpensive and easily discardable Snowplow stack for testing your tracker and schema changes Learn about Snowplow without having to set up a horizontally-scalable, highly-available production-grade pipeline This release focuses on...

Snowplow 91 Stonehenge released with important bug fix

17 August 2017  •  Ben Fradet
We are pleased to announce the release of Snowplow 91 Stonehenge. This release revolves around making EmrEtlRunner, the component launching the EMR steps for the batch pipeline, significantly more robust. Most notably, this release fixes a long-standing bug in the way the staging step was performed, which affected all users of the Clojure Collector (issue #3085). This release also lays important groundwork for our planned migration away from EmrEtlRunner towards separate snowplowctl and Dataflow Runner...

Snowplow 90 Lascaux released, moving loading step onto EMR

26 July 2017  •  Anton Parkhomenko
We are tremendously excited to announce the release of Snowplow 90 Lascaux. This release introduces RDB Loader, a new EMR-run application replacing our trusty StorageLoader, as proposed in our Splitting EmrEtlRunner RFC. This release also brings various enhancements and alterations in EmrEtlRunner. Read on for more information on R90 Lascaux, named after the Upper Paleolithic cave complex in southwestern France: RDB Loader Other improvements Upgrading Roadmap Getting help 1. RDB Loader 1.1 The rationale for...

Elasticsearch Loader 0.9.0 released

21 July 2017  •  Ben Fradet
We are thrilled to announce version 0.9.0 of Elasticsearch Loader, our component that lets you sink your Kinesis streams of Snowplow enriched events to Elasticsearch. This release adds support for Elasticsearch 5 and other important features such as the possibility to use SSL when relying on the REST API of Elasticsearch and the ability to sign requests when using Amazon Elasticsearch Service. In this post, we will cover: Support for Elasticsearch 5 Security features Bug...

Loading and analyzing Snowplow event data in Neo4j

17 July 2017  •  Dilyan Damyanov
Back in 2014 we published a series of blog post on using Snowplow event data in the graph database Neo4j. Three years on, they’re still among our most popular blog posts. (See below for links to the original posts.) A lot has changed since then. Neo4j has strengthened its position as a leading graph database solution. Its query language, Cypher, has grown with the platform. It has changed to the point where some of the...

Kinesis S3 0.5.0 released

07 July 2017  •  Ben Fradet
We are proud to be releasing version 0.5.0 of Kinesis S3, our project dedicated to sinking Kinesis streams, including Snowplow raw and enriched event streams, to S3. This release revolves around community-driven improvements as well as the modernization of the project. This post will cover: Fix silent suppresion of failures Community contributions Project modernization Roadmap Contributing 1. Fix silent suppression of failures We’ve uncovered a situation where failures prior to the serialization of the records...