The third Snowplow Meetup London take place last Wednesday evening. The event was focused on real-time event data processing. It’s been more than two and a half years since we started working on the Snowplow real-time pipeline and it is great that in the last few months usage of that technology has really started to sky rocket!
Simply Business’s Dani Sola kicked the event off with a look at how Simply Business use their real-time event stream to personalise the web user experience for users visiting from a radio campaigns, and to provide comprehensive near-real time reporting about the performance of their call center teams. (Simply Business do a great job of tracking their users across both web and call center seamlesslly with Snowplow.) Dani went into a fair amount of technical detail including a detailed description of why and how they use Spark Streaming to process their event data, and how the real-time visitor data is made easily available to user-facing applciations via a visitor API.
Our very own Alex followed up with a talk on Tupilak: our fabric for monitoring, autoscaling and alerting real-time pipelines. At Snowplow we run many real-time pipelines on behalf of our clients and Tupilak is a platform we’ve developed to enable us to do this effectively at scale. Tupilak enables us to visualize, reason about and manage complicated real-time data processing topologies. We rely on it to alert us to pipeline issues (e.g. processing lag) and automatically scale pipelines to efficiently handle traffic spikes.
We’ll be in New York next week for the second Snowplow Meetup New York! Get in touch if you’d like to meetup whilst we’re there.