How DataOps can meet the challenges of managing behavioral data

Yali Sassoon

This is the first in a series of posts on DataOps, with a specific focus on how DataOps approaches can be used to meet challenges working with behavioral data (e.g. web and mobile data).

In this post we start by exploring the types of challenges that manifest in organizations when working with behavioral data that can lend themselves to a DataOps approach. In future approaches we will flesh out how those DataOps processes and methodologies work, and how Snowplow technology can support them.

Introduction

Casey was exhausted. One of only a handful of data product managers at Global News Inc, she found she was constantly being pulled back and forth between the many different team members at the firm that cared about data. In particular, she frequently found herself being pulled in one direction by the development teams that wanted to use behavioral data, and the different squads that she had to coordinate across to ensure consistent data systems across Global News’ more than 100 different apps. If only she could spend less time getting people aligned on what data they needed, and where to find it, she could actually spend some time making time supporting teams drive value from the data.

Nowhere were the challenges Casey faced more evident than in the case of coordinating work between the “Changing Times” mobile team, the Global News Business Intelligence team, and the newly spun up personalization team:

Both the Business Intelligence and Personalization teams were very keen to incorporate Changing Times in their reporting and personalization solutions – given the importance of Changing Times to Global News as a whole. However, that was proving much more challenging than integrating other titles. Successive efforts either to incorporate the data from Changing Times in the company wide reporting, or to test the personalization product on the Changing Times app, kept failing because of data integration issues.

The Business Intelligence and Personalization teams kept complaining that the Changing Times data quality was poor: every time they agreed a specification for the different data they required from the Changing Times team, they’d find that the data delivered would not be complete, and this was because the Changing Times team were making very frequent changes to the mobile app, so that successive versions would generate different data sets shaped in slightly different ways. Casey had investigated and validated that this was the case – the Changing Times app was very fast evolving, with frequent changes being made to the data collected.

On questioning the Changing Times squad, however, they were frustrated that the “slow moving” Business Intelligence and Personalization teams were trying to “trap” them into specific approaches to data collection when they needed the freedom to evolve their application as fast as possible, based on the experiments they were running: this was a key driver for the success of the app in the first place! “Please don’t let us sit in another long planning meeting with the BI or Personalization team” the Changing Times product manager would beg Casey. Some of us actually need to make money in this place!”

Challenges for Casey and the teams working with behavioral data at Global News Inc

The type of situation described at “Global News Inc.” is common in the data operations at many companies today. Delivering enterprise data projects involving behavioral data, for example:

End up being difficult for data professionals because:

These are exactly the sort of challenges that the DataOps practice was developed to solve

These challenges around behavioral data are specific examples of some of the more general data management challenges that have prompted implementing DataOps as a discipline. The conditions that give rise to these challen
ges are:

Data and analytics leaders can leverage DataOps approaches to meet these challenges

The recent DataOps report from Gartner recommends that Data and Analytics Leaders:

To learn more, we recommend readers download the Gartner white paper here.

In this series of follow-up posts from the Snowplow Team, we will bring to life how DataOps approaches can be brought to bear to solve challenges around behavioral data, in particular, and how Snowplow technology can help support those practices. Upcoming posts include:

Related articles