This is a 5-part series
Click below to navigate to the next chapter:
Chapter 1 The challenges of working in data in 2021
Chapter 2 A guide to data team structures with examples
Chapter 3 Breaking communication barriers with a universal language
Chapter 4 Reducing data downtime with data observability
Chapter 5 How data storytelling can make your insights more effective
Download the full eBook Treating data as a product
It’s an exciting time to be working in data. The opportunity for data teams to make an impact is huge. They have more tools at their disposal, with exciting technology hitting the market each day, than ever before.
Hiring in data, despite a slowdown in 2020, continues to be hot, with new job titles emerging like the Analytics Engineer and growing investment in data capabilities, and the market is projected to be worth $103 billion by 2023.
|Watch the webinar: People and process in data with Alex Dean and Scott Breitenother|
But it’s not all roses. Building and managing a modern data capability is a tremendous challenge. Aside from the technology involved, it requires working closely with technologists with a broad range of skill sets. It means working with (i.e. winning over) internal teams and stakeholders and educating them around the value of data. Sometimes it’s a challenge just to get a seat at the table and make the case for data in the business.
And even with all the right things in place: the tools, the people and buy-in from the right stakeholders, there is only so much one team can achieve. Small data teams can find themselves stretched, under-resourced while overwhelmed by company demands.
To get a better understanding of the challenges of working in data, we spoke to some of our customers, asking about the key pain points in their everyday working lives. Throughout the article, you’ll find featured snippets and quotes from conversations with our customers.
The answers below were not gathered in a formal or scientific way, nor is the list of challenges exhaustive. Nonetheless, we hope they shed some light on the experience of a data professional in 2021.
N.B. Our quotes below have been anonymized and paraphrased to protect the privacy of our interviewees.
Internal customers and responsibilities
During our conversations, the difficulties of working with internal customers made up around 21% of the challenges mentioned.
When we think of ‘customers’ we might imagine people buying products or services. But pretty much all the data professionals we spoke to listed their primary ‘customers’ as internal colleagues. That could mean working with finance, marketing, sales teams, operational teams, or sometimes even the C-suite.
A crucial part of working with those teams was winning their trust. The data professionals we talked to spoke about the need to win over these colleagues, build strong relationships with them and serve them efficiently.
In a sense, data teams have to act a bit like internal detectives – investigating what their internal customers need and building a plan to deliver it.
Time is of the essence. There is pressure for data teams to deliver data efficiently to their data consumers. Marketing and product teams don’t want to wait around for data they need to execute use cases. Equally important is getting the data at the right latency.
But above all, data teams mentioned that the data needs to be clear and coherent. It needs to be a single source of truth, so all teams can be on the same page.
Evaluating and purchasing tools made up over 10% of the challenges mentioned by our customers. With technology constantly evolving in the data industry, perhaps this comes as no surprise.
A quick look at Indicative’s recent post on the ecosystem of modern data infrastructure shows you that the challenge of choosing the right tools for the data stack is no joke. Even with Indicative’s selection and helpful breakdown of each tech category, it’s clear from their diagram that modern data teams face a number of choices for how they build out their data capability.
Our customers explained that tool evaluation (and selection) was an ongoing challenge for them. They told us that picking the right tools demands constant research, investigation, trialing and careful planning to ensure their teams are well equipped.
When it comes to buying tools, our customers were clear that they preferred to put in the research themselves before contacting a sales team. Not only does this save them time in the long run, it also gives them the opportunity to investigate not just pricing and features, but the ‘softer’ side of the tools – e.g. is there a community? Are there other teams using this solution who I can reach out to?
People and communication challenges
Challenges around people and communication were brought up the most, making up around 29% of the difficulties our data professionals mentioned.
Communication was actually a common thread in all of our conversations with our customers. It boils down to the need to be able to educate and inspire confidence in internal clients – demonstrating the value in data and data processes.
And data professionals, despite the stereotypes, are good communicators. One customer summed it up perfectly:
According to the customers we spoke to, technical ability isn’t enough. Data professionals need to back it up with strong communication skills and the ability to to guide people towards a common goal.
That’s not to say that technical skills are not important. But for data leaders, it’s also about amplifying your team’s abilities with appropriate team structures and processes in place. Put another way, the best tools cannot be leveraged without the right people.
One of the biggest challenges for data teams is to prove their value, or the value of their work. Our customers mentioned the constant battle to win buy-in from their colleagues and stakeholders.
Sometimes it’s about showcasing a new, exciting way of working with data. But often (and more challenging) it’s the case that data leaders need to convince colleagues that an investment in a certain data project is worth the time and resources. In both cases, communication and the ability to ‘sell’ the value in data are key.
At other times, education around data is crucial. Working with data means handling a vital business asset that should be treated as such, especially when sensitive data and customer information (PII) is involved.
And while ‘self-service’ data is often the dream, data professionals are tasked with coaching the rest of the organization on how they can find, understand and work with their data. While some in the business (e.g. developers and engineers) may already be data-proficient, that’s not always the case for other stakeholders.
Without education around data practices, resources can go to waste and teams can grow frustrated with their data product. As one customer found, it’s a long term challenge when internal teams are ‘sold the dream’ by a packaged solution, only for it to go wrong in the longer term. This could have been avoided if the data team was involved earlier in the evaluation process.
Working in data is a constant challenge
Between the continual demands of the organization, ongoing tool evaluations, hiring, decision making and dealing with internal stakeholders – data teams have it tough.
Thankfully the data industry is maturing. Each year there are more tools available, more budding communities of helpful contributors and examples of content (hopefully including this one!) to guide data teams towards success.
Sadly there’s no silver bullet to the challenges of working in data. But as one customer summed it up, one key element might lie in hiring, educating and equipping the right people, arguably the most important part of your data capability.
Despite the challenges, data teams are still driving huge value in their organizations, from empowering their colleagues with insights to turning game-changing use cases into reality.
At Snowplow, we want to make it as easy as possible for data professionals to manage and work with their behavioral data.
If you can relate to any of the above challenges and would like to discuss them further, feel free to get in touch or reach out to me at Simon (at) snowplowanalytics.com.