The term ‘data mesh’ is being bandied about more and more in the tech sector, but what does it mean? And why should businesses care about it?
We sat down with Jens Neuse, Tyk’s Technical Director, to find out about what data mesh means in the business world…
What is data mesh?
Jens: It’s a term that comes from the consulting space. It can be a little overloaded – it’s actually a very simple concept
Many of the articles out there describe it as distributed Business Intelligence. Of course, this begs the question, what is non-distributed Business Intelligence?! Let’s work forward from there…
Business Intelligence itself is very simple. Think of something like Spotify. It gives you a way to stream music. Let’s say you want to build up some intelligence on how people are engaging with the product, what they’re doing, which tracks or artists are popular and so forth.
To find all that out, you need to ingest data while people are using Spotify. Whenever they click play or skip a track or conduct any other activity within Spotify, you take the data and ingest it. Then, you need to aggregate all that data somewhere. After that, you can produce data on things like top artists and most played tracks.
What you do is create all of those events, all that data, then you feed it into a system – which we could call Business Intelligence (BI). It’s essentially a giant database. Then, business analysts can query that database and try to find out what’s actually happening at Spotify.
That could mean everything from identifying the most popular tracks and artists, to more complicated stuff like, “I’m Jens, I like these genres of music, I have these friends on Facebook, recommend something that I would like to listen to.”
That’s the kind of stuff that BI can do. BI implementations were initially proprietary systems, but that’s seen as an older approach now.
More modern approaches go a slightly different way. To build your “data lake” you build databases on platforms like Postgres, which are more open. They’re no longer proprietary systems. They need a bit more work but generally it’s still much the same pattern. You have the music player team, they ingest the data, and they send it to a central system.
In the central system, the data needs to be clean, it needs to be mapped, perhaps joined with other data, and then fed into the data lake, or BI, or whatever you want to call it.
People can then begin to analyse the data, build machine learning models and build dashboards on top. For the artist, for example, you could build them a dashboard so they could see their most popular tracks, things like that.
It’s rather like a pipeline: user-generated data comes into the system, it’s aggregated, cleaned and fed into another system and from there it can be analysed, with valuable stuff created on top of it.
For a long time, there was a pattern where – from a bird’s eye view – the people with the most understanding of the music player were not the ones processing, cleaning or analysing the data. They basically just sent it off like a long log file. Somebody else would be aggregating it.
This creates the first problem. The people who really understand the data from all these different domains (the app, the website, the smartwatch and whatever else) just send the data off to the BI. Somebody else then has to process this data, work out if it’s valid, if it makes sense, whether some of it should be dismissed, how it needs to be cleaned up.
In the end, this results in a situation where the ownership of the data is spread across the organisation. It’s not owned by those who are best placed to use it effectively.
So that’s basic BI. All the different parts of the organisation send data to it, it’s analysed, and something is put on top of it.
It’s a centralised system – a monolith that sits in the centre of the company – and that might not be ideal. Shouldn’t the people who really understand the player also be the ones to analyse the data? Or maybe even provide an API to the other parts of the company that’s really easy to understand, clean and aggregated?
That’s what data mesh is all about. It’s not just about technology, but about changing the way the organisation thinks. It’s about treating the data as a product in and of itself. If you think about the data as a product, you don’t just have the player, you have the knowledge about how people use the player. You create a product out of that.
You don’t necessarily have to sell the data-product. It may just be part of your internal business domain. But if you treat it like a product inside the company, you can make it valuable to other teams in a way that’s easy to use.
So with data mesh the player becomes two products: the one the outside world uses, and the other, which is the intelligence you gain from offering your service.
What advantages does this give to businesses?
Jens: If you have a log file of all the click events, and you also own the player domain, it’s very clear to you what all the data means. You can then aggregate the data and put it into an API or even just on a spreadsheet. You can make that data really easy to work with.
Without data mesh, you tend to have a centralised BI team that ingests data from all over the company, cleans it and feeds it into a gigantic database. If you’re in the player team, and you want to get some insights into how people are using the player, you create a ticket for that centralised team and ask a BI analyst to query that big proprietary database. You wouldn’t know how to use that database, as you only send the log files to it!
So you raise a support request for someone to analyse the data that you understand and they don’t!
There’s a risk of creating a bottleneck with this approach, because there are lots of teams creating lots of data. And then there’s the centralised BI team. They get a lot of tickets and then have to determine which they should prioritise.
With data mesh, everything that belongs to one domain could be kept in that one team, instead of having a central team through which everything flows.
So, data mesh allows you to speed up all of those processes?
Jens: Not only speed up – you can also increase quality. With data mesh, the people who create, aggregate and analyse the reports are the people who really understand the data.
Centralised BI is increasingly becoming a problem. Data mesh is a solution to that problem. It’s not like your actual goal as a business is to “have a data mesh.” Your goal is to not have some centralised data bottleneck. That’s the reality.
What is Tyk doing to help with all of this?
Jens: We have a technology called Universal Data Graph (UDG).
UDG allows you to take APIs from different teams and combine them into one unified interface – one unified API.
If you don’t have a centralised system, and you want to create a decentralised approach where each different team creates their aggregates/their analysed views, and then exposes that as an API, that can create a different problem.
Instead of just sending the logs and having a central team aggregating everything, you do the aggregation at team level. Every small team provides an internal API to access their data. With this decentralised approach, instead of sending a ticket to a central team, you’d have to go around each individual team, use their internal API, and figure out what you can build on top of the data and how you can interact with it.
It’s an improvement in some ways, but really it’s just a different problem. Each API could be different, and you have a whole load of APIs to deal with. This is the problem that Universal Data Graph solves by providing a universal interface.
Let’s think about UDG and our data mesh example. The player team would create a data API that offers views into how people use the player, what tracks they play, etc. We take this API and connect it to the Universal Data Graph. Then we go to the website team – who also have a data API – and connect that to the UDG as well. And we do that with all the other teams.
The end result is that you get one single API – the Universal Data Graph – that gives you answers to all the possible questions. It distributes your requests to all the teams. So, you don’t have to interact with all the individual APIs, you just have to interact with this one aggregated or composed API.
So, it looks like you have a centralised BI, but the implementation of the BI is within the teams – it’s distributed. The individual teams control their own APIs, but you get one central access point.
It sounds like a much smarter way to query and access that data!
Jens: It is. We solve two problems with UDG. First, we solve the bottleneck of the centralised team. That then creates the problem around the accessibility of those distributed APIs, which Universal Data Graph solves as well.
As well as solving them, does the data mesh approach create any problems of its own?
Jens: Well, distributed systems are hard! If you have a centralised BI system, we’ve talked about the bottleneck, which is a trade-off, but on the flip side, it means there’s only one system you have to deal with. You send a support ticket to that BI team, they give you a view or an end point or something, it’s just one service.
If you distribute everything, networks can fail, you need to ensure that every part of it is secure, you need to think about authentication. Different APIs for different teams could be capable of handling different loads, so you need to think about rate limiting or throttling. And maybe you have some kind of internal system to monetise the value of your APIs.
With a centralised architecture, some of this is much easier to handle. With a distributed architecture, you’re entering “API management land!” It’s not just that you need something like UDG to put it all together, you also need Tyk’s core capabilities to manage your APIs in a scalable way.
The core capabilities of Tyk can already handle all of those things – the security, rate limiting, throttling etc. We can then put UDG on top to mesh together all the APIs.
Are there any industry sectors that data mesh particularly suits?
Jens: It’s a good fit for any kind of company that has multiple products spread across different teams, whenever each of those products generates some kind of data flow.
For example, in banking, there are lots of events like people logging into the banking system, sending and receiving money, FX transfers and so on.
There are also lots of digital consumer products – things like Tinder or TikTok – where you have billions of events, probably every second, because people click and interact with the apps.
With anything where people interact with a digital system that generates a lot of data, you need to be able to understand the data you generate.
If you were speaking to an organisation that was grappling with this stuff, what would your tips be to help them understand whether data mesh is right for them?
Jens: They should certainly think about the implications of building a centralised BI system. When a company thinks “we’re generating a huge amount of data, we need something to help us deal with it,” their first approach is usually to explore the idea of building a Business Intelligence system.
The problem is that they might not realise that a centralised BI could lead to the issues we’ve talked about. At a small scale, it’s not obvious it will lead to them. But as you add more ingestion points, more people that want to use it, it could become unscalable.
On the other hand, moving the responsibility of BI onto the teams means more work for those teams. It’s an organisational challenge that you have to solve, but we can always apply learnings from our adoption of microservices, and different solutions will suit different companies.
I’d advise people to think about the long-term strategy: what’s the long-term goal for what you want to do with the data? It can make a lot of sense to have the aggregation and analysis happening inside each domain – with the team that understands it best.
You can then build a framework that allows each individual team to build “tiny BIs” in a fixed way that’s compatible with the company’s other BIs. Then you just need a tool to glue it all together: Universal Data Graph.
What often happens is that each team builds their “tiny BI” in a completely different way, that’s not easily compatible with the others. There’s a risk it can still feel like lots of different things glued together that don’t really belong together.
So maybe, instead of having this centralised BI team, you could have a “free floating” Data Platform team that helps each individual domain build a compatible system that fits well together with the other ones. There are various different ways to approach this, with data mesh being one that’s becoming increasingly popular right now.
