Data Engineers: Stuck in the Middle?

|Chief Scientist
October 4, 2020

A data engineering manager sent me this snippet from Stealers Wheel’s famed “Stuck in the Middle with You”:

“Losing control, yeah, I'm all over the place,
Clowns to the left of me, jokers to the right,
Here I am, stuck in the middle with you”

The reference described her feeling like she was stuck in the middle of a conflict, and stretching herself in efforts to please two opposing factions who actually wanted the same thing: success for their businesses.

Engineering Wants Data Innovation

“Let’s throw it all in a data-lake, and we’ll figure it out later” is an enticing concept, which drove companies to store a lot of data, as it became cheaper to keep data, more tools to analyze it were created, and new business models to profit off it were developed. Data science and analytics teams are using data repositories kept in data warehouses and data lakes for innovation and competitive edge.

Engineering teams want to have it all including:

Increasing efficiency by using more data stores. Sometimes it is because of better operational efficiency, sometimes it is because there is a better fit for specific use cases, but the bottom line is that there is a growing demand to use new data storage and processing technologies.
Storing more data and making use of it. For example, a research team wants to add many more types of data from different sectors of the organization, so they can analyze it to find new trends and boost innovation.
Allowing access to more teams. For example, allowing access to more raw data for support engineers can help them solve problems quicker without disrupting development teams.

These needs, and more, are beneficial for the organization. They will enable it to serve the customers faster and better, innovate and gain a competitive advantage, and make great operational decisions.

Restraining the Beast

On the other corner, there are teams who want to restrain this rush towards data innovation. They are also doing it for the good of the business, as they want to keep it safe and in compliance with rules and regulations. Examples of these teams include:

Risk & Compliance teams who want the company to be in compliance with regulations and adhere to different security frameworks to lower its risk exposure.
Security teams who want to keep internal and external threats at bay.
Data Analytics teams who want to improve data governance level, understanding where the data is, what types of sensitive data exist, improve data quality, and more.
Privacy teams who want to make sure the organization is meeting privacy regulations and requirements.

Data Engineering: Stuck in the Middle

And now we are getting to our data engineering manager, who has limited resources and would like to please both “factions”. She thoroughly understands the advantages of using whatever data technologies are most effective, and she is even excited about trying the latest innovations and pushing the business forward.

On the other hand, she has to allocate resources to the projects that will be involved in adopting new technologies, such as:

Making sure data access control is working well not only in allowing access to specific data stores but also in granting authorization to the different data types within those data stores, and especially protecting access to sensitive information.
Meeting different compliance requirements, such as mapping where each type of data resides, who has access to what data, and who is accessing it.
Increasing transparency in regards to what data access (log analysis, etc).

The feeling she has is that R&D teams view her as a disabler, while security and risk teams view her as a liability. While she perfectly understands each side’s motives, she can only do so much with her resources, which leads to frustration.

Let’s have both!

Here at Satori, we understand this pain, and we are working to solve it so that data engineering can achieve both goals. Our product allows the following:

Enabling data access control that is decoupled from the specific data store infrastructure, so data authorization can be transparent and portable even when adding additional data store technologies.
Reducing risks of data leaks and other security risks across different data stores by applying out-of-the-box-security policies.
Quickly implementing regulatory and compliance needs of different data stores so that the requirements can be met without delaying them with costly projects.

Want to learn more about Satori? Contact us today to arrange a product demo and see how it can help you simultaneously achieve both data innovation and security and compliance.