How do we create business value at our data team

In companies where data is not the primary source of revenue, it may be less clear how data teams contribute to overall success. Some data team processes may even have a delayed effect on increasing company performance. However, data teams can provide business value in two ways: generating revenue and reducing costs. Using examples from our team, I will explain how data teams can create business value. In few words, it could be described as follows:

• Serverless solutions, idempotent operations, and third-party components can reduce the total cost of ownership.
• Making data easily accessible and utilizing expert systems can boost a company’s revenue.
• Data teams should prioritize creating value for their organization by considering data as the product.

Reducing TCO

Serverless

At our data team, we strive to minimize expenses and find better infrastructure solutions by using serverless services. Serverless architecture has the advantage of only generating costs when functions are used, making it ideal for batch processes and business process support like infrastructure management and high-demand response.

For example, instead of running a web service host constantly, we use Cloud Functions or Cloud Run. We also use a temporary cluster with worker nodes that auto-scale during processing instead of maintaining a powerful Hadoop-Spark cluster for heavy processing.

Serverless solutions decrease the total cost of ownership (TCO) of our solutions. I estimate that the serverless approach can save us the cost of three additional specialists. Demand for these specialists is much lower with the serverless approach, and interaction with them is rare. We save on:

• One person for deploying and rolling out components to production.
• One person for server maintenance, hosting, network configurations, certificates, security, and firewalls.
• One person for telemetry, monitoring, and system recovery.

Cost savings come not only from reducing staff, but also from having a well-integrated ecosystem of services. This allows for a greater focus on business-related features rather than operational challenges.

Reducing the Cost of Errors

The time it takes to fix a system failure is a loss for the company and can negatively impact performance. Instead of moving forward, the team may stay in the same place or even fall behind. Mistakes and system failures are inevitable, so it’s better to mitigate their effects on the business rather than try to eliminate them completely. A malfunction that doesn’t cost anything isn’t truly a malfunction.

We keep this in mind as we develop our processes and features. For example, we intentionally build our operations to be idempotent, so that they can be run multiple times in some cases. This allows us to automatically repeat tasks in case of transient failures.

We also back up all processes that produce continuous results, like real-time data processing, with batch on-demand processing. This allows us to recreate any necessary pieces of data at any time.

Another important aspect of our data processing system is that we rely on third-party components whenever possible. These components serve as primitives for specific purposes, saving us time on development and testing. For example, we use Google’s Transfer Service and its Transfer Service Agent for reliable data transfer from on-premise to the cloud.

Generating Revenue

In companies where data teams are not the primary drivers of revenue, the impact of their work may not be immediately apparent to those unfamiliar with all of the processes involved. However, in such environments, data teams can contribute to revenue generation by enhancing decision-making processes and providing valuable insights to business experts.

Data as a Product

Making data easily accessible to business analysts and stakeholders can boost the company’s revenue. This involves organizing data and providing tools that allow people to quickly obtain the datasets and analytics they need. It also includes tools that enable stakeholders to work directly with data, without having to go through multiple people for every report and query.

Our team uses systems that allow analysts to analyze data without requiring expertise in SQL. One example is Metabase, which is similar to Tableau and is user-friendly for non-technical users. It enables analysis of data from multiple sources and automatically analyzes data by dimensions.

It is essential to keep in mind that for the data team, data is the product, and not just pipelines or infrastructure. When evaluating tools and infrastructure development, we consider whether the change will improve the quality of the data, make it more accessible, and facilitate querying.

Data-driven solutions

Augmenting decision-making processes is another way for data teams to bring value to businesses. Expert systems utilize machine learning or other data-driven algorithms to multiply the number of executed decisions by 100 times and keep quality of decisions at a high level.

Our team has developed a fraud prevention system that employs multiple rules to identify all suspicious cases and flag them for our experts. The system can detect patterns of suspicious user activity or unusual behavior, and identify technical issues in the data supplied by other companies. This helps prevent data issues at the earliest stage possible.

Here are some potential other ideas of data-driven solutions:

• Personalized customer offerings based on their preferences and past behavior.
• Predictive scaling of infrastructure based on current usage patterns, trends, and historical data.

If anyone has additional ideas for how data teams can create business value, please feel free to reach out and share your thoughts.