Wintershall Dea Development of a company-wide data platform

How can we take production and other company data from where it originates and make it available to everyone who wants to work with it? How can we dismantle silos and leverage the veritable treasure trove of data that has been amassed over the years in various formats on various systems? How can we provide users with an overview of existing data and its connections?

Wintershall Dea was facing these and other questions when the group began working with inovex in 2017. The companies worked together to develop an incremental solution, beginning with a SQL data warehouse developed by a small team. By the time Wintershall and Dea merged in 2019, however, it became apparent that it was necessary to “think bigger” and to develop a centralised data platform for the entire organisation.

In addition to its main headquarters, the Wintershall Dea group also includes numerous business units in various countries. These operate relatively independently − including using their own technologies and data formats. As well as production data, therefore, data from areas such as finance, health and safety, environment, and repairs were also to be included in the platform.

The application landscape is very heterogeneous, and, in the initial stages of the project, it included a large number of non-automated reporting processes based on document exchange. These processes used a series of manual steps to reformat and transmit the data, making them very labour-intensive and error-prone.

Although this enabled a large number of separate connections to be made between individual applications, the data silos remained in place.

In order to remedy this, to automate processes, and to make the existing data accessible to the entire company, 2019 saw the start of the development of a central data hub, which Wintershall Dea and inovex are continuing to expand.

Solutions

Wintershall Dea and inovex’s joint vision involves developing a state-of-the-art, cloud-based data platform which brings together company data from a wide variety of source systems, processes it, and makes it available for analysis or further processing.

One important aspect of developing the platform is maintaining metadata and contextualizing the data. This enables end users to easily gain an overview of which data is available, where it originates, how it is being used, and which other data it is linked to.

In realising this vision, the companies are working agilely with a cross-functional team. Over the years, this team has been strengthened by the addition of both internal developers from Wintershall Dea and experts from inovex.

During the project’s peak phases, Wintershall Dea provided an internal product owner for the entire hub, three internal product owners for sub-areas (one of whom also functioned as a data engineer), and an additional internal data engineer. inovex provided six data engineers and an inovex scrum master. Currently, in addition to a product owner and a scrum master, there are three internal developers and three inovex developers working on the project.

Tech Stack

The data hub is based on Azure Stack. The project also involves a data lake, which is filled using pipelines orchestrated via Azure Data Factory. Data transformation is performed through specially configured Python functions or via Synapse.

A database containing all the metainformation on the pipelines ensures an even greater degree of automation. Selected users are given access to the database via a proprietary app and can make configuration changes themselves or trigger pipeline runs via an interface.

The various specialist areas and business units in Wintershall Dea use different names for the same data and assets. For this reason, the companies are consolidating their master data. The project includes a database which is maintainable via an app and which enables both naming information and individualised asset hierarchies to be maintained. Both database frontends were developed using Angular.

To make it easy to call up data from the data lake, inovex made available interfaces/APIs (REST and GraphQL) written in TypeScript. For several use cases, the IT project experts are working closely with the stakeholders to develop Power BI datasets which can be used as a basis for reporting.

To ensure that the data and the lake could be searched and enriched with metadata, Microsoft Purview was used as a data catalogue. This provides both developers and end users with an overview of the ways in which each type of data enters the data hub, how the data is being used, and how it is categorised.

Current Situation

The data hub currently contains more than 1.7 million files and around 12 terabytes of data from 120 different sources in 10 countries. It covers 23 different specialist areas and feeds data to hundreds of Power BI reports. The data hub team’s role here is to develop and maintain around 150 pipelines.

inovex supports a range of analytics use cases − including those pertaining to health and safety and to production monitoring and optimisation. In doing so, the experts work closely with Wintershall Dea’s data science team, as well as with users who create reports and dashboards.

In addition, inovex enables an entire range of hub use cases, in which the data hub distributes data from a source system to one or more target systems − for example, in finance and controlling. At the same time, inovex makes interfaces (APIs) available to enable users to query a variety of sensor data.

This involves connecting data from various systems and making it available for further processing. The underlying data can, of course − in accordance with appropriate authorisations − also be used in other reporting contexts and in conjunction with other key figures.

Added Value

A centralized data platform like the one created jointly by the Wintershall Dea / inovex partnership provides a number of benefits. It enables interested parties to easily access large quantities of data from various sources and to combine it with other data and further process it. The automations save time and make the associated reporting processes significantly less error-prone. The self-service functions offered by both the Purview Data Catalog and the apps enable stakeholders to take an active role. They can describe data, create hierarchies, and trigger their own pipelines.

Future Prospects

Wintershall Dea is currently facing a great deal of change, as parts of the company are being sold to Harbour Energy. During this transition, inovex experts will continue to work with the company to monitor and improve the data hub.

We are very pleased about the appreciation of our work, which is reflected in the fact that the Data Hub will be part of the Transitional Services that Harbour is intensively reviewing.