Data Engineering Challenges: Working with Personal Data

The training addresses the challenges that arise for data engineers from the GDPR for their data products. Together, we will work on the targeted use of technologies to successfully overcome these challenges [on a cloud platform].

Request now

Zeichnung: Frau zeigt auf Whiteboard mit Graphen. Mann steht daneben mit Schloss-Symbol in der Hand.

Home / Academy / Data Engineering Challenges: Working with Personal Data

At a glance

General information

1 day practical training, in-house

Target group

Data Engineers, Data Scientists, Product Owner

Application examples

Providing added value from data in a legally secure way

Handling anonymization, deletion requests, retention periods, and metadata management right from the start

Description

The training sessions are usually held in German. Please contact us if you are interested in training sessions in English.

The General Data Protection Regulation (GDPR) came into force in 2018 with the aim of safeguarding and standardising the data protection rights of EU citizens. It provides companies and organizations with a legal framework which defines how they can collect, process, and store personal data.
To avoid having to make time-consuming corrections later, it is important to consider the legal data protection requirements when designing your own data platform.
Although the various provisions of the GDPR could be perceived as barriers to innovative solutions, current data engineering approaches and concepts offer excellent opportunities to provide companies with added value from their data in a legally secure way.

Based on the general requirements of the GDPR as they apply to personal data, this training course teaches participants different methodological and technical concepts for handling anonymization, deletion requests, retention periods, and metadata management. The course takes a hands-on approach which enables participants to try out practical concepts that have proven themselves in productive real-world projects. In each case, the course focuses on a specific use case and its corresponding architecture.

After completing this course, participants will be able to assess their data products for potential GDPR-related issues and use targeted technologies to successfully overcome the challenges [on a cloud platform].

The training environment will be set up on GCP. However, the technologies used can also be deployed on the other public clouds (or there are similar services).

The training is not legal advice.

Agenda

GDPR & Requirements for Data Engineers
Deletion Concepts & Architectures
- Handling of personal data (PII, data privacy)
- Encryption/anonymization (hashing, SHA-256, k-anonymity)
- Privacy-aware table design (metadata, tagging, schemas)
- Retention policies, archival processes
Hands-On Implementation
- Big data updates & deletes using Spark & Delta Lake
- Metadata management using data catalogues (data discovery)
Data Governance
- Mandatory processes within your organization
- IAM & rights concept

Typical questions we answer:

Which requirements regarding the storage and processing of personal data result from the GDPR?
How should a data platform be designed to take into account GDPR requirements?
How can deletion requests from users be technically implemented?
Which approaches to anonymising personal data are feasible?
How do I answer requests for information from users?

Data Engineering Challenges: Working with Personal Data

signed certificate of completion
in-house training
Customization available (agenda, tech stack, language, etc.)
small training groups

Request now

Why inovex Academy?

Our offer

The inovex Academy has set itself the task of passing on knowledge about methods and technologies that we already use successfully in our projects.

Curated content

Our trainers create a customized training offer based on your requirements.

Customizable tech stack

In exclusive trainings, we can consider your tech stack for the training content.

Individual assistance

If needed, we can tailor the training to a specific use case of your company and work directly based on your data.

Trainers

Our trainers are field-tested experts in their areas of expertise. Through their work in projects, they expand their knowledge day by day and pass on this know-how in their trainings - application-oriented and practice-oriented.

Kolja Maier

Kolja Maier is a Data & ML Engineer at inovex. He designs and implements data products that create targeted business value. His focus is on the implementation and production of data and ML applications using cloud technologies and best practices from software development

Marcel Spitzer

Google Cloud Certified Data Engineer Badge

Databricks Machine Learning Practitioner Associate Certificate

Databricks Developer Associate Certification

Marcel Spitzer is a Data Engineer at inovex. He is involved in the development of streaming and batch pipelines for data processing in distributed systems and uses machine learning to make data products smart

Our training approach

From the needs analysis to the awarding of certificates, we offer customized training courses, flexibly designed and carried out according to your requirements.

If you are interested in in-house training, we will start by identifying your needs and discussing your objectives. This discussion forms the basis for an initial offer.

As soon as the framework data has been clarified, our trainers start adapting the training content. Many of our training courses have a modular structure and offer the opportunity to design the agenda flexibly. Training courses that prepare for certifications, on the other hand, are less flexible. Here, however, you can set the content focus according to your wishes.

You will receive all relevant information in advance of the training. The training will then take place in the room of your choice and at the agreed time. Our trainers will adapt to your requirements.

After completing the training, all participants receive a certificate confirming their participation. You will also have the opportunity to give us feedback on the content and the course. We are always happy to receive praise and suggestions for improvement.