Real-time Architectures with Spark and Kafka
Training course on the architectural principles involved in operating big data systems
At a glance
General information
2 days practical training, in-house
Target group
Software Architects, Software Developers
Application examples
Operating big data systems capable of processing large amounts of real-time data
Making real-time data highly available for queries
Description
The training sessions are usually held in German. Please contact us if you are interested in training sessions in English.
Training course on the architectural principles involved in operating big data systems
With the number of use cases increasing, big data systems are now facing the challenge of processing data streams instead of files. As this circumstance necessitates transitioning from a batch processing to an event streaming model, it often means reorganizing (or even redeveloping) the entire architecture of a big data system.
This training course examines the architectural principles needed to operate big data systems capable of processing large amounts of real-time data and making it highly available for queries. In order to explore these, course participants will use Spark and Kafka to set up a sample big data system capable of processing the Wikipedia edit stream, a real-time data stream containing every single edit made to every single Wikipedia article.
Agenda
- Event streams (brokers, topics, partitions in Kafka)
- Stream processing (transformations, processing patterns, error handling (at-least-once vs exactly once)
- Offloading / archiving large volumes of data (Lambda Architecture, Flume, Kafka Connect, Camus/Gobblin)
- Storing analysis results (caches (HBase, Cassandra, Riak, Redis), dashboards (ES, Kibana), handling historical data)
Typical questions we answer:
- What are the key components of event streams in Kafka and how are brokers, topics and partitions defined in this context?
- What concepts and practices are important in the stream processing environment - including transformations, processing patterns and dealing with errors (e.g., at-least-once vs. exactly once)?
- How does the offloading and archiving of large amounts of data work within the Lambda architecture and what role do tools such as Flume, Kafka Connect, Camus and Gobblin play in this?
- What options are available to store and retrieve the results of stream processing operations? How can caches (such as HBase, Cassandra, Riak and Redis) and dashboards (such as Elasticsearch and Kibana) be used for this purpose?
- How do you deal with historical data in an event stream processing architecture and what challenges can arise when storing and accessing this data?
- signed certificate
- in-house training
- Customization possible (agenda, tech stack, language, etc.)
- small training groups
Why inovex Academy?
Our offerThe inovex Academy has set itself the task of passing on knowledge about methods and technologies that we already use successfully in our projects.
Curated content
Our trainers create a customized training offer based on your requirements.
Customizable tech stack
In exclusive trainings, we can consider your tech stack for the training content.
Individual assistance
If needed, we can tailor the training to a specific use case of your company and work directly based on your data.
Our training approach
From the needs analysis to the awarding of certificates, we offer customized training courses, flexibly designed and carried out according to your requirements.
If you are interested in in-house training, we will start by identifying your needs and discussing your objectives. This discussion forms the basis for an initial offer.
As soon as the framework data has been clarified, our trainers start adapting the training content. Many of our training courses have a modular structure and offer the opportunity to design the agenda flexibly. Training courses that prepare for certifications, on the other hand, are less flexible. Here, however, you can set the content focus according to your wishes.
You will receive all relevant information in advance of the training. The training will then take place in the room of your choice and at the agreed time. Our trainers will adapt to your requirements.
After completing the training, all participants receive a certificate confirming their participation. You will also have the opportunity to give us feedback on the content and the course. We are always happy to receive praise and suggestions for improvement.
Frequently Asked Questions
Will I receive a certification as a result of the training?
When does the training start?
Do I get an invitation? When do I get it?
Supplementary information
Collin Rogowski
Head of inovex Academy
I look forward to your inquiry.
Collin Rogowski
We are your partner for successful training
We would be happy to talk to you personally about your concerns. Get in touch now!
- Customized training courses for your company
- Over 25 years of experience