Description:

Delve into the internals of Apache Spark by understanding the design decisions and implementation details behind the scenes. By looking at various use cases, we will get to understand the problems that Spark solves in the Data Engineering ecosystem, the way in which it solves them, and the tradeoffs that come with the power and flexibility it provides to developers.

Attendees will walk away with a knowledge of the stages involved in executing a Spark job, helping them write more efficient jobs, pinpoint issues quickly while debugging failing jobs, and identify configuration issues in the Spark cluster.

Agenda:

A brief look at the history of the Data Engineering ecosystem
Understand the problems that Spark was created to solve
The fundamental aspects in writing a Spark job
Go behind the scenes to look at how Spark processes data
Understand the various touchpoints and their performance implications
Identifying potential problems and optimisations

Meet our Speaker(s)

Dakshin K

Solution Consultant, Sahaj AI Software

Dakshin has spent the better part of the last 3 years working with Spark and Data pipelines in production, gaining insights into how these tools operate under the hood, equipping him to write efficient data pipelines and identify performance issues quickly. He is also an open source contributor to some tools in the Data Engineering space, including Apache Airflow and Marquez. He looks forward to engaging with fellow attendees and contributing to the advancement of the community’s expertise in Apache Spark.