About the Event
Join Sahaj Software’s Ioseb Laghidze as he shares battle-tested techniques to infuse rigour and reliability into your Databricks projects, drawing directly from real-world software engineering practices. This session is designed to empower data engineers and teams to move beyond basic validation and achieve true data engineering excellence.
About the Talk
Databricks offers a flexible and powerful environment for building data pipelines, yet it doesn’t always provide an opinionated framework for testing, unlike some other tools. This can leave engineers uncertain about the best ways to validate their work and ensure the integrity of their data solutions. In this session, Ioseb will guide you through practical ways to integrate structured testing into your Databricks projects. By borrowing proven techniques from software engineering, you’ll learn how to:
Modularize PySpark Transformations: Break down complex PySpark logic into smaller, testable units, making unit testing straightforward and effective.
Set Up Containerised Environments: Master the creation of local development environments using containers, enabling you to work with PySpark, Delta, and Kafka in a consistent and isolated manner.
Run Integration Tests: Implement integration tests that mimic real-world data scenarios, providing comprehensive validation of your end-to-end data flows.
We’ll also discuss how these essential practices seamlessly fit into a modern CI/CD workflow, ultimately building greater confidence in your production pipelines.
What You’ll Learn
Clean, Testable PySpark: Learn to write PySpark code that is no longer a black box. Discover methods to make your transformations clean, modular, and easily unit-testable, significantly improving code quality and maintainability.
Local Production Mirroring: Understand how to set up containerized environments that accurately mirror your production setup locally, incorporating PySpark, Delta Lake, and Kafka. This allows for robust development and testing in a controlled environment.
Simulate Real-World Flows: Gain practical insights into simulating real-world data flows to proactively identify and catch issues before they ever reach production. This proactive approach minimizes risks and ensures pipeline stability.
Automated Checks and Feedback Loops: Explore strategies for automating checks and establishing effective feedback loops within your CI/CD workflow. These practices build confidence in your production pipelines, ensuring they are robust and reliable.
Who Should Attend
Whether you’re looking to scale your data pipelines, enhance your team’s QA practices, or simply ensure the reliability of your Databricks solutions, this session offers the practical knowledge and techniques you need to succeed.
