Videos
Welcome to our Video Library, where you can access all the insightful videos. Our collection covers a range of topics, from foundational concepts to advanced strategies, providing you with a comprehensive understanding of the latest trends and innovations from past Devday, Conferences, etc.
Apache Spark Under The Hood
February 28, 2025
Then vs. Now of Build Tools
February 28, 2025
1:11:31 Empowering Data Decisions: A Data Engineering Solutions for Integrated Insights
February 17, 2025
1:12:47 Container Security: Understanding the Security Landscape
November 14, 2024
00:05:26 Sahaj Software Corporate Video
October 22, 2024
00:03:03 Client Voice: Technology innovations to transform Talon Outdoor and the Out of Home industry
October 22, 2024
00:04:13 Client Voice: A new ISA platform for Vemo Education to address the US student loan crisis
October 22, 2024
00:03:44 Client Voice: Disrupting OOH media with game-changing solutions
October 22, 2024
00:16:00 The Beginner’s Guide to All Things Sahaj
October 22, 2024
00:07:10 Sahaj – Lending Sixth Sense to a Business with Data Science
October 22, 2024
00:04:59 Sahaj X Citizens GBR – Securing the Great Barrier Reef with Engineering and Collaboration
October 22, 2024
00:26:24 A Playbook For Kickstarting Your Data Journey
October 18, 2024
01:51:40 Dev Day | Understanding the internal workings of databases
October 18, 2024
01:51:14 Uncover the mysteries of IAC
October 18, 2024
01:29:49 The Power of Right Abstractions – The Story of HTTP
October 18, 2024
01:00:43 Testing in Flutter
October 18, 2024
02:32:43 Strangulating infrastructure to containers
October 18, 2024
01:10:09 Starting your journey as Developer
October 18, 2024
00:37:11 Solving real world enterprise problems using Blockchain
October 18, 2024
01:06:50 Shipping LLM Addressing Production Challenges
October 18, 2024
01:29:49 Open language resources and strategies for building speech systems in under-resourced languages
October 18, 2024
01:27:16 MLOps, Data Drift and Concept Drift in Production
October 18, 2024
01:04:29 Mental models used in nation-scale digital certificate platform
October 18, 2024
00:59:49 LLMs in action: Strategies for crafting solutions
October 18, 2024
00:28:46 Decentralized Finance: Business Opportunities and Trend III
October 18, 2024
01:19:04 JVM GC Tuning in containerised environment
October 18, 2024
01:15:20 Intro to Apache Spark 2.x & running a Spark cluster in the cloud
October 18, 2024
00:57:31 From VMs to containers and back again
October 18, 2024
01:07:15 From Concept to Code: Building a Q&A Bot Live with LLM
October 18, 2024
01:18:42 LLM Unleashed: Deep dive into fundamental concepts of LLM
October 18, 2024
00:47:06 Clean Code 101
October 18, 2024
00:37:07 Decentralized Finance: Business Opportunities and Trends IV
October 18, 2024
01:05:19 Decentralized Finance: Business Opportunities and Trends II
October 18, 2024
00:43:03 Decentralized Finance: Business Opportunities and Trends I
October 18, 2024
01:09:42 Classifying Indian addresses for the e-commerce domain
October 18, 2024
01:16:06 Building a search engine for interactive search
October 18, 2024
01:21:56 Automatic Speech Processing for Voice AI
October 18, 2024
00:59:50 LLM Solutions Navigating the landscape
October 18, 2024
Apache Spark Under The Hood
By Dakshin K
February 28, 2025
Summary- Apache Spark addresses limitations of MapReduce by leveraging in-memory processing and lazy evaluation, reducing costly disk I/O and enabling complex workflows.
- Lazy evaluation optimizes performance by consolidating operations into stages, minimizing intermediate data shuffling between cluster nodes.
- Spark's execution model uses narrow (partition-local) and wide (shuffle-dependent) transformations, with stages created at shuffle boundaries impacting job efficiency.
- Data immutability and incremental processing are emphasized for cost-effective reprocessing, auditability, and historical data retention in production pipelines.
- Partitioning strategies critically affect performance: skewed data distribution can cause bottlenecks, while proper repartitioning improves parallelism.
- Spark SQL and DataFrame APIs simplify distributed computation by abstracting low-level complexities, though understanding execution plans remains vital for optimization.
- Cluster memory management and spill-to-disk mechanisms balance large dataset processing with hardware constraints, requiring careful monitoring.
Generated using GPT-4-o-min.