Performance Tuning

Post Image

Prompting Is Architecture: Why Understanding Systems Matters More Than Ever

A Spark story about scale, grain, and why speed makes system awareness non-negotiable.Image generated using ChatGPTRecently, I ran into a Spark job that looked perfectly fine. I used Claude in this case to generate code, which wasSyntactically correctSemantically correctEasy to read and reason aboutThis worked on the sample dataset. Assuming Spark will manage the scale, we pushed it to production. However, on prod, it took more than 3 hours to complete on a 40-node EMR cluster. It was trying to...

Read
Post Image

Designing OLAP Schemas by Query Shape

How we learned that schema purity matters less than matching the natural grain of your queries.Image generated using ChatGPTBefore we talk schema design, we need to discuss the problem. Without the domain context, the technical decisions won’t make sense.Think of the digital billboards you see on highways, in malls, airports, or gyms. Each physical screen is called a .Advertisers don’t just ask — How many people see this screen?They ask:: Physical advertising screens. In the US alone, there are millions...

Read
Post Image

Scaling Stateful Connections to Millions

In a world of high-traffic IoT systems, delivering real-time data updates is critical for enhancing user experience. A scalable architecture is essential to manage continuous streams of information from connected devices, be it for smart home applications, industrial monitoring, or connected vehicles. Here, we’ll discuss a general technical approach to building a high-scale, reliable event-streaming platform, using a Connected Vehicle Platform (CVP) as an example.We are building a high-scale connected vehicle platform for one of the leading automobile manufacturers. Currently,...

Read
Post Image

K8s Security Context: Optimizing fsGroup for Faster Pod Starts ⏰

In a data pipeline architecture, it is often challenging to efficiently read a batch of messages from Kafka and write them to AWS S3 while reducing storage costs. To achieve this, Apache Flume was employed, an effective tool for collecting and moving large amounts of log data in real-time. This use case revolves around reading messages from Kafka in bulk, and then buffering them temporarily before transferring them to S3 in larger batches. This batching mechanism helps reduce API calls...

Read