Data Platforms

Post Image

To ETL or Not to ELT: Choosing the Right Approach for Data Processing and Management

Deciding between Extract, Transform, Load (ETL) and Extract, Load, Transform (ELT) is one of the most important decisions a data engineer has to make. Transforming data before loading (ETL) or after loading (ELT):can shape how efficiently we handle datahow flexible we are in adapting to changing requirementsand how well we scaleAs a data engineer, I've had many opportunities to work with various data pipelines and tools. A commonly asked question is 'Should we use ETL or ELT?'. Both methods have...

Read
Post Image

Scaling Stateful Connections to Millions

In a world of high-traffic IoT systems, delivering real-time data updates is critical for enhancing user experience. A scalable architecture is essential to manage continuous streams of information from connected devices, be it for smart home applications, industrial monitoring, or connected vehicles. Here, we’ll discuss a general technical approach to building a high-scale, reliable event-streaming platform, using a Connected Vehicle Platform (CVP) as an example.We are building a high-scale connected vehicle platform for one of the leading automobile manufacturers. Currently,...

Read
Post Image

Data storage patterns, versioning and partitions

When you have large volumes of data, storing it logically helps users discover information and makes understanding the information easier. In this post, we talk about some of the techniques we use to do so in our application.In this post, we are going to use the terminology of AWS S3 buckets to store information. The same techniques can be applied on other cloud, non cloud providers and bare metal servers. Most setups will include a high bandwidth low latency network...

Read