Platform Engineering

Scaling Stateful Connections to Millions

In a world of high-traffic IoT systems, delivering real-time data updates is critical for enhancing user experience. A scalable architecture is essential to manage continuous streams of information from connected devices, be it for smart home applications, industrial monitoring, or connected vehicles. Here, we’ll discuss a general technical approach to building a high-scale, reliable event-streaming platform, using a Connected Vehicle Platform (CVP) as an example.We are building a high-scale connected vehicle platform for one of the leading automobile manufacturers. Currently,...

Read

Understanding NextAuth by Fixing a Session Expiry Bug

Last week, we were fixing some of the critical bugs in our internal application. Given that we are nearing our 1.0 release (this is a greenfield project for us!), some of us are polishing the application and removing any bugs we find. Among the bugs we fixed, we also fixed a notable one which was one of the most longstanding but subtle bugs we have seen so far, and it was related to Session Expiry.The problem was straightforward: a user...

Read

Cassandra’s Tunable Consistency Model: A Game-Changer for Distributed Systems ⚙️🚀

In today’s world, databases are the backbone of information flow, much like roads guiding traffic 🚗. Just as roads differ in structure, databases rely on varying consistency models to manage data. In distributed systems, where data is replicated across multiple nodes or data centers, ensuring that all users see the same data simultaneously can be challenging, especially during high traffic or network failures 🚦. These conditions can cause nodes to temporarily hold different data copies, making synchronization across the system...

Read

K8s Security Context: Optimizing fsGroup for Faster Pod Starts ⏰

In a data pipeline architecture, it is often challenging to efficiently read a batch of messages from Kafka and write them to AWS S3 while reducing storage costs. To achieve this, Apache Flume was employed, an effective tool for collecting and moving large amounts of log data in real-time. This use case revolves around reading messages from Kafka in bulk, and then buffering them temporarily before transferring them to S3 in larger batches. This batching mechanism helps reduce API calls...

Read

A mobile money app powers digital wallets of multiple telecom service providers

A multi-tenant cloud-ready platform drives market leadership, leveraging modern design paradigms. Comviva is a mobile banking solutions provider and caters to over two billion platform users globally. The company's digital mobile wallet is used by telecom providers to provide financial transaction capabilities on telecom networks largely focused in Africa and Middle East, South America, and Southeast Asia markets. To secure market leadership, the company developed a multi-tenant cloud-ready platform, in partnership with Sahaj Software, to power mobile wallets for various...

Read

An Artisan’s Way of Data Fetching in React

In React applications, one of the most frequently executed tasks is to use the backend APIs and fetch data using fetch/axios/similar libraries and render it using the component. So, generally, the code has the following parts:React state to store the result of the API call as well as state to show the beautiful spinnerFunctional call to the API (most probably in useEffect)Statements in useEffect hook to update the state of the resultCatch statement to show errors related to API (alert...

Read

Unifying SEEK’s Complex Data Ecosystem for Business Transformation

SEEK is a market leader in online employment marketplaces with a multinational presence spanning Australia, New Zealand, Hong Kong and South East Asia. In addition, SEEK has minority investments in China, South Korea and several other countries. In 2014, SEEK acquired leading employment marketplaces in South East Asia, JobsDB and Jobstreet. We partnered with Seek to implement the Unification programme - an ambitious plan to merge diverse systems from various geographies into its existing Australia and New Zealand platforms. A...

Read

Solving the Silent SNS Alarm Issue in SQS Workflow: An Interactive Dive into a Step Function Solution

In the realm of SQS workflow management, I encountered a perplexing issue with SNS alarms. Previously, I relied on SNS alarms to notify me when messages populated in my SQS queues, especially when they found their way into the Dead Letter Queue (DLQ). However, the alarms seemed to stay silent when my workflow triggered multiple times, causing messages to accumulate in the DLQ. Upon closer inspection, the root cause surfaced: the SNS alarm remained in an ALARM state, because the...

Read

Creating a habit: Refactoring

For more visibility posting this on medium. The original article is posted on my website. :) I want to start this blog with Kent Beck quote I am not a great programmer. I am just a good programmer with great habits Having these great habits while developing software is essential. These habits show your commitment to your work and the quality of the software you want to deliver. In this blog, we are going to talk about one of those habits. . Before we...

Read

Making Sense of Parser Combinators

Initially, I started this post with the protocol I wrote a parser for, but since the post became big and complicated — it was probably worth breaking it down into a smaller piece. Our aim is to write a simple parser from scratch and parse a number from a String. For this post let’s scope it down to positive numbers (unsigned integers). The first question that arises is: why do we need to start from scratch? We could utilise existing...

Read