Data Versioning

Data Quality in the Training Stage of ML Pipelines

Data quality refers to the degree to which a dataset meets the following key requirements for reliable machine learning: Accuracy: The correctness and precision of data values in representing real-world facts Example: Temperature readings from sensors match actual temperatures within acceptable margins. Temperature in Kashmir in January cannot read 30 degrees celsius. Completeness: The presence of all necessary data points and features Example: No missing values in crucial fields. Consistency: The uniformity of data across the dataset Example: Consistent date...

Read

Ensuring Data Quality in ML Pipelines using DVC

Ensuring data quality is crucial for building reliable ML models. In this guide, we integrate data quality checks into an ML pipeline using DVC (Data Version Control) to enable versioning, tracking, and automated validation of datasets before model training.Machine Learning pipelines automate the end-to-end ML process, handling everything from data ingestion to model deployment. They help break down complex workflows into manageable stages.Important stages of the pipeline:Data Quality checkData preprocessingFeature engineeringModel training and evaluationModel serving and monitoringThis article focuses on...

Read

Reproducibility in ML: The Role of Data Versioning

Once you’ve run machine learning models in production, becomes one of the non-negotiable aspects of delivery since the baseline performance of the model has already been defined. The ability to replicate results, debug issues, and improve models requires that all experiments are easily traceable. Unlike traditional software development, where versioning source code is sufficient to reproduce builds, ML workflows can degrade significantly with changes to the data, its distribution as well as changes to any preprocessing steps.In this article, we’ll...

Read

Data storage patterns, versioning and partitions

When you have large volumes of data, storing it logically helps users discover information and makes understanding the information easier. In this post, we talk about some of the techniques we use to do so in our application.In this post, we are going to use the terminology of AWS S3 buckets to store information. The same techniques can be applied on other cloud, non cloud providers and bare metal servers. Most setups will include a high bandwidth low latency network...

Read