Dataset Tracking With DVC: A Practical Guide for MLOps

Wed, 02 Oct 2024 14:36:11 +0200

In the rapidly evolving field of machine learning, managing and tracking datasets is crucial for reproducibility, collaboration, and efficient workflow. Data Version Control (DVC) has emerged as a powerful tool in the MLOps toolkit, allowing data scientists and ML engineers to version their datasets alongside their code. This blog post will guide you through setting up DVC with a DigitalOcean Spaces backend, an S3-compatible storage solution, to track your datasets effectively.

Why Dataset Tracking Matters

Before we dive into the setup, let’s briefly discuss why dataset tracking is essential in machine learning workflows:

Reproducibility: Ensures that experiments can be replicated with the exact same data.
Collaboration: Enables team members to share and sync datasets effortlessly.
Versioning: Allows tracking of data changes over time, similar to code versioning.
Storage Efficiency: Stores only the changes, not duplicate copies of entire datasets.
Integration: Seamlessly integrates with existing Git workflows for code.

Now, let’s get started with setting up DVC in your project.

Setup DVC

Step 1: Initialize DVC

Before you start, make sure you have DVC installed:

pip install dvc dvc-s3

Begin by initializing DVC in your project root:

dvc init

This command creates a .dvc directory to store DVC-specific files and configurations.

Step 2: Configure the Remote Storage

Next, we’ll configure DVC to use DigitalOcean Spaces as our remote storage. Add the following to the .dvc/config file:

[core]
 analytics = false
 remote = mapllm
['remote "mapllm"']
 url = s3://mapllm/data
 endpointurl = https://ams3.digitaloceanspaces.com
 profile = mapllm

This configuration tells DVC:

To use a remote named “mapllm”
The S3 bucket URL where data will be stored
The endpoint URL for DigitalOcean Spaces
The AWS credential profile to use

Step 3: Set Up Credentials

For security, we’ll store our DigitalOcean Spaces credentials in the AWS credentials file. Add the following to your ~/.aws/credentials file:

[mapllm]
aws_access_key_id = YOUR_ACCESS_KEY_ID
aws_secret_access_key = YOUR_SECRET_ACCESS_KEY

Replace YOUR_ACCESS_KEY_ID and YOUR_SECRET_ACCESS_KEY with your actual DigitalOcean Spaces credentials.

Note: Keep these credentials secure and never commit them to your repository.

Tracking Data with DVC

Now that we’ve set up DVC and configured our remote storage, let’s start tracking some data.

Adding Data to DVC

To track a data file with DVC, use the dvc add command:

dvc add data/cities.parquet

This command will:

Create a .dvc file (data/cities.parquet.dvc) that contains a reference to the data file.
Add the original data file to .gitignore to prevent it from being tracked by Git.

You should see output similar to this:

100% Adding...|████████████████████████████████████████████████████████████████|1/1 [00:00, 2.01file/s]
To track the changes with git, run:
 git add data/.gitignore data/cities.parquet.dvc
To enable auto staging, run:
 dvc config core.autostage true

Committing and Pushing Data

After adding your data to DVC, you need to:

Commit the DVC tracking file to Git:

git add data/cities.parquet.dvc
git commit -m "Add cities data"

Push the actual data to your remote storage:
```
dvc push
```

Pulling Data

Later when clonning the repo in a new machine, assuming AWS credentials are in place all you have to do is to pull the data from the remote storage is to use the following command:

dvc pull

This process separates the concerns of version control: Git tracks the metadata and DVC handles the large data files.

A few tips

Regular Updates: Whenever you update your dataset, re-run dvc add and dvc push to keep everything in sync.
Branching: Use Git branches for different versions of your data, just like you would with code.
Pulling Data: When collaborating, use dvc pull to fetch the latest version of the data after a git pull.
Checking Status: Use dvc status to see if your local data is in sync with the remote storage.
Data Exploration: DVC allows you to switch between different versions of your data easily, facilitating A/B testing and model comparison.

Setting up DVC with a DigitalOcean Spaces backend provides a robust solution for dataset tracking in your MLOps workflow. By versioning your data alongside your code, you ensure reproducibility, enhance collaboration, and maintain a clear history of your dataset evolution.

Solving Memory Issues When Loading Parquet Files into Pandas DataFrames

Wed, 02 Oct 2024 12:10:22 +0200

Introduction

Have you ever encountered unexpected memory issues when loading a seemingly small Parquet file into a Pandas DataFrame? You’re not alone! In this article, we’ll explore a common but often overlooked problem that can cause your Python process to consume gigabytes of memory when working with Parquet files containing dictionary columns.

The Problem: Unexpected Memory Consumption

Recently, I faced a perplexing issue while working with a 145MB Parquet file containing over 2 million rows. When attempting to load this file into a Pandas DataFrame using the standard method:

import pandas as pd
df = pd.read_parquet('path/to/my/file.parquet')

To my astonishment, the process consumed more than 90GB of memory before being terminated by the operating system. In 2024, a 145MB file shouldn’t be considered large, so what was causing this extreme memory usage?

Investigating the Cause

After some debugging, I discovered that the culprit was a single column with a dictionary data type. This column contained dictionaries with unpredictable keys - potentially thousands of unique keys across the dataset.

Understanding Parquet’s Dictionary Encoding

Parquet uses a technique called dictionary encoding to efficiently store and compress columns with repeated values. When storing a pandas DataFrame as a Parquet file, this encoding works well. However, when the data is sparse and most keys are not frequently reused, loading the Parquet file back into memory can cause issues.

When Parquet loads a dictionary-encoded column with sparse data, it creates a map for each entry containing all unique keys, with most values set to None. For millions of rows, this results in a massive memory footprint - in our case, over 90GB!

The Solution: Converting Dictionary Columns to JSON Strings

Since I didn’t need to perform queries on this problematic column, the solution was surprisingly simple: convert the column to a string representation before saving to Parquet. Specifically, using json.dumps to serialize the dictionaries:

import json
import pandas as pd

# When saving the DataFrame to Parquet
df['column_name'] = df['column_name'].apply(json.dumps)
df.to_parquet('path/to/output.parquet')

# When reading the Parquet file
df = pd.read_parquet('path/to/output.parquet')
# If you need the column as dictionaries again:
df['column_name'] = df['column_name'].apply(json.loads)

This one-liner drastically reduced memory usage, allowing the file to load into a DataFrame without issues.

Key Takeaways

Be cautious when using dictionary columns in Parquet files, especially with sparse data.
Consider converting dictionary columns to JSON strings if you don’t need to query their contents directly.
Always profile your data and memory usage when working with large datasets.
Understand the underlying storage mechanisms of file formats like Parquet to optimize your data pipeline.

Conclusion

While Parquet’s dictionary encoding is generally beneficial for data compression and query performance, it can lead to unexpected memory issues in certain scenarios. By being aware of these potential pitfalls and applying simple solutions like converting to JSON strings, you can significantly optimize your data processing workflows.

Data Science | Sasan Jafarnejad | AI Researcher