programmers-horizontal

Our Blog

Artificial Intelligence (AI)
AGILE & Development
DIGITAL MODERNIZATION
Data & Analytics
Case Studies
Agile & Development
Introduction to Machine Learning Operations (MLOps)
As artificial intelligence and its applications become more prevalent in digital products and services, it is crucial...

11 MIN READ

November 11, 2024

11 MIN READ
As artificial intelligence and its applications become more prevalent in digital products and services, it is crucial to develop appropriate methods and studies for the verification, implementation, management, and monitoring of these models in production environments. Essentially, with the practice of MLOps, we aim to minimize the “technical debt” in Machine Learning implementations.

Introduction to MLOps

Differences Between MLOps and DevOps

Basic Principles of Machine Learning Operations

Versioning

Feature Store

Automation

Monitoring

Tools

Introduction to MLOps

MLOps is an operational approach designed to simplify the process of deploying an experimental model into a production environment and managing it effectively. Based on DevOps practices, it treats machine learning (ML) models as reusable software artifacts. Among all the improvements that adopting this approach can bring, I believe the most directly impactful on business value are: Adaptation to Change – It can be difficult to keep up with the evolution of business requirements. The data flowing into the models can change constantly. Therefore, having a system that is easy to maintain for such cases is essential in various situations. Collaboration – The inability to find common ground for communication among all stakeholders in a project, including data scientists, operations directors, and business leaders, can be a significant obstacle to AI initiatives.

Differences Between MLOps and DevOps

MLOps is based on DevOps practices, so there are many similarities in various stages of their structures. However, there are some important differences: Experimentation – ML teams need to adjust hyperparameters, data, and models while tracking their experiments to ensure reproducible results in their respective labs. Testing – In an MLOps system, it is necessary to train, test, and evaluate the models to identify their performance. This makes testing more complex, as each step must be validated and meticulously documented. Here’s a comparison of the tests typically performed in both practices:

Machine Learning (ML) systems require extensive testing and monitoring.

Automated Deployment – ML models require more coordination and automated processes for deployment. This necessitates a multi-step pipeline that can retrain the model, evaluate it, and monitor its performance in production. Production Monitoring – It is expected that ML models will experience performance degradation. Models often perform worse in production than during training due to differences between the training data and the inputs of new data. CI/CD/CT – In ML, the pipeline must address additional concerns, including data validation, data schemas, models, and their performance. Thus, continuous testing (CT) is a new element in the MLOps pipeline, which automatically retrains and delivers models based on inputs from the production environment.

Basic Principles of Machine Learning Operations

Thus, when developing a draft architecture for your MLOps system, we can define some of its main characteristics:

Versioning

The ability to replicate the same code base so that multiple people can work on the same project simultaneously is a great benefit.

“To ensure that the experimentation of the data science team leads to a production model for the project, it is important that key factors are documented and reusable.”
(“Introducing MLOps,” Treveil and Dataiku Team)

Therefore, what should be properly versioned:

    • Assumptions: The decisions and assumptions of the Data Scientist must be explicit.
    • Randomness: It needs to be under some type of control to ensure reproducibility, for example, by using a “seed.”
    • Data: The same data from the experiment must be available.
    • Configurations: Repeat and reproduce experiments with the same configurations as the original.
    • Environment: It is crucial to have the same runtime configurations among all data scientists.

Feature Store

The goal of a Feature Store is to process data from various data sources simultaneously and transform it into features that will be consumed by the pipelines.

“Feature Stores are reliable tools for managing features for research and training using an Offline Store, as well as to manage the feeding of features for a model served in production using an Online Store.”
(“Introducing MLOps,” Treveil and Dataiku Team)

Offline Store: A store composed of pre-processed batch data features, used to build a historical source of features that can be utilized in the model training pipeline.
Online Store: A store made up of data from the Offline Store combined with real-time pre-processed features from streaming data sources.

Automation

The automation of machine learning pipelines is highly correlated with the maturity of the project. There are several stages between model development and deployment, and much of this process relies on experimentation.

MLOps Level 0: Manual Process: A complete experimentation pipeline executed manually using rapid application tools.
MLOps Level 1: ML Pipeline Automation: Automation of the experimentation pipeline, including data and model validation.
MLOps Level 2: CI/CD Pipeline Automation: Automatically build, test, and deploy ML models and ML training pipeline components.

    • CI (Continuous Integration): Whenever code or data is updated, the ML pipeline is re-executed. This is done in a way that everything is versioned and reproducible.
    • CD (Continuous Deployment): Continuous deployment is a method for automating the deployment of the new version to production or any environment, such as testing.

Monitoring

The performance of machine learning models can vary over time. Therefore, once a model is deployed, it needs to be monitored to ensure it operates as expected. But what should be monitored?

Model Monitoring

Some of the key points we should pay attention to are the components and concepts of:

    • Performance: Based on a set of metrics.
    • Data: Inconsistencies or errors due to various transformations.
    • Explainability: Being able to explain a model’s decision is vital.
    • Bias: Monitoring for undesirable biases or trends in the model’s outputs
    • Drift: Properties may change over time, leading to concept drift.

Tools

Finally, there are numerous tools you can use to develop systems based on MLOps architectures. In Azure, for example, you can leverage Azure Machine Learning, which provides an integrated environment for managing the complete lifecycle of machine learning models, including building, training, deploying, and monitoring. Azure DevOps is also an excellent choice for CI/CD automation specific to ML projects.

In AWS, you can use Amazon SageMaker, a comprehensive solution that facilitates the building, training, and deployment of machine learning models at scale. AWS CodePipeline and AWS CodeBuild can also be configured to automate training and deployment steps for models.

In Google Cloud, Vertex AI stands out as a unified platform for developing machine learning models, encompassing everything from data preparation to model deployment. Lastly, GCP also offers Cloud Build for CI/CD automation, which integrates seamlessly with other Google Cloud services for effective ML lifecycle management.

RELATED POSTS

Recent Post

Stay up to date on the latest trends, innovations and insights.