When machine learning (ML) engineers or data scientists work on any ML solution, they progress through a cycle of building, deploying, and scaling the solution. Data and infrastructure are two of the major areas of focus in building a pipeline for better model performance. Larger amounts of data lead to a higher level of model performance and a greater need for infrastructure.

The machine learning infrastructure stack is the combination of all components required to build, deploy, and scale the machine learning systems. ML infrastructure includes all kinds of hardware, software, and other utilities.

As a machine learning engineer or a data scientist, you’ll work on a lot of projects throughout your career. These projects will present you with a multitude of technology options, including those covering varied functionalities or only a few or individual functionalities. Tools can be open source or paid, and some may require a high level of expertise or be simple enough for beginners to get started with. All these tools and technologies depend on your use cases and your level of experience, and having a proper understanding of the ML infrastructure stack will make it easier for you to choose suitable technologies for your project.

In this article, you will learn about each piece of ML infrastructure stack in detail and explore some of the tools and technologies recommended for certain categories.

The importance of understanding the ML infrastructure stack 

Usually, the ML infrastructure stack comprises three layers:

 This layer comprises all the tools required to do the data preprocessing, from data gathering to data analysis, data cleaning, and data visualization. Tools such as 

 This layer is responsible for building the machine learning models for your use cases. These use cases include model experimentation, logging, and version control. Tools like 

 Once you are done with all the data preprocessing and model experimentation, you get a final model that is handed over to the deployment team as an artifact. This layer contains all the tools required to deploy the models, such as 

Now that you know what the ML infrastructure stack is and have an understanding of the three layers that comprise it, let’s take a look at some of the important use cases of the ML stack:

It allows you to choose which tools you want to use, as well as how they connect.

 When machine learning engineers work on any use case, they typically go through a series of steps wherein each step requires different tools and technologies. In the development of a project pipeline, each step is closely interlinked with the others—the output from one stage becomes the input for the next stage. Thus, it’s important that these tools be selected in such a way that they are easily embedded together to complete the project pipeline.

It provides you with the facilities required to do effective ML operations.

 Machine learning solutions are built with the help of effective data analysis, data visualization, model experimentations, and successful deployment. Tools like Python and SQL make it easy for ML engineers to do the data gathering and analysis part, while tools like 

 make visualizations easy and effective. 

 technologies help ML engineers easily create machine learning and deep learning models. Finally, platforms like 

It allows you to compare different tools.

 When someone develops any traditional software, there are many programming languages to choose from to write that application. Similarly, for a machine learning use case, each stage has several tool options based on your specific needs. By comparing different tools, you can select the best combination for your particular use case.

The various pieces of the ML infrastructure stack 

The traditional way of developing a machine learning pipeline is to create a model using an ML stack. Here, data scientists and machine learning engineers develop models that are then passed to the operations team for deployment. However, miscommunication among teams can sometimes lead to project failure. A new technology is now being used by many organizations to tackle this issue: 

 is the combination of both machine learning and operations, and it makes ML pipeline development easy. Technologies used in both the ML stack and the MLOps stack are almost identical, though MLOps does incorporate some extra tools to make pipeline development more robust.

Let’s take a look at the different pieces of the MLOps stack to get a better understanding of the role of each tool involved in ML pipeline development.

Data is the foundational element when developing machine learning solutions. If you do not have a good amount of data—or any data at all—having the right tool or technology to create the solution is irrelevant.

 and have your data ready to use, you need to analyze the data well before commencing the model building phase. Data collected for any use case cannot be used directly, as it may not be uniform or it could contain some corrupted values. The data is first cleaned to remove those corrupted values, and then the data is transformed into a format suitable for model training. For data cleaning and transformation, there are several tools available.

 is an open-source programming language often preferred by data scientists. It provides different libraries that can be directly imported into a program, which reduces the effort of implementing different concepts and machine learning algorithms. Another choice for a programming language by a data scientist is the 

While working with tabular data, you need to read it using Excel and CSV files. The 

 library not only helps you in reading these files but also provides a lot of functions that you can perform on those data frames, such as merging, replacing values, searching elements, and so on.

Once your data is cleansed and transformed, your next task is to find a machine learning model that performs best on your data. Getting to the best model involves trying out different models with a different set of parameters. This is referred to as 

, which is done with the help of a programming language; you need to write different code snippets to try out different models. To write the code, data scientists commonly use two different tools:

 is a web-based platform that allows you to write and compile code in small blocks called 

. These notebooks are very helpful for data cleaning and data exploration phases, as you need to look at your data and graphs after each line.

 are generally preferred by software engineers, but data scientists and ML engineers also use them at the end of the model-experimentation phase since applications can’t be directly deployed in the form of notebooks. Code editors help developers convert code snippets from notebooks into Python files. There are plenty of options available for code editors; you can check out a list of those widely used by data scientists and machine learning engineers 

Imagine you are working on a project and have tried a particular model out for a day. The next day, your system restarts, which means you have to repeat the data cleaning and data transformation phase all over again. 

 can help you avoid having to redo all that work. After preprocessing your data, you can store it in a feature store, giving you the opportunity to reuse it whenever it’s needed. Some examples of feature stores include the following:

 is provided by AWS and can be easily integrated with its other services like 

 is built around Spark DataFrames. It provides a UI for easy interaction and also supports SQL querying.

There are various other options available for feature stores, which you can check out 

Data scientists typically don’t work alone; many ML solutions are developed in a team environment. If you’re working on one module of a project while other members are working on other modules, at the end of the day, all the changes need to be merged to form the system. However, you also need to manage different versions of your applications, as you may include new features in your app over time. These processes cannot be done manually using file transfer or a local file store. However, using version-control tools, you can achieve both of these functionalities.

 is a very popular tool for managing applications, as it provides version control and excellent team collaboration. You can learn more about it 

 is another version-control tool that is easily integrated with Jira and has built-in CI/CD. You can learn more about it 

A machine learning pipeline is a series of steps needed to build, deploy, and scale your model. It’s a very crucial thing for an organization, as it comprises all the stages required to create web-based or API-based applications. Each stage’s completion may require different libraries and dedicated hardware resources in order to make the ML project successful. To learn more about machine learning pipelines in detail, you can refer to 

 is one of the great tools used by various enterprises for ML model experiment tracking and effective team collaboration. Learn more about all the functionalities that MLflow offers 

 is mainly used for the deployment purpose of machine learning applications. If you want to deploy your applications as containers using Kubernetes, Kubeflow is a great choice.

 is a repository that is used for the version control of trained machine learning models. They are responsible for keeping track of ML models as they move from team to team, starting from training and all the way to deployment and retirement. It’s very difficult for data scientists to keep track of all the things they have tried throughout a project lifecycle; the model registry helps them to document all the models and parameters by providing each with a unique ID. Examples of popular model registries include the following:

 is a part of multiple offerings from Amazon’s SageMaker. It can be used to easily store models and their different parameters at any time during development.

 is an open-source model registry. It has multiple offerings like model versioning, stage transitions, annotations, and more. All these features help in managing the machine learning models easily during development.

To learn more about model registries in detail, take a look at 

When data scientists develop machine learning modules, those modules can be made available using APIs to be incorporated into other applications as well. Consider an example: You have created a face-detection model for marking employee attendance in an office, and later you are working on a new use case like 

KYC (“know your customer”) verification

. For this new project, you don’t need to start from scratch to implement facial detection—you can easily import your previously created face-detection model for this new use case. 

 is simply the process of hosting machine learning applications on the cloud (or on-premise) and making them accessible in the form of APIs so that they can be incorporated into other applications.

To learn about the best tools used for model serving—such as 

Once you are done with the model development and deployment, the work isn’t over. There may be scenarios where your model is performing very well in training but is failing miserably in the real world of data. The only way to know for sure is to monitor it continuously for a specific period from deployment. This process is called 

, and it is the best way to ensure an acceptable level of performance from your model.

For a list of tools used for model monitoring—such as 

When you train a machine learning model, there is a lot of information generated, including model parameters, evaluation metrics, data-set versions, pipeline output, weight files, model-architecture files, and so on. This information is called 

, and it needs to be stored so that you can easily use it again as needed.

There are several tools available for this purpose, like 

 to see which one might be a good fit for you.

In the real world, you’ll often see use cases where positive class samples occur far less often than negative class samples. One great example of this is in credit card fraud detection. In this use case, you might notice that the number of true fraud cases is significantly lower than the number of nonfraudulent cases. Here, negative/nonfraudulent classes are called 

 classes, and positive/fraudulent classes are called 

 classes. These highly skewed proportions are referred to as 

. Due to these imbalanced classes, machine learning models are prone to 

One of the solutions to this problem is to use 

 for minority classes. In the process of synthetic-data generation, new data points are generated for minority classes in order to bring more balance to the class samples. To learn more about synthetic data and how you can generate it, you can refer to 

 provides a wide variety of solutions for machine learning, including synthetic-data generation, data cleansing, and the relabeling and tuning of model architecture to improve the overall performance of your ML models.

For more information on other synthetic-data generation tools—such as 

In this article, you learned what an ML infrastructure stack is and what tools and technologies come together to create it. When data scientists and machine learning engineers work on any use case, their focus is typically on the problem statement and the machine learning models to solve that problem and not necessarily the hardware and software needed to do so well. However, as you’ve seen in this guide, selecting the right infrastructure is essential to developing and deploying any solution (machine learning or traditional software) to make an ML project successful.

As you’ve seen, choosing the right tools for your ML infrastructure stack is not a one-time process that works for all use cases within your organization—each type of machine learning solution requires a different kind of tech stack. Be sure to check out the resources mentioned in this article to learn more about the different tools available to create the right combination of technologies for your particular use case.

* A previous version of this article listed the company name as Unbox, which has since been rebranded to Openlayer.

Posts from Gourav Singh Bais

Understanding each piece in the ML infrastructure stack

Making sense of ML systems

Gourav Singh Bais

Writing Openlayer posts since

Number of posts