Islam | Machine Learning Model Serving Patterns and Best Practices | E-Book | www.sack.de
E-Book

E-Book, Englisch, 336 Seiten

Reihe: Packt Publishing

Islam Machine Learning Model Serving Patterns and Best Practices

A definitive guide to deploying, monitoring, and providing accessibility to ML models in production
1. Auflage 2024
ISBN: 978-1-80324-253-8
Verlag: De Gruyter
Format: EPUB
Kopierschutz: 0 - No protection

A definitive guide to deploying, monitoring, and providing accessibility to ML models in production

E-Book, Englisch, 336 Seiten

Reihe: Packt Publishing

ISBN: 978-1-80324-253-8
Verlag: De Gruyter
Format: EPUB
Kopierschutz: 0 - No protection



Serving patterns enable data science and ML teams to bring their models to production. Most ML models are not deployed for consumers, so ML engineers need to know the critical steps for how to serve an ML model.
This book will cover the whole process, from the basic concepts like stateful and stateless serving to the advantages and challenges of each. Batch, real-time, and continuous model serving techniques will also be covered in detail. Later chapters will give detailed examples of keyed prediction techniques and ensemble patterns. Valuable associated technologies like TensorFlow severing, BentoML, and RayServe will also be discussed, making sure that you have a good understanding of the most important methods and techniques in model serving. Later, you'll cover topics such as monitoring and performance optimization, as well as strategies for managing model drift and handling updates and versioning. The book will provide practical guidance and best practices for ensuring that your model serving pipeline is robust, scalable, and reliable. Additionally, this book will explore the use of cloud-based platforms and services for model serving using AWS SageMaker with the help of detailed examples.
By the end of this book, you'll be able to save and serve your model using state-of-the-art techniques.

Islam Machine Learning Model Serving Patterns and Best Practices jetzt bestellen!

Autoren/Hrsg.


Weitere Infos & Material


Table of Contents - Introducing Model Serving
- Introducing Model Serving Patterns
- Stateless Model Serving
- Continuous Model Evaluation
- Keyed Prediction
- Batch Model Serving Pattern
- Online Learning Model Serving
- Two-Phase Model Pattern
- Pipeline Pattern Model Serving
- Ensemble Model Serving Pattern
- Business Logic Pattern
- Exploring Tensorflow Serving
- Using Ray Serve
- Using BentoML
- Serving ML Models using a Fully Managed Cloud Solution


1


Introducing Model Serving


While machine learning (ML) surprises us every day with new, stunning ideas and demos, a burning question remains: Often, we see demos of models on different blogs, books, YouTube videos, and so on, and we remain hungry to use the models ourselves. This is where model serving comes into the picture. Model serving is how we make our models available for use.

In this chapter, we will learn the definition of model serving, the importance of model serving, the challenges that make model serving difficult, and how people currently serve models, and see some of the available tools used for model serving.

By the end of this chapter, we will understand what model serving is, why model serving is needed, what makes it different from traditional web serving, and how people currently deploy/serve models.

In this chapter, we are going to cover the following main topics:

  • What is serving?
  • What are models?
  • What is model serving?
  • Understanding the importance of model serving
  • Challenges of serving models
  • Using existing tools to serve models

Technical requirements


This chapter does not require you to follow along with any hands-on exercises. However, there are some examples used from the BentoML official site: https://docs.bentoml.org/en/latest/tutorial.html.

If you want to try those examples on your local machine, please feel free to install a local version of BentoML following the simple installation instructions here: https://docs.bentoml.org/en/latest/installation.html.

Basically, you need to install BentoML like other Python packages using the following command:

pip install bentoml

Feel free to follow the quick get-started link to understand the steps involved in model serving, which are highlighted in a later section in this chapter.

What is serving?


Serving is an important step for ensuring the business impact of the applications when we develop the life cycle of application development. The application we have developed needs to be available to the user so that they can use it. For example, let’s say we have developed a game. After the development, if the game just stays on the developer’s machine, then it is not going to be of any use to the users. So, the developer needs to bring the game to the users by serving it through a serving platform such as Apple Store, Google Play Store, and web servers.

So, serving can be seen as a mechanism to distribute our applications/services to end users. The end users can be different based on the applications/services we develop. Serving creates a bridge of communication between the two parties: the developer and the users. This bridge is vital for the business success of our application. If we don’t have people using our service, then we are not gaining any business value or impact from the applications we’ve developed. That’s what we have seen in the past when big corporate companies’ servers go down for some time: they incur a huge amount of loss. Facebook (Meta) lost ~65 million US dollars due to its outage for some hours in October 2021, as per Forbes: https://www.forbes.com/sites/abrambrown/2021/10/05/facebook-outage-lost-revenue/?sh=c1d7d03231ad.

The development-to-serving process usually forms the life cycle of the service or application.

For example, let’s consider the life cycle of web application development in .

Figure 1.1 – Web development life cycle

The developer develops the website. Then, the website is served using a web server. Only after that are the users able to use the website. The cycle can continue through collecting feedback from users, improving the website, and serving again on the web server.

Now we know what serving is, let’s look at what we are actually serving.

What are models?


There are a lot of definitions of models from the perspective of various domains. When we define a model or use the term model in this book, we will consider it in the context of ML. A model in ML can be seen as a function that has been fine-tuned through training, using some well-engineered data so that the function can recognize and distinguish patterns in unseen data.

Depending on the problem and business goal, a different model can be used, for example, a linear regression model, a logistic regression model, a naive Bayes model, or a decision tree model. These models’ underlying logical representations are different from each other. However, we use the generic term and the problem domain and name of the ML algorithm to give us a clear picture of what the model is, how it was trained, and how the model is represented. For example, if we are told that the model is for linear regression, then we know that it was trained by minimizing a cost function iteratively using the training data, and it can be saved by storing the regression parameters’ coefficients and intercepts. Similarly, other models will have different algorithms for training and storing. For a deep learning model, we might have to use forward propagation and backward propagation for training, and for storing we might have to store the weights and biases of all the layers.

The trained model can be stored in different formats to load later for serving and inference. Some popular formats in which to save a model are as follows:

  • ONNX
  • YAML
  • Protobuf
  • Pickle
  • JSON
  • H5
  • TFJS
  • Joblib

However, model-serving tools usually require the models to be saved in a particular format. So, they provide a function to save the model in its desired format. There are also tools and libraries to convert models from one format to another. For example, in , we see that an AlexNet model that is pre-trained in PyTorch is loaded and exported to ONNX format in a file named alexnet.onnx.

PyTorch files

It’s worth knowing that PyTorch saves the model using the Python pickle (https://docs.python.org/3/library/pickle.html) library. For further reading on PyTorch strategies for saving and loading models, please check out their official documentation: https://pytorch.org/tutorials/beginner/saving_loading_models.html.

Figure 1.2 – Example code converting a PyTorch pre-trained AlexNet model to ONNX format

Note

is an example from the PyTorch official website: https://pytorch.org/docs/stable/onnx.html#example-alexnet-from-pytorch-to-onnx.

Now we should have a good idea about models and how each model is represented and stored. The following section will introduce us to model serving.

What is model serving?


Like serving a website, we need to serve the trained model so that the model can be used for making predictions to perform business goals. Web/software serving is already at a mature stage. So, we have sophisticated, agreed-upon tools and strategies to serve software. However, ML model serving is still in the phase of growth, and new ideas and tools are coming almost every day.

Model serving can be defined as bringing a model to production by deploying it to a location and providing some access points for users to pass data for prediction and get prediction results.

Model serving usually involves the following steps:

  1. Saving the trained model: The format in which the model needs to be saved can be different based on the serving tool. So, usually, the serving tools provide a function for saving the model to ensure the model is saved in a format needed by the library.

Let’s use BentoML as an example. We’ll cover BentoML in more detail in , but in the following code snippet taken from the BentoML official site, https://docs.bentoml.org/en/latest/tutorial.html, we see that the popular model-serving library BentoML provides a save function for each of the ML frameworks. During serving using BentoML, we have to call the save method on the appropriate framework. For example, if we have developed a model using sklearn, we need to call the bentoml.sklearn.save_model( method to save the model in BentoML format from sklearn format:

import bentoml from sklearn import svm from sklearn import datasets # Load training data set iris =...


Islam Md Johirul :

Md Johirul Islam is a Data Scientist and Machine Learning Researcher at AWS. He has a PhD in Computer Science and is also an adjunct professor at Purdue University. His expertise are focused on designing explainable, maintainable and robust data science pipeline applying the software design principles and helping organizations deploy Machine Learning models into production at Scale.



Ihre Fragen, Wünsche oder Anmerkungen
Vorname*
Nachname*
Ihre E-Mail-Adresse*
Kundennr.
Ihre Nachricht*
Lediglich mit * gekennzeichnete Felder sind Pflichtfelder.
Wenn Sie die im Kontaktformular eingegebenen Daten durch Klick auf den nachfolgenden Button übersenden, erklären Sie sich damit einverstanden, dass wir Ihr Angaben für die Beantwortung Ihrer Anfrage verwenden. Selbstverständlich werden Ihre Daten vertraulich behandelt und nicht an Dritte weitergegeben. Sie können der Verwendung Ihrer Daten jederzeit widersprechen. Das Datenhandling bei Sack Fachmedien erklären wir Ihnen in unserer Datenschutzerklärung.