Welcome to the 31st issue of the MLOps newsletter (hint: it’s a special one!). 

Nihit and I have been busy building a company – Refuel.ai! We’re building a platform to clean, label and enrich datasets at human-level quality, using Large Language Models (LLMs). Great data leads to great models, but clean, labeled data is a huge bottleneck for machine learning teams. 

We just launched out of stealth (media coverage) and are excited to share more details with you. In this issue, we’ll cover a benchmark for LLM-powered data labeling and Autolabel, a library to label, clean, and enrich data with your favorite LLM. Feel free to join us on Discord if you want to discuss LLMs or data labeling (or anything else) with us! 😊

Thank you for subscribing. If you find this newsletter interesting, tell a few friends and support this project ❤️

LLMs can label data as well as humans, but faster

Motivation

LLMs are an incredibly powerful piece of technology. While it is well known that LLMs can write poems and solve the BAR and SAT exams, there is increasing evidence suggesting that when leveraged with supervision from domain experts, LLMs can label datasets with comparable quality to skilled human annotators

What we did

We evaluated the performance of LLMs vs human annotators for labeling text datasets across a range of tasks (classification, question answering, etc) on three axes: quality, turnaround time, and cost. We tested techniques for boosting LLM accuracy such as few-shot prompting, chain-of-thought reasoning, and confidence estimation, and propose a live benchmark for text data annotation tasks, to which the community can contribute over time.

Results

• State-of-the-art LLMs can label text datasets at the same or better quality compared to skilled human annotators, but ~20x faster and ~7x cheaper.

• For achieving the highest quality labels, GPT-4 is the best choice among out-of-the-box LLMs (88.4% agreement with ground truth, compared to 86% for skilled human annotators). For achieving the best tradeoff between label quality and cost, GPT-3.5-turbo, PaLM-2 and open-source models like FLAN-T5-XXL are compelling.

• Confidence-based thresholding can be a very effective way to mitigate the impact of hallucinations and ensure high label quality.

Conclusion

We think that LLM-powered labeling is the future for a vast majority of data cleaning, annotation, and enrichment efforts, and will continue to track the performance of many LLMs across a diversity of tasks in this ongoing benchmark. 

You can read this in tweet thread form here:  https://twitter.com/nihit_desai/status/1669752203949793281?s=20

Autolabel: Python library to label, clean, and enrich datasets with LLMs

We open-source Autolabel, a Python library to label, clean, and enrich text datasets with any Large Language Model (LLM) of your choice. With a few lines of code, you’ll be labeling data at extremely high accuracy, but in a tiny fraction of the time compared to human annotation! Join us on Discord if you have any questions, or open an issue on Github to report a bug. If this is interesting, feel free to give us a star! ⭐

Thanks

Thanks for making it to the end of the newsletter! This has been curated by Nihit Desai and Rishabh Bhargava. If you have suggestions for what we should be covering in this newsletter, tweet us @mlopsroundup or email us at mlmonitoringnews@gmail.com. 

If you like what we are doing, please tell your friends and colleagues to spread the word. ​​❤️

Welcome to the latest issue of the MLOps newsletter. In this special (mini) edition of the newsletter we have some exciting news to share!

MLOps: From Models to Production, a new course that @Nihit has been developing in partnership with the team at co:rise, is almost ready to go live! This will be a 4 week course, with the first iteration starting July 11th (07/11/2022). More details below 👇

New Course | MLOps: From Models to Production

Summary

Over the last decade, Machine Learning has become ubiquitous, powering platforms and applications across a variety of domains. And yet, we are likely only at the beginning of the ML adoption journey. In order to realize the promise of Machine Learning, we believe it is important to combine the knowledge from Machine Learning research with skills and best practices around building effective real-world ML systems. With that in mind, we’d like to share a new course @Nihit has been developing in partnership with the team at co:rise.

Who is this course for?

• Software engineers who want to build production systems that integrate ML

• Data scientists who want to learn about the production ML lifecycle 

• Students/recent college grads who want hands-on experience building and shipping ML applications

This course is meant to complement a foundation course in Machine Learning/Deep Learning and assumes familiarity with basic machine learning concepts. 

Logistics

• This will be a 4-week course, with the first iteration starting July 11th (07/11/2022). We expect to run the course a few times each year. 

• To enroll or learn more about the course, you can visit the course webpage. You can use the code NIHIT10MLOPS for a discount. 

• Also, check out other courses in the ML track that you might find interesting.

Scholarships

We are also offering a limited number of scholarships! If you're interested to apply, or know someone who is, here is the application link

Thanks

Thanks for making it to the end of the newsletter! This has been curated by Nihit Desai and Rishabh Bhargava. If you have suggestions for what we should be covering in this newsletter, tweet us @mlopsroundup or email us at mlmonitoringnews@gmail.com. 

If you like what we are doing, please tell your friends and colleagues to spread the word. ​​❤️

In this issue, we cover a collection of learnings from ML platforms at Netflix, Doordash, and Spotify; share a fantastic new opportunity for data scientists with the OnDeck community; discuss challenges in building explainable machine learning models for healthcare; link to a Twitter thread on building hybrid human/machine learning systems for content moderation and more.

Thank you for subscribing. If you find this newsletter interesting, tell a few friends and support this project ❤️

Medium | Lessons on ML Platforms — from Netflix, DoorDash, Spotify, and more

This is a wonderful article by Ernest Chan where he analyzes the broad components of ML Platforms at large companies such as Netflix, Doordash, and Spotify.

Why are we talking about ML Platforms anyway? 

Your data scientists produce wonderful models, but they can only deliver value once the models are integrated into your production systems. How do you make it easy for the data scientists to repeatedly deliver value? What do you build, what do you buy, and what tools do you need to solve your organization’s problems specifically?

The goal of an ML platform is to accelerate the output of data science teams. Let’s look at the common components of an ML platform (at least from a large tech company perspective). 

Components of an ML Platform

• Feature Store

• Workflow Orchestration

• Model Registry

• Model Serving

• Model Quality Monitoring

It turns out that most of these components have been built in-house so far. This is for many different reasons: these companies had to solve their problems before much ML tooling existed (and we are still very much in the early innings of MLOps), organizational reasons (how engineers at large companies are rewarded), etc. 

What does this mean? 

Well, first of all, most companies are not one of these large companies, so your situation will likely be different. 

The first thing that stands out is workflow orchestration component is the one where an out-of-the-box (and open-source) solution is most commonly used. This makes sense since workflow orchestration tools are part of a generic software/data toolkit, and are more mature. Airflow seems to be dominant here. 

The Model Registry is the next component that has some established tools, such as MLflow. However, Feature Stores, Model Serving, and Model quality monitoring seem to have been built in-house in pretty much every case. Aside from most tools being new, there are a few reasons for this: all these components have “stringent production requirements”, need to support diverse use cases within an organization, and can require tight integrations with the rest of the tech stack in the company. 

If this is interesting to you, check out the full article and the beautiful set of references compiled at the end of it. 

Community | On Deck Data Science

We wanted to share an exciting opportunity with all of you. Both of us have been members of the On Deck community for the past few months, and cannot recommend it highly enough. We have met some incredible people, and learned a lot from the community.

While we are part of the On Deck Deep Tech cohort, they have a new cohort called On Deck Data Science, and here is a blurb from their team. If you have any questions about our experience, drop us a note. 

Enter On Deck Data Science (ODDS):

ODDS is a continuous community for ambitious Data Science leaders who want to maximize their impact and accelerate their careers alongside a highly-curated network of peers.

The fellowship brings together experienced data science leaders who have delivered results for organizations and customers at the highest level.

Members get access to:

• Community: Develop meaningful relationships with peers and mentors. Curated 1:1 connections, mastermind sessions and mentorship matchmaking - the hard work is done for you.

• Networking: ODDS is a side door to 10x your network of peers. You can find exciting job opportunities, investors to fund your next idea or your next star hire. You get access to the entire On Deck network (like an internal LinkedIn) instantly when you join.

• Professional Development: Acquire specialized frameworks, knowledge and skills to add to your toolkit via live sessions and fireside chats with incredible guests and an extensive library of content.

Applications close on Feb 13th, so don’t miss your chance to join some of the most accomplished data scientists in the business. Apply now!

Paper | The false hope of current approaches to explainable artificial intelligence in health care

We have covered explainability for ML multiple times (see here and here), but this was a thought-provoking paper that talks about some of the pitfalls of these techniques (especially in healthcare). 

What are the authors saying?

It has been argued that explainable AI will engender trust with the health-care workforce, provide transparency into the AI decision making process, and potentially mitigate various kinds of bias… we advocate for rigorous internal and external validation of AI models as a more direct means of achieving the goals often associated with explainability, and we caution against having explainability be a requirement for clinically deployed models.

Current explainability approaches and their gaps

Explanations for decisions can fall into two categories – through models that are inherently explainable (simple models such as linear regression) or through post-hoc explainability techniques (saliency maps, LIME, SHAP, etc). 

While inherently explainable models might seem appealing, there can be confounding variables in the mix, and when the number of variables grows, “information overload” can make explanations tricky. 

Heat maps (or saliency maps) highlight how much of a region contributed to a decision in imaging use cases. However, as seen in a pneumonia diagnosis example, the authors claim that:

“the hottest parts of the map contain both useful and non-useful information (from the perspective of a human expert), and simply localising the region does not reveal exactly what it was in that area that the model considered useful.”

Why does this happen? 

This interpretability gap of explainability methods relies on humans to decide what a given explanation might mean. Unfortunately, the human tendency is to ascribe a positive interpretation: we assume that the feature we would find important is the one that was used (this is an example of a famously harmful cognitive error called confirmation bias).

Another issue is that explanations have no performance guarantees. Most tests “rely on heuristic measures” and qualitative measures rather than explicit scores. Since explanations are often a simplification of the original model, they are very likely a less accurate version of the trained (and hard to explain) model, which makes this process harder. 

Final Thoughts

Rather than seeing explainability techniques as producing valid, local explanations to justify the use of model predictions, it is more realistic to view these methods as global descriptions of how a model functions. If, for example, a clinical diagnostic model appears to perform well in a specific test set but the heat maps show that the model is consistently distracted by regions of the images that cannot logically inform the diagnosis, then this finding can indicate that the test set itself is flawed and that further forensic investigation is required.

Here is the full paper if this was an interesting read!

AWS News Blog | Top Announcements of AWS re:Invent 2021 

AWS re:Invent 2021 had a ton of interesting announcements from AWS, but they were light on AI/ML improvements this year around (compare this with our re:Invent coverage from last year). 

MLOps Highlights:

• Amazon Sagemaker Studio Lab: This is a free service that gives people access to a working Jupyter instance for any experimentation - appears very similar to Google Colab. 

• Amazon Sagemaker Inference Recommender: This service recommends what instance types you should be running your inference workloads on. 

• Amazon SageMaker Ground Truth Plus: This service (currently in pilot) appears to be a higher quality labeling service compared to Mechanical Turk, and seems to be a competitor to the Scale AI’s of the world. 

• Amazon Sagemaker Training Compiler: This service optimizes deep learning training code to run faster on Sagemaker GPU instances (if you do use this, let us know what your experience is like)

There are a few more, but we’ll leave it up to you to read about them here. We do worry about the length of Sagemaker service names at this point - we wonder if we will be covering Amazon Sagemaker Deep Training Speed-Booster Plus next year…

Washington Post | New technology mandate in infrastructure bill could significantly cut drunken driving deaths

We try to track interesting news from the policy world where ML may have a part to play, and this certainly caught our eye. 

What is this and why is it important? 

The recent infrastructure bill from the US Congress has a mandate that would require new cars to have technology that would stop drunk people from driving. As the article reports:

More than 10,000 people died in crashes involving an alcohol-impaired driver in 2019, according to the National Highway Traffic Safety Administration…A recent study by the Insurance Institute for Highway Safety concluded the technology could reduce deaths by 9,400 people a year if widely deployed.

How would it work? 

The technology involved in preventing drunk driving isn’t finalized yet (and probably won’t be for a few years), but one of the ideas being floated is to:

“rely on cameras that monitor drivers for signs they are impaired, building on systems that automakers are using to ensure people relying on driver assistance technologies don’t lose concentration.”

Whether it plays out this way or not, this definitely seems like an ML application that has the potential to save a lot of lives. 

Twitter | Lessons from ML Systems for Content Moderation

 Nihit: I worked on systems for content moderation at Facebook. In a recent thread, I shared some learnings & observations around what makes this a challenging problem. 

Detecting & taking enforcement action on illegal/undesirable content is an important problem for most online platforms as they scale. This is done typically with human-in-the-loop Machine Learning systems. The content moderation domain presents some unique challenges when building machine learning models - bootstrapping labels, subjective annotation guidelines that can lead to label noise, adversarial drift, and the need for adaptive enforcement. If some of these sound interesting or relevant, definitely check out the thread! 

Thanks

Thanks for making it to the end of the newsletter! This has been curated by Nihit Desai and Rishabh Bhargava. If you have suggestions for what we should be covering in this newsletter, tweet us @mlopsroundup or email us at mlmonitoringnews@gmail.com. 

If you like what we are doing, please tell your friends and colleagues to spread the word. ​​❤️

Welcome to the 29th issue of the MLOps newsletter. This one is going to be fairly image-rich - hope you enjoy it! 🖼️

In this issue, we briefly cover the latest State of AI Report, look into the key trends from Kaggle’s Data Science and ML survey, explore Reddit’s model deployment architecture, deep dive into a paper on augmenting human annotations in microscopy, and much more.

Thank you for subscribing. If you find this newsletter interesting, tell a few friends and support this project ❤️

State of AI Report 2021

We covered the State of AI Report 2020 last year here and the authors are back with their latest analysis of the most interesting developments in AI. It’s a whopping 188 slides and ~5 hours of reading and digesting, but probably worth the effort. While it’s broken up into sections on research, talent, industry, politics, and predictions, we will only focus on the slides on AI in industry. 

It’s nice to see that the lessons from ML in production have made it back into the world of research -- recently, one of the topics we have covered a lot is the move towards data-centric AI. 

Similarly, changes in distribution between training and prediction-time is a common topic in industry, and it was interesting to see the WILDS benchmark featured here. 

A bunch of other topics that we have covered before made an appearance-- dynamic benchmarking (and Dynabench in particular) to quickly improve models, underspecification when the same model with different random seeds behave differently, and pervasive “bad data” issues (highlighted in the research around ML for Covid). 

We loved the stories of applied ML; from Ocado’s deep learning being used for 98% of stock replenishment decisions for online grocers to Viz.ai’s stroke detection software helping 1 patient every 47 seconds in the US, to reinforcement learning used at the Greek border (another story we covered here) for Covid testing, to computer vision aiding in disaster relief, transformer models being used to accurately forecast electricity demand and so much more!

There is a lot in here that we can’t fully do justice to -- check out the slides or the Twitter thread from one of the authors:

Kaggle | 2021 Data Science and Machine Learning Survey

Kaggle recently shared results from their annual survey of data scientists and machine learning engineers. The survey focuses on trends in the data science landscape in industry. It is quite insightful and we recommend reading it in its entirety. A few key takeaways for us:

• Continuing democratization of Data Science: Advanced degrees (Masters or higher) are still the norm (>60% of respondents) but going into data science after a bachelor's degree is becoming increasingly more common (~35% of respondents vs only 20% in 2018).  Only 12.2% of survey respondents reside in the US and 24.4% in India, followed by a long tail.

• Online learning continues to be popular: Given the rapidly progressing landscape, a large number of data scientists use one or more online learning resources to stay up to date. Coursera (57.8%) and Kaggle (39%) remain the most popular online learning destinations.

• Algorithms and Model architectures: Strong year-over-year growth in the use of large complex models such as transformers, but linear models (linear regression, logistic regression) and decision trees are still among the most widely used and deployed ML models.

• Machine Learning Frameworks: It’s all Python. Scikit-learn is still the most widely used framework for building and deploying ML models (>80% of respondents). Pytorch is continuing to grow in popularity (>33% of respondents). 

• ML experimentation tools: Repeatable and well-documented experimentation tracking has received a lot of attention in the ML community. But according to the survey, >57% of respondents do not use any ML experimentation tools. This shows that we are in the early innings of ML tooling ecosystem maturity.

• AutoML usage growing steadily: AutoML is becoming an increasingly common part of the ML development lifecycle. Google Cloud AutoML is the most widely used AutoML framework at 23.4% of respondents.

Reddit Engineering Blog | Evolving Reddit’s ML Model Deployment and Serving Architecture

Problem

Machine Learning powers a lot of the product experiences on Reddit - from the personalization of content feed for users, to optimization of push notifications, content moderation, and more. ML systems at Reddit serve thousands of inference requests every second. The engineering team at Reddit undertook a redesign of the production ML system to keep up with usage growth and help with scalability, and this article shares the goals and details of this effort.

Legacy ML Stack

Reddit’s legacy ML stack was based on baseplate.py (Reddit’s python web services framework): Machine learning models were deployed as model classes inside of their application service called Minsky. When launching a new model, a machine learning engineer would typically end up touching the application code to download the new model upon application start, load it into memory and implement the feature transformation and inference logic for serving inference traffic. 

As you might have noticed, this process has several limitations that are highlighted in the article. One that especially stands out is that there is a tight coupling across models: if a single model encounters an exception the entire service instance will crash. Moreover, this coupling between the model inference and application layer means limited ability to monitor online model characteristics.

Gazette Inference Service

The team at Reddit solved some of the limitations of previous architecture with the Gazette Inference Service. This is a dedicated service for online model inference with a single endpoint: Predict. The request simply needs to specify the model, version, and how to fetch features that go into the model. The service is dockerized and deployed with Kubernetes. While it currently supports Tensorflow models, they are working on expanding support to other frameworks. 

The article highlights how Gazette overcomes the limitations of the previous setup, especially scalability and isolation between different models. In general, this progression of locally coupled models in the application layer → dedicated inference service sounds quite familiar. We look forward to future learnings and improvements shared by the team as they build out Gazette.

Paper | Biological data annotation via a human-augmenting AI-based labeling system

This is an interesting paper that showcases some of the challenges in acquiring high-quality annotations in domains where large datasets can be generated easily, but the right tools for annotating don’t currently exist. 

Background

Supervised learning—in which computational models are trained using data points (e.g., histopathology image; raw microscopy image) and data annotations (e.g., “cancerous” vs “benign”; stained microscopy image)—have been central to the success of CV in biology. Biologists have the distinct advantage of being able to generate massive amounts of data—a single microscopy image can yield a gigabyte of visual data for algorithms to learn from. A disadvantage, however, is the difficulty and cost of obtaining complete annotations for datasets.

While practitioners use computational techniques for augmenting data (rotations, distortions, changes in color), these are often a poor substitute for human-quality annotations. This led to the authors presenting a human-augmenting AI-based labeling system, or HALS. 

What is it?

HAL provides a data annotation interface with three deep learning models (segmentation model, classifier model, and an active learner) that work in tandem to:

1. learn the labels provided by an annotator

2. provide recommendations to that annotator designed to increase their speed, and

3. determine the next best data to label to increase the overall quality of annotations while minimizing total labeling burden.

Every microscopy image (these are very large images) is first pre-processed by running it through the segmentation model, and then as human annotators start providing labels for small regions, the classifier starts to learn a model for the regions being labeled. After a small number of labels have been acquired for each class, the classifier starts to suggest labels for regions that annotators haven’t seen yet. The annotator can choose to accept or reject such proposed labels. Simultaneously, the active learner breaks the entire image into square patches, converts them into feature vectors, and recommended new regions for the annotator to look at. 

If you’re interested in further details on the specific datasets, model pretraining, active learning algorithm used, etc, we recommend going through the paper. 

Results

Using four highly repetitive binary use-cases across two stain types, and working with expert pathologist annotators, we demonstrate a 90.6% average labeling workload reduction and a 4.34% average improvement in labeling effectiveness.

“Workload reduction” has an interesting definition -- the fraction of AI-generated labels that were accepted immediately by the annotators, and it would be interesting to explore the actual time saved (something the authors do mention in the paper as well). 

“Labeling effectiveness” was measured by the AUC score on a validation set, and it was nice to see a modest bump in the true performance of the final ML model as well.

New Resources for Machine Learning

MAD Podcast 

We recommend checking out the MAD Podcast (Machine Learning, AI, and Data), co-hosted by Michael Harper and Honor Chan, featuring industry experts and with a focus on data quality.

Embeddinghub by Featureform 

Embeddings (dense vector representations) are a fundamental building block for Machine Learning. Embeddinghub is an open-source database for storing and searching embeddings built by FeatureForm. EmbeddingHub uses RocksDB to durably store embeddings and metadata, and HNSWLib to build approximate nearest neighbor indices. Additionally, it supports versioning and rollbacks for embeddings which can be very useful for production applications. You can learn more about the project on Github.

Twitter | Inferentia chips

Sudeep Pillai has a great thread testing the AWS Inferentia chips, custom-designed for ML inference. He finds that many ResNet and Transformer style models have pretty good performance, with some models returning sub-10-ms inference latencies. We’ll look forward to reading a more detailed post on this in the future!

Thanks

Thanks for making it to the end of the newsletter! This has been curated by Nihit Desai and Rishabh Bhargava. If you have suggestions for what we should be covering in this newsletter, tweet us @mlopsroundup or email us at mlmonitoringnews@gmail.com. 

If you like what we are doing, please tell your friends and colleagues to spread the word. ​​❤️

Welcome to the 28th issue of the MLOps newsletter. We really enjoyed writing this one, hope you enjoy it too! 

In this issue, we briefly cover Nihit’s interview with Eugene Yan, discuss Matt Turck’s ML, AI, and Data landscape, share a cool ML-based invite spam detection from Slack, and dive into a fascinating reinforcement learning system for COVID testing.

Thank you for subscribing. If you find this newsletter interesting, tell a few friends and support this project ❤️

Eugene Yan | Applying ML

Nihit: I recently did an interview with Eugene Yan as part of his ApplyingML initiative (which is a fantastic collection of resources - papers, tools, playbooks, mentor interviews - for building machine learning systems). In the interview, I share my thoughts on the machine learning infra and tooling landscape, and my experience building ML systems at Facebook. You can check out the transcript here (or read about interviews with other mentors)

As Eugene put it, his main motivation is to collect “ghost knowledge” that resides within the community but is seldom officially documented:

Knowledge that is present somewhere in the epistemic community, and is perhaps readily accessible to some central member of that community, but it is not really written down anywhere and it's not clear how to access it.

Matt Turck | Red Hot: The 2021 Machine Learning, AI and Data (MAD) Landscape

The Data Landscape (now the MAD Landscape) post from Matt Turck is a great read every year. We covered the 2020 post last year, and it was fun to look back on it and compare it to 2021. 

How has this last year been?

As the authors write:

It’s been a hot, hot year in the world of data, machine learning and AI. 

This is easily evidenced by the denser landscape chart (and our coverage of the ecosystem over the past year) but also observed by the recent, successful IPOs in the space (Snowflake, Confluent, UIPath, C3.ai, Sentinel One) and the increased funding for AI startups ($38B in the first half of 2021 compared to $36B in all of 2020). 

We have seen tremendous growth in the number of startups and tools, leading to a dizzying set of options for companies and practitioners (which is mostly good news). As the authors put it:

"...we believe that companies will continue to work with multiple vendors, platforms and tools, in whichever combination best suits their needs.  

The key reason: the pace of innovation is just too explosive in the space for things to remain static for too long.  Founders launch new startups, Big Tech companies create internal data/AI tools and then open source them, and for every established technology or product, a new one seems to emerge weekly. 

For newer categories such as MLOps, it has also led to a crowded market:

as VCs aggressively invested in emerging sectors up and down the data stack, often betting on future growth over existing commercial traction, some categories went from nascent to crowded very rapidly – reverse ETL, data quality, data catalogs, data annotation and MLOps.  

2021 for MLOps

It’s rare that we get to quote ourselves, but here was one of our predictions from last year:

we believe there will be many more categories (which will be much more crowded) in the coming years.

That’s certainly proven to be true. This year, we saw new categories for:

• Feature Stores

• Model Building

• Deployment and Monitoring

• Synthetic Media

• Data Quality & Observability

The older categories for Data Generation & Labelling, Data Science Notebooks, Data Science Platforms, ML Platforms, Speech, Computer Vision, and NLP continued to grow in terms of companies. We aren’t going to reference any specific companies here -- there are simply far too many for us to do them justice. 

Final Thoughts

There are many interesting and important problems to be solved in the world of data and ML, and the density of talent trying to address these challenges is the most exciting aspect to us! 

We definitely recommend giving the entire article a read, especially to get a sense of the broader data ecosystem.

Paper | Efficient and targeted COVID-19 border testing via reinforcement learning

This is a fascinating paper that describes a Reinforcement Learning system deployed at the border of Greece. 

What’s the Problem? 

Throughout the COVID-19 pandemic, countries relied on a variety of ad-hoc border control protocols to allow for non-essential travel while safeguarding public health: from quarantining all travelers to restricting entry from select nations based on population-level epidemiological metrics such as cases, deaths or testing positivity rates.

This has been a difficult problem for many countries -- whether to allow travelers in, which passengers to allow in, and what testing requirements to place on them. The circumstances change rapidly; outbreaks can occur in countries much faster than rules and policies can be created. The team built a reinforcement learning system that was deployed across all 40 ports of entry to Greece - from airports to land crossings to seaports. 

Without going into details, the system consisted of the following components:

1. Asking incoming travelers to fill out a form per household containing information about their travel and demographics. 

2. Estimating prevalence of COVID using recent test results across a discrete set of interpretable traveler types (based on country, region, age, and gender)

3. Allocating a scarce number of tests among the travelers while balancing two objectives: maximize the number of infected asymptomatic travelers identified (exploitation) and better learn their prevalence of COIVD for traveler types it does not currently have precise estimates (exploration)

4. Conduct tests, and log data as quickly as possible to get better estimates for Step 2

What were the Results? 

The results are amazing. The authors compare the system’s performance against counter-factual modeling and found that it identified 1.85 times as many asymptomatic, infected travelers as random testing (with up to 2-4 times as many during peak travel), and 1.25-1.45 times as many asymptomatic, infected travelers as testing policies that only utilized epidemiological metrics. 

Lessons from Design and Deployment

• Data minimization: Designing the algorithm with the philosophy of requiring minimal data about travelers, in order to comply with GDPR (even when there is a “tradeoff between privacy and effectiveness”)

• Prioritize interpretability: For example, they used empirical Bayes to communicate that large confidence intervals suggest higher risk. Similarly, their usage of gittins indices to provide a simple metric of risk for a traveler type made intuitive decisions easier. 

• Design for flexibility: The system required substantial financial and technical investment, and it needed to be flexible to accommodate unexpected changes. For example, they were able to quickly define new traveler types when vaccine distribution started without altering any other components. 

Our Thoughts

Problems such as this are fraught with complications of bias, data privacy, and other real-world concerns. We have covered extensively when ML goes wrong (see our coverage of ML + Covid here and here), but this is a fantastic example of how we can build better systems when ML is deployed appropriately.

Some more reactions to this work:

Slack Engineering Blog | Blocking Slack Invite Spam With Machine Learning

In this article, Aaron Maurer shares some learnings and insights from building invite spam detection models at Slack. We especially liked the journey (no spam detection → rules based → ML model-based) shared in this article and is something that many teams and companies will find relatable as they go through the ML adoption maturity journey. 

Problem

One of the first things you do after creating a new Slack team is to invite people from your community/research group/company etc to join. This is done by entering emails of people you’d like to invite, and then Slack will send out invites on your behalf. However, in some cases, spammers abuse this system to send out spam invite emails.

Rule-based spam detection

As with many real-world ML applications, Slack’s early versions of detecting and enforcing invite-based spam were rules-based, based on attributes like IP addresses, the occurrence of certain phrases or words in the invite matches, etc. As can be expected, this worked reasonably well in preventing spam invites but was labor-intensive (to have to create new rules on an ongoing basis) and the rules created many false positives.

Model-based spam detection

• Feature set: The team used the existing feature set that was used in the hand-tuned rules. This is quite intuitive for most ML applications that are trying to replace hand-tuned rules: oftentimes, the rules (even though they are noisy) can serve as a good set of features for your first model

• Labels: As highlighted in the post, getting ground truth labels (“is the invite a spam or not”) is not available for all invite emails. Even though it would be possible to get these labels via human review, there is a time and money cost to it. The team instead chose to train the model on a proxy observation: whether an invite would be accepted or not within 4 days (90% of invites are accepted in that timeframe)

• Model: A sparse logistic regression model, that outputs a score proportional to the probability that the invite is spammy. 

• Online serving: Slack has an in-house service for serving model predictions. To serve predictions from a machine learning model online, an ML engineer would implement a lightweight Python class. The service deploys it as a microservice through Kubernetes, which can be queried from the rest of our tech stack

Impact

As outlined in the post, the switch to model-based detection resulted in a big decrease in false positives, while maintaining high recall: 

The machine learning model was much better at preventing false positives. Only 3% of the invites it flagged ended up being accepted when allowed through, while around 70% of the invites flagged by the old model actually ended up being accepted

Towards Data Science | The 6-Minute Guide to Scikit-learn’s Version 1.0 Changes 😎

Scikit-learn, the widely used Python machine learning library, recently released version 1.0. This article by Jeff Hale highlights the changes, including bug fixes, new features, and API cleanups. We summarize the key takeaways below:

1. OneHotEncoder: This feature preprocessing step now supports handle_unknown='ignore' to accept values it hasn’t seen before (especially helpful for using in production where it can encounter values it does not have encoding for), and dropping categories within a feature that you might want to ignore.

2. Pandas: When passing a dataframe as input to a transformer, it stores the feature columns in feature_names_in_. Additionally,  get_feature_names_out has been added to the Transformer API to return names of output features. 

3. SGDOneClassSVM: This new class is an SGD implementation of one-class SVM (commonly used for outlier or out-of-distribution detections). 

4. Normalization in Linear models: The “normalize” option in linear_model.LinearRegression is deprecated and will be removed in version 1.2. This behavior can now be reproduced with a sklearn pipeline that stitches together the StandardScaler and the linear model.

5. Sklearn metrics: Several metrics such as ConfusionMatrixDisplay and PrecisionRecallDisplay now expose two class methods from_estimator and from_predictions allowing to create a confusion matrix plot using an estimator or the predictions. Equivalent methods such as metrics.plot_confusion_matrix  and  metrics.plot_precision_recall_curve are deprecated and will be removed in version 1.2

Twitter | Lessons for Recommender Systems

In this Twitter thread, Karl Higley shares some great insights about building large-scale recommendation systems. Worth clicking and reading through it!

Thanks

Thanks for making it to the end of the newsletter! This has been curated by Nihit Desai and Rishabh Bhargava. If you have suggestions for what we should be covering in this newsletter, tweet us @mlopsroundup or email us at mlmonitoringnews@gmail.com.

If you like what we are doing, please tell your friends and colleagues to spread the word. ❤️

Welcome to the 27th issue of the MLOps newsletter. It is officially one year since we started writing this newsletter, and we are incredibly grateful for your support. We are excited for the next year to come! 🎉

In this issue, we cover a paper on the machine learning challenges in medical imaging, discuss interesting research regarding machine unlearning, share Doordash’s strategies for matching supply and demand, and much more.

Thank you for subscribing. If you find this newsletter interesting, tell a few friends and support this project ❤️

Paper | How I failed machine learning in medical imaging - shortcomings, and recommendations

This is a fascinating paper that highlights some of the challenges when applying machine learning in the world of medical imaging. 

As the authors put it:

there is a staggering amount of research on machine learning for medical images, as many recent surveys show. This growth does not inherently lead to clinical progress. The higher volume of research can be aligned with the academic incentives rather than the needs of clinicians and patients. As an example, there can be an oversupply of papers showing state-of-the-art performance on benchmark data, but no practical improvement for the clinical problem.

It’s Not All About Larger Datasets

Large labeled datasets have solved many problems in computer vision. However, very few clinical questions can be easily posed as discrimination tasks, and even when they do, larger datasets can often fail to “solve” such questions. 

An example is that of early diagnosis of Alzheimer’s Disease. Even with a lot of effort and collection of lots of data:

the increase in data size did not come with better diagnostic accuracy, in particular for the most clinically-relevant question, distinguishing pathological versus stable evolution for patients with symptoms of prodromal Alzheimer’s. Rather, studies with larger sample sizes tend to report worse prediction accuracy.

Datasets Reflect An Application Only Partly

Available datasets only partially reflect the clinical situation for a particular medical condition, leading to dataset bias... Dataset bias occurs when the data used to build the decision model (the training data), has a different distribution than the data representing the population on which it should be applied (the test data). 

There are many sources of dataset bias; from a cohort not representing the range of possible patients, to imaging procedures introducing biases. A particularly harmful bias is when spurious correlations can appear in clinical images, for example, when dermatologists place a mark next to lesions. Labeling errors can also introduce biases. Expert human annotators may give different labels with systematic biases, but multiple annotators are seldom available.

Metrics That Do Not Reflect What We Want

Suitable metrics for reporting performance on a dataset can change over time, and important metrics, such as calibration, can be missing from research. Also, metrics that are used may not be synonymous with practical improvement. 

Similarly, when comparing such metrics to baseline, baselines may be poorly chosen. For example, choosing an underpowered baseline may create an “illusion of progress”. The opposite problem, of not reporting a simple problem that would have been effective on a dataset is also leaving our critical information. 

Finally, the evaluation error on a dataset might be higher than the performance gains, even as more and more effort is spent on diminishing returns. ​​More Than Beating The Benchmark

Good machine-learning benchmarks are more difficult than they may seem...we want to look at more than just outperforming the benchmark, even if this is done with proper validation and statistical significance testing. One point of view is that rejecting a null is not sufficient, and that a method should be accepted based on evidence that it brings a sizable improvement upon the existing solutions.

Conclusions

This paper was a great (although not the most cheerful) read. It’s important to keep in mind some of the challenges with machine learning research, especially in complex domains like medical imaging. We remain super excited about what ML technology can bring to us in this discipline!

Wired | Now That Machines Can Learn, Can They Unlearn?

What is Machine Unlearning? 

Machine Unlearning is a new area of research that aims to make models selectively “forget” specific data points it was trained on, along with the “learning” derived from it i.e. the influence of these training instances on model parameters. The goal is to remove all traces of a particular data point from a machine learning system, without affecting the aggregate model performance.

Why is it important? 

Work is motivated in this area, in part by growing concerns around privacy and regulations like GDPR and the “Right to be Forgotten”. While multiple companies today allow users to request their private data be deleted, there is no way to request that all context learned by algorithms from this data be deleted as well. Furthermore, as we have covered previously, ML models suffer from information leakage and machine unlearning can be an important lever to combat this. 

In our view, there is another important reason why machine unlearning is important: it can help make recurring model training more efficient by making models forget those training examples that are outdated, or no longer matter.

Initial Explorations

• This paper by researchers from the universities of Toronto and Wisconsin-Madison introduces a framework to expedite the unlearning process by limiting the influence of a data point in the training procedure. 

• A more recent paper by researchers at Cornell, University of Waterloo, and Google studies the generalizability of various machine unlearning approaches. It also proposes a new unlearning algorithm that improves “deletion capacity” (fraction of the training dataset that can be deleted with a bounded loss to performance) for convex loss functions.

This is a new area of research. The article highlights a couple of papers that have been published recently around this (that we shared above). Our general take is that we need a lot more exploration and rigorous testing before concluding the real-world applicability of machine unlearning.

Doordash Engineering Blog | Managing Supply and Demand Balance Through Machine Learning

Doordash recently shared their approach to managing supply and demand balance on their platform with machine learning. The article does a good job of explaining the problem, challenges, and their approach and we recommend reading it entirely. We highlight some salient takeaways below.

Delivery-level vs Market-level balance

The goal of the system is to balance supply (of Dashers) and demand (of delivery orders). But at what granularity should this balance be optimized (and measured)? The ideal scenario would be to have the system balance supply and demand at a delivery/order level - i.e. every order has a Dasher available at the most optimal time, and every Dasher meets or exceeds their targeted pay per hour. However, a more tractable problem is for the system to balance at the market level i.e. ensure that as many Dashers as are approximately necessary and sufficient to meet aggregate order demand but each delivery/Dasher’s outcome might be less than the optimal.

Problem Formulation

Forecasting: As shared in the article, the primary metric used to measure this is “number of delivery hours” i.e. the system tries to balance the number of Dasher-hours required to make deliveries while keeping delivery time low and Dasher utilization high. In this way, the demand forecasting problem is mapped to a regression problem. As highlighted in the article, the team used gradient boosting as the underlying model architecture, through the LightGBM framework.

Optimization: Predictions from the forecasting ML model are fed into a Mixed-integer programming (MIP) optimizer whose objective is to decide incentive rewards to minimize undersupply of Dashers subject to certain configurable constraints. 

Reliability Considerations

The article shares some interesting and valuable learnings on tradeoffs between model performance and reliability. Their forecasting model, for example, relies on a minimal set of features by design and it reduces the complexity of ETL pipelines and the likelihood of feature drift, improving transparency:

We could encode hundreds of features to build a model that has high performance. Although that choice is very appealing and it does help with creating a model that performs better than one that has a simple data pipeline, in practice it creates a system that lacks reliability and generates a high surface area for feature drift

Another point highlighted in the article is around long-chain ETL dependencies, something that we have observed in our own work experiences. ETL jobs can fail or be delayed for any number of reasons. When inputs to your models are a final result of a long chain of such dependencies, it can become very unreliable due to increased surface area for failures. Designing your input feature computations to have minimal intermediate dependencies can go a long way to improve reliability.

Emerging Tech Brew Blog | To regulate AI, try playing in a sandbox

The last couple of years have been especially news-filled on the AI regulation front. 

Last May, Norway announced plans to create an AI regulatory sandbox, and the recently proposed EU legislation mentions the word “sandbox” 38 times. The goal of such regulatory sandboxes is to:

…allow organizations to develop and test new technologies in a low-stakes, monitored environment before rolling them out to the general public. 

What happened?

After Norway’s data protection authority, Datatilsynet, announced plans to create an AI regulatory sandbox, it accepted four participants (from both the public and private sector) to work in 3-6 month-long engagements with them. These participants were facing regulatory conundrums such as “applying principles like transparency, fairness, and data minimization to AI systems”. 

Generally, the goal of Norway’s AI regulatory sandbox is to facilitate compliance with some of these trickier provisions of the GDPR. It’s not looking to create a process that all tech developers go through, but rather to produce helpful precedent in fuzzy legal areas, and communicate those findings to organizations building AI systems. Its approach relies on a series of hands-on workshops and extended conversation and negotiation between tech developers and regulators.

Our Thoughts

These are still very early days for both the usage of AI in industry and regulating such technology. As different countries (and regions such as the EU) try different strategies to tackle the very real risk that AI systems pose, we will get to learn from the approaches that deliver the best results.

Snowflake | Unstructured Data Support

Snowflake recently announced support for unstructured data - now users can access, share, load, and process unstructured files of data in Snowflake. 

First previewed at the Snowflake Summit earlier this year as being in ‘private preview’, you can read more about it in this documentation. The release notes and documentation are a little bit light on what’s possible to do with unstructured data beyond storing and accessing this data in Snowflake, but there is an upcoming webinar on September 22 (signup link) that might be relevant if you wish to learn more about this.

Github | Python Package for Online ML

River looks like a neat Python library for online machine learning. As they put it:

​​River's ambition is to be the go-to library for doing machine learning on streaming data.
Machine learning is often done in a batch setting, whereby a model is fitted to a dataset in one go. This results in a static model which has to be retrained in order to learn from new data… With River, we encourage a different approach, which is to continuously learn a stream of data. This means that the model process one observation at a time, and can therefore be updated on the fly. This allows to learn from massive datasets that don't fit in main memory. 

You can learn more by reading this paper or by watching this video.

Thanks

Thanks for making it to the end of the newsletter! This has been curated by Nihit Desai and Rishabh Bhargava. If you have suggestions for what we should be covering in this newsletter, tweet us @mlopsroundup or email us at mlmonitoringnews@gmail.com. If you like what we are doing please tell your friends and colleagues to spread the word.

In this issue, we deep-dive into inferring concept drift, share a paper on outlier detection using self-supervised learning, discuss NLP applications to summarize legal documents, cover a recent article about customization in B2B ML applications, and more.

Thank you for subscribing. If you find this newsletter interesting, tell a few friends and support this project ❤️

Fast Forward Labs | Inferring Concept Drift Without Labeled Data

This is a wonderful introduction to the problem of concept drift (and how to infer it) from the team at Fast Forward Labs. We have covered concept drift before, but we recommend going through this post for a detailed overview, replete with experiments and nice visualizations. In the meantime, here’s a summary. 

Motivation

The authors put it best:

After iterations of development and testing, deploying a well-fit machine learning model often feels like the final hurdle for an eager data science team. In practice however, a trained model is never final, and this milestone marks just the beginning of the perpetual maintenance race that is production machine learning. This is because most machine learning models are static, but the world we live in is dynamic.

What is Concept Drift? 

This phenomenon in which the statistical properties of a target domain change over time is considered concept drift.

This has two parts, feature drift and “real concept drift”. 

Feature drift refers to changes in the input variables (i.e. changes in P(X)). Such changes to input data over time may or may not affect the actual performance of the learned ML model, as seen in the image below. 

Real concept drift refers to changes in the learned relationships between the inputs and the target variables (i.e. changes in P(y|X)). This type of drift always causes a drop in model performance. “Real concept drift” can happen at the same time as Feature drift - as seen in the image below. 

Supervised Drift Detection

Often, teams will:

monitor a task-dependent performance metric like accuracy, F-score, or precision/recall. If the metric of interest deviates from an acceptable level (as determined during training evaluation on the reference window), a drift is signaled.

At this point, teams will retrain the model with fresh data, and performance levels will improve. However, there is a flawed assumption here: that true labels are instantaneously available after inference. Annotations can be expensive to obtain, sometimes requiring hired domain expertise. Aside from the cost, labels can sometimes take a long time to become available -- for example, it can take days to months for fraud to be reported or defaults on a loan to occur. 

In such cases, detection of drift without any actual labels can be very helpful for ML teams. 

Unsupervised Drift Detection

Without ground truth labels, any drift detection will be prone to some errors - both false positives (signaling drift when there is no impact on model performance) and false negatives (missing crucial concept drift problems). However, some techniques that might prove helpful:

• Statistical test for change in feature space: Compare a time window for each input feature with a reference time window (or from the training data) to see if they come from the same distribution (using tests like the Kolmogorov–Smirnov test)

• Statistical test for change in response variable: Similar to input features, we can run the KS test on the predicted probability values on fresh data and compare it with predicted probabilities from the past.

• Statistical test for change in margin density of response variable: In the previous techniques, we were looking at changes in the entire probability distribution. However, changes in the probability values might not matter too much when the model is very confident (p < 0.1 or p > 0.9). In those cases, we might look for changes within a margin of probability values close to the decision threshold. 

For more details (and their experimental setup on a toy problem), we recommend reading the entire post here. This is a really important topic that is going to gather more mindshare as more production ML systems are built.

Google AI: Discovering Anomalous Data with Self-Supervised Learning

Anomaly detection (or outlier detection) is a common application of Machine Learning with use cases across multiple domains - detecting fraudulent financial transactions, manufacturing defects, tumors in X-rays, and so on. Researchers at Google recently published a novel approach (subsequently published in ICLR ‘21) for anomaly detection based on self-supervised representation learning.

The Approach

Google’s approach is based on a two-stage framework for deep one-class classification:

1. In the first stage, the model learns self-supervised representations from one-class data.

2. In the second stage, a one-class classifier, such as OC-SVM or kernel density estimator, is trained using the learned representations from the first stage.

In addition to the above framework, their work also includes a distribution-augmented contrastive learning algorithm to learn self-supervised representations that are more suited to outlier detection downstream tasks.

The Results

The researchers experimented with two representative self-supervised representation learning algorithms, rotation prediction, and contrastive learning, and evaluated the performance of one-class classification on the commonly used datasets in computer vision, including CIFAR10 and CIFAR-100, Fashion MNIST, and Cat vs Dog. Images from one class are given as inliers and those from remaining classes are given as outliers.

For a more in-depth technical overview, refer to their ICLR ‘21 paper Learning and Evaluating Representations for Deep One-class Classification. Additionally, the accompanying code for this paper can be found on GitHub.

Stanford HAI | Is Natural Language Processing ready to take on legal hearings?

We’ve covered Stanford HAI’s work previously in our newsletter. In a recent article, Stanford researchers Catalin Voss and Jenny Hong discussed the opportunity and challenges associated with applying natural language understanding techniques in reviewing legal hearings.

Problem

A parole hearing is a hearing to determine whether an inmate should be released from prison to parole supervision to serve the remainder of the sentence. During this hearing, a parole commissioner and a deputy review all the relevant history and life circumstances of the candidate, and then decide whether or not to grant parole. As outlined in the article, each such hearing generates a ~150-page transcript of the entire conversation so it can be reviewed if needed later on. In California alone, there are on average 5000-6000 parole hearings each year. 

The governor’s office and parole review unit are tasked with checking parole decisions, but they lack the resources to read every transcript, so as a matter of practicality, they generally only read transcripts for parole approvals. If parole is denied, unless an appellate attorney or another influential stakeholder pushes for a review, the transcript is usually just archived.

Recon Approach

While manually reading the transcripts in their entirety is not feasible, the authors propose using NLP to “read” and summarize these transcripts (flag important factors and data points for each case). This can help scale parole review, and analyzing the outcomes of parole reviews at scale can help understand whether the process is fair/what kinds of biases exist. Further, it can help flag individual cases that seem like outliers. This proposal is discussed in more detail in a forthcoming paper available here.

NLP Challenges

While massive language models like BERT and GPT-3 have shown performance improvements across a large variety of tasks, the article outlines some existing challenges that would need to be solved for applying NLP to this problem:

1. Ability to maintain “state” over long texts. Legal transcripts can often be in the range of tens of thousands of words

2. Synthesizing information across various pieces of text to answer concrete questions

3. Multi-step (multi-hop) reasoning and query planning to answer questions.

Our view

It is far more common to read about the potential dangers and biases of applying machine learning and AI to new real-world use cases. It is refreshing, in our view, to read about this proposal which aims to reduce bias and improve the fairness of the parole process which currently has limited transparency and oversight. There was a conversation about this on Twitter:

Towards Data Science |  Should I Train a Model for Each Customer or Use One Model for All of My Customers?

In this article, Yonatan Hadar discusses an interesting challenge faced by many ML teams building B2B software: what are the pros and cons of training a single model across all customers vs training a model per customer (and all strategies that lie in between)? 

Here, we dissect the lessons from the post and add some of our own thoughts. 

What’s the goal?

At a high level, the goal for any ML team is to create business value through models that perform well (i.e. generalize well) when deployed to production. 

Some good properties of ML systems:

• Models should work well for all customers

• Engineering complexity should be minimized -- training, optimizing, deployment, and monitoring should be as easy as possible

• Onboarding new customers should be easy (no “cold start”)

• Legal and security constraints should be met 

One Model Per Customer

This often performs the best for each customer, since the data distribution between train and test is closest when dealing with just one customer. In certain domains, this might be the only option available to teams, especially when mixing of data across customers isn’t allowed. 

However, there is less data per customer, and teams have to be more careful when adding a new customer (is there enough data available, should simple heuristics be used while data is collected, etc). The engineering complexity might also be much higher since many more models need to be trained (and complexity only rises if model types and hyperparameter tuning strategies can be changed across customers). 

One Global Model

From an engineering perspective, this is usually the simplest approach. The model is typically trained on a much larger dataset (comprising data from many, if not all, customers) and there is no additional training to be done when onboarding a new customer (the model is always ready to go). 

However, the model is now trained on potentially different distributions (from different customers) leading to an overall loss of accuracy. There might also be subtle problems in the model that affect a small subset of customers (thus not impacting overall KPIs), and without proper monitoring will be hard to debug and fix. 

One Model Per Customer Segment

This has the potential to be the best of both previous worlds. If segments are chosen appropriately, the data distributions will be relatively consistent across customers in a segment, and each segment may also end up with a large training dataset. With the right number of segments, the engineering complexity of the system can also be tuned. 

On the flip side, choosing segments correctly is difficult since there are many ways to segment customers (geography, industry, etc). If done improperly, one might end up in a situation of the worst of both previous worlds. 

Global Model + Transfer Learning

This is only applicable to deep learning models, but this has the potential for high performance on all customers with much fewer data points. 

However, deep learning might not be the right solution in many cases (for example, when you need very low serving latency or easy explainability). This method is still fairly new, so some exploration might be needed for your use cases. 

Our Take

ML teams are often operating with constraints that are unique to their industry and company. While there are many good ways to build and deploy ML systems, having an understanding of the trade-offs involved is useful.

Twitter | Bad Labels in Public Datasets

In this Twitter thread (and associated blog post), Vincent Warmerdam explores how easy it is to find incorrect labels in a publicly available dataset, even one that was curated by researchers from Stanford and Google. 

On this Google Emotions dataset, the author trains a simple high bias, low variance model and then ranks examples where a very low confidence score was given to the correct class. This resulted in plenty of mislabeled examples, such as in the image below:

This is similar to (and inspired by) the work of the team behind labelerrors.com, where they show problems with many popular datasets such as CIFAR, MNIST, etc. If you’re directly using publicly available data, you might want to use some simple heuristics to ensure that label quality is at an acceptable level. 

Thanks

Thanks for making it to the end of the newsletter! This has been curated by Nihit Desai and Rishabh Bhargava. If you have suggestions for what we should be covering in this newsletter, tweet us @mlopsroundup or email us at mlmonitoringnews@gmail.com. If you like what we are doing please tell your friends and colleagues to spread the word.

Welcome to the 25th issue of the MLOps newsletter.

In this issue, we cover Tesla AI Day, a short paper about gaps in feature stores, updates about the NIST proposal to reduce bias in AI, tips on ML monitoring, and a tech review about the challenges of deploying AI tools for diagnosing COVID.

Thank you for subscribing. If you find this newsletter interesting, tell a few friends and support this project ❤️

Tesla | AI Day

We’ve covered updates about Tesla’s AI for self-driving cars in a prior edition of the newsletter. Tesla recently held an AI Day event to share its research and innovations across the stack for solving self-driving and the general real-world robotics perception & planning problem. It was a fascinating event and highlighted the scope and scale of Tesla’s AI efforts. We recommend watching it in its entirety. A brief summary of major innovations presented: 

Network Architecture for Perception & Planning

• Andrej Karpathy gave an overview of how deep learning models for perception have evolved over the last four years, starting from residual nets for lane-keeping to a fairly complex multi-task architecture trained to predict in the vector space (as opposed to static images)

• Fusion of sensor data across all 8 car cameras and training the network to do the final prediction on the joint data. This brought a step function improvement compared to the earlier (arguably more straightforward) approach of making predictions on streams from each camera and then combining the predictions.

• Perception on 4D video for modeling time progression (e.g. which objects are moving and which ones are stationary; what is the speed and trajectory of moving objects)

• Tesla has started using neural networks not just for perception but also to improve the efficiency of searching through the action space for better planning and control.

Data Labeling & Augmentation

• Annotating in the vector space means you annotate once, simultaneously creating labels for streams from all 8 input cameras across many timestamps.

• Autolabeling is used to massively scale up the size of labeled data that the neural net can learn from. The details of how autolabeling is done (by leveraging video clips across different cars in the same physical location) are shared in this part of the talk.

• Tesla leverages simulations for constructing training data for rare edge cases that are very unlikely in the real world.

• Interesting note: Tesla hires >1000 human annotators in-house to work full-time on just data labeling tasks. 

Dojo

• Dojo is Tesla’s next-generation chip & data center architecture for fast training of large neural networks. It is still under development and expected to be rolled out next year.

• The “exapod” setup, which brings together 3000 of these D1 chips, should enable 1.1 exaflops of configurable-FP8 compute.

• From an ML engineer’s point of view, training on Dojo would involve just a one-line change to the code which makes it super easy to switch over. 

Paper | Managing ML Pipelines: Feature Stores and the Coming Wave of Embedding Ecosystems

This is a short paper that discusses one of the shortcomings of today’s conception of feature stores -- that of dealing with embeddings as features, given that feature stores have typically been built for tabular data. 

What is a Feature Store? 

Feature Stores provide a centralized repository of reusable features during ML training and inference (you can also read our discussion here, here, and here). Features can be constructed using tabular data or stream data and are saved for training and serving time:

Users provide simple definitional metadata, e.g., the feature update cadence and a definition SQL query, and upload the definition to the FS. When the underlying data changes, the FS orchestrates the updates to the features based on the user-defined cadence.

For streaming features, users provide aggregation functions that are applied on the raw streaming features. The aggregated features are persisted to the online store and logged to the offline store…

Once a model is deployed, features need to be continuously provided to deployed models even as the feature data is updated over time. To provide low latency feature serving, FSs are typically a dual datastore: one for offline training (e.g., SQL warehouse) and for online serving (e.g., in-memory DBMS).

Where’s the gap? 

Many ML systems today rely on embeddings that are derived from raw data (see examples from Twitter and Doordash). These embeddings are learned in a self-supervised manner; there are no labels and traditional metrics for data quality don’t apply here. These embeddings also need to be saved for training and inference time, and the dual datastores might not be an ideal fit for such embeddings. Finally, monitoring and detecting drifts with embeddings is a very different challenge compared to tabular data. 

Solutions

In the past, we have covered vector databases (such as the Google Vertex Matching Engine covered here). These datastores are optimized for storage, retrieval, and search over embeddings and are a more natural fit for embeddings. 

We are excited to see new technologies develop in this space to address the varieties of data types used by the ML community! 

NIST Proposal: Reducing Risk of Bias in AI

Potential bias in machine learning models is a well-recognized problem in academia and industry today. In an effort to counter this, the National Institute of Standards and Technology (NIST) is advancing a proposal to identify and reduce the downstream risk of bias in AI. 

The draft proposal is shared in NIST Special Publication 1270. The approach proposed in this report consists of three distinct stages, and inputs from accompanying stakeholders at each stage: 

1. PRE-DESIGN: where the technology is devised, defined and elaborated 

2. DESIGN AND DEVELOPMENT: where the technology is constructed 

3. DEPLOYMENT: where technology is used by, or applied to, various individuals or groups. 

NIST is accepting comments on the document until Sept. 10, 2021. Authors seek to use these responses to help shape the agenda of collaborative virtual events in the coming months. Comments can be submitted by downloading and completing this template form and sending it to ai-bias@list.nist.gov

Nubank Blog | ML Model Monitoring – 9 Tips From the Trenches

We always enjoy reading articles from ML teams in industry -- their lessons are hard-won and extremely applicable. This week we cover tips on ML monitoring from the Latin American company Nubank (now the largest digital bank in the world by number of customers: 40 million).

Why ML Monitoring?

“Machine learning (ML) models are very sensitive pieces of software; their successful use needs careful monitoring to make sure they are working correctly.

This is especially true when business decisions are made automatically using the outputs of said models. This means that faulty models will very often have a real impact on the end-customer experience.

Models are only as good as the data they consume, so monitoring the input data (and the outputs) is critical for the model to fulfill its real objective: to be useful to drive good decisions and help the business reach its objectives.”

Lessons

• Averages don’t tell the full story: Monitoring average values for numerical features can lead to an incomplete picture. Tools often don’t know how to deal with missing (or null data) and assume that data problems will be large enough to move averages significantly. It’s better to monitor percentiles (99th, 95th, 90th, 10th, 5th, 1st, etc) along with missing value rates for all features. 

• Break monitoring into subpopulations for better insights: It can be easier to understand data by breaking it up into subpopulations that are monitored separately. 

• Consistency reduces the mental burden of monitoring: Monitoring tools and dashboards should be consistent and standardized. A single tool for monitoring is best with consistent naming of files/datasets/dashboards and these artifacts should be ordered based on priority to end-users. 

• Patterns for alerting: Real-time alerts (emails, Slack messages) can easily end up being “too noisy” (going off too often and people not taking them seriously) or “not sensitive at all” (not going off even when they should). The authors recommend always including the time frame and specific data points, for example:

  “Average value of Feature X in model Y for the past 15 minutes was too high (expected between 0.4 and 0.5, but got 100.0 instead)”

• Monitor monitoring jobs/routines themselves (meta-monitoring): Model monitoring tools are just another piece of software and they can stop working from time to time. It’s good to monitor execution times for monitoring jobs and have heartbeat-style alerts. 

Conclusion

As the authors say:

“These are a couple of tips we found useful for monitoring several ML models here at Nubank. 

They are used in a variety of business contexts (credit, fraud, CX, Operations, etc) and we believe they are general enough to be applicable in other companies too.”

MIT Technology Review | Hundreds of AI tools have been built to catch covid. None of them helped.

A recent article published in MIT Technology Review highlights the challenges of building AI-powered diagnosis and treatment tools for COVID-19, but many of the observations and learnings can be generalized to healthcare.

The Problem 

Early in March/April 2020, hospitals around the world scrambled to treat potential COVID-19 cases. In order to help hospitals scale and better allocate resources, many AI efforts were initiated to diagnose the disease. A recent report by Turing Institute in the UK however concluded that AI tools had very little to zero positive impact in helping hospitals fight COVID.

This conclusion is largely in line with a similar one reached by Derek Driggs, a machine-learning researcher at the University of Cambridge. They looked at deep learning models for diagnosing covid from medical images (X-rays, CT scans). Out of the 415 such published tools, they concluded that none were fit for being deployed in clinics (we had covered this briefly here).

What caused it? 

In a nutshell, data quality. The article illustrates some failure cases due to data quality and data skew issues that seemed to systematically plague these models. A couple of examples:

• Some approaches unknowingly used datasets that contained medical images of children (where the prevalence of COVID is much lower). As a result, the model inadvertently learned to classify kids vs adults

• The source of ground truth labels was the doctor’s diagnosis of whether the chest scans showed signs of covid (versus actual ground truth from test results such as PCR). This introduced noise and bias in the data.

What can we learn? 

The article summarizes suggestions from healthcare professionals about what we can learn and how to fix this problem:  

• Transparency and data sharing: Disclosing the source of your data, and if possible releasing the datasets along with the models. While this might not always be possible in healthcare contexts, at the very least sharing the source with doctors and healthcare professionals who intend to use these models in the field can be helpful for them to make informed judgments.

• Data standardization: Lack of standardization makes it hard for healthcare data from different sources (hospitals, countries) to be integrated. 

• Model validation: Validating models from other teams might not be the most glamorous work, but it can help take “tech from ‘lab bench to bedside.’”

Twitter: ML Strategy tip

We love this tweet by Brandon Rohrer. We have both built “very simple models” (read: heuristics) in the past to solve problems, and cannot agree with the sentiment more. As ML practitioners in industry, our job is to create business value -- only sometimes does it require ML! 

Thanks

Thanks for making it to the end of the newsletter! This has been curated by Nihit Desai and Rishabh Bhargava. If you have suggestions for what we should be covering in this newsletter, tweet us @mlopsroundup or email us at mlmonitoringnews@gmail.com. If you like what we are doing please tell your friends and colleagues to spread the word.

In this issue, we cover an interesting approach to autonomous vehicles at Porsche, discuss strategies for the efficient serving of ML models, explore the bootstrapping of labels with weak supervision and share some recent resources.

Thank you for subscribing. If you find this newsletter interesting, tell a few friends and support this project ❤️

The Big Loop: artificial intelligence and machine learning

Porsche Engineering recently shared this exploration of how an autonomous vehicle could learn from experience and react intuitively to new circumstances. The article describes a very specific use case -- detecting lane changes in front of a car. While this article doesn’t talk about any other elements of self-driving, the process they describe is interesting and reminiscent of Tesla, which we discussed earlier. 

The Problem

Their vehicles are equipped with an enhanced Adaptive Cruise Control (ACC) system which ensures that a safe distance is maintained from the vehicle in front, detecting early when other road users are cutting in, etc. With ACC:

“A likely lane change is detected half a second to a second earlier – the equivalent of 30 metres of driving on the motorway,” explains Dr. Joachim Schaper, Senior Manager AI and Big Data at Porsche Engineering.

The model needs to be continuously improved, but going through all the data that is being generated is too consuming. 

“…we only want to record the data that really helps the system move forward,” says project manager Philipp Wustmann, an expert in longitudinal and lateral control at Porsche Engineering. “That's no easy task, because radar sensors and cameras generate an immense amount of data, most of which is not relevant to the function under consideration.”

Solution

They follow the following process to improve the model:

• Find scenes where the ACC is not reacting optimally in the car (using some heuristics) and send them to their servers in the cloud

• Augment this with additional simulated scenes that generate extra training data without more drives

• Train a new model and validate it on unseen data

• Push the model to the car and allow the new version of the model to be activated by the driver

They are now using this technical approach for other development projects, and we believe that elements of this approach are much more broadly useful. 

Efficient Machine Learning Inference

This is a fascinating article that discusses strategies for serving multiple ML models in scenarios where latencies matter. 

Why is this important? 

Let’s imagine your team is responsible for serving multiple ML models, and having low latency inference is important for your business. There could be a wide variation in the kinds of models and use cases they are satisfying.

Some models are large while others are small...Other models get periodic bursts of traffic, while others have consistent load. Yet another dimension to think about is the cost per query of a model: some models are expensive to run, others are quite cheap.

Typically, teams will provision for one model per host -- this ensures (relatively) predictable latency since you just have to track per-host throughput. This can then be horizontally scaled for peak traffic per model, and you should have sufficient capacity. However, these provisioned servers are often in excess of what is necessary at most times, and if you choose to reduce costs by reducing the size of the VM, that often has a negative impact on latency. 

So what should one do?

Multi-model serving, defined as hosting multiple models in the same host (or in the same VM), can help mitigate this waste. Sharing the compute capacity of each server across multiple models can dramatically reduce costs, especially when there is insufficient load to saturate a minimally replicated set of servers. With proper load balancing, a single server could potentially serve many models receiving few queries alongside a few models receiving more queries,  taking advantage of idle cycles.

They then go on to show numbers from a real-world example. They gather the traffic in queries per second (QPS) for 19 models over a period of a week. 

As can be seen, most models have infrequent traffic (and some spikes), but one model has high constant traffic and large spikes. When these models are deployed on a standard VM (in Google Cloud, which is where the example is from), the 99th percentile latencies for the models fall into different ranges (depending on the type of the model) ranging from 1ms to 1000ms.

Now, if all the models were to be hosted on a single, much larger VM, the 99th percentile latencies range between 4ms - 40ms (see the article for all the charts). Keeping all 19 models in memory consumes 40GB of RAM, but it is easy to find machines in the cloud that easily satisfy that requirement. All the more, this is done while keeping monthly costs 1-2 orders of magnitude lower. 

Your mileage may vary (especially if you have only models with consistently high QPS), but these results are worth keeping in mind. 

Conclusion

The authors put it well:

Multi-model serving enables lower cost while maintaining high availability and acceptable latency, by better using the RAM capacity of large VMs. While it is common and simple to deploy only one model per server, instead load a large number of models on a large VM that offers low latency, which should offer acceptable latency at a lower cost. These cost savings also apply to serving on accelerators such as GPUs.

If you’re also responsible for deploying a large number of models and need to serve inferences in real-time, this advice may be for you. One caveat: this approach leads to a coupling between models to some extent. Any correlation between the traffic experienced by a few models will have an outsized impact on the system.

Bootstrapping Labels via ___ Supervision & Human-In-The-Loop

In a recent article, Eugene Yan shared a summary of prevailing approaches to bootstrap labeled training data in real-world machine learning applications  - an underappreciated but important problem in our view. It discusses semi-supervised, active, and weakly-supervised learning, along with examples from DoorDash, Facebook, Google, and Apple.

• Semi-supervised learning: This approach combines a small amount of labeled data with a larger amount of unlabeled data during training, and aims to improve upon the performance that can be obtained from just the labeled data (supervised learning) or unlabeled data (unsupervised learning). 

• Active Learning: Active learning aims to select the most interesting/informative unlabeled examples that should be labeled to improve the model performance. While multiple metrics can be used in active learning, one that’s commonly used is to select examples that are the “hardest” for the model (one where the model is the most uncertain). Active learning is often used to solve the “cold start” problem and set up a feedback loop to iteratively improve the performance of ML models. In this blog post, Doordash shared details of their human-in-the-loop active learning system for menu item tagging (a version of which we covered here). In this paper, researchers at Facebook AI shared details of an approach to couple active learning with similarity search around positive labels in cases of skewed datasets. This is especially relevant when the underlying detection problem of interest has a low prevalence (e.g. integrity problems like bullying, hate speech, misinformation) and we covered this in our last issue.

• Weak Supervision: Related to semi-supervised learning, weak supervision aims to combine multiple (often noisy and imprecise) sources of labels to generate an additional source of information. These sources could include heuristics, regex rules, prior trained models, etc. In this joint paper, researchers at Google and Snorkel share some case studies of Snorkel Drybell, a weak supervision system deployed internally at Google.   

Putting it all together

Given a large variety of constraints and priorities in real-world ML scenarios, it’s unlikely that there will be a single “globally optimal” path to acquire good quality and a large volume of labels over time. Nonetheless, the article shares some heuristics for when to apply which technique: On day 1, perhaps using heuristics or weak supervision is a good approach to get your product working end to end. Afterward, some combination of active learning and label denoising might work well as evidenced in the several examples in the article.

New Resources for Machine Learning and MLOps

We’ve recently come across a few useful resources that we’d like to share:

Approximate Near-Neighbor search benchmarks:  We’ve covered applications related to approximate nearest-neighbor (ANN) similarity search earlier in a prior issue of the newsletter. This Github repo aims to benchmark various implementations of ANN search on a collection of public datasets. We liked the attention to usability in this repository -- datasets are already pre-generated and each implementation has a docker container. If you’re considering implementing ANN for your use case, we highly recommend checking this out to understand which implementation might best suit your needs.

Shapash: A neat Python library for ML interpretability based on SHAP and LIME. This library allows data scientists or end-users of a machine learning model to generate local and global explanations for predictions (features contributions) and understand feature correlations. 

QuestDB: QuestDB is an open-source database for time-series or events data with a focus on performance. In benchmarks shared on their website, QuestDB outperforms other popular alternatives like InfluxDB and might be suitable for your use case especially if you’re looking for real-time query-ability on a high volume of data.

Twitter | Is pushing ML models to production really the hard part? 

Neal Lathia’s recent Twitter thread highlighted the two (and one might say somewhat at odds) points of view in the MLOps world, namely: 

1. The biggest challenge is shipping models to production

2. Shipping models to production is the easy part: wrap it behind an API and call it from your service (for e.g. using Flask or FastAPI)

The thread generated interesting discussions. We recommend reading the various reply threads and conversations to get a complete picture, but we highlight a couple that we found especially relevant: 

• Batch and point predictions might need different approaches (thread)

• The biggest question for companies is really the value added by deploying machine learning (thread)

• While deploying a model behind a flask API (or equivalent) is easy, questions around scaling, authentication and reliability still need to be solved (thread)

Thanks

Thanks for making it to the end of the newsletter! This has been curated by Nihit Desai and Rishabh Bhargava. If you have suggestions for what we should be covering in this newsletter, tweet us @mlopsroundup or email us at mlmonitoringnews@gmail.com. If you like what we are doing please tell your friends and colleagues to spread the word.

Welcome to the 23rd edition of the MLOps newsletter. 

In this issue, we cover updates on AI regulations and frameworks in the EU and the US, a recent paper on web-scale active learning, details about LinkedIn’s internal platform for model observability, and a summary of Google’s new scalable vector similarity search.

Thank you for subscribing. If you find this newsletter interesting, tell a few friends and support this project ❤️

AI Regulations in the EU and the United States

Demystifying the Draft EU AI Act

A few months ago, we had shared the AI regulations from the EU when they were still being considered (and some of the early reactions). 

Now, Michael Veale has put together a fascinating deep dive into the Artificial Intelligence Act. We recommend reading through the Twitter thread (or their paper for the full analysis) -- our overwhelming takeaway is that writing policy is hard! 

Let’s look at a couple of the specific shortcomings in the AI Act they write about:

• The Act prohibits manipulative systems. In their formulation, being “manipulative” requires “intent” from the system and the system must exert influence through a “vulnerability” such as age, disability, or through “subliminal techniques”. However, there is no clause for whether the user was harmed in any way or not. They also explicitly exclude manipulation from systems based on ratings and reputation. The end result is that almost all online systems will be excluded from being judged under this Act (and, of course, no vendor will admit an “intent” to manipulate).

• For high-risk AI systems (biometric identification, law enforcement, etc), the Act lays out “essential requirements” including data quality criteria of accuracy, representativeness, and completeness. However, the European Commission plans to ask two private organizations who will create a paid “harmonized standard”. Such private organizations are heavily lobbied and could drift away from the “essential requirements”, which is likely to make this regulation difficult to enforce. 

United States GAO AI Accountability Framework

This is a report from the Government Accountability Office (GAO) that details an accountability framework for federal agencies when it comes to using AI responsibly. 

We won’t go into the details here, but their focus on data quality, governance, monitoring, and performance tracking is spot on!

Paper | Efficient Active Learning with Similarity Search

In a recent paper, researchers from Stanford, Facebook AI, and the University of Wisconsin-Madison introduced SEALS (Similarity Search for Efficient Active Learning and Search).

Problem

​​For web-scale datasets, active learning approaches are often intractable due to size - billions of unlabeled examples. Existing approaches search globally for the optimal examples to label, scaling linearly or even quadratically with the size of unlabeled data. In effect, the time complexity of active learning for such use cases is O(dataset size * number of sampling iterations). 

Proposed Solution

This paper improves the computational efficiency of active learning and search methods by restricting the candidate pool for labeling to the nearest neighbors of the currently labeled set. The intuition behind this idea is that learned embeddings from generalized pre-trained models can cluster many rare concepts. This structure can then be exploited to improve the efficiency of active learning: only considering the nearest neighbors of the currently labeled examples in each selection round allows us to eventually get to all, or most positive labels for a given concept, without having to exhaustively scan over all of the unlabeled data.

Results

The paper evaluated the proposed selection strategy on three datasets: ImageNet, OpenImages, and an anonymized and aggregated dataset of 10 billion publicly shared images on the web. We recommend reading the paper to learn more about the experiment details and parameters for each dataset, but the overall takeaway was that this approach achieved similar mean average precision and recall from downstream classifiers trained on the labeled datasets, as the traditional global approach (baseline) while reducing the computational cost by up to three orders of magnitude.

Risks

As identified by authors in the paper, this gain in computational efficiency does come at the cost of some added complexity that should not be ignored:

1. This approach requires a similarity search index that can index and search over the entire dataset. There are some open-source implementations of approximate nearest neighbor search that can be considered here.

2. There is a risk that embeddings used for this task do not effectively cluster examples that belong to a given concept. Just restricting labeling selections to nearest neighbors of the currently labeled data might not be an effective strategy.

LinkedIn Engineering Blog | Model Health Assurance at LinkedIn

This recent article by LinkedIn shares some of the design principles and details of Model Health Assurance, LinkedIn’s in-house platform for ML model monitoring and observability (which is, in turn, part of its Pro-ML platform for training and serving ML models online). We share some highlights from the article below:

• Motivation: Monitoring ML models post-deployment is a recurring problem for ML engineers at LinkedIn. Health Assurance (HA) is a platform to provide engineers with tools and systems that help them identify issues with production models faster and help debug them.

• Core functionality: HA plugs into the online model inference code path to measure a variety of system indicators (e.g. traffic volume, latency, resource usage of hosts, etc) and data indicators (feature statistics, prediction drift, etc). 

• Consumption: A batch job computes feature and prediction statistics for all deployed models and pushes them to Pinot (an open-source distributed OLAP data store) and ThirdEye (LinkedIn’s in-house monitoring & alerting tool) which then alerts the downstream team that owns the model. 

• Easy onboarding: HA has two types of functionalities: a basic set of metrics & alerts that are autoconfigured and supported out of the box for any model that goes to production; and another set of custom metrics & features to track for a specific model (configured by ML engineers).

New Product Alert: Vertex Matching Engine by Google

Google Cloud recently announced the release of Vertex Matching Engine, a completely managed cloud solution for fast and scalable approximate nearest neighbor (vector similarity) search. 

What is Nearest Neighbor Search? 

Embedding representations, a learned mapping of raw content to a dense n-dimensional space, are a foundational tool for any ML engineer. While embeddings are useful for all kinds of machine learning tasks as general feature representations, one class of powerful applications they enable is embedding-based search.

Compared to traditional tf-idf based retrieval which searches a database of documents based on token match, an embedding-based search can rely on matching queries and documents based on semantic properties learned by the embeddings. To answer a query, we map the query to the embedding space and then find, among all database embeddings, the ones closest to the query (i.e. nearest neighbor search).

Overview of Vertex Matching Engine

While many models used to generate embeddings are open-source to used (e.g. Universal Sentence Encoder for text or ResNet for images), using them for search/similarity applications is still hard because traditional databases are not optimized for nearest-neighbor search queries. 

Vertex Matching Engine is a managed solution by Google cloud for fast and scalable approximate nearest neighbor search. It is powered by ScaNN (the same technology that it uses internally for Google search, youtube, etc). It enables real-time searching - O(millisecond) latency - over O(billions) database size. Based on empirical observations from teams within Google, Vertex Matching Engine can achieve 95-98% recall compared to brute force nearest neighbor search. Furthermore, GCP handles all the scaling requirements based on database size and query load and allows for index updates with zero downtime. 

To Get Started

If you’re interested in experimenting with Vertex Matching Engine or are evaluating it for a potential production use case, you can take a look at the documentation here, or this sample notebook.

Twitter Thread: Dealing with Uncertainty in ML

Sarah Hooker shares two interesting papers that were presented at ICML. Both papers explore what happens when real-world, noisy data is brought to bear on the ML training process. 

The first paper discusses the process of up-weighting samples with a higher loss during training time with the assumption is that the model has more to learn from such examples. However, they find that when data is corrupted (or the level of noise is high enough), up-weighting can end up hurting the final trained model. High-quality data is a prerequisite for such training acceleration techniques. 

The second paper breaks down error cases into reducible errors (outliers and edge cases) and irreducible errors (super noisy or mislabeled examples). If such errors can be categorized, then model training could be improved by cleaning the noisy examples and augmenting the outliers or atypical examples.

Thanks

Thanks for making it to the end of the newsletter! This has been curated by Nihit Desai and Rishabh Bhargava. If you have suggestions for what we should be covering in this newsletter, tweet us @mlopsroundup or email us at mlmonitoringnews@gmail.com. If you like what we are doing please tell your friends and colleagues to spread the word.

Welcome to the 22nd issue of the MLOps newsletter. 

In this issue, we cover OpenAI’s paper introducing Codex (their fine-tuned GPT language model for code generation that powers Github CoPilot), share Uber’s CI/CD for deploying production ML models, discuss concept drift challenges with production ML models, and share Andrej Karpathy’s recent tweets about challenges with designing data labeling workflows.

Thank you for subscribing. If you find this newsletter interesting, tell a few friends and support this project ❤️

Paper | Evaluating Large Language Models Trained on Code

Github recently announced Copilot, an AI-powered pair programming tool. Check out the official webpage or this intro video if you want to get a gist (ha!). Copilot is powered by Codex, which is a GPT language model fine-tuned on code from Github, and the subject of this paper from OpenAI.

Overview

This paper makes two important contributions:

1. It introduces Codex, a GPT language model that is finetuned to generate code using publicly available code from GitHub. It evaluates Codex on its Python code code-writing capabilities, given a docstring as the “prompt”.

2. It introduces HumanEval, a new evaluation dataset and associated code released by OpenAI to measure the functional correctness of generated code programs from docstrings.

Codex: Under the hood

• Training data: As shared in the paper, Codex is fine-tuned on code publicly available on Github. The training dataset for the model was collected in May 2020 from 54 million public software repositories hosted on GitHub. After applying heuristic filters to filter out likely auto-generated code files, the total size of the training data was 159 GB. 

• Task: The paper focuses on the task of generating standalone Python functions from docstrings and evaluating the correctness of code samples automatically through unit tests.

• Evaluation Methodology: To benchmark the model, the authors create a dataset of 164 original programming problems with unit tests. As outlined in the paper, the complexity of these programs can be compared to easy interview questions. This dataset is released and publicly viewable on their Github repo. The metric used is “pass rate @ K”: K code samples are generated per problem, and a problem is considered solved if any sample passes all the unit tests for that problem.

• Performance: Codex fine-tuned on Github data solves 28.8% of the problems with just 1 sample per problem. Further, the authors find that repeated sampling is an effective strategy for producing working solutions to difficult prompts. Using this method, the model solves 70.2% of problems with 100 samples per problem.

Limitations & Risks

In the latter half of the paper, the authors examine and discuss Codex’s limitations and risks. We definitely recommend reading it in more detail in the paper, and summarize the major takeaways here:  

• Sample efficiency: Codex’s training dataset is basically all publicly available Python code on GitHub, which is hundreds of millions of lines of code. This is many orders of magnitude more than the amount of code a human software engineer will encounter over their career.

• Error-prone: As noted in the paper, programs generated by Codex often have syntax errors, and invoke functions or call class attributes that are undefined.

• Alignment problem: Codex, like all language models, generates code output that is drawn from the training distribution. This means the code output, in some real-world scenarios might add negative value if the input prompt contains subtle bugs or errors

• Bias & representation: Codex can be prompted in ways to generate denigratory outputs as code comments.

Codex is a fairly new tool in the software engineer’s toolkit. Github Copilot, for example, is not yet publicly available to use. Over the next decade, as the capabilities of tools like Copilot improve, we believe they could result in a multiplier on engineering productivity (especially for tasks that are below a certain threshold of complexity) assuming that the risks are sufficiently mitigated.

Uber Engineering Blog | Continuous Integration and Deployment for Machine Learning Online Serving and Models

It’s always fascinating to read about the ML systems at large consumer tech companies -- they are often solving problems that most companies will never face. Certain elements from this post by the Uber team feel like that -- let’s jump into it!

Problems

Uber relies on a large number of ML models to deliver a good customer experience. As they  scaled their services, they needed to address the following MLOps challenges:

• Develop a CI/CD system for their ML model deployments so that a large volume of model deployments could happen on a daily basis. 

• Dealing with the latencies of fetching and loading models and the memory footprint of old models in their Prediction Service. 

• Rolling out models for a subset of traffic, or in shadow mode (where predictions are not being directly used but captured for experimentation/analysis)

• CI/CD for their Prediction Service itself is key to prevent issues such as incompatibilities between models and the Prediction service.

Model Deployment

Historically, the Uber team used to include model artifacts in the Docker image for their Prediction service. With the growth in model deployments, they started dynamic loading of models where the Prediction service would periodically check the local model version with the latest version in the Model Artifact and Config Store, and fetch the latest model as needed. 

Before a model is pushed into the Model Artifact and Config Store, the artifacts are validated and the compiled model file is used to run predictions on some sample data to ensure that model is working. 

The deployment process can be tracked by ML Engineers and a health check on each model is automatically deployed. 

Model Auto-Retirement

While a model retirement API is provided, engineers can forget to retire old models. This leads to unnecessary memory consumption. The Uber team built an expiration date into their model deployment process such that old models, if unused, would be automatically retired with alerts sent to the relevant teams. 

Model Rollout and Shadowing

The Uber team found that different ML engineers chose to roll out models with different strategies -- gradual rollout to production traffic to shadowing current model for a period of time. They ended up building these mechanisms into the Prediction Service directly such that users could easily set up model rollout and shadowing strategies. This also allowed simpler sharing of features between primary and shadow models from their Feature Store, allowing auto-shadowing of models and better infrastructure management during times of high traffic. 

Continuous Integration and Deployment of Prediction Service

It’s not enough for models themselves to have CI/CD, but the service that is running predictions across all models required full CI/CD as well. This is important to make sure that code changes or dependency changes don’t lead to compatibility issues when the model is being loaded or being run. The Uber team had a three-step validation process: staging and canary integration tests against a non-production environment followed by a gradual rollout to production workloads. 

Our Take

There are interesting lessons here for companies that might be scaling to a large number of models across a large amount of traffic. We’d love to hear from you about what resonated with you!

Forbes Technology Council | Why Machine Learning Models Crash And Burn In Production

This is a great post from David Talby, the CTO of John Snow Labs, a company that builds ML and NLP solutions for the healthcare world. Even though the post was written in 2019, the lessons remain relevant, especially since they’re backed by his own experiences. 

What’s the problem?

“One magical aspect of software is that it just keeps working. If you code a calculator app, it will still correctly add and multiply numbers a month, a year, or 10 years later. The fact that the marginal cost of software approaches zero has been a bedrock of the software industry’s business model since the 1980s.

This is no longer the case when you are deploying machine learning (ML) models. The moment you put a model in production, it starts degrading.”

The author says that this performance degradation is largely down to a change in the data in the real world, or concept and data drift (a topic we have covered extensively). 

Real-world example

The author was trying to predict 30-day readmission rates, which is a well-studied problem thanks to the US Medicare’s Hospital Readmissions Reduction Program. However, in their specific project, they found that:

“A predictive readmission model that was trained, optimized and deployed at a hospital would start sharply degrading within two to three months. Models would change in different ways at different hospitals — or even buildings within the same hospital.”

There were many reasons for this drop in performance. Changing fields in electronic health records would make some workflow better, but leave old fields blank. Important codes would change when a different lab was used and starting to use a new type of insurance would change the type of patients going to the ER. 

Lessons

• Online measurement of accuracy: We need a better understanding of the performance of models in production, and logging real-world results is “an elementary requirement.”

• Mind the gap: We need to watch out for “gaps between the distributions of your training and online data sets”. 

• Online data quality alerts: If the input data distribution changes in a meaningful way, an alert should go to the operations team - it might also be time to retrain your model. 

Here is another great post on the lessons from deploying ML models in healthcare from the author.

Jay Alammar | Behavioral Testing of ML Models

Evaluating models using an aggregate metric on a test set (like accuracy, F1-score), while helpful to benchmark research approaches, is often insufficient for production machine learning applications. This video by Jay Alammar discusses  Beyond Accuracy: Behavioral Testing of NLP Models with CheckList, a paper we covered previously in our newsletter as well. The paper (and the video) introduces us to the practice of behavioral testing for ML models (analogous to unit tests for traditional software).

The Problem

The traditional practice of evaluating models using an aggregate standardized metric (like accuracy, F1-score) on a standardized test set is helpful to benchmark research approaches. However, it runs into a few problems in the real world:

1. Overestimation: Traditional approach of testing ML models on a held-out subset of the complete labeled dataset, usually does not generalize to examples “in the wild”.

2. Resolution: A single metric produces at best a low-resolution picture of model performance. Given two models, say A and B with test set accuracies of 65% and 68%, it is often insufficient to tell which one should be launched in production. Some follow-up questions that ML practitioners often care about:

   1. How was the test dataset constructed? Does this look like my production data in any way?

   2. What are the examples that model A gets right, but B gets wrong? And vice-versa?

   3. There are these 20 examples that any model going to production HAS to get right. How do models A and B perform on these?

The Solution

Behavioral tests can give us a higher resolution evaluation of a model's capabilities. While the video focuses specifically on behavioral testing for NLP models, the idea we believe can generalize very well to domains beyond NLP. 

• Minimum Functionality Tests: Equivalent to unit tests on tiny test datasets. Often test the model’s performance on obvious test cases or cases where the cost of incorrect prediction is extremely high. 

• Invariance Tests: Test the model’s performance on “label preserving” perturbations to the input. A variation of this is often used to do adversarial robustness testing 

• Directional Expectation Tests: Test whether the model’s predictions move in a manner that is directionally consistent with one’s expectations. 

In addition to this video, which gives a good high-level introduction to behavioral testing, you can also check out a talk by the paper’s author here from the Stanford MLSys Seminar Series.

New ML Resource | Course on Transformers from Hugging Face

This looks like a super interesting course from the Hugging Face team. It focuses on Transformer models, which are slowly becoming the standard in NLP, and delves into both the theory and usage with their transformers library. The first part of the course is out now and will help you get started with either Tensorflow or PyTorch, and more advanced lessons will become available later this year. 

For folks who are thinking about deploying this in AWS, here is a handy tutorial for running a serverless inference service using AWS Lambda. 

Twitter | Challenges with annotating data for ML

Andrej Karpathy, Director of AI at Tesla, has shared some of the details and (fun) challenges with building Tesla’s self-driving suite of capabilities on multiple occasions. We covered his talk about the Tesla data engine in a previous edition of the newsletter. His recent tweets about designing good labeling/annotation workflows to collect labeled data for training models are worth reading. 

The challenges he highlighted - writing good labeling docs (aka the “algorithm” human labelers should follow), training labelers, designing tools to do this task with good quality and throughput, deciding which data should be labeled - are all relatable and something we have experienced first hand at some of the companies we’ve worked at. A couple of recent examples of products trying to tackle this problem that we’ve come across include LabelBox and Scale Nucleus.

Thanks

Thanks for making it to the end of the newsletter! This has been curated by Nihit Desai and Rishabh Bhargava. If you have suggestions for what we should be covering in this newsletter, tweet us @mlopsroundup or email us at mlmonitoringnews@gmail.com. If you like what we are doing please tell your friends and colleagues to spread the word.

Welcome to the 21st issue of the MLOps newsletter. 

In this issue, we cover some tips from Google Cloud on choosing the right MLOps capabilities, share what continuous delivery for ML systems looks like, deep dive into the performance of language models over time, discuss the implications of AWS terms of service, and much more. 

Thank you for subscribing. If you find this newsletter interesting, tell a few friends and support this project ❤️

Google Cloud Blog | Getting started with MLOps: Selecting the right capabilities for your use case

In a previous edition of the newsletter, we covered VertexAI, a managed machine learning platform on top of the Google Cloud Platform for companies to train, deploy and maintain their AI models. As a follow-up, Google has released a structured framework to think through what a mature production MLOps stack can look like. In this article, Christos Aniftos shares valuable insights about how to map the various capabilities on offer to your company’s use cases. 

• Pilots: These use cases are about rapid iteration and testing a proof of concept. The capabilities most helpful for such applications are experimentation tracking and data processing & transformation capabilities.

• Production/Mission Critical: These applications are on the critical path of serving customer needs and creating business value. Failures can have a significant negative impact (legal, ethical, reputational, or financial risks). In such cases, robust offline evaluation is important (to identify poor performance or potential bias before a model goes live). Additionally, production monitoring is critical to keep an eye on post-launch performance

• Reusability & Collaboration: In cases where the same data source, feature sets, or model architecture powers multiple production ML use cases, it is helpful to standardize and have a single source of truth for these assets. E.g. a feature store helps the processes of registering, storing, and serving features for multiple ML models. 

• Ad-hoc vs Frequent retraining: For use cases where ML models are not recurringly trained, production monitoring is critical to detect drifts, outliers, and anomalies. For use cases where recurring training is part of the ML workflow, it is important to have end-to-end and reproducible ML workflows that stitch together various parts of the ML development and evaluation lifecycle. Additionally, online experimentation might be required too depending on your use cases.

The article goes into a few more examples and we recommend reading it. Our takeaway from the article, which we have expressed previously when covering the MLOps ecosystem as well, was that there isn’t a “one-size fits all” solution stack for production machine learning. Given multiple companies and open source tools specializing in one or few aspects of the ML development cycle, and the need for flexibility, we think initiatives like AI Infrastructure Alliance can help define a common standard of interoperability among these products in the long term.

Martin Fowler Blog | Continuous Delivery for Machine Learning

The process for developing, deploying, and improving ML applications is more complex compared to traditional software. Continuous Delivery has been the approach to bring automation and standardization to create a reliable process to release software into production. In this article, Martin Fowler discusses Continuous Delivery for Machine Learning (CD4ML), the discipline of bringing Continuous Delivery principles and practices to Machine Learning applications.

Why is it hard?

ML applications have three underlying components that can cause the end output to change:  the code (application code), the model (learned parameters), or the data (training or validation dataset used for model training). Their interaction is often complex and hard to predict which is what makes the problem harder compared to traditional software.

Components of CD4ML

1. Discoverable and Accessible Data: it is important that the data is easily discoverable and accessible. The harder it is to find the data, the longer it will take for them to build useful models. This means having a well-defined and functioning infrastructure that makes the flow of logging/production data to a warehouse, data lake, or traditional database.

2. Reproducible Model Training: It is critical to be able to reproduce and version artifacts in the ML training workflow (e.g. data clearing & transformation, splitting into training vs validation sets, etc). Tools like DVC (Data Science Version Control) and Pachyderm that we have covered previously can be a possible solution to solve this problem. 

3. Model Serving: There are three possible ways to deploy a model to do predictions online - embedded in the end application, deployed as a service (exposed as an API that is called by the consuming application), or published as data (which is ingested as a data stream by the consuming application). 

4. Testing: This step is equivalent to running quality checks, unit & integration tests for software releases. However, in the case of ML applications, it is important to test not just the code but also the model and data. E.g. data validation checks to ensure schema consistency or statistical properties; model quality with validation checks with predefined unit test cases or offline performance thresholds. 

5. Experiments Tracking: To understand the performance of various experimental models that might be running online and their associated stability and performance. Tools like MLflow that we have covered previously can be a good option to accomplish this

6. Model Monitoring: Once the model is deployed in production, we need to understand how it performs and close the data feedback loop. This consists of logging model inputs and predictions and creating relevant alerts, detecting outliers and drifts, monitoring various sub-populations of the traffic for possible model bias, etc. 

The article takes the example of a sales forecasting application to walk through a possible CD4ML setup and we highly recommend checking it out in detail. You can also check out the associated code repository on Github.

Paper | Pitfalls of Static Language Modelling

This is a really interesting paper from researchers at Deepmind (one of whom is a subscriber to this newsletter!) that discussed the performance degradation of large language models over time. Given the explosion of large language models in the past few years (BERT, GPT-3, etc), and how excitedly they’re being adopted by practitioners, it’s important to understand their performance characteristics and potential failure points. 

The Problem

As the authors put it:

Our world is open-ended, non-stationary and constantly evolving; thus what we talk about and how we talk about it changes over time. This inherent dynamic nature of language comes in stark contrast to the current static language modelling paradigm, which constructs training and evaluation sets from overlapping time periods.

A language model is a probability distribution over utterances (such as words or characters) that is learned from a set of observations. A good language model assigns a high probability to an utterance in the future, such as predicting the next word in a sentence. 

Many NLP applications (such as question and answering systems) rely on language model pretraining and “require up-to-date factual knowledge of our ever-changing world.” Now imagine answering questions like “How many people in the world have died from COVID-19?” or “Who is the current President of the United States?” and you’ll see that the answer depends on when the question was asked. Using a language model that was trained on data from the past might be a recipe for failure. 

Findings

The authors trained language models on news and scientific datasets, and tested the models in two ways: first on documents that were in the same time period as the training data, and then on documents that were in the future, relative to the training data. For all datasets, they saw a significant increase in perplexity (which means that it was harder to predict the future utterance). 

Further, they observed that:

the model performs increasingly badly when it is asked to make predictions about test documents that are further away from the training period, demonstrating that model performance degrades more substantially with time.

They also noticed that the drop in performance was much more significant when trying to predict words that were nouns - both proper nouns and common nouns (imagine a newly elected politician), or when the documents were about rapidly changing topics like sports or when they discussed emerging topics like Covid or 5G. 

They also found that training even larger language models doesn’t address this problem. 

Solutions?

One way to solve this problem is to retrain the language models frequently, but this is an expensive proposition (as an example, training a model like GPT-3 costs a few million dollars). 

The authors report that a technique such as “dynamic evaluation” could help. Dynamic evaluation is a form of online learning that updates the parameters of an already trained model as new data becomes available. They find that this provides a benefit with successfully predicting emerging new words and topics. However, this might not be good enough -- dynamic evaluation might lead to the language model forgetting important concepts from the past (“catastrophic forgetting”). 

It’s best to be careful when using these language models, especially if they were trained a while ago.

Techmonitor Blog | AWS Customers are Opting in to Sharing AI Data Sets with Amazon Outside their Chosen Regions and Many Didn’t Know

This is a post from July 2020 that remains quite relevant today. It gets into the weeds of AWS service terms and agreements but the legal implications are super interesting. 

What happened?

As they report:

The cloud provider is using customers’ “AI content” for its own product development purposes. It also reserves the right in its small print to store this material outside the geographic regions that AWS customers have explicitly selected.

Digging into the service terms of AWS, “AI Content” is “Your Content that is processed by an AI Service”. AI Services include AWS Services such as Amazon Comprehend (which provides NLP algorithms such as Sentiment Analysis), Amazon Polly (text to speech), Amazon Transcribe (speech to text), Amazon Translate (machine translation), etc. 

AWS’s service terms allow them to not only store the data that folks run through such AI services but move it to different AWS regions (which has data privacy implications) and use that data for the improvement of AWS’s AI services. It’s unlikely that most users of such AI services are aware that their data is being used in such a way, and AWS’s terms also say that AWS’s customers are responsible for informing their End Users of such usage! Sneaky!

What can you do?

Well, you can opt out of such usage by AI services. If you are using AWS’s AI services on any sensitive data, you might want to ask your DevOps team to enable this policy.

Resources

New Library Alert: Dud

Dud is a tool for storing, versioning and reproducing large files alongside source code (and it has cool design principles). As the creators say:

Dud is heavily inspired by DVC. If DVC is Django, Dud aims to be Flask. Dud is much faster, it has a smaller feature set, and it is distributed as a single executable.

If you’re dealing with large datasets, and want a quick and easy tool, this might be it!

New Library Alert | Kats

In an earlier issue, we had shared a Python library for time series forecasting called GreyKite. Kats is a new Python library from Facebook that looks really neat - it provides forecasting, seasonality and outlier detection, and feature extraction capabilities for time series data.

Competition for data-centric AI by Andrew Ng

We have previously covered Andrew Ng’s talk about striking the right balance between data and modeling improvements in real-world ML applications. Landing.AI and DeepLearning.AI, two initiatives cofounded by Andrew Ng recently announced the Data-Centric AI Competition. Traditionally, ML competitions invite participants to innovate on model architectures while keeping the dataset fixed. However, this competition aims to incentivize innovation in the creation and curation of high-quality data for ML by inverting the format. We welcome this competition and look forward to covering the progress and innovations that come out of it in future editions of the newsletter.

“Machine learning has matured to the point that high-performance model architectures are widely available, while approaches to engineering datasets have lagged. The Data-Centric AI Competition inverts the traditional format and instead asks you to improve a dataset given a fixed model.”

Thanks

Thanks for making it to the end of the newsletter! This has been curated by Nihit Desai and Rishabh Bhargava. If you have suggestions for what we should be covering in this newsletter, tweet us @mlopsroundup or email us at mlmonitoringnews@gmail.com. If you like what we are doing please tell your friends and colleagues to spread the word.

Welcome to the 20th issue of the MLOps newsletter. 

In this issue, we will cover a McKinsey report on an AI playbook for executives, analyze a recent paper on curating data for NLP research, share some recent MLOps resources we’ve come across, and discuss ML for COVID detection.

Thank you for subscribing. If you find this newsletter interesting, tell a few friends and support this project ❤️

McKinsey Report | The Executive AI Playbook

We recently came across this interesting resource from McKinsey for executives who are thinking about AI and the impact it can have on organizations. While we would take the $$ numbers for AI opportunities with a grain of salt, there might be some compelling arguments you might want to use in your organization. 

For example, “breakaway” companies (organizations achieving better scale and value than others) are 13X more likely to spend >25% of their budget on IT. (Let us know if you’re able to get the budget for a new MLOps tool using this! 😃)

Or that “breakaway” companies are 3X more likely to have well-defined analytics roles and career paths. 

If you’d like to see more in-depth articles from McKinsey, read this article on market sizing for AI in different industries, this article on how to scale analytics and ML at your company, and this one on red flags with analytics in your organization.

AWS Blog | “Amazon SageMaker Named as the Outright Leader in Enterprise MLOps Platforms”

Well, it’s not uncommon for companies to toot their own horn. However, we don’t see it this blatantly from AWS, so we thought we’d share it.

AWS claims (well, it’s actually a marketing research company called Omdia) that Amazon Sagemaker is the top Enterprise MLOps platform today. It’s no secret that Sagemaker is a great ML platform, and the speed at which they’ve added functionality is breathtaking. Over the course of four years, they’ve managed to add all the features you can see below.

This is what the marketing research report had to say:

“AWS is the outright leader in the Omdia comparative review of enterprise MLOps platforms. Across almost every measure, the company significantly outscored its rivals, delivering consistent value across the entire ML lifecycle. AWS delivers highly differentiated functionality that targets highly impactful areas of concern for enterprise AI practitioners seeking to not just operationalize but also scale AI across the business.”

Our Take:

This report only does a comparison between the cloud ML platforms and the end-to-end ML platforms. We are seeing a lot of new MLOps tools which are very specialized (just for labeling, or deployment, or monitoring), and we expect teams to make the best decisions for their very unique circumstances. This might naturally lead to teams picking the best-in-breed solution for each part of the ML workflow, rather than a generic ML platform. We are excited to see how this market develops.

Paper | Changing the World by Changing the Data

We have previously covered Andrew Ng’s talk about striking the right balance between data and modeling improvements in real-world ML applications. Here we cover a related paper by Anna Rogers arguing about the importance of data curation for robust, inclusive, and secure NLP models. 

NLP community is currently investing a lot more research and resources into the development of deep learning models than training data. While we have made a lot of progress, it is now clear that our models learn all kinds of spurious patterns, social biases, and annotation artifacts. 

Arguments for curating data

• Bias: Human written text contains all kinds of social biases based on gender, race, religion, class, age, etc. Models can learn these biases and even amplify them if we don’t curate the datasets that models learn from

• Privacy: Memorization of training data is a known issue in machine learning, including possibly personally identifiable information. This can be a privacy risk.

• Performance gap with respect to human-level NLU: The distributions of data in the current NLP resources such as web texts do not seem to provide enough signal for current models to do human-level language understanding

• Security: Certain concerns from adversarial attacks could be mitigated by having greater control over datasets.

Arguments against curating data

• Faithfully representing the world as it is: A language model should reflect how the language is used in the real world. Any data curation means that the input distribution to the model does not faithfully reflect the real world.

• Dataset is already the entire data universe: There are cases where the training data is not drawn from some distribution but represents the entirety of the data universe.

• An algorithmic approach to correct biases: Perhaps the way to tackle models learning biases is not to curate the data but to curate model training

• Against the long-term direction of AI: As the paper describes it, “The great promise of DL (Deep Learning) was to stop trying to define everything, and let the machine to identify and leverage patterns from huge datasets”. Explicitly curating data seems to go against this ethos.

Our take

Irrespective of how one might feel about the role of data curation in NLP (and ML in general), let’s first agree on the desired outcome: we do want more robust, capable, and generalizable models, and we do not want models to learn/amplify stereotypes and biases. At the end of the day, most ML practitioners are pragmatists and will adopt any technique so long as it helps solve these problems at a reasonable cost. We share the author’s view: in most real-world ML applications, the constraints are about data quality & quantity.

New Resources for MLOps

We’ve recently come across a few useful resources that we’d like to share with our readers:

• MLOps course: We have shared MadeWithML in a previous newsletter, and here we wanted to highlight a new course on their website focused on all aspects of “ML in production”. It is fairly hands-on, accompanied by tutorials, code snippets, and relies on open-source tools. We’d recommend checking it out if you are looking to get your hands dirty with anything MLOps.

• Practical MLOps Book: This book, written by Noah Gift and Alfredo Deza, takes you through what MLOps is, how it differs from DevOps, and how to put it into practice to operationalize your machine learning models. 

• MLOps World 2021 Event: The second annual MLOps World event is being held virtually this year from June 14-17. In addition to invited talks, we are also looking forward to the interactive workshops. You can check out the full list here.

New Product Alert | Bodywork

Bodywork looks like a really useful product to help deploy ML projects to Kubernetes. Many of the challenges involved with bringing ML projects to production are DevOps related, and tools that can automate the deployment process with simple APIs or commands are very handy. 

Here is a very well-written article by Alex Ioannides, one of the creators of Bodywork, that shows how a probabilistic model can be “trained” using Bayesian inference and then easily deployed using Bodywork. We learned some fun, new ML concepts by reading through this post!

Twitter: Can you detect COVID-19 using Machine Learning?

We covered this story in an earlier issue, but it’s important enough to discuss again.

Here is an insightful thread by Vladimir Haltakov, a machine learning engineer based in Germany, about the challenges with applying Machine Learning to detect COVID-19 from data such as X-rays and CT scans (many of these learnings can be generalized to healthcare more broadly). Three key takeaways for us were:

1. Researchers from Cambridge considered 415 papers on the topic published from January to October 2020. 0 had any potential for clinical use!

2. Most papers examined had problems with the quality and quantity of data - datasets were small, unbalanced and in some cases biased (e.g. Some papers used a dataset that contained non-COVID images from children and COVID images from adults).

3. For ML practitioners and medical professionals alike, if you’re interested to explore the applicability of machine learning to COVID detection, this Kaggle challenge is a good start.

Thanks

Thanks for making it to the end of the newsletter! This has been curated by Nihit Desai and Rishabh Bhargava. If you have suggestions for what we should be covering in this newsletter, tweet us @mlopsroundup or email us at mlmonitoringnews@gmail.com. If you like what we are doing please tell your friends and colleagues to spread the word.

Welcome to the 19th issue of the MLOps newsletter. 

In this issue, we will cover an insightful perspective on the MLOps tooling landscape, dive into a recent announcement from Google, discuss explainability in real deployments, share the news on a proposed piece of legislation, and much more.

Thank you for subscribing. If you find this newsletter interesting, tell a few friends and support this project ❤️

Lj Miranda Blog | Navigating the MLOps tooling landscape

Here, we will be summarizing Lj Miranda’s wonderful 3-part series on navigating the MLOps tooling landscape (Part 1, Part 2, Part 3). He starts by looking into who these tools are for, then categorizes the tools into a neat 2x2, and finally lays out a framework for deciding what tools to use and why. 

Who are these tools for and what is the eventual goal?

Miranda focuses on two personas: the ML Researcher and the Software Engineer. The ML Researcher wants to focus on training models and creating new features, which is what she is trained for. The software engineer, on the other hand, wants to have a seamless process for getting models into production. 

The software engineer is dealing with the loop on the left, while the ML Researcher is dealing with the loop on the right. Together this constitutes the ML lifecycle, and the aim for MLOps tooling is to improve this lifecycle. 

How do you categorize these MLOps tools? 

Miranda looks at two axes to divide up all MLOps tools. First, what type of artifact do they help produce -- software or models? Second, what is the scope of the tool -- piecemeal, that is affecting only one or two processes in the ML lifecycle or all-in-one, providing an end-to-end ML solution? 

This leads to four quadrants:

• Cloud Platforms: This includes general-purpose cloud providers such as AWS, GCP, Azure along with other Big Data focused platforms such as Cloudera and Paperspace. Miranda also includes Cloud-based ML Platforms such as AWS Sagemaker here, although we would choose to include them in the next category. 

• ML Platforms: This includes tools that address multiple components in the ML lifecycle such as ClearML, Valohai, and orchestration framework specifically addressing the ML process, such as Kubeflow and Metaflow. 

• Specialized ML tools: These are tools that address a very specific component in the ML lifecycle. Weights and Biases for experiment management, DVC for data versioning, Prodigy for data annotation, etc. 

• Standard SWE tools: This includes orchestration tools such as Airflow and CI/CD tools such as Jenkins. 

How to pick MLOps tools?

Miranda introduces the Thoughtworks Technology Radar as a methodology to decide which tools to adopt, which ones to trial, which to assess, and which to hold out on. We’ll let you read the post if you’re interested in the full analysis but will share his build vs buy criteria here. 

• Don’t build MLOps tooling if it isn’t your core business -- leave it to the companies focusing 100% of their effort on the problem. 

• Build integrations and connectors between tools -- this allows you to personalize your tooling choice for your business use cases and best utilize this nascent industry. 

• Buy specialized ML tools first -- they are easier to plug in and out and work best with existing workflow in your organizations. This is sage advice!

Google Cloud launches Vertex AI, a managed Machine Learning platform

At Google I/O this year, Google announced Vertex AI, a new managed machine learning platform on top of GCP for companies to train, deploy and maintain their AI models. Andrew Moore, vice president, and general manager of Cloud AI told Techcrunch: 

“We had two guiding lights while building Vertex AI: get data scientists and engineers out of the orchestration weeds, and create an industry-wide shift that would make everyone get serious about moving AI out of pilot purgatory and into full-scale production”

Official VertexAI documentation on Google Cloud provides a good introduction to Vertex AI feature sets, pricing, etc and we recommend reading it if you are considering building new ML models or migrating your ML models to the cloud. We share some key highlights below: 

• Training with minimal code: With the help of AutoML ML engineers and data scientists can build models in less time. Additionally, companies can take advantage of a centrally managed registry for all datasets across data types (vision, natural language, and structured data).

• Data quality: Vertex Data Labeling is a data labeling service to generate custom labels for collecting data to train models.

• Support for open source ML frameworks: Vertex AI integrates with frameworks such as TensorFlow, PyTorch, and scikit-learn via custom containers for training and prediction.

• MLOps: With services like Vertex Pipelines (for continuous model monitoring) and Vertex Feature Store (for serving and sharing ML features across models) companies can deploy and maintain ML models in production.

GCP has had support for most of the features announced as part of VertexAI even earlier (e.g. AI platform pipelines, AI platform data labeling). In part, this announcement is a rebranding exercise to bring these disparate services under the same umbrella and improve interoperability among them. Overall though, we quite welcome Google’s announcement especially related to MLOps. This suggests to us that model monitoring and maintenance is an important priority for Google Cloud AI.

Paper | Explainable Machine Learning in Deployment

We have discussed explainability in previous issues, such as here when we talked about using k-NNs for explaining and improving model behavior. This week, we cover a paper that explores how organizations use explainability in their day-to-day operations. 

What is this Paper? 

In this paper, we explore how organizations have deployed local explainability techniques so that we can observe which techniques work best in practice, report on the shortcomings of existing techniques, and recommend paths for future research.

This was done by conducting interviews with fifty people from approximately thirty organizations.

How is Explainability Used Today? 

There are different sets of stakeholders who require explainability:

• Executives: High-level executives often desire explainability as a goal for their systems, but it can end up being an item to be crossed off for their team. 

• ML Engineers: These end up using explainability to debug what the model has learned or inspect how it performs for certain data points. 

• End Users: This is the most intuitive consumer of an explanation (since they are directly impacted by the model predictions) and more explainable models lead to a more transparent product. 

• Other stakeholders: This could include regulators, domain experts, annotators, etc

Some of the needs being fulfilled by explainability today:

• Model debugging: Data scientists want to understand why a model performs poorly on certain inputs, or if adding new features and removing old features would improve performance. 

• Model monitoring: Teams often want to understand whether drift in certain features would impact their model performance (and knowing feature importance is key). 

• Model transparency: Explanations increase model transparency, giving end users more trust in the product while satisfying regulations. This can help communicate with business teams and respond to customer complaints. 

• Model audit: In financial organizations, all deployed ML models must go through an internal audit to satisfy regulatory requirements such as SR 11-7. Explainability can provide guidance on “conceptual soundness” by evaluating the model on multiple data points. 

Takeaways and Concerns

The judgment of domain experts (ie labels from experts) is still considered ground truth. Explanation-based methods have a long way to go -- these can suffer from spurious correlations (something we covered previously here) and causal understanding remains challenging.

ML Engineers mostly use explainability techniques to:

identify and reconcile inconsistencies between the model’s explanations and their intuition or that of domain experts, rather than for directly providing explanations to end users.

There are other technical challenges -- computation of explanations can be slow (exponential in input dimensions, in the case of Shapley values), it can be tricky to produce feasible counterfactual explanations and certain explanations can lead to privacy risks (by exposing details about the model behavior or training data). 

Finally, being a new discipline, organizations are yet to figure out the right frameworks for deciding how they’re going to use explainability, who is it useful for, and when. We expect to see continued research in this field and will continue to cover stories of adoption when we can.

Regulation | The Algorithmic Justice and Online Platform Transparency Act of 2021

Senator Edward J. Markey and Congresswoman Doris Matsui recently introduced the Algorithmic Justice and Online Platform Transparency Act of 2021, a copy of which can be found here. The legislation specifically relates to algorithms used by companies like Facebook and Twitter to determine which content and advertisements to show to users. Congresswoman Doris Matsui noted in the above article: 

“For far too many Americans, long-held biases and systemic injustices contained within certain algorithms are perpetuating inequalities and barriers to access. The Algorithmic Justice and Online Platform Transparency Act is an essential roadmap for digital justice to move us forward on the path to online equity and stop these discriminatory practices. I look forward to working with Senator Markey and urge all of my colleagues to join us in this effort.”

Highlights

Key highlights of the legislation can be categorized into two buckets. It should be noted that in the bill and the highlights described below, the term “algorithm” is really more like a machine learning model (or ensemble of models) combined typically with some business rules that typically constitute large-scale data products such as search engines, news feeds, etc.

(1) Preventing Harm to Users:

• Prohibit platforms from using algorithmic that discriminate on the basis of protected categories such as race, age, and gender

• Platforms may not employ algorithms if they fail to take reasonable steps to ensure these algorithms achieve their intended purposes.

• Create an inter-agency task force in the government to investigate potential discriminatory algorithms used online.

(2) Transparency: 

• Platforms will be required to explain to users how they use algorithms and what data is used to power them.

• Platforms will be required to maintain details of how they build algorithms for review by the FTC.

• Platforms will be required to publish annual reports of their content moderation practices.

Connection to Section 230

As noted in this article, this legislation is a fresh approach to bringing more accountability in platforms’ policies and processes around content moderation by not touching Section 230. While there is agreement across the political spectrum about Section 230 needing reform, the details surrounding the reform have become a contested partisan issue. By not touching or seeking to reform Section 230, this bill might have a better shot at getting support across the aisle.

Anthropic: An effort to build reliable, interpretable and steerable AI systems

Anthropic, an AI safety research company, recently came out of stealth with an announcement about their mission & funding (a $124M Series A!!).

Problem

Large AI systems today can also be unpredictable, unreliable, and opaque. As stated in the announcement above, Anthropic’s goal is to “make progress on these issues”, primarily focusing on research. Some areas that are within the scope of their research include natural language understanding, reinforcement learning, and interpretability.

Team

While the announcement was light on Anthropic’s research and product plans, they have a stellar team of scientists and engineers many of whom have worked at OpenAI and Google Brain previously on initiatives like GPT-3, Circuit-Based Interpretability, Multimodal Neurons, and Scaling Laws. We wish the team success on this journey and look forward to covering their research in the coming months.

Learn about Explainability

Here’s a great compilation of resources on explainability from Hima Lakkaraju, Assistant Professor at Harvard University.

Thanks

Thanks for making it to the end of the newsletter! This has been curated by Nihit Desai and Rishabh Bhargava. If you have suggestions for what we should be covering in this newsletter, tweet us @mlopsroundup (open to DMs as well) or email us at mlmonitoringnews@gmail.com

If you like what we are doing please tell your friends and colleagues to spread the word.

Welcome to the 18th issue of the MLOps newsletter. First things first, we are excited to show you some design changes to the newsletter. We have moved away from the default Substack orange, and have a newly designed logo (which you can see above) with its own dark version on Twitter. Let us know how you feel about it by replying to this email or commenting on the Substack post. ✏️

In this issue, we share a new MLOps course and a virtual event where you can meet us, discuss the effects of long-tail events on ML systems, deep dive into research on teaching ML models to forget, and more.

Thank you for subscribing. If you find this newsletter interesting, tell a few friends and support this project ❤️

Andrew Ng on Coursera | Machine Learning Engineering for Production (MLOps) Specialization

In a previous issue of our newsletter, we covered Andrew Ng’s talk about MLOps where he outlines the importance of striking the right balance between data and modeling improvements in real-world ML applications. In partnership with Coursera and DeepLearning.AI, Andrew Ng is teaching a course focused on MLOps. We are very happy to learn about this development and highly recommend checking it out:

The Machine Learning Engineering for Production (MLOps) Specialization covers how to conceptualize, build, and maintain integrated systems that continuously operate in production. In striking contrast with standard machine learning modeling, production systems need to handle relentless evolving data. Moreover, the production system must run non-stop at the minimum cost while producing the maximum performance. In this Specialization, you will learn how to use well-established tools and methodologies for doing all of this effectively and efficiently.

Event Reminder | #datalift No 5 - Productionize data analytics and machine learning

#datalift is an initiative from the AI Guild which brings together a community of AI practitioners, and business leaders to help bridge the gap between proof-of-concept and productionization. They’re hosting an event on May 28 with interesting speakers and opportunities to network with folks interested in ML deployments. 

We’re excited to announce that the MLOps Roundup will have a virtual booth at the event -- come say hi to us if you’re attending!

DoorDash Blog | Improving ETA Prediction Accuracy for Long-tail Events

This is a really interesting blog post from the Doordash Engineering team that discusses the impact of long-tail events on a specific ML system at Doordash and how they improved their system to address this problem. 

What are long-tail events?

Long-tail events are often problematic for businesses because they occur somewhat frequently but are difficult to predict. We define long-tail events as large deviations from the average that nevertheless happen with some regularity. Given the severity and frequency of long-tail events, being able to predict them accurately can greatly improve the customer experience. 

In the context of Doordash, their customers see an ETA before they get food delivered. This ETA sets an expectation for their customers for when their order will arrive, and while this system works well for most cases, a small number of late deliveries can have an outsized negative impact (we all know what it’s like to be “hangry” - remember the Snickers commercials?). 

The post makes a distinction between tail events and outliers:
Outliers tend to be extreme values that occur very infrequently. Typically they are less than 1% of the data...On the other hand, tail events represent occurrences that happen with some amount of regularity (typically 5-10%), such that they should be predictable to some degree.

Challenges with tail events?

Given that tail events happen 5-10% of the time, there is a sizable opportunity in improving the ML system for these events. 

However, it’s challenging because there is often not as much ground truth or factual information for an ML model to learn generalized patterns. It can also be difficult to obtain leading indicators that are correlated with the occurrence of a tail event. A simple example of this is in the case of an online retailer, a social media post from an influencer might cause a sudden spike in the demand for a product. It is almost impossible to predict something like this. 

For Doordash, they could choose to always overestimate the ETA time as a safeguard against tail events, but this hurts revenue. Many people will choose not to get food delivered if ETA crosses some threshold. So the only recourse is to improve predictions on tail events. 

Solutions

The Doordash team tried a few different options to address the challenges. First, they used bucketing and target encoding (read this good intro to target encoding) for certain continuous-valued features (like marketplace health). This gave the model an easier path to learning the effect of such features on the target value. 

Second, they introduced real-time features that captured real-time signals about the outcome they cared about. 

For example, we look at average delivery durations over the past 20 minutes at a store level and sub-region level. If anything, from an unexpected rainstorm to road construction, causes elevated delivery times, our ETAs model will be able to detect it through these real-time features and update accordingly.

Finally, they tweaked their loss function from a linear to a quadratic loss function, which penalizes large deviations much more strongly. 

Results

Based on the experiment results, we were able to improve long-tail ETA accuracy by 10% (while maintaining constant average quotes). This led to significant improvements in the customer experience by reducing the frequency of very late orders, particularly during critical peak meal times when markets were supply-constrained.

The lessons they share:

First, investments in feature engineering tend to have the biggest returns. 

Secondly, it’s helpful to curate a loss function that closely represents the business tradeoffs.

Facebook AI Blog | Teaching AI to Forget at Scale

A Transformer is a type of deep learning model architecture that utilizes the mechanism of attention to selectively focus on certain parts of the input that the model thinks are relevant for making the final prediction. While transformers have revolutionized language understanding, they suffer from the inability to scale to large pieces of text, primarily because of computational costs. Facebook recently published new research (paper, code) that allows models to “forget” or expire information it has learned in the past that might no longer be relevant, thus allowing attention mechanisms to scale to much longer sequences of inputs.

What is it?

In the paper above, authors introduce Expire-Scan, a new technique to allow neural networks to expire information it has learned in the past that might no longer be relevant. While the problem this paper solves has long been acknowledged,  the main challenge so far has been that “expiring a given piece of information” is a discrete operation i.e. not differentiable. Expire-Span assigns an expiration value to each hidden state it has learned in the past and recomputes this value at each time step. In this way, the expiring information is a learnable parameter in the model.

Our take

As mentioned in the paper and in the article above, Expire-Scan is promising to scale attention to long pieces of text (or sequential inputs more generally - e.g. image frames in a video). It is also quite interesting to consider potential use cases of “memory expiration” for problems such as models encoding bias of the underlying datasets. For example, is it possible to set up training loss so that models learn to forget spurious (and often biased correlations) it has learned from data? Can we improve generalizability and tackle data/concept drifts by ensuring models forget information that might no longer be relevant? This is an exciting and novel idea, and we look forward to future research directions that build upon this work.

New Tool Alert | GreyKite from LinkedIn

LinkedIn recently open-sourced Greykite (link to Github), a Python library for time series forecasting. As part of the library, they are also open-sourcing the core algorithm used for predictions based on time series data called ‘Silverkite’ (link to the paper if you’re interested). The authors have shown this algorithm to work well for cases like time-varying trends and seasonality, which we assume LinkedIn has to deal with quite frequently. The open-source article linked above, and the Github repo are quite comprehensive and deep-dive into multiple use cases. We recommend checking it out if you’re dealing with time-series data!

Twitter Fun | Types of ML/NLP papers

“All memes are wrong, but some are interesting” (or was it “All models are wrong, but some are useful”?). This one was too real to not share! 😃

Thanks

Thanks for making it to the end of the newsletter! This has been curated by Nihit Desai and Rishabh Bhargava. If you have suggestions for what we should be covering in this newsletter, tweet us @mlopsroundup (open to DMs as well) or email us at mlmonitoringnews@gmail.com

If you like what we are doing please tell your friends and colleagues to spread the word.

In this issue, we cover some recent developments on AI regulations, discuss the similarities and differences between data warehouses and feature stores, and share our thoughts on a recent paper about dataset generation using language models.

Thank you for subscribing. If you find this newsletter interesting, tell a few friends and support this project ❤️

A Quick Update on the EU AI Regulations

In our last issue, we had discussed the AI regulations that the EU had been considering. This week, the EU Commission released its proposed AI Framework to address the risks associated with AI. It still needs to be adopted by the European Parliament and member states, but it is likely to pass through successfully. However, if this follows a path similar to GDPR, it will be a few years before it is enforceable (the earliest estimate is the second half of 2024). 

From Reuters, European tech chief Margrethe Vestager:

"On artificial intelligence, trust is a must, not a nice to have. With these landmark rules, the EU is spearheading the development of new global norms to make sure AI can be trusted"

Our very short take

AI risks are very real and need good regulation. That being said, we do need to be careful about poorly written regulation, and we hope that there will be significant discussion in the next few months and years about AI risks. 

FTC | Aiming for truth, fairness, and equity in your company’s use of AI

The FTC recently provided guidance to companies on building AI applications with an aim for truth, fairness, and equity, that we wanted to share with our readers. This guidance comes almost a year after a previous FTC recommendation about the use of AI technologies and shows perhaps that the topic continues to be a high priority. 

Existing laws

The guidance includes a reminder on the three laws that are important to this conversation:

(1) Section 5 of the FTC Act. The FTC Act prohibits unfair or deceptive practices. That would include the sale or use of – for example – racially biased algorithms.

(2) Fair Credit Reporting Act. The FCRA comes into play in certain circumstances where an algorithm is used to deny people employment, housing, credit, insurance, or other benefits.

(3) Equal Credit Opportunity Act. The ECOA makes it illegal for a company to use a biased algorithm that results in credit discrimination on the basis of race, color, religion, national origin, sex, marital status, age, or because a person receives public assistance.

Recommendations

In the guidance, the FTC also issued recommendations for companies to follow when building AI applications. For ML researchers and practitioners, these might not be very new but it is great to see these practices and recommendations get more visibility.  You can check out the full list of recommendations on FTC’s website but we share our thoughts below: 

• Focus on foundations: The guidance focuses on the importance of starting with datasets that are unbiased and representative of the world in which the models will operate

If a data set is missing information from particular populations, using that data to build an AI model may yield results that are unfair or inequitable to legally protected groups.

• Focus on transparency and practices: The guidance focuses on the importance of transparency in how datasets and models are used, including third-party audits, unambiguous terms of service to inform users, and not making exaggerated claims about the model’s real-world performance. 

As your company develops and uses AI, think about ways to embrace transparency and independence … by conducting and publishing the results of independent audits, and by opening your data or source code to outside inspection.

• Cited examples: One thing that struck us while reading the article was the citations of bias in AI, and actions against these violations in the past: racial bias in the healthcare algorithms, a complaint against Facebook, and recent action against Bronx Honda car dealership. To us, these examples and the strong language associated with them, indicate that FTC is serious about enforcing these laws in the context of AI applications. 

Reactions

This series of tweets by University of Washington School of Law professor Ryan Calo is a good read. 

Medium | Feature Stores vs Data Warehouse

In a recent post, Jim Dowling highlights the similarities and differences between data warehouses and feature stores. The feature store is a data warehouse of features, used for training and inference of ML models. Both, data warehouse and feature store, are a central store of data and both have pipelines to ingest data from one or more sources. However, as noted in the post, there are some important differences in terms of architecture and use-cases.

Dual Use-Cases of Feature Stores

Unlike data warehouses, which are used typically for offline batch processing of data for gathering insights, reporting, etc, feature stores support a dual use-case:

• Offline use-case (column-oriented) for training models and offline batch inference

• Online use-cases (row-oriented) for online inference i.e. individual predictions.

Data validation in Feature Stores 

A data warehouse stores data in tables along with predefined schemas and column constraints. By contrast, feature stores typically have more relaxed constraints and at the same time, they can store more abstract data (e.g. an array of floating-point values as an input feature vector of a model). For feature stores, it is best to do data validation and quality checks before ingestion. The article mentions example data validation tools like Great Expectations, which we have covered earlier in our newsletter as well. 

As model training is very sensitive to bad data (null values, outliers cause numerical instability, missing values), feature data should be validated before ingestion

Feature Statistics Monitoring

As we have covered previously in our newsletter, data and feature drifts are important to monitor for real-world ML applications. Many feature stores can facilitate comparisons of online traffic against the training or validation sets for the detection of such drifts.

Event | #datalift No 5 - Productionize data analytics and machine learning

#datalift is an initiative from the AI Guild which brings together a community of AI practitioners, and business leaders to help bridge the gap between proof-of-concept and productionization. They’re hosting an event on May 28 with many interesting speakers and opportunities to network with folks who are interested in ML deployments. We’ll be attending and hope to see you there!

Paper | Generating Datasets with Pretrained Language Models

What’s the problem? 

One of the major challenges in machine learning (especially when working with unstructured data like text, images, videos) is getting good representations for data. In NLP, this has typically meant embeddings, which were popularised by the Word2Vec paper from 2013. While there have been many innovations (such as large pre-trained language models like GPT-3) that have changed the NLP landscape over the last few years, generating good sentence-level embeddings for downstream tasks remains difficult. 

As the authors say:

To obtain high-quality sentence embeddings from pretrained language models (PLMs), they must either be augmented with additional pretraining objectives or finetuned on a large set of labeled text pairs. While the latter approach typically outperforms the former, it requires great human effort to generate suitable datasets of sufficient size.

How do they solve this problem?

This paper introduces a method of leveraging pre-trained language models to:

generate entire datasets of labeled text pairs from scratch, which can then be used for regular finetuning of much smaller models.

This is done by prompting the language model (LM) in a very specific way (we’ll let you read the paper for full details) and generating tokens until full sentences have been produced. For example, as seen in the image above, the input could be the LM was the Task + Sentence 1, with Sentence 2 being the output. 

This is neat because the LM is producing example pairs of sentences in a completely unsupervised fashion, and the sentences are going to be fairly fine-tuned to your dataset (if Sentence 1 is from your dataset) while Sentence 2 is produced using the knowledge of the LM. 

What does this mean? 

While we won’t focus on the results from this paper too much, we believe that this is an interesting direction for the future. Companies are often dealing with large amounts of unlabeled data where human labeling would be expensive. If large LMs can somehow be leveraged to create datasets or provide labels with reduced human intervention, we could radically speed up the time-to-production for many projects.

Crossing the Deep Learning Chasm: Why the prototype to production journey is hard

We came across this series of tweets by François Chollet, creator of Keras, about why the prototype to productionization journey for deep learning applications is hard.

This struck a chord with us:

Every app demo based on GPT-3 follows this pattern. You can build the demo in a weekend, but if you invest $20M and 3 years fleshing out the app, it's unlikely it will still be using GPT-3 at all, and it may ever meet customer requirements

Thanks

Thanks for making it to the end of the newsletter! This has been curated by Nihit Desai and Rishabh Bhargava. If you have suggestions for what we should be covering in this newsletter, tweet us @mlopsroundup (open to DMs as well) or email us at mlmonitoringnews@gmail.com

Thanks again, and if you like what we are doing please tell your friends and colleagues to spread the word.

In this issue, we begin with some news out of the EU on AI regulation, followed by a fascinating paper on explainability using nearest neighbours. Next, we discuss a few talks from Scale’s Transform conference and deep dive into an insightful Andrew Ng talk on taking a data-centric view of AI. 

Thank you for subscribing. If you find this newsletter interesting, tell a few friends and support this project ❤️

The Verge | The EU is considering a ban on AI for mass surveillance and social credit scores

What happened?

From the Verge:

“The European Union is considering banning the use of artificial intelligence for a number of purposes, including mass surveillance and social credit scores. This is according to a leaked proposal that is circulating online…”

This would lead to certain use cases being regulated (similar to privacy rights under GDPR) by member states of the EU, and if companies sell prohibited software, they “could be fined up to 4 percent of their global revenue” (and not European revenue). Here is the draft of the regulation, if you’re interested. 

What do the regulations include?

• A ban on AI for “indiscriminate surveillance,” including systems that directly track individuals in physical environments or aggregate data from other sources

• A ban on AI systems that create social credit scores, which means judging someone’s trustworthiness based on social behaviour or predicted personality traits

• Special authorization for using “remote biometric identification systems” like facial recognition in public spaces

• The creation of a “European Artificial Intelligence Board,” consisting of representatives from every nation-state, to help the commission decide which AI systems count as “high-risk” and to recommend changes to prohibitions

Reactions

As mentioned in the Verge article, Daniel Leufer, Europe policy analyst at Access Now says:

“The descriptions of AI systems to be prohibited are vague, and full of language that is unclear and would create serious room for loopholes”.

From Twitter:

Our Take

We understand how difficult it is to write good regulation, especially in domains that involve new technologies, such as artificial intelligence. And while it is critical that AI/ML companies are regulated appropriately to protect consumers of AI software from harm, we do believe that it needs to be done carefully and thoughtfully. Poorly written regulation can itself have many ill-effects: difficult to enforce consistently, loopholes, push companies away from deploying in the EU, making the market anti-competitive for smaller companies, etc. 

We will be following this closely to see what changes are made to this draft proposal, with a potential announcement on April 21st. 

Andrew Ng Talk | MLOps: From Model-centric to Data-centric AI

Andrew Ng probably needs no introduction for the readers of this newsletter. He recently gave a talk about the importance of striking the right balance between data and modelling improvements in real-world ML applications. It is an hour long and filled with practical insights and learnings and we recommend watching it fully. Overall, the message and theme of this talk is music to our ears and one of our motivations for writing this newsletter.

Model v/s data-centric approaches to AI

• Much of the high visibility work, especially in research and academia, happens in the realm of improving model architectures, learning schemes etc whereas the benchmark training and test sets are standardized and static (think ImageNet, MNIST, SNLI dataset etc). This is the opposite of most real-world settings where the data is messy, noisy, ever-changing.  

• In Ng’s view, this has ensured we’ve had great progress in the realm of model-centric AI but we don’t place nearly enough importance on data collections & quality in most applications. 

How is MLOps different from DevOps?

• Real-world AI systems (Software 2.0) are a combination of Software and Data.

• In Andrew Ng’s view, the rising and nascent field of MLOps can play a role in AI systems that’s equivalent to DevOps in the traditional software world. 

• Unlike software 1.0 which is mostly feed-forward (build → package → deploy), AI systems have feedback loops which means that MLOps plays a critical role in establishing best practices and processes around establishing these feedback loops from post-deployment back to model training and data collection. 

MLOps as a way to systematize best processes around data quality

• In Andrew Ng’s view, the biggest value an MLOps team can provide is to ensure consistently high-quality training data.

• In this sense, MLOps plays an important role across the model development and deployment lifecycle: how to get enough training data? How to ensure label consistency? How to clean and transform data to improve model performance? How to track concept and data drifts in deployed models? 

James Le | Learnings from Scale Transform 2021

James Le always does an incredible job of capturing notes from the conferences that he attends (last year we had included his post on the Toronto ML Summit), and his latest post from Transform, Scale’s conference doesn’t disappoint. 

We recommend reading the entire post, but here we wanted to cover a couple of the talks that were interesting from an MLOps perspective. 

AI at Facebook Scale

This was a talk by Srinivas Narayanan, who is the Director, Applied Research at Facebook AI. The talk goes into a few different challenges faced by a company at Facebook’s scale.

Data Challenge: Scaling Training Data

One of the biggest challenges in building AI systems is getting the right and sufficient training data. Getting labeled data for supervised learning can be difficult, expensive, and in some cases, impossible. Facebook approaches this by focusing on techniques beyond supervised learning.

For example, at Instagram, FB was able to train an object-classification system using 3.5B images along with the hashtags there were shared with, which is a technique called weak supervision. This allowed them to leverage a much larger volume of training data, leading to state-of-the-art results. 

Similarly, they were able to use self-supervision (where the system learns to fill in the blanks for inputs with missing pieces) to train cross-lingual language models and audio models. Again, with a smaller amount of labelled data, they are able to achieve good results. 

Tooling Challenge: Building the AI Platforms

As can be seen in the image, Facebook has built out a rich roster of internal tooling to support their AI efforts. As James writes:

• On the left, you’ll see tools for preparing data in the right format.

• In the middle, you’ll see the pieces for building and training models — going bottom up all the way from hardware, whether it’s CPUs or GPU’s. These include frameworks like PyTorch that ease the model building environment, libraries that are specific to each domain, and the models that are used in products.

• And on the right, once you have the trained models, you have the right tools and systems for deploying them in production, whether it’s in a data center or locally on the device.

And to make the research to production flow smooth, Facebook invested heavily in PyTorch as a single framework. 

Bias Challenge: Creating AI Responsibly

As James writes:

Not introducing or amplifying bias and creating unfair systems is a challenge because it’s not as simple as using the right tools. Fairness is a process. At each step of the implementation process, there is a risk of bias creeping in — in data and labels, in algorithms, in the predictions, and in the resulting actions based on those predictions.

At each step, Facebook attempts to surface the fairness risks, resolve questions and document the decisions that were made. 

Applied ML at Doordash

Andy Fang, co-founder and CTO at DoorDash discussed a couple of case studies of how DoorDash uses AI to improve its business. The one that we’ll cover is their process for creating a rich item taxonomy. 

DoorDash has tens of millions of items across all its restaurants and tens of thousands of new items are added every day. These have to be sorted and categorized so that users can quickly understand the offerings on the DoorDash platform, and make the best (ie tasty) decisions for themselves.

DoorDash managed to achieve this with a rich taxonomy, models that categorize each item into this taxonomy and a human-in-the-loop system that reduces annotation costs while allowing them to grow their taxonomy efficiently. 

There are three critical rules for defining annotation tags:

• Make sure that there are different levels of item tagging specificity that don’t overlap. Let’s say for coffee, you can say it’s a drink, you can say it’s non-alcoholic, or you can say it’s caffeinated. Those are three separate labels that don’t overlap and categorization with each other.

• Allow annotators to pick “others” as an option at each level. Having “others” is a great catch-all option that allows DoorDash to process items tagged in this bucket to see further how they can add new tags to enrich their taxonomy.

• Make tags as objective as possible. They want to avoid popular or convenient tags — things that would require subjectivity for an annotator to determine.

Also, when defining tasks for human annotators, it is important for the tasks to be high-precision and high-throughput. 

High precision is critical for accurate tags, while high throughput is critical to ensure that the human tasks are cost-efficient.

DoorDash’s taxonomy + simple binary/multiple-choice questions help them achieve high precision for their models with less experienced annotators and less detailed instructions. Finally, a separate QA loop on the annotations maintains high annotation quality. 

Paper | Explaining and Improving Model Behavior with k Nearest Neighbor
Representations

This recent paper from Salesforce Research explores the idea of improving model explainability by searching for similar examples from the model’s training data. 

Highlights

• The paper proposes using k-nearest neighbors in some representation space to identify training examples that are most similar to the example that the model is currently producing a prediction for. These examples from the training set (and the labels associated with them) can be interpreted as the ones most responsible for the model’s prediction.

• In addition, the paper presents a generic implementation of the space in which k-NN should be computed for this application: the deep learned model’s output hidden layer representation. This intuitively makes sense in that this representation is learned specifically for the training task, and in this way, the meaning of “nearness” is learned contextually to the downstream task for which the model is doing inference. 

• As an additional benefit, the paper shows that oftentimes looking at nearest neighbors of misclassified examples can surface label noise (incorrectly labelled examples) in the training data.

Our take

Explainability is a problem that has received a lot of attention and visibility in the last few years, both by researchers and ML practitioners alike. And for good reason: increasingly we’re seeing applications of machine learning outside of their digital-only sandboxes and out into the physical world (think self-driving cars, healthcare, autonomous drones). The technique presented in this paper appeals a lot to us - it is conceptually simple to understand and offers a way to explain model predictions by reasoning by analogy. A couple of open questions/interesting directions to explore:

1. Can k-NN based explanation techniques be extended to tasks beyond classification, for instance, question answering, textual entailment or machine translation?

2. Can the representations learned for this kNN search have other applications in the domain of MLOps and monitoring? For instance, measuring data drift? 

Fun | Machine Learning Pipelines are hard

Thanks

Thanks for making it to the end of the newsletter! This has been curated by Nihit Desai and Rishabh Bhargava. If you have suggestions for what we should be covering in this newsletter, tweet us @mlopsroundup (open to DMs as well) or email us at mlmonitoringnews@gmail.com.

In this issue, we highlight a talk on self-driving cars at Tesla, discuss a partnership between Hugging Face and AWS, share a post on embedding stores, dive into a paper on ML-in-databases and more. 

Thank you for subscribing. If you find this newsletter interesting, tell a few friends and support this project ❤️

AI for Self-driving Cars at Tesla: The importance of feedback loops

Andrej Karpathy, Director of AI at Tesla, gave a talk last year at ScaledML Conference on how Tesla is solving the ML challenges to provide Full Self Driving in their cars. It is a fascinating talk, both in terms of concepts in deep learning and vision and the systems and engineering aspects of making this work at Tesla’s scale. We recommend listening to the talk. For this article, we want to especially focus on the data feedback loops that help Tesla collect what is arguably the best data on self-driving, and how it helps their models improve over time. 

Highlights

• Over the past decade, neural networks have repeatedly shown their ability to solve complex computer vision problems. However, when the same set of model architectures and training techniques are available to everyone, data is the most important variable in terms of improving visual recognition in self-driving cars

• In the limit, this means that if the “right datasets” can be collected, the models will successfully learn to recognize the things we need. The right data is not just about quantity but also quality: 

“What’s important is not just the scale of the dataset, but covering all possible use cases”

• Because of the size of the Tesla fleet driving in the wild, Tesla has the ability to collect this data at scale to solve the long tail of corner cases (occluded Stop signs, construction sites, potholes, etc) that are critical to solve but hard to get data for.

• Andrej mentioned that the platform allows the team to collect data when a driver’s actions disagree with the self-driving model’s predictions but also fanout to the fleet to collect “similar” images to the ones in a seed dataset: 

• This “Data Engine” is a feedback loop system that allows Tesla to know exactly where the models currently don’t do a good job, collect data at scale to solve it, and move on to the next problem in the long tail of problems to get to 99.999+ % reliability.

Our take

We’re big fans of the approach outlined by Andrej and the Tesla team in this talk. We believe that the data collection <> model improvement feedback problem is not just unique to cars or tesla but something that should ideally be solved for all real-world ML applications. We think this is broadly an unsolved problem (to the extent that it is even recognized as a problem) and why we’re excited and optimistic about the role MLOps and monitoring tools can play.

Hugging Face Blog | The Partnership: Amazon SageMaker and Hugging Face

What happened?

Hugging Face and AWS have announced a strategic partnership:

“to make it easier for companies to leverage State of the Art Machine Learning models, and ship cutting-edge NLP features faster.

Through this partnership, Hugging Face is leveraging Amazon Web Services as its Preferred Cloud Provider to deliver services to its customers.”

This is a big move from Hugging Face, less than a month after raising a $40M Series B round.

What does this involve?

First, Hugging Face and AWS Sagemaker will provide Hugging Face Deep Learning Containers optimized for PyTorch and TensorFlow training that will work well with different EC2 instances. This will also mean that Hugging Face models that are trained/fine-tuned with Sagemaker will only be charged by the number of seconds of compute used.

Second, there will be a Hugging Face extension to the Sagemaker Python SDK, which will simplify creating and managing training jobs in AWS. For a quick demo of what this looks like, check out this YouTube video.

Third, the Hugging Face extension to Sagemaker will work seamlessly with existing Sagemaker functionality for Data Parallelism, Model Parallelism and Hyperparameter tuning when training models. This extension will also simplify sending metrics into Sagemaker’s own metrics store or CloudWatch. 

What’s next?

They already have an integrated solution for training, so what could be next?

“We are working on offering an integrated solution for Amazon SageMaker with Hugging Face Inference DLCs in the future - stay tuned!”

Our Take

This is pretty exciting news. Transformer models in NLP show tremendous promise and Hugging Face and AWS Sagemaker are making it very simple to train such models. We would recommend reading through the article and skimming through some of the resources if you’re interested in learning more. 

In our last issue, we had highlighted the AI Infrastructure Alliance which had many startups coming together to build the canonical stack for ML (as an alternative to the offerings from the Big Cloud providers). This news from Hugging Face and AWS shows that the world of MLOps is in a bit of a free-for-all, which can only be a good thing for ML practitioners.

Neal Lathia | Embedding Stores

Neal Lathia, the Director of Machine Learning at Monzo, recently wrote this article on using “embedding stores” at Monzo. We share our takeaways and thoughts on the article here. 

Highlights

• When dealing with unstructured data (e.g. raw text, images, videos), training models from scratch is often not feasible in real-world scenarios because of two reasons. First, privacy considerations mean that we might not have access to the raw data. Second, unstructured data is incredibly high dimensional and training high-quality models on these takes more compute resources, larger datasets and more time. In such cases, embeddings (models that map the raw input to a low dimensional dense vector representation) can be a great tool.

• Pretrained models for text and images are available out of the box (e.g. torchvision.models, or Hugging Face). By exposing these models as an API endpoint, the team built a system to generate and log embeddings. 

• The team then trained the ML models for their end use case by using these embeddings as feature vectors. This ensured that the data was used in a privacy-compliant way and the model training and iterations were faster compared to training end to end models from scratch. 

• The tradeoff here, of course, is the inability to backpropagate into the embedding model to fine-tune the embeddings for their specific downstream tasks.

Thoughts & Open Questions

• The general approach highlighted in Neal’s article is a great summary of the best practices emerging in industry regarding content understanding: a generic “trunk” model that learns good general-purpose embeddings of raw content, combined with a multitude of task-specific downstream models that are built on top of the embedding representation and fine-tuned for the specific task (e.g. Copy.AI powered by GPT-3,  Monzo uses HuggingFace).   

• As alluded to in the article, embeddings don’t entirely solve the privacy concerns as they aren’t strictly one-way hashes. As has been shown in prior research (and something we covered here), it is possible to recover the raw input partially from embeddings in some cases so it is important to evaluate this explicitly for your individual use case.

• One limitation that we believe is yet to be solved (at least we’re not aware of a good solution) is a clean way to support versioning and experimentation on underlying embeddings, especially if multiple downstream models depend on it as input features. 

Paper | Deep Integration of Machine Learning Into Column Stores

This is a paper from the team that built DuckDB and MonetDB, two of the most exciting new databases of the last couple of years. If you’re interested in learning about DuckDB, check out this YouTube video from one of the creators. Now, let’s discuss the paper itself. The authors say that they:

“integrate unchanged machine learning pipelines into an analytical data management system. The entire pipelines including data, models, parameters and evaluation outcomes are stored and executed inside the database system. Experiments using our MonetDB/Python UDFs show greatly improved performance due to reduced data movement and parallel processing opportunities.”

What is the problem they are trying to solve?

The authors say that current ML workflows have the following problems when it comes to data management:

• Managing large datasets as flat files is error-prone and multiple people working on such datasets leads to further issues

• Loading data from structured data formats (such as CSV and XML) is inefficient, and often data needs to be loaded from multiple times

These problems can be solved by existing relational database management systems (RDBMS), but integrating analytical tools with databases has proven tricky. This is because:

• The standard approach of storing data on a separate database and communicating over a socket connection is a bottleneck with large amounts of data

• On the other hand, in-database processing techniques are cumbersome, and rewriting analytical pipelines into SQL remains a research problem. 

Contributions of the Paper

In this paper, they show:

• Classification models (such as the ones provided by scikit-learn) that can be trained within a column-store RDBMS using a combination of a Python User-Defined Function (UDF) and SQL (see image earlier for an example)

• Storage of models within the database, which can then further be used for testing and future predictions

• Performance benefits of running the end-to-end analytical workflow within the database. This is compared against reading raw data from files or a separate database, followed by pre-processing, model training and inference in a Python environment. 

Our Take

Given how much data is being stored in analytical databases (Snowflake, Amazon Redshift, Google BigQuery), there seems to be growing interest in bringing non-SQL workloads to these databases. BigQuery appears to be the furthest ahead today - check out the following example of creating a Deep Learning model using TensorFlow directly inside BigQuery:

However, whether such approaches will take off remains an open question. If you’re working on problems involving tabular data, this might be worth a try. We remain intrigued by these ideas and will follow developments closely.

Covid-19 + AI

This is a slightly depressing Twitter thread from Eric Topol. He discusses an article from Nature Machine Intelligence which shows that even with more than 2000 studies involving machine learning models on detection and prognostication of Covid-19 from Chest X-Rays and CT images, not a single one is of “potential clinical use due to methodological flaws and/or underlying biases.” 

While the enthusiasm to do interesting work in Machine Learning remains high, producing valuable research remains difficult. We hope that clear documentation and well-defined processes become the norm both in industry and academia!

Thanks

Thanks for making it to the end of the newsletter! This has been curated by Nihit Desai and Rishabh Bhargava. This is only Day 1 for MLOps and this newsletter and we would love to hear your thoughts and feedback. If you have suggestions for what we should be covering in this newsletter, tweet us @mlopsroundup (open to DMs as well) or email us at mlmonitoringnews@gmail.com

In this issue, we look at charts and numbers from the AI Index Report 2021, cover some fascinating research from OpenAI, news of the AI Infra Alliance, Vector Databases and much more. Thank you for subscribing. If you find this newsletter interesting, tell a few friends and support this project ❤️

 Artificial Intelligence Index Report 2021

This is an annual report from the Stanford’s Institute for Human-Centred AI. Clocking in at 222 pages, it spans everything from R&D and technical performance of AI, trends with AI education and ethics and the impact of Covid-19 on AI development. Here we’ll highlight a subset of topics, but we’d recommend skimming through the report if you have the time. 

Research and Technical Performance

• The number of AI publications grew by 34.5% from 2019-2020, compared to 1.9.6% from 2018-2019.

• Performance of generative AI systems (text, audio, images) has improved to a sufficiently high degree that humans have a hard time telling the difference between synthetic and non-synthetic outputs for certain applications. That being said, there has been a significant improvement in the results for the Deepfake Detection Challenge created by Facebook.

• Performance on image benchmarks seems to be flattening out, which suggests that harder benchmarks are needed (as an example, we saw 98.8% Top-5 accuracy on ImageNet when using additional training data)

• NLP continues to make massive gains against the state-of-the-art with advances like GPT-3, but also with systems achieving near-human performance on language understanding tasks such as SuperGlue. 

• AlphaFold from DeepMind made a significant breakthrough in the challenge of protein folding -- generally speaking, AI has had a major impact on biology and healthcare. 

AI and the Economy

• “Drugs, Cancer, Molecular, Drug Discovery” received the greatest amount of private AI investment in 2020, with more than USD 13.8 billion, 4.5 times higher than 2019.

• More private investment in AI is being funneled into fewer startups. 2020 saw a 9.3% increase in the amount of private AI investment from 2019 (compared to 5.7% in 2019 from 2018), though the number of newly funded companies decreased for the third year in a row.

• Brazil, India, Canada, Singapore, and South Africa are the countries with the highest growth in AI hiring from 2016 to 2020. 

• Despite the economic downturn caused by the pandemic, half the respondents in a McKinsey survey said that the coronavirus had no effect on their investment in AI, while 27% actually reported increasing their investment.

• Check the chart below for the growth in AI job postings in different industries:

AI Policy and National Strategies

• More than 30 countries have published documents discussing national AI strategy since Canada was the first to do so in 2017. 

• US Federal civilian (non-defense) agencies allocated ~1.1B USD for AI research and development, while some reports suggest that the Defense R&D spend on AI might be closer to 5B USD

• In the US, the most recently ended 116th Congress (January 3, 2019 – January 3, 2021) was the most AI-focused congressional session in history, with the number of AI mentions in legislation and congressional reports being more than triple of the 115th Congress.

Final Thoughts

At a very high-level in the world of AI, the chart seems to be going “up and to the right” along almost all meaningful dimensions. The next decade will remain an exciting time to contribute to AI research, applications and strategy.

Multimodal Neurons in Artificial Neural Networks

In the last issue of our newsletter, we briefly touched upon the multimodal neurons research by OpenAI. We cover this topic in more detail here. Recently, OpenAI released CLIP, a new state of the art visual understanding model that outperforms existing vision systems on datasets like ImageNet and ObjectNet. As a follow up, OpenAI shared their observations around the existence of “multimodal neurons” in CLIP - 

We’ve discovered neurons in CLIP that respond to the same concept whether presented literally, symbolically, or conceptually.

Typically, single neurons in an artificial neural network would fire for a visual cluster of ideas - e.g. “edge detectors”, “face detectors” etc. But this finding is novel, in that the neuron responds to a semantic cluster of ideas represented across a variety of forms (sketch, picture, text etc) by forming abstractions. However, as noted in the paper, the degree of abstraction in multimodal neurons can present new attack vectors and introduce sources of bias that haven’t manifested in previous systems.

Typographic Attacks

Authors observed that excitations of the multimodal neurons in CLIP can be controllable by its response to images of text. This introduces a simple attack vector they call “typographic attacks”: Fooling the model into classifying an image by overlaying adversarial text unrelated to the underlying image. For example, in the image shared above, ovulating “$$$” over an image of a poodle, the model is fooled into thinking the image is a piggy bank:

“The finance neuron [1330], for example, responds to images of piggy banks, but also responds to the string “$$$”. By forcing the finance neuron to fire, we can fool our model into classifying a dog as a piggy bank.”

Bias and overgeneralization

Another unintended consequence of abstraction is new sources of bias stemming from overgeneralization: the model learns associations between concepts because of the associations present in underlying training data. As noted in the article, some such associations uncovered during CLIP’s evaluation:

• “Middle East” neuron [1895] has an association with terrorism; 

• “immigration” neuron [395] fires when the input contains Latin America. 

• Neuron [1257] fires for both dark-skinned people and gorillas.

Our take 

Both, adversarial attacks like the typographic attack mentioned above, and bias from overgeneralization, present challenges to real world adoption. While attacks like the typographic attacks can be at least formulated as adversarial learning, problems arising from bias and overgeneralization are even more challenging: While the examples shared above give us anecdotal evidence of bias, it is hard to measure or quantify this because the exhaustive set of all possible biases is impossible to anticipate in advance. We thank the authors for sharing these insights, and agree with their view on the importance of building a robust toolkit to study interpretability:

We believe that these tools of interpretability may aid practitioners the ability to preempt potential problems, by discovering some of these associations and ambiguities ahead of time.

AI Infrastructure Alliance: Building the canonical stack for ML

What is it ? 

We’ve written in the past about the wide variety of tools and products that comprise the enterprise AI landscape today (check out the Awesome MLOps repo, MLOps Tooling landscape). As a first order approximation, we can consider two axes along which to categorize these solutions:

1. Part of the ML lifecycle (data labeling, model training, inference, monitoring) 

2. Breadth of support (tied to a single cloud, support across cloud providers, VPC etc)

With this cambrian explosion of tools, teams often struggle to stitch together the right set of tools to create an end-to-end ML training & deployment workflow. With the aim to tackle this issue, more than 20 startups are coming together to form the AI Infrastructure Alliance. Their goal is to define the “canonical stack” for enterprise Machine Learning - a set of common practices and standards for cross-platform support that these companies can build towards. 

Dan Jeffries (who works at Pachyderm) will serve as director of the alliance and has previously written about the problem motivation in this post on a Canonical Stack (CS) for machine learning. This article, reporting on the alliance quotes Dan Jeffries:

“ In a conversation with VentureBeat, Jeffries referred to the endeavor for small to medium-size businesses in AI as a “rebel alliance against the empire” that will serve as an alternative to offerings from Big Tech cloud providers, which he characterized as “building an infrastructure just to lock you in.””

Our take

We think the problem of defining common standards & practices, allowing startups to build towards interoperability is a real one. We believe if this gains adoption then over time, larger cloud providers like AWS Sagemaker will want to bake these standards into the platform’s ML offerings, thus leveling the playing field and reducing complexity for the end customer. All in all, a big accelerant for ML adoption. 

The alliance so far has laid out its goals in broad strokes. We look forward to more concrete technical details about the “canonical stack” and standards of interoperability surrounding it that are proposed.

We do wonder whether the Big Tech cloud providers (think Sagemaker and GCP Cloud ML) are really the “empire”? While they have MLOps offerings, there are startups both large and small that are trying to attack different parts of the pie. Will the “rebel alliance” be strong enough given that there are competitive dynamics between them? Will this alliance continue to be accepting of newer rebels? We have so many Star Wars themed questions.

Having said this, we continue to believe that AI & ML are going to create tremendous value over the next two decades and the journey is likely to be a positive sum for most participants. 

Domain-Specific MLOps: Tying Monitoring to Business Outcomes

Lina Weichbrodt, a Machine Learning engineer based in Germany, recently wrote an insightful article on the importance of domain-specific metrics when monitoring ML models. We recommend reading it in its entirety but share our key takeaways and thoughts.

The case for domain-specific metrics 

Monitoring ML models, much like monitoring any online software service, is important for a real-time “pulse check”. One question we need to answer as part of any ML monitoring solution is “what should we measure”. In addition to the already well-defined and well understood metrics around data, concept drift etc, Lina makes the case that it is important to track metrics that are as closely related to business/product outcomes as possible. For instance, at Spotify, the homepage personalization team tracked “Rank of a user’s most used carousel”: if a new model ranks the user’s favorite carousel low or a sudden drop in the rank occurs it indicates a problem.

Insights lie at the extremes 

When designing domain-specific metrics for monitoring ML models, it helps to think of possible failure cases and design metrics that can capture the occurrence of failures even though they may be infrequent. As an analogy from DevOps, we track not just the average or median latency but also the P95 and P99 latency as this can help detect problems earlier.

Metric Maturity Cycle

This insight holds not just for metrics related to ML monitoring, but really any metric you want to build and operationalize: it helps to think of the maturity cycle:

• Step1: Research & Analysis - first “implementation” of a metric. The goal is to verify correctness and that tracking it can be valuable if productionized.

• Step2: Offline/batch compute - typically how most metrics first get productionized. There is a data pipeline that runs at regular intervals to fetch the upstream data, compute the metric and store it for consumption downstream (in dashboards, email reports etc)

• Step3: Realtime - the ideal stage for any metric. Computation is real-time, often using a streaming data abstraction. Realtime metrics are often accompanied by triggers and alerts that can proactively notify relevant stakeholders in case of any outliers. 

New product: Vector database from Pinecone

We recently came across a company called Pinecone and the product that they're building, which is a vector database. This definitely feels like a missing piece of MLOps infrastructure. Often, data scientists are working with vectors and while vectors can be saved and retrieved relatively easily in relational and noSQL databases, a “vector database” can go above and beyond by providing similarity search as a service. Another analogy for this is: ElasticSearch for vectors (rather than raw text). 

While an ML team can build something similar in-house by deploying a library like Faiss or Annoy (which are both pretty good for nearest-neighbour search), if a company can provide a hosted version (and take care of sometimes painful fine-tuning with these libraries), that might be valuable. 

You can read a few of the examples (Movie recommendations or Image search) that the Pinecone team provides if you want to learn more

ML in Covid-19 Research: A sobering Twitter thread

Thanks

Thanks for making it to the end of the newsletter! This has been curated by Nihit Desai and Rishabh Bhargava. This is only Day 1 for MLOps and this newsletter and we would love to hear your thoughts and feedback. If you have suggestions for what we should be covering in this newsletter, tweet us @mlopsroundup (open to DMs as well) or email us at mlmonitoringnews@gmail.com

In this issue, we cover a wonderful article on feature stores, explore information leaks in embedding models, continue our coverage of the AWS Well-architected framework whitepaper, discuss the impact of Covid on banks and more.

As always, thank you for subscribing. If you find this newsletter interesting, tell a few friends and support this project ❤️

Eugene Yan | Feature Stores - A Hierarchy of Needs

This is a really insightful read from Eugene Yan. We covered Feature Stores last year, and since then, the chatter around Feature Stores has only grown. Back then, based on this blog post from Tecton, we mentioned:

A feature store is an ML-specific data system that:

• Runs data pipelines that transform raw data into feature values

• Stores and manages the feature data itself, and

• Serves feature data consistently for training and inference purposes

Eugene breaks down feature stores by viewing them through a “hierarchy of needs” lens. This means that feature stores fulfil several needs and that “some needs are more pressing than others” and thus have to be addressed first. Let’s look into these:

• Access needs: At the most basic level, a feature store needs to provide users access to feature data and other information about features. This makes it easier for data scientists to discover work done in the past, understand how/when to use features and experiment easily. 

• Serving needs: Next, the models need to be served in production and features need to be made available to these models with high throughput and low latency. This might involve the movement of feature data from analytical data stores (Snowflake) to faster, “more operational” data stores (Redis). There might be data transformation needs addressed in this layer as well. 

• Integrity needs: The feature data being served to models in production needs to be fresh, accurate and available at all times. Advanced features would allow point-in-time correctness (providing a snapshot of how the data looked at an earlier time) and an understanding of train-serve skew. 

• Convenience needs: Features stores should be simple to work with for data scientists and software engineers. It should be intuitive to use features during development and serve them in production. Features should be easy to discover and when distributions change, users should know about it. 

• Autopilot needs: Tedious work for feature engineering should be simplified — backfilling data, monitoring and alerting, etc. 

Discussion

We encourage you to read the original article if you want to dive deeper - it is superbly written and provides enlightening examples from major feature store deployments (Gojek, Doordash, Airbnb, Monzo, Uber, Netflix and others). However, here are a few things we find particularly interesting. 

Feature Sharing

At Gojek, Eugene says:

Data engineers and scientists create features and contribute them to the feature store. Then, ML practitioners consume these ready-made features, saving time by not having to create their own features. 

But then he goes on: 

Not sure how I feel about this productivity gain, given my preference for data scientists to be more end-to-end.

This resonates with us. There is a balance between data scientists fully relying on available features as “raw data” vs experimenting, transforming data based on their intuition and really owning the models they build. We wonder if there are factors such as company size or the number of unique models owned by a team or the cost of recreating features that play a part in how much “feature sharing” is prevalent. 

Development vs Production data

A common pain point for data scientists working in dev environments is that they don’t have access to production data. This leads to issues such as train-serve skews going unnoticed, or worse, data leaks in features. Having a feature store that can bridge these gaps is a huge win. 

What is Day 1 vs Day 100 for Feature Stores? 

There are open questions about when should teams start thinking about “feature stores”? There are aspects of a feature store (access to features, serving them in models) that are necessary when deploying a model, but should a team building their first model really think about buying/building a feature store? 

Companies with relatively well-developed ML teams have likely thought about several of these needs already and might have their “unique” infrastructure that supports them. Is it even reasonable for them to integrate an external solution? Or is it worth it? If so, what is the path for them? 

We would love to hear from you if you have thoughts on this!

Paper | Information Leakage in Embedding Models

Background

Embedding models learn mappings from the space of raw input data to an n-dimensional space (typically much lower than the dimensionality of the input), while preserving important semantic information about the inputs. Learning pre-training embeddings from large amounts of unlabeled data (often called “trunk models”) that can then be fine-tuned for downstream tasks has become a common practice in achieving state-of-the-art results on a variety of machine learning tasks. 

The problem

This paper demonstrates that embedding models can often learn vector representations that leak sensitive information about the raw data. It introduces three classes of adversarial attacks to study the kind of information leaked by embedding models:

1. Inversion Attack: Think of embedding model as a functional mapping F: input data → n-dim output space. Embedding vectors can be inverted to partially recover the input data, given the embedding. Two types of attacks are studied under this scenario: white-box attack (where embedding model’s parameters are known) and black-box attack (where we only have the input raw data and output embedding representations). The paper shows that popular sentence embedding models like BERT allow recovery of between 50%–70% of the input words! 

2. Attribute Inference Attack: Oftentimes, for privacy reasons, we may not want to reveal/log/store the raw input to a model, but only an n-dimensional embedding representation. However, the paper shows that embeddings can reveal sensitive attributes (metadata) about a raw input just from the embeddings. For example:

   attributes such as authorship of text can be easily extracted by training an inference model on just a handful (10-50 examples per author) of labeled embedding vectors. 

3. Membership inference Attack: In this kind of attack, the goal is to figure out if a given data point was part of the training set of an embedding model. The paper shows that, especially for infrequent/outlier training data inputs, it can be very easy to figure this out.

Proposed Mitigation

The paper proposes and evaluates adversarial training techniques to minimize information leakage in embedding models. In this training scheme, embeddings are trained jointly with an adversarial loss: a simulated adversary A, whose goal is to infer any sensitive information, is trained jointly with an embedding model M, while M is trained to maximize the adversary’s loss and minimize the learning objective. If you’re interested in a deep dive into the technical details, we highly recommend reading the paper.

Embedding Models are gaining widespread use in real-world AI applications, and for good reason: To the extent that we can train models to understand content and context about the real world, and map it to a low dimensional representation, it makes the downstream task of task-specific modeling simpler and more efficient. However, papers like the one above highlight the potential dangers, one we think will be important to address and mitigate before these models can be adopted in privacy-sensitive domains like healthcare.

AWS Whitepaper | AWS Well-Architected Framework - Machine Learning Lens [Part 2]

In our previous issue, we introduced the Machine Learning Lens on AWS’s Well-Architected Framework. As they say:

In the Machine Learning Lens, we focus on how to design, deploy, and architect your machine learning workloads in the AWS Cloud.

In this issue, we will go over the five Pillars of the Well-Architected Framework for a machine learning solution. These five pillars are:

• Operational Excellence

• Security

• Reliability

• Performance Efficiency

• Cost Optimization

The original document goes into significant detail, but here we will cover the considerations that were most interesting to us. 

Operational Excellence

• Establish cross-functional teams (since different personas are involved)

• Identify the end-to-end architecture and operational model early (this ensures that business and technical objectives are aligned)

• Establish a model retraining strategy (data drift is a given)

• Document findings throughout the process and version all input and artifacts (reproducibility!)

Security

• Restrict access to ML systems and ensure data governance (inference, model and data access in all environments available only to the right users)

• Enforce data lineage (makes it easier to trace source data in case of errors)

• Enforce regulatory compliance (privacy concerns and HIPAA, GDPR need to be respected)

Reliability

• Manage changes to model inputs through automation (since data and code can both change, automation ensures reproducibility)

• Train once and deploy across environments (models shouldn’t be retrained when moving them from dev to prod, since minor data changes/randomness can influence behaviour)

• Ensure that inference services can scale easily

Performance Efficiency

• Optimize compute for your ML workload (training and inference workloads often require different hardware)

• Define latency and network bandwidth performance requirements for your models (if real-time inference is needed in applications, latency requirements are critical)

• Continuously monitor and measure system performance (collection of system, service and business metrics for ML workloads gives us directions for improvement)

Cost Optimization

• Define overall ROI and opportunity cost for ML projects ahead of time (is a team of two data scientists worth it or can using predictions from some API suffice)

• Experiment with small datasets (fail fast / de-risk projects early)

• Account for inference architecture based on consumption pattern (batch predictions vs real-time predictions need different hardware)

Final Thoughts

While this is a lot to take in, it can be valuable to look at end-to-end projects from the lens of these five pillars. We will continue to evaluate different frameworks for ML projects, but would love to hear from you on this as well!

How has COVID-19 affected the performance of ML models used by UK banks

The full report is available here, and we recommend clicking through and going through the insightful charts. 

High-level overview

We asked banks how Covid-19 (Covid) had affected their use of machine learning and data science. Although these technologies will continue to have many benefits, over a third of banks reported a negative impact on model performance as a result of the pandemic.

Learnings

• While half of the banks reported an increase in their perceived importance of Machine Learning (ML) and Data Science (DS) for them, only a third reported an actual increase in the number of planned or existing projects. 

• About 35% of banks reported a negative impact on ML model performance as a result of Covid. They note that is likely because of “major movements in macroeconomic variables, such as rising unemployment and mortgage forbearance”, which required ML models to be “recalibrated”. 

• Smaller banks showed a positive impact of Covid in their use of third-party solutions for different parts of their ML stack. While we don’t quite know the “magnitude” of impact, it reflects both a willingness to invest in ML applications and the openness to use third-party solutions. 

Our Take

While the number of institutions surveyed isn’t very high (17 UK banks, 9 banks based outside the UK and 6 insurers although representing 88% of all UK banks assets), we believe the information to be directionally correct. While there are more and more use cases that fit ML applications well, the financial sector being more heavily regulated means that the safe deployment of models will be critical. 

New tool alert: Sagify

We recently came across this neat MLOps tool that we wanted to share. Sagify is a command-line utility to train and deploy ML models on AWS Sagemaker. Think of it as a higher-level abstraction on top of AWS Sagemaker that hides away much of the low-level details. If you’re interested in a quick demo of the tool, check out this video or this intro article. 

Fun: How to Break ML Models - Exhibit 9456

OpenAI recently released a fantastic post about Multimodal Neurons - parts of the neural network that respond to a “concept” in the input content in the same way, across modalities like text, image, video. We’ll cover this article in depth in the next edition of our newsletter but here we wanted to share this funny (and scary) anecdote highlighting just how easy it is to mount adversarial attacks. Just because the model can read a piece of text in the input saying “iPod” the model thinks it is actually looking at an iPod!  

Thanks

Thanks for making it to the end of the newsletter! This has been curated by Nihit Desai and Rishabh Bhargava. This is only Day 1 for MLOps and this newsletter and we would love to hear your thoughts and feedback. If you have suggestions for what we should be covering in this newsletter, tweet us @mlopsroundup (open to DMs as well) or email us at mlmonitoringnews@gmail.com