Supervised Learning Explained And How To Choose The Right Model

by | Oct 20, 2022 | Machine Learning

Machine learning can be broadly classified into three major types: supervised, unsupervised, and reinforcement learning. We will go into supervised learning in more detail in this article and compare it to the other two types.

Supervised learning is a machine learning approach used for problems where the data is labelled examples or data points with features and a corresponding label.

Supervised learning is used where labelled data is available

Supervised learning is used where labelled data is available

The objective of supervised learning algorithms is to learn, for example, input-output pairs, a function that maps inputs to outputs. This may sound rather technical but bear with us. An example will quickly provide clarity.

A simple example of supervised learning

Let’s say we would like to find out the weight of a new client joining a gym so that we can adjust equipment accordingly during their first visit. However, the customer success team at the gym has found that clients feel uncomfortable disclosing their weight, and prospective clients tend to discontinue the onboarding process at this stage.

As a result, the gym no longer asks for this data, and the trainer now needs to adjust the equipment based on the information provided and has decided to deduce a person’s weight based on their height. He samples his current client base and plots the results. He can then draw a line of best fit, also known as a linear model. Now, whenever a new client joins, the trainer can “predict” or “estimate” their weight based on their height.

Fitted line plot for a linear model

Although this might sound like an easy example, all supervised machine learning techniques follow this same process. Firstly, we consider the problem. What output are we trying to achieve? Our output is going to be the weight. Then we start collecting data; in our case, personal data related to the current clients. Note that it is essential to use the existing client base and not just a larger population sample, as we want our data to represent the problem we are solving.

Note that the definition of “a representative sample” is often a contentious issue in every machine learning discussion.

Making predictions

We then determine the input features. In our example, this was the height, but the model could be further improved by adding other features already in the dataset. Age and gender come to mind. The features should generally contain enough information to predict the output accurately. Problems also occur when we add too many features, so more is not always better here.

Our final step is then to choose an algorithm and train the model. In our case, the “training” is finding the line of best fit through our scattered data points. The line of best fit, or the regression line, is our model.

To use the model, we can provide a height and get the corresponding weight on the regression line.

Different types of supervised machine learning algorithms

There wasn’t any actual training involved in the previous example. Still, other more complicated models are trained until they are capable of recognising the underlying relationships and patterns between the input labels and the output labels. This enables the model to produce accurate labelling results when presented with data it has never seen before.

Supervised learning algorithms can primarily generate two kinds of results: classification and regression.

Classification algorithms

A classification algorithm divides inputs into a certain number of categories or classes. Categories are based on the labelled data it was trained on. Classification algorithms can be used for binary classifications, such as classifying customer feedback as positive or negative and classifying emails as spam. Another common classification issue solved by supervised learning is feature recognition. It can identify handwritten letters and numbers or classify drugs into numerous categories.

Regression models

Regression tasks are distinct because they anticipate a numerical relationship between the input and output data from the model. These models can predict things like real estate prices based on zip codes, click rates in online ads based on the time of day, or customer willingness to pay for a particular product based on age.

The following are some examples of algorithms frequently used in supervised learning programs:

When choosing a supervised learning algorithm, a few things should be considered. The first is the bias and variance within the algorithm, as there is a fine line between being flexible enough and too flexible. Another is the complexity of the model or function that the system is trying to learn. Finally, as noted, the data’s heterogeneity, accuracy, redundancy, and linearity should also be analysed before choosing an algorithm.

Choosing a machine learning algorithm

As we have seen, there are numerous supervised learning algorithms available, each with its advantages and disadvantages, and as such, there isn’t a single learning algorithm that solves all supervised learning issues. These are, however, four major points to consider when choosing an algorithm.

Bias-variance trade-off

We introduced the terms “bias” and “variance” earlier. From this, it should have become clear that there is a trade-off between the two. Imagine that we have a variety of equally helpful training data sets at our disposal. When trained on each data set, a learning algorithm is considered biased for a given input if it consistently predicts the incorrect output. On the other hand, an algorithm has a high variance for a particular input if it predicts different output values when trained on different training sets.

The sum of the bias and the variance of a learning algorithm is related to the prediction error, and so there is always a tradeoff between bias and variance.

An algorithm with low bias is flexible and so can fit the data well. However, if it is too flexible, it will fit each training data set differently and hence have high variance. The bias-variance trade-off is necessary for many problems, so most supervised learning algorithms allow you to adjust this.

Complexity and amount of training data

The second thing to think about is how much training data is available compared to how complex the “true” function is. If the learning algorithm is simple, an “inflexible” learning algorithm with high bias and low variance can learn the actual function from a small amount of data. On the other hand, if the true function is rather complex, it can only be understood using a lot of training data and a “flexible” learning algorithm with low bias and high variance.

The dimensionality of the input space

A third issue is the dimensionality of the input space. Even though only a few of those features are necessary for the actual function to work, learning the function can be difficult if the input feature vectors have high dimensionality. This is because the many “extra” dimensions could confuse the learning algorithm and make it more unpredictable.

As a result, tuning the classifier to have a high bias and low variance is typically required for large-dimensional input data. If you can manually remove unimportant features from the input data, the learned function’s accuracy will likely be improved in real-world applications.

Numerous feature selection algorithms seek to separate the relevant features from the irrelevant ones.

Noise in the output values

The level of noise in the target variables (which represent the desired output values) is the fourth issue. If the desired output values are frequently incorrect (due to human or sensor error), the learning algorithm should avoid trying to find a function that matches the training examples exactly. When the data is fitted too precisely, overfitting occurs.

overfitting occurs when data is fitted too precisely

Overfitting occurs when you fit data too precisely.

Even with no measurement errors, you can overfit a function if it is too complex for your learning model. In this case, the un-fittable portion of the target function “corrupts” your training data, referred to as deterministic noise. It is possible to reduce noise in output values in practice using several techniques. For example, we can apply early stopping to prevent overfitting or identify and remove noisy training examples before training the supervised learning algorithm. The removal of potentially noisy training samples before training can statistically significantly reduce generalisation error.

Other factors to consider

When selecting and using a learning algorithm, there are additional factors to take into account:

  • Data heterogeneity. Some algorithms are more straightforward to use than others when the feature vectors contain features of various types (discrete, discrete ordered, counts, and continuous values). The input features must be numerical and scaled to similar ranges for many algorithms, including support-vector machines, linear regression, logistic regression, neural networks, and nearest neighbour techniques (e.g., to the [-1,1] interval). Decision trees have the benefit of being able to handle heterogeneous data with ease. 
  • Redundant data. Some learning algorithms, like linear regression, logistic regression, and distance-based methods, will perform poorly if the input features contain redundant information (for example, highly correlated features) because of numerical instabilities. Regularisation can frequently be imposed to address these issues.
  • Interactions and non-linearities. Algorithms based on linear functions (e.g., logistic regression, support vector machines, naive Bayes) and distance functions (e.g., nearest neighbour methods, support-vector machines with Gaussian kernels) typically perform well when each feature contributes independently to the output. However, because decision trees and neural networks are created specifically to find these interactions, they perform better when complex interactions exist between features. You can also use linear methods, but when doing so, the interactions must be manually specified.

You can compare various learning algorithms when thinking about a new application and experimentally determine which one performs best on the current issue. It can take a lot of time to adjust a learning algorithm’s performance. Given fixed resources, spending more time gathering more training data and informative features is frequently preferable to spending more time fine-tuning the learning algorithms.

Supervised vs unsupervised learning

What is unsupervised learning?

The algorithm’s learning process is the main distinction between supervised and unsupervised learning. Unlabeled data is provided to the algorithm as a training set in unsupervised learning. There are no correct output values, in contrast to supervised learning, because the algorithm finds patterns and similarities within the data rather than correlating it to some outside measurement. Unsupervised learning algorithms can operate to learn more about the data and discover intriguing or unexpected results that people weren’t looking for. Applications of clustering (the process of identifying groups within data) and association benefit greatly from unsupervised learning (predicting rules that describe the data).

Benefits and limitations

Compared to the unsupervised method, supervised learning models have some benefits, but they also have drawbacks. Because humans have provided the basis for the decisions, supervised learning systems are more likely to reach conclusions that humans can understand.

However, supervised learning systems have trouble adjusting to new information using a retrieval-based approach. Bicycles, for instance, would have to be incorrectly categorised if a system with categories for cars and trucks were presented. However, if the AI system were generative (that is, unsupervised), it might not be able to identify the bicycle. Still, it would be able to identify it as being in a different category.

To achieve acceptable performance levels, supervised learning also typically needs a significant amount of accurately labelled data. Labelled data is, unfortunately, not always available. This issue is not present in unsupervised learning, which can also use unlabeled data.

Semi-supervised learning

Semi-supervised learning might be the best learning strategy when supervised learning is required, but we lack high-quality data. This learning model, which sits between supervised and unsupervised learning, accepts partially labelled data. This means that most of the data is unlabeled.

Similar to unsupervised learning, semi-supervised learning finds the correlations between the data points and then uses the labelled data to mark those data points. The entire model is then trained using the recently applied labels.

Given the limited amount of labelled data, supervised learning algorithms would not be able to perform as intended in many real-world problems. However, semi-supervised learning has been shown to produce accurate results. As a general rule, semi-supervised learning can be applied to data sets with at least 25% labelled data.

How we used supervised machine learning

Natural language processing (NLP) lends itself well to many supervised learning problems. Whether extracting company names, financial data points or specific relevant paragraphs in much larger documents. Supervised algorithms can accurately extract this data as long as we provide some examples to the algorithm.

A concrete example would be spam filtering in an email inbox. We label some emails as spam and others as relevant as we go through our inboxes. You can then use this labelled data to predict whether new emails coming into the inbox are spam by running them through a trained spam classification algorithm. The output can then direct the incoming mail to the correct folder.

spam detection is a typical supervised learning problem

An email spam classification algorithm is a typical supervised learning problem.

Classifying spam is a simple example, but in the real world, we often encounter complicated scenarios. Are advertisements and promotions spam? What is relevant for one person might be irrelevant for the next. So, even simple examples quickly become complicated when trying to implement them in the real world. The algorithm not only needs to be trained for every person but also continuously over time. We find relevant today might become irrelevant to us in a month.

So even though supervised machine learning problems are commonly used, a lot goes into the design, selection, and maintenance process. Understanding the trade-offs in this process will help you achieve better outcomes and manage expectations accordingly.

About the Author

Neri Van Otten

Neri Van Otten

Neri Van Otten is the founder of Spot Intelligence, a machine learning engineer with over 12 years of experience specialising in Natural Language Processing (NLP) and deep learning innovation. Dedicated to making your projects succeed.

Recent Articles

One class SVM anomaly detection plot

How To Implement Anomaly Detection With One-Class SVM In Python

What is One-Class SVM? One-class SVM (Support Vector Machine) is a specialised form of the standard SVM tailored for unsupervised learning tasks, particularly anomaly...

decision tree example of weather to play tennis

Decision Trees In ML Complete Guide [How To Tutorial, Examples, 5 Types & Alternatives]

What are Decision Trees? Decision trees are versatile and intuitive machine learning models for classification and regression tasks. It represents decisions and their...

graphical representation of an isolation forest

Isolation Forest For Anomaly Detection Made Easy & How To Tutorial

What is an Isolation Forest? Isolation Forest, often abbreviated as iForest, is a powerful and efficient algorithm designed explicitly for anomaly detection. Introduced...

Illustration of batch gradient descent

Batch Gradient Descent In Machine Learning Made Simple & How To Tutorial In Python

What is Batch Gradient Descent? Batch gradient descent is a fundamental optimization algorithm in machine learning and numerical optimisation tasks. It is a variation...

Techniques for bias detection in machine learning

Bias Mitigation in Machine Learning [Practical How-To Guide & 12 Strategies]

In machine learning (ML), bias is not just a technical concern—it's a pressing ethical issue with profound implications. As AI systems become increasingly integrated...

text similarity python

Full-Text Search Explained, How To Implement & 6 Powerful Tools

What is Full-Text Search? Full-text search is a technique for efficiently and accurately retrieving textual data from large datasets. Unlike traditional search methods...

the hyperplane in a support vector regression (SVR)

Support Vector Regression (SVR) Simplified & How To Tutorial In Python

What is Support Vector Regression (SVR)? Support Vector Regression (SVR) is a machine learning technique for regression tasks. It extends the principles of Support...

Support vector Machines (SVM) work with decision boundaries

Support Vector Machines (SVM) In Machine Learning Made Simple & How To Tutorial

What are Support Vector Machines? Machine learning algorithms transform raw data into actionable insights. Among these algorithms, Support Vector Machines (SVMs) stand...

underfitting vs overfitting vs optimised fit

Weight Decay In Machine Learning And Deep Learning Explained & How To Tutorial

What is Weight Decay in Machine Learning? Weight decay is a pivotal technique in machine learning, serving as a cornerstone for model regularisation. As algorithms...


Submit a Comment

Your email address will not be published. Required fields are marked *

nlp trends

2024 NLP Expert Trend Predictions

Get a FREE PDF with expert predictions for 2024. How will natural language processing (NLP) impact businesses? What can we expect from the state-of-the-art models?

Find out this and more by subscribing* to our NLP newsletter.

You have Successfully Subscribed!