Machine Learning Techniques, Lesson 1: Three Kinds of Questions a Machine Can Learn to Answer

What you'll learn: the three big families of machine learning problems, and how to tell which one you're facing.

The three families we'll meet again and again: pick a category, pick a number, or find groups.

Your phone suggests the word you were about to type. Your email quietly drops a message into the spam folder. Nobody sat down and wrote a rule for every sentence you might send. So how does the machine know?

That's what this series is about. Over fifteen short lessons we'll open the box and look at how machine learning really works, with small experiments you can run yourself. Today we start with the map: the three kinds of problems almost everything else is built on.

Learning from examples, not rules

Think about how a child learns to tell cats from dogs. You don't hand over a rulebook about ear shape and whisker length. You point and say "cat", "dog", "dog", "cat", and after enough examples the child just gets it.

Machine learning works the same way. Instead of writing the rules ourselves, we show a program lots of examples and let it find the patterns. The pattern it finds is called a model, and using that model on something new is called making a prediction.

Almost every problem falls into one of three families. Knowing which one you have is half the battle.

The three families

Classification means choosing a category. Is this email spam or not? Is this flower an iris setosa, versicolor or virginica? Each training example comes with the right answer attached (a label), and the model learns to give the right label to new cases.

Regression means predicting a number. How much will this house sell for? How many people will visit the shop tomorrow? It is also learned from labelled examples, but the label is a quantity, not a category. A handy test: if the answer could be 3.7, it's regression.

Clustering means finding groups when nobody has told you what the groups are. Imagine a shop with ten thousand customers and no categories at all. A clustering method can notice that some people buy in bulk once a month and others pop in daily for small things. Because no answers are given, this is called unsupervised learning. The first two are supervised learning, since the examples come with answers to learn from.

The same 150 flowers twice. On the left the species are known. On the right a clustering method found three groups without ever seeing them.

Look at the two plots. On the left we know each flower's species. On the right the computer was never told. It still carved out a small separate group, which matches one species well, but it blurred the other two where they overlap. That's typical: clustering gives you a useful first look, not a final truth.

Try it yourself

This snippet runs one example from each family, using small datasets that ship with scikit-learn. You'll need Python with scikit-learn installed.

from sklearn.datasets import load_iris, load_diabetes
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier
from sklearn.linear_model import LinearRegression
from sklearn.cluster import KMeans
from sklearn.metrics import adjusted_rand_score

X, y = load_iris(return_X_y=True)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0, stratify=y)

# 1. Classification: labels are given, predict a category
clf = KNeighborsClassifier(n_neighbors=5).fit(Xtr, ytr)
print("Classification accuracy:", round(clf.score(Xte, yte), 3))

# 2. Regression: predict a number (diabetes progression score)
Xd, yd = load_diabetes(return_X_y=True)
Xa, Xb, ya, yb = train_test_split(Xd, yd, test_size=0.3, random_state=0)
print("Regression R^2:", round(LinearRegression().fit(Xa, ya).score(Xb, yb), 3))

# 3. Clustering: the species labels are never shown to the model
km = KMeans(n_clusters=3, n_init=10, random_state=0).fit(X)
print("Cluster sizes:", sorted(int((km.labels_ == k).sum()) for k in range(3)))
print("Agreement with true species:", round(adjusted_rand_score(y, km.labels_), 3))

Here's what I got when I ran it:

Classification accuracy: 1.0
Regression R^2: 0.393
Cluster sizes: [38, 50, 62]
Agreement with true species: 0.73

Some honest reading of those numbers. The classifier got every flower in the held-out test set right, but iris is a famously easy dataset, so don't expect that on messy real data. The regression score of 0.393 is far more modest: a plain straight-line model explains only about 39 percent of the variation in the diabetes measure on the test data. And the clustering agrees with the true species to a degree of 0.73 on a scale where 1 means a perfect match, which fits what the plot showed.

When to use which, and where it goes wrong

Ask yourself what you want out. A category means classification. A number means regression. A set of groups you don't have names for means clustering. If you have no examples with answers at all, the first two aren't options yet.

The common trap is trusting any model more than its data deserves. A model can only learn what the examples show it, so biased or sloppy examples give biased or sloppy predictions. We'll spend the next lesson on exactly that: getting data ready.

Recap: machine learning finds patterns in examples, and most problems are about picking a category, picking a number, or finding groups.

Next time: Lesson 2, Data prep: cleaning, features, train and test splits.

Which everyday app do you use that you think is classification, regression or clustering behind the scenes? Tell me in the comments.

Sources: scikit-learn user guide (datasets and estimators used in the code).