Chrysolites Tech Academy

Machine Learning Fundamentals · Course book

Chapter 1: A First Tour of Machine Learning

Pairs with Lesson 1: Machine Learning for Everybody

Machine learning is the craft of getting a computer to improve at a task by studying examples rather than following rules a programmer wrote by hand. A spam filter that learns from emails people have marked as junk, a model that estimates house prices from past sales and an app that groups shoppers by their habits are all built the same way: collect data, choose a model, let the model adjust itself to fit the data, then check how well it does on examples it has never seen. This chapter gives you the vocabulary and the mental models for that whole loop. It introduces features and labels, the difference between classification and regression, how data is split and prepared, and then a tour of the classic algorithms, from nearest neighbours and Naive Bayes to support vector machines, neural networks, clustering and principal component analysis. Every idea comes with a small worked example you can check by hand.

Features, labels and kinds of learning

Every machine learning dataset is a table. Each row is one example, such as one patient visit, one flower or one house sale. Each column describes something about that example. The columns the model reads are called features, or inputs, and the column it tries to predict is the label, or target. In a house-price dataset the features might be floor area, number of bedrooms and distance to the city centre, and the label is the sale price.

Features come in two broad types. Numerical features are quantities, such as age or temperature, where differences and averages make sense. Categorical features are names of groups, such as a country or a colour. A model cannot do arithmetic on the word "red", so categories are usually turned into numbers with one-hot encoding: one column per category, holding 1 for the category that applies and 0 for the rest. Ordered categories, such as small, medium and large, can instead be mapped to 1, 2 and 3, because their order carries meaning.

Kind of learningWhat the data containsTypical questionExample algorithms
Supervised: classificationFeatures and a categorical labelWhich group does this example belong to?K-nearest neighbours, Naive Bayes, logistic regression, SVM
Supervised: regressionFeatures and a numerical labelWhat number should we predict?Linear regression, regression neural networks
UnsupervisedFeatures only, no labelWhat structure or groups are hidden here?K-means clustering, PCA
ReinforcementActions, states and rewardsWhich action earns the most reward over time?Q-learning, policy methods

Classification predicts a category. When there are exactly two categories, such as spam and not spam, it is binary; with three or more, such as types of flower, it is multi-class. Regression predicts a continuous number, such as tomorrow's temperature. The same dataset can support both: predicting a house's exact price is regression, while predicting whether it will sell within a month is classification.

Training, splitting and preparing data

Training a model means adjusting its internal numbers, its parameters, so that its predictions match the labels as closely as possible. The gap between prediction and truth is measured by a loss function. For regression a common choice is mean squared error, the average of the squared differences; for classification it is often cross-entropy, which punishes confident wrong answers heavily. Training is a loop: predict, measure the loss, nudge the parameters to reduce it, repeat.

A model that scores well on the data it trained on may simply have memorised it. To get an honest measure, the data is split before training begins. The training set, often 60 to 80 percent of the rows, is used to fit the parameters. The validation set is used to compare settings and models while you are still making decisions. The test set is kept locked away and used only once, at the end, to estimate how the model will do on genuinely new data. If you keep checking the test set while tuning, it quietly becomes another training set and its score stops being trustworthy.

Training: 600 rows (60%)Validation: 200Test: 200 Dataset of 1,000 rows, shuffled before splitting fit the parameterschoose settingsfinal score, once
Figure 1.1 — A 60/20/20 split. Each part has one job, and the test set is used only at the very end.

Preparing the data matters as much as choosing the algorithm. Rows are shuffled so that each split is representative. Missing values are filled in or the rows dropped. Features measured on very different scales, such as income in thousands and age in tens, are scaled so that each has a mean of 0 and a standard deviation of 1; otherwise algorithms that measure distances will be dominated by the feature with the biggest numbers. When one class is much rarer than another, as with fraud among normal payments, the training data can be rebalanced by oversampling the rare class, so that the model does not learn to ignore it.

Classifiers you can reason about

The k-nearest neighbours (KNN) classifier is the most intuitive algorithm in the field. It does no real training at all; it stores the labelled examples. To classify a new point, it measures the distance to every stored point, takes the k closest, and lets them vote. Distance is usually the ordinary straight-line, or Euclidean, distance: the square root of the sum of squared differences in each feature.

Worked example. Five labelled points: A (1, 2) red, B (2, 5) blue, C (4, 4) blue, D (5, 1) red and E (3, 2) red. Classify the new point (3, 4). The distances are A: √(4 + 4) = 2.83; B: √(1 + 1) = 1.41; C: √(1 + 0) = 1.00; D: √(4 + 9) = 3.61; E: √(0 + 4) = 2.00. With k = 1 the nearest point is C, so the answer is blue. With k = 3 the neighbours are C, B and E: two blue, one red, so blue again. With k = 5 every point votes: three red, two blue, so the answer flips to red. The choice of k changes the result, which is why it is tuned on the validation set.

Naive Bayes takes a probabilistic route. Bayes' rule says that the probability of a class given the evidence is proportional to the probability of the evidence given the class, multiplied by how common the class is to begin with. The "naive" part is the assumption that features are independent of one another once the class is known. That is rarely exactly true, but it lets the model multiply simple per-feature probabilities together, so it trains in a single pass over the data and works surprisingly well for text, where each word is a feature.

Logistic regression, despite its name, is a classifier. It computes a weighted sum of the features, called z, and passes it through the sigmoid function, 1 / (1 + e−z), which squashes any number into the range 0 to 1. The output is read as the probability of the positive class, and a threshold, usually 0.5, turns it into a decision. Training finds the weights that make the probabilities match the labels.

z = 0 gives 0.5 −6−3036 1.00 z (weighted sum of features)
Figure 1.2 — The sigmoid curve used by logistic regression. Sample values: z = −2 gives 0.119, z = 1 gives 0.731 and z = 2 gives 0.881.

A support vector machine (SVM) looks for the boundary between two classes that leaves the widest possible gap, or margin, to the nearest points on either side. Those nearest points are the support vectors, and they alone decide where the boundary goes. When the classes cannot be separated by a straight line, the kernel trick lets the SVM behave as if the data had been lifted into a higher-dimensional space where a flat boundary does work, without ever computing that space explicitly.

Measuring a classifier

Accuracy, the share of predictions that are correct, is the obvious score, but it can mislead. If only 1 percent of payments are fraudulent, a model that always says "not fraud" is 99 percent accurate and completely useless. A fuller picture comes from the confusion matrix, which counts the four possible outcomes.

Predicted positivePredicted negative
Actually positiveTrue positives (TP): 40False negatives (FN): 5
Actually negativeFalse positives (FP): 10True negatives (TN): 45

From these 100 test examples we can compute four scores. Accuracy is (TP + TN) / total = 85 / 100 = 0.85. Precision, the share of predicted positives that really are positive, is TP / (TP + FP) = 40 / 50 = 0.80. Recall, the share of real positives the model found, is TP / (TP + FN) = 40 / 45 = 0.889. The F1 score balances the two: 2 × precision × recall / (precision + recall) = 2 × 0.80 × 0.889 / 1.689 = 0.842. Which score matters most depends on the cost of each mistake: a screening tool should favour recall, so it misses few cases, while a filter that deletes messages should favour precision, so it rarely removes a good one.

Neural networks, regression and unsupervised learning

A neural network is built from simple units called neurons. Each neuron takes a weighted sum of its inputs, adds a bias and applies an activation function, such as the sigmoid or the ReLU, which outputs zero for negative inputs and passes positive inputs through unchanged. Neurons are arranged in layers, and the output of one layer feeds the next. Training uses gradient descent: the network works out how the loss would change if each weight moved slightly, a calculation called backpropagation, and then moves every weight a small step in the direction that lowers the loss. The size of that step is the learning rate. Libraries such as TensorFlow handle the calculus for you, so building a classifier becomes a matter of describing the layers, choosing a loss and calling a training function.

Linear regression fits a straight line, y = mx + b, through numerical data by choosing the slope and intercept that minimise the squared errors. For the points (1, 2), (2, 4), (3, 5), (4, 4) and (5, 6), the mean of x is 3 and the mean of y is 4.2. The slope is the sum of (x − 3)(y − 4.2), which is 8.0, divided by the sum of (x − 3)2, which is 10, giving 0.8. The intercept is 4.2 − 0.8 × 3 = 1.8, so the best line is y = 0.8x + 1.8. A single neuron with no activation function computes exactly this kind of weighted sum, which is why the same problem can also be solved with a tiny neural network, and why adding hidden layers lets a network fit curves that a straight line cannot.

Unsupervised learning works without labels. K-means clustering splits the data into k groups. It places k centre points, assigns every example to its nearest centre, moves each centre to the average of its assigned examples, and repeats until nothing changes. Principal component analysis (PCA) reduces the number of features. It finds the directions along which the data varies the most and re-expresses each example using only the top few of those directions. Two hundred correlated measurements might shrink to ten components that keep most of the information, which speeds up later models and makes it possible to plot the data in two dimensions to look for patterns.

Check your understanding

  1. A dataset records each customer's age, country and monthly spend, and whether they cancelled their subscription. Which columns are features and which is the label? Is predicting cancellation classification or regression?
  2. Why should the test set be used only once? What goes wrong if you check it after every change to the model?
  3. Using the five points in the KNN worked example, classify the new point (4, 2) with k = 3. Show the distances you computed.
  4. A classifier has 30 true positives, 20 false positives, 10 false negatives and 140 true negatives. Compute its accuracy, precision, recall and F1 score.
  5. Explain why a model that is 99 percent accurate may still be useless, and name a score that would reveal the problem.
  6. Describe in your own words the two steps that K-means repeats, and say what makes it stop.

Install Chrysolites Tech Academy

Android / Chrome / Edge: open the browser menu and choose Install app or Add to Home screen.

iPhone / iPad (Safari): tap the Share button, then Add to Home Screen.