Chrysolites Tech Academy

Machine Learning Fundamentals · Course book

Chapter 3: Building a Classifier from Scratch

Pairs with Lesson 3: Machine Learning Without Libraries

Most machine learning tutorials start by importing a library, and a single line of code trains a model. That is efficient, but it hides everything that matters. In this chapter we take the opposite approach and build a complete classifier from nothing but plain JavaScript and a web browser, the same "no black box" approach the video takes. The running example is a set of simple hand-drawn sketches, such as cars, fish and houses, which a program learns to recognise. Along the way you will see how raw data becomes features, why features must be scaled, how a chart reveals what a model can and cannot learn, how the nearest-neighbour classifiers work line by line, and how to measure the result honestly. When you later use a library such as scikit-learn, you will know what each function call is doing for you.

From drawings to data

Every machine learning project starts with data, and in this case the data has to be collected. A small drawing app in the browser lets a person sketch with the mouse or a finger. Each sketch is stored as a list of paths, one for each stroke drawn without lifting the pen, and each path is a list of points, pairs of x and y coordinates. A drawing of a house might be two paths, one for the walls and roof and one for the door, holding a few hundred points in total.

The app asks each person to draw the same set of objects, labelled in advance, so every sketch arrives with its correct class. Saved as JSON, one sample might look like the example below, shortened for space.

{
  "id": 17,
  "label": "fish",
  "student": "contributor-3",
  "paths": [
    [[112, 80], [118, 84], [127, 90]],
    [[160, 95], [158, 101]]
  ]
}

Collecting data this way teaches some important lessons quickly. People draw differently, so the same object appears in many sizes and styles. Some samples are wrong or unfinished, so data needs cleaning: a reviewer flags and removes sketches that are blank or plainly not the requested object. Keeping a record of which person drew which sample also matters, because if one person's drawings end up in both the training data and the test data, the model can learn that person's style rather than the object, and the test score becomes too optimistic.

Feature extraction and scaling

A classifier cannot compare raw lists of points of different lengths, so each sketch is reduced to a short, fixed list of numbers called features. Choosing them is feature extraction, and it is where human insight about the problem enters the system. Simple, cheap features for sketches include the following.

FeatureHow to compute itWhat it captures
Path countThe number of strokesHow many separate parts the drawing has
Point countThe total number of points across all pathsRoughly how much was drawn
WidthLargest x minus smallest xHow wide the drawing is
HeightLargest y minus smallest yHow tall the drawing is
ElongationWidth divided by height, or a similar ratioWhether the shape is long and thin or tall and narrow
RoundnessHow close the outline is to a circleWhether the shape is curved or angular

Computing a feature is usually a few lines of code. The function below finds the width of a sketch by tracking the extreme x values. The second function, counting paths, is even shorter.

function getWidth(paths) {
  const xs = paths.flat().map((p) => p[0]);
  return Math.max(...xs) - Math.min(...xs);
}

function getPathCount(paths) {
  return paths.length;
}

Features measured in different units cause a subtle problem. Width might range from 0 to 500 pixels, while path count ranges from 1 to 10. Any algorithm that measures distance between samples will be dominated by the feature with the bigger numbers. The fix is scaling, and the simplest form is min-max normalisation: subtract the smallest value of the feature and divide by its range, so that every feature lies between 0 and 1. The minimum and range must be computed from the training data only and then reused, unchanged, on new samples.

Worked example. An unknown sketch has width 200 and 2 paths. One training car has width 230 and 8 paths; one training fish has width 260 and 2 paths. Without scaling, the distance to the car is √(302 + 62) = 30.6 and to the fish is √(602 + 02) = 60, so the nearest neighbour is the car, even though the path counts, 2 against 8, are very different. Now scale, with width ranging from 0 to 500 and paths from 1 to 10. The unknown becomes (0.40, 0.11), the car (0.46, 0.78) and the fish (0.52, 0.11). The distances become 0.669 to the car and 0.120 to the fish, so the answer flips to fish. Scaling let the path count finally have a say.

Seeing the data before modelling it

Before training anything, it pays to look at the data. With two features, each sketch becomes a point on a scatter chart, coloured by its label. Building the chart by hand on an HTML canvas, rather than with a charting library, means writing code to map data values to pixel positions, draw axes, and support panning and zooming, which deepens your feel for what scaling does: the chart itself is a scaling from data space to screen space.

unknown fishcarhouse width (scaled 0 to 1) path count (scaled 0 to 1) 01
Figure 3.1 — A scatter chart of scaled features, coloured by class. Well-separated clusters suggest a nearest-neighbour classifier will work; overlapping ones warn that new features are needed.

A chart like this answers practical questions immediately. If the classes form separate clusters, simple models will succeed. If two classes overlap heavily, no algorithm can separate them using these features, and the right response is to invent better features rather than reach for a more complicated model. Outliers, such as a "car" sitting deep among the fish, often turn out to be mislabelled or badly drawn samples worth removing.

Nearest neighbour and k-nearest neighbours

The nearest neighbour classifier is the simplest classifier that works. To label a new sample, compute its distance to every training sample and copy the label of the closest one. In JavaScript the core is a short loop.

function nearest(point, samples) {
  let best = null;
  let bestDist = Infinity;
  for (const s of samples) {
    const d = Math.hypot(point[0] - s.point[0], point[1] - s.point[1]);
    if (d < bestDist) {
      bestDist = d;
      best = s;
    }
  }
  return best.label;
}

Relying on a single neighbour is fragile: one mislabelled or unusual training sample can capture a whole region of the chart. The k-nearest neighbours classifier fixes this by sorting all the training samples by distance, taking the closest k and returning the most common label among them. With k = 1 it is the nearest neighbour classifier; larger values of k give smoother boundaries that ignore isolated noise, but a value that is too large lets the biggest class outvote smaller nearby ones. Choosing an odd k avoids ties between two classes.

Seeing the classifier as a picture makes its behaviour concrete. Colour every pixel of the chart by the label the classifier would give a sample at that position, and you get a decision boundary map. With k = 1 the coloured regions are jagged, with small islands around stray points; as k grows the islands disappear and the boundaries smooth out.

Evaluating honestly

A classifier that is tested on the samples it memorised will appear perfect: with k = 1, every training sample's nearest neighbour is itself. A meaningful score needs a separate testing set. Here, the split should be by person rather than by sample, so that every sketch from a given contributor lands entirely in training or entirely in testing. Then the test checks whether the model recognises objects drawn by people it has never seen, which is what will happen in real use.

Accuracy is the number of correct test predictions divided by the number of test samples. Suppose 480 test sketches are classified and 384 are correct: the accuracy is 384 / 480 = 0.80, or 80 percent. With eight equally common classes, random guessing would score 1 / 8 = 12.5 percent, so 80 percent shows real learning. A confusion matrix, with one row per true class and one column per predicted class, shows which pairs get mixed up, such as cars and houses, pointing to the features that need improving.

Once the ideas are clear, the same pipeline can be expressed in Python with scikit-learn in a few lines: a scaler, a KNeighborsClassifier and a call to compute accuracy. Doing it once from scratch is what makes those lines meaningful, because each one now corresponds to code you have written and understood yourself.

Practice tasks

  1. Write a getHeight function in the same style as getWidth, and a getPointCount function that adds up the points in every path.
  2. Repeat the scaling worked example with a third training sample, a house with width 380 and 5 paths. Which class is nearest to the unknown sketch after scaling?
  3. Explain why the minimum and range used for scaling must come from the training set only, and what goes wrong if you compute them from all the data.
  4. Change the nearest function into a knn function that takes k as a parameter and returns the majority label among the k closest samples.
  5. A classifier gets 252 of 360 test sketches correct across six equally common classes. Compute its accuracy and compare it with random guessing.
  6. Describe why splitting by contributor gives a more honest test score than splitting individual sketches at random.

Install Chrysolites Tech Academy

Android / Chrome / Edge: open the browser menu and choose Install app or Add to Home screen.

iPhone / iPad (Safari): tap the Share button, then Add to Home Screen.