Chrysolites Tech Academy

Deep Learning with PyTorch · Course book

Chapter 2: Tensors, Gradients and Linear Regression

Pairs with Lesson 2: Tensors, Gradients and Linear Regression

Every deep learning model, however large, is built from the same three ingredients: tensors that hold the data and parameters, gradients that say how to change the parameters, and a loop that applies those changes over and over. This chapter builds all three from first principles in PyTorch, using linear regression as the running example because it is small enough to compute by hand. You will create tensors and see how PyTorch tracks the operations performed on them, let automatic differentiation compute gradients and check them against calculus, then train a linear regression model from scratch with nothing but tensor operations. Finally, you will rebuild the same model using PyTorch's built-in layers, losses, datasets and optimisers, the tools every later chapter relies on.

Notebooks, tensors and NumPy

Deep learning work is usually done in Jupyter notebooks, documents that mix runnable code cells, their outputs and written explanations. Notebooks can run on your own computer or on free cloud platforms that also offer graphics cards, and they make it easy to experiment one step at a time, inspecting every intermediate result.

A PyTorch tensor is a number, vector, matrix or higher-dimensional array. Creating one from a Python number or list is a single call, and every element shares one data type. A tensor created from the list [[5., 6], [7, 8], [9, 10]] has shape (3, 2): three rows and two columns. Tensors support arithmetic element by element, matrix multiplication with the @ operator, and many mathematical functions.

Two details save a lot of confusion later. First, the data type matters: a tensor made from whole numbers holds integers, while neural network weights need floating-point numbers, so writing 5. rather than 5, or asking for torch.float32 explicitly, avoids type errors when the two meet. Second, PyTorch supports broadcasting: when two tensors of different shapes are combined, the smaller one is stretched to match where that makes sense. Adding a tensor of shape (2,) holding 1 and 10 to the (3, 2) tensor above adds 1 to every value in the first column and 10 to every value in the second, giving [[6, 16], [8, 18], [10, 20]]. Broadcasting is exactly what lets a single bias vector be added to the predictions for every sample in a batch.

PyTorch tensors are deliberately similar to NumPy arrays, the standard numerical arrays in Python, and convert in both directions with torch.from_numpy and .numpy(). The conversion shares memory where possible, so it is fast. Why not just use NumPy? Two reasons: PyTorch tensors can live on a graphics card, where large operations run many times faster, and PyTorch can automatically compute gradients of any calculation made with tensors.

Gradients with autograd

A gradient is the rate at which an output changes as one input changes, the derivative from calculus. When a tensor is created with requires_grad=True, PyTorch records every operation that uses it. Calling .backward() on a final result works back through that record and stores, in each tracked tensor's .grad attribute, the derivative of the result with respect to it. This is autograd, automatic differentiation.

Worked example. Let x = 2, and create w = 3 and b = 1 with gradients tracked. Compute y = w × x + b = 7. By calculus, the derivative of y with respect to w is x = 2, and with respect to b is 1. After y.backward(), w.grad holds 2 and b.grad holds 1, exactly as expected. Because x was created without gradient tracking, x.grad is empty. If y had been w2 × x + b instead, the derivative with respect to w would be 2wx = 12.

One behaviour surprises everyone once: gradients accumulate. Calling backward again adds the new gradients to those already stored, rather than replacing them. That is useful in some advanced situations, but in an ordinary training loop the gradients must be reset to zero after each update, or every step would use a growing sum of all previous gradients.

Linear regression from scratch

Linear regression predicts each output as a weighted sum of the inputs plus a bias. With inputs arranged as a matrix X, one row per sample, weights W and biases b, the predictions for every sample at once are X @ WT + b. The model starts with random weights; training adjusts them until the predictions match the targets.

The quality of the predictions is measured by the mean squared error (MSE): subtract target from prediction for every sample, square the differences, and average them. The gradient of the MSE with respect to each weight has a simple form: two divided by the number of samples, times the sum over samples of the error multiplied by the matching input. Autograd computes this automatically, but working it once by hand makes it concrete.

Worked example. Three samples have two inputs each, (2, 1), (1, 3) and (3, 2), with targets 5, 8 and 8. Start with both weights and the bias at 0. Every prediction is 0, the errors are −5, −8 and −8, and the MSE is (25 + 64 + 64) / 3 = 51. The gradient for the first weight is (2/3) × (−5 × 2 − 8 × 1 − 8 × 3) = (2/3) × (−42) = −28. For the second it is (2/3) × (−5 − 24 − 16) = −30, and for the bias (2/3) × (−21) = −14. With a learning rate of 0.01, each parameter moves against its gradient: the weights become 0.28 and 0.30 and the bias 0.14. The new predictions are 1.00, 1.32 and 1.58, and the MSE falls to 33.95.

Repeating this is gradient descent. In PyTorch, each step computes the predictions, computes the loss, calls loss.backward(), then, inside a torch.no_grad() block so the update itself is not tracked, subtracts the learning rate times each gradient and resets the gradients to zero. An epoch is one pass over the training data. On the three-sample example, the loss falls quickly at first and then ever more slowly as the weights approach the exact answer, weights 1 and 2 and a bias of 1.

10010.01 050100150200 epochMSE (log scale) 1.16 at epoch 10
Figure 2.1 — Loss during training on the three-sample example, on a logarithmic scale. Each tick down is a hundredfold reduction; progress slows as the weights near the exact solution.

After 200 epochs the weights are about 1.06 and 2.04 and the bias about 0.79, with a loss near 0.002. They are still creeping towards the exact values, which shows a typical pattern: most of the improvement comes early, and the last small gains take many more steps. A larger learning rate would converge faster, up to the point where steps overshoot and the loss starts to grow.

The same model with PyTorch built-ins

Writing everything by hand is the best way to understand it, but PyTorch provides tested building blocks that make the same model shorter and less error-prone.

From scratchBuilt-in replacementWhat it adds
Weights and bias tensors with gradientsnn.Linear(2, 1)Creates and initialises the parameters for you
Hand-written MSEF.mse_lossA standard, efficient loss function
Manual subtraction of gradientstorch.optim.SGDUpdates every parameter with step() and clears with zero_grad()
Indexing input and target tensorsTensorDatasetPairs inputs with targets as one dataset
Looping over all data at onceDataLoaderServes shuffled mini-batches

With these, a training function takes the number of epochs, the model, the loss function, the optimiser and the data loader. For each epoch it loops over the batches: compute predictions, compute the loss, call backward, call step, call zero_grad. Printing the loss every ten epochs confirms progress.

model = nn.Linear(2, 1)
opt = torch.optim.SGD(model.parameters(), lr=1e-2)
loader = DataLoader(TensorDataset(inputs, targets), batch_size=5, shuffle=True)

for epoch in range(100):
    for xb, yb in loader:
        loss = F.mse_loss(model(xb), yb)
        loss.backward()
        opt.step()
        opt.zero_grad()

Using mini-batches rather than the whole dataset at each step is called stochastic gradient descent. Each batch gives a noisy estimate of the true gradient, but the steps are much cheaper, and the noise can even help the model avoid poor solutions. Shuffling the data each epoch ensures every batch is a fair mixture. With 15 samples and a batch size of 5, each epoch makes 3 updates instead of 1.

Once trained, the model is used by simply calling it on new inputs: model(new_inputs) returns the predictions. Comparing predictions with targets for the training samples is a quick sanity check, and printing model.weight and model.bias shows what was learned. Because those parameters were created by nn.Linear with gradient tracking switched on, predictions made purely for inspection are best wrapped in torch.no_grad(), which saves memory and makes it clear that no training is happening.

The same pattern, a model, a loss, an optimiser and a loop over batches, carries through every chapter that follows. Only the model and the loss will change as the problems get harder.

Practice tasks

  1. With x = 4, w = 2 and b = 3, compute y = w2x + b and the gradients autograd would report for w and b.
  2. Continue the worked example for one more step from weights 0.28 and 0.30 and bias 0.14, and check that the loss falls to about 22.64.
  3. Explain what would go wrong if you forgot to zero the gradients after each update.
  4. A dataset has 1,000 samples and a batch size of 64. How many updates happen per epoch, counting a final partial batch?
  5. What shape is the weight tensor of nn.Linear(3, 2), and how many parameters does the layer have in total?
  6. Rewrite the from-scratch training step using nn.Linear, F.mse_loss and torch.optim.SGD.

Install Chrysolites Tech Academy

Android / Chrome / Edge: open the browser menu and choose Install app or Add to Home screen.

iPhone / iPad (Safari): tap the Share button, then Add to Home Screen.