Linear algebra is the math of collections of numbers that move and combine together — a vector, a matrix, a transformation. It sounds abstract until you notice how much of it is already sitting quietly inside code you've probably written: an RGB colour is a 3-vector, a CSS transform is a matrix, a "feature vector" fed into a machine learning model is, literally, a vector. This course is about making that quiet, already-present math explicit and usable on purpose, rather than leaving it as something that happens to work.

What Linear Algebra Actually Studies

Where Discrete Mathematics Fundamentals is about counting, logic, and discrete structure, linear algebra is about a different question entirely: given a collection of numbers arranged as a vector or a matrix, what operations can be done to it, and what do those operations mean geometrically? A vector can represent a position, a direction, a colour, or a list of features — the same handful of operations (add, scale, multiply by a matrix) apply regardless of what the numbers represent.

Five Concrete Connections to Code Already On This Site

Every topic in this course maps onto something already covered elsewhere on this site, usually without the underlying linear algebra ever being named:

What This Course Won't Cover

A few genuinely related topics are deliberately left for their own future courses under this same Maths for Programmers subject, rather than folded in here as extra chapters:

• Calculus-based optimization — how gradient descent actually derives its update rule, or how backpropagation's chain rule works, get their own future Calculus & Optimization course, even though eigenvalues (Chapter 9) sit right next door to that territory
• Formal vector space theory — axiomatic proofs about abstract vector spaces over arbitrary fields stay out of scope; Chapter 8 covers span, basis, and dimension, but kept concrete and code-grounded rather than proof-heavy
• Numerical stability at scale — why a huge, ill-conditioned matrix can quietly produce garbage results in floating-point arithmetic is real and important, but belongs to this subject's own future Numerical Methods & Floating-Point Computation course

Where This Course Is Headed

Hands-On Exercises

Chapter 1 introduced a vector as "an ordered list of numbers, often a point or a direction." This chapter makes that concrete: the handful of operations vectors support, what each one actually does to the arrow it represents, and — the operation that does the most real work in practice — the dot product, which turns out to be the mathematical basis for "how similar are these two things."

Vector Addition — Tip to Tail

Adding two vectors is done component by component: a + b = [a₁+b₁, a₂+b₂]. Geometrically, this is the "tip to tail" picture — place the second vector's tail at the first vector's tip, and the sum is the arrow from the very start to the very end.

A worked example used throughout this chapter: a = [3, 4] and b = [1, 2]. So a + b = [4, 6] — verified above.

Scalar Multiplication — Stretching, Shrinking & Reversing

Multiplying a vector by a plain number (a scalar) scales every component by that number: k·a = [k·a₁, k·a₂]. Geometrically, a scalar greater than 1 stretches the arrow, a scalar between 0 and 1 shrinks it, and a negative scalar reverses its direction entirely while still scaling its length.

Magnitude — How Long Is the Arrow

The magnitude (or "norm," written |a|) of a vector is its length, computed with the Pythagorean theorem generalized to as many dimensions as the vector has:

For a = [3, 4]: |a| = √(3² + 4²) = √(9 + 16) = √25 = 5 — the classic 3-4-5 right triangle, just relabelled as a vector.

Normalization — Keeping Only the Direction

Dividing a vector by its own magnitude produces a unit vector — a vector of length exactly 1 that points in the same direction as the original, with the "how far" information stripped away and only "which way" left behind. This matters constantly in graphics and game code, where a direction is often needed independent of any particular distance.

Normalizing a = [3, 4] (magnitude 5): a / |a| = [3/5, 4/5] = [0.6, 0.8]. Checking the result is genuinely a unit vector: √(0.6² + 0.8²) = √(0.36 + 0.64) = √1 = 1 ✓.

The Dot Product — Measuring "How Aligned"

The dot product of two vectors is computed algebraically by multiplying corresponding components and summing the results: a · b = a₁b₁ + a₂b₂ + ... + aₙbₙ. It produces a single number, not a vector — and that number turns out to have a precise geometric meaning.

The geometric meaning connects the dot product directly to the angle θ between the two vectors:

Using a = [3, 4] and b = [1, 2]: a · b = 11, |a| = 5, |b| = √5 ≈ 2.236, so cos(θ) = 11 / (5 × 2.236) ≈ 0.984, giving θ ≈ 10.3° — a and b point in nearly, but not exactly, the same direction.

Vectors in Code — By Hand vs. NumPy

Every operation above was written from scratch using plain Python lists, which is worth doing once to see exactly what's happening. In real code, especially anything performance-sensitive or higher-dimensional, the NumPy library provides all of these operations directly and considerably faster:

Hands-On Exercises

Chapter 2's dot product takes two vectors and returns a single number. The cross product is a different kind of operation entirely: it takes two 3D vectors and returns a third vector — one that's perpendicular to both of the originals. It's specifically a 3D operation, which is itself worth understanding, not just the formula.

The Formula

For a = [a₁, a₂, a₃] and b = [b₁, b₂, b₃], the cross product is:

It looks arbitrary at first, but there's a pattern: each output component skips the matching input index (the first output component has no a₁/b₁ in it, and so on), and the two products in each pair are subtracted in a "cross" pattern. Chapter 7's own determinant will give this same formula a cleaner, more memorable form once determinants are available.

The Right-Hand Rule

The direction of a × b is found with the right-hand rule: point your right hand's fingers along a, curl them toward b, and your thumb points in the direction of the result. This is also why order matters — the cross product is anti-commutative:

Geometric Meaning: Perpendicular, With a Meaningful Length

Two things are true about a × b at once, and both matter in practice:

1. Direction: it's perpendicular to both a and b — confirmed by the fact that (a × b) · a = 0 and (a × b) · b = 0 always hold, using Chapter 2's own dot product as the perpendicularity test.
2. Magnitude: |a × b| = |a| |b| sin(θ) — which is exactly the area of the parallelogram that a and b span.

Worked example: a = [2, 1, 0], b = [1, 0, 2].

Testing for Parallel Vectors

Since sin(θ) = 0 exactly when θ is 0° or 180°, the cross product of two parallel (or anti-parallel) vectors is always the zero vector, regardless of their lengths. This gives a direct, practical test: if a × b = [0, 0, 0], the two vectors point along the same line.

2D vs. 3D: There's No True 2D Cross Product

The cross product as defined above genuinely requires three dimensions — "perpendicular to both a and b" only pins down a unique direction (up to sign) when there's a third axis to be perpendicular into. In 2D, a lot of code still uses a "cross product," but it's really a shortcut scalar, not a true cross product:

Where This Shows Up: Surface Normals

In 3D graphics, a flat triangular face is usually stored as three corner points. To light that face correctly, the renderer needs its normal — a vector pointing straight out from the surface. That's computed by taking two of the triangle's edges as vectors and crossing them:

Cross Products in Code — NumPy

Hands-On Exercises

Chapters 2 and 3 worked entirely with vectors — single ordered lists of numbers. This chapter introduces the other core object of linear algebra: the matrix, a rectangular grid of numbers arranged in rows and columns. A matrix can represent a table of data, an image's pixel grid, or — most importantly for this course — a rule for transforming vectors, which Chapter 5 covers in full.

What a Matrix Is

A matrix with m rows and n columns is called an "m × n matrix." Each individual number is an entry, identified by its row and column position.

Matrix Addition & Scalar Multiplication — The Easy Part

These work exactly like the vector versions from Chapter 2: entry by entry. Addition requires both matrices to have the exact same shape.

Worked example: E = [[2, -1], [0, 4]], F = [[1, 3], [-2, 5]].

Matrix Multiplication — The Real Operation

This is the operation that actually does the interesting work, and it does not mean multiplying entry-by-entry the way addition does. To find the entry at row i, column j of the result, take the dot product (Chapter 2 again) of row i from the first matrix and column j from the second.

Worked example: A = [[1, 2], [3, 4]], B = [[5, 6], [7, 8]] — both 2×2, so the result is 2×2.

So A × B = [[19, 22], [43, 50]].

Why Matrix Multiplication Isn't Commutative

Computing B × A instead — same two matrices, reversed order — gives [[23, 34], [31, 46]], a completely different result from A × B = [[19, 22], [43, 50]].

The dimension rule can make the non-commutativity even starker. With a non-square 2×3 matrix C and a 3×2 matrix D, C × D is a valid 2×2 matrix — but D × C is also valid, and produces a 3×3 matrix. Same two matrices, same multiplication rule, two entirely different-shaped results, purely from the order they're multiplied in.

The Identity Matrix

The identity matrix I is the matrix equivalent of the number 1: multiplying any matrix by it (in either order, where the shapes allow) leaves that matrix completely unchanged. It has 1s down the main diagonal and 0s everywhere else.

Confirmed with the earlier example: A × I = [[1, 2], [3, 4]] — exactly A, unchanged.

Matrices in Code — NumPy

Hands-On Exercises

Chapter 4 ended with a forward reference: a matrix is "a rule for transforming vectors." This chapter makes that literal. Multiplying a matrix by a vector — a special case of Chapter 4's own matrix multiplication, where the second matrix has just one column — produces a new vector, and a small, well-known family of matrices produce specific, useful transformations: scaling, rotating, reflecting, and shearing.

Matrix-Vector Multiplication — The Same Rule, One Column Wide

A vector v = [3, 4] can be written as a 2×1 matrix, a single column: [[3], [4]]. Multiplying a 2×2 matrix by it uses exactly Chapter 4's row-by-column rule — each output entry is the dot product of a matrix row and the vector's single column.

Scaling

A scaling matrix has the scale factors on the diagonal, zeros elsewhere:

With S = [[2, 0], [0, 0.5]] and v = [3, 4]: S v = [2×3, 0.5×4] = [6, 2] — twice as wide, half as tall.

Rotation

A rotation by angle θ (counter-clockwise) uses:

For θ = 90°: cos(90°) = 0, sin(90°) = 1, so R(90°) = [[0, −1], [1, 0]]. Applied to v = [3, 4]: R v = [0×3 + (−1)×4, 1×3 + 0×4] = [−4, 3].

Reflection

Reflecting across the x-axis flips the sign of y only; reflecting across the y-axis flips the sign of x only:

Shear

A shear slides one axis's coordinates sideways in proportion to the other axis, turning a rectangle into a parallelogram without changing its area:

With k = 1: Sh v = [1×3 + 1×4, 0×3 + 1×4] = [7, 4] — x shifted by an amount proportional to y, y unchanged.

Composing Transformations — And Why Order Matches Chapter 4

To apply two transformations in sequence, multiply their matrices together — and because Chapter 4 already established that matrix multiplication isn't commutative, the order genuinely changes the result. R × S means "apply S first, then R" (read right to left, matching how it's applied to a vector: (R × S) v = R (S v)).

Reusing S = [[2, 0], [0, 0.5]], R = [[0, −1], [1, 0]], and v = [3, 4]:

Homogeneous Coordinates — Why Translation Needs a Trick

Every transformation above is linear — algebraically, a 2×2 matrix multiplication always sends the origin [0, 0] to itself (M × [0,0] = [0,0], for any M). Translation — sliding every point by a fixed offset — genuinely moves the origin, so no plain 2×2 matrix can represent it at all.

The fix is homogeneous coordinates: pad every 2D vector with an extra 1, turning [x, y] into [x, y, 1], and use a 3×3 matrix. The extra row and column let a translation "leak" into the result through that constant 1:

Translating v = [3, 4] (as [3, 4, 1]) by (tₓ, tᵧ) = (5, −2): T × [3, 4, 1] = [1×3 + 0×4 + 5×1, 0×3 + 1×4 + (−2)×1, 1] = [8, 2, 1] — the point (8, 2), exactly (3+5, 4−2), as expected.

Hands-On Exercises

A system of linear equations is just several linear equations sharing the same unknowns, all required to hold at once. Chapters 4 and 5 already built the exact notation for this: a system is A x = b, where A is a matrix of coefficients, x is the vector of unknowns being solved for, and b is the vector of right-hand-side values.

From Equations to a Matrix

Consider the system:

This is A x = b with:

Gaussian Elimination — Systematic Row Reduction

Rather than guessing or using substitution tricks, Gaussian elimination works on the augmented matrix — A with b tacked on as an extra column — using three row operations that are guaranteed never to change the solution set:

• Swap two rows
• Multiply a row by a nonzero scalar
• Add a multiple of one row to another row

The goal is to use these operations to eliminate variables from rows, one at a time, until the system is easy to solve by simple substitution.

The second row now reads 0x − 3y = −9, which only involves y — the x term has been eliminated. This is called row echelon form: each row's leading (leftmost nonzero) entry sits strictly to the right of the row above it.

Back-Substitution

Once the system is in echelon form, solve from the bottom row upward, substituting each known value into the row above it:

So x = 2, y = 3. Checking against the original second equation: 2(2) − 3 = 4 − 3 = 1 ✓.

When There's No Solution

Consider instead:

Eliminating: R1 → R1 − 2·R2 gives [0, 0 | −6] — the row 0x + 0y = −6, which is never true for any x or y. Geometrically, these are two parallel lines (both have the same slope, x + y = k for different k) that never intersect. Whenever elimination produces a row that's all zeros on the left but nonzero on the right, the system has no solution.

When There Are Infinitely Many Solutions

Now consider:

Eliminating: R2 → R2 − 2·R1 gives [0, 0 | 0] — the row 0x + 0y = 0, which is always true, for any x and y. The second equation was never independent information at all — it's exactly the first equation multiplied by 2. Whenever elimination produces a row that's entirely zero on both sides, that equation contributed nothing new, and there's a free variable: letting x = t for any value t, y = 3 − t — every point on the line x + y = 3 is a valid solution.

Systems in Code — NumPy

Hands-On Exercises

Chapter 6 ended with a forward reference: a single number, computed directly from a matrix, that instantly reveals whether a system of equations has a unique solution. That number is the determinant. This chapter defines it, gives it a genuine geometric meaning, and uses it to build the matrix inverse — the closest thing a matrix has to "dividing by" itself.

The Determinant Formula

For a 2×2 matrix, the determinant is refreshingly simple:

This is exactly the calculation Chapter 6 used to flag a "no unique solution" system, and it's exactly Chapter 3's own 2D "pseudo-cross-product" (a₁b₂ − a₂b₁) — stacking two vectors as the rows of a matrix and taking the determinant is the same number as taking their 2D cross product.

Geometric Meaning: Area Scaling

A matrix, per Chapter 5, transforms every point in the plane. The unit square (corners at [0,0], [1,0], [0,1], [1,1], area exactly 1) gets mapped to some parallelogram — and |det(A)| is exactly that parallelogram's area. The determinant is the transformation's area-scaling factor.

The Determinant as a Singularity Test

If det(A) = 0, the transformation squashes the entire plane down onto a line (or a single point) — all area is destroyed, mapped to zero. A matrix with a zero determinant is called singular. This is precisely why Chapter 6's two "no unique solution" systems both had determinant 0: their coefficient matrices collapse the plane, so there's either no point that lands exactly on b, or a whole line of points that do.

The Matrix Inverse

The inverse of a matrix A, written A⁻¹, is the matrix satisfying A A⁻¹ = A⁻¹ A = I — the matrix equivalent of a reciprocal. For 2×2 matrices, it has a direct formula built from the determinant:

Worked example, reusing Chapter 6's own system matrix A = [[1, 1], [2, -1]], det(A) = -3:

Verifying A × A⁻¹ = I: [[1×⅓+1×⅔, 1×⅓+1×(-⅓)], [2×⅓+(-1)×⅔, 2×⅓+(-1)×(-⅓)]] = [[1, 0], [0, 1]] ✓.

Solving a System With the Inverse

Since A x = b and A⁻¹ A = I, multiplying both sides by A⁻¹ gives x = A⁻¹ b directly — no elimination needed, if the inverse is already known. Reusing Chapter 6's exact system (A = [[1,1],[2,-1]], b = [5, 1]): A⁻¹ b = [⅓×5 + ⅓×1, ⅔×5 + (-⅓)×1] = [2, 3] — exactly the x = 2, y = 3 Chapter 6 found by elimination.

Determinants & Inverses in Code

Hands-On Exercises

This chapter is the most abstract in the course, so it stays deliberately concrete: no axioms, no formal proofs — just the four ideas (span, linear independence, basis, dimension) explained through the same kind of worked, checkable examples used throughout, reusing Chapter 6's row reduction and Chapter 7's determinant as the actual tools that answer these questions.

Linear Combinations & Span

A linear combination of vectors is just Chapter 2's addition and scalar multiplication, applied together: c₁v₁ + c₂v₂ + ... for any scalars c₁, c₂, .... The span of a set of vectors is the set of every linear combination reachable from them — everywhere those vectors, scaled and combined in every possible way, can reach.

Linear Independence — Testing With Chapter 7's Determinant

A set of vectors is linearly independent if none of them is redundant — none can be written as a combination of the others. For exactly two vectors in 2D, Chapter 7's determinant gives an instant test: nonzero determinant means independent (and their span is the whole plane); zero determinant means dependent (redundant — their span collapses to a line).

Linear Independence for More Vectors — Chapter 6's Row Reduction

The determinant shortcut only works for exactly two vectors in 2D. For any other number of vectors, or higher dimensions, the general tool is exactly Chapter 6's row reduction: stack the vectors as rows of a matrix and eliminate. Every row that reduces to all zeros represents a vector that added no new information — it was already reachable from the others. The number of surviving nonzero rows is called the rank of the set.

Worked example: u₁ = [1, 2, 1], u₂ = [0, 1, 2], u₃ = [1, 3, 3]. Notice first that u₁ + u₂ = [1, 3, 3] = u₃ exactly — a strong hint this set is dependent.

Row 3 reduced to all zeros — confirming u₃ really was redundant. The rank is 2, not 3: this set of three vectors spans only a 2D plane sitting inside 3D space, not the full 3D space.

Basis

A basis for a space is a set of vectors that is both linearly independent and spans the entire space — the minimal, non-redundant "building block" set that reaches everywhere. The simplest example is the standard basis: i = [1, 0] and j = [0, 1] for the 2D plane — independent (det = 1 ≠ 0), and obviously spanning everything, since any point [x, y] = x·i + y·j.

Testing whether c = [1, 0, 0], d = [0, 1, 0], e = [1, 1, 1] form a basis for 3D space, by row reduction:

All three rows survive — rank 3, fully independent. Since there are exactly three independent vectors in 3D space, they automatically span all of it: {c, d, e} is a valid basis.

Dimension

Span, Basis & Rank in Code

Hands-On Exercises

Most vectors, transformed by a matrix, get both moved to a new direction and scaled. This chapter is about the special exceptions: directions a given transformation only scales, never rotates off their own line. Those special directions — and how much they get scaled by — turn out to be some of the most useful numbers in all of linear algebra.

The Definition

The simplest possible example reuses Chapter 5's own reflection matrix, Rx = [[1, 0], [0, -1]] (reflection across the x-axis) — since it's diagonal, its eigenvalues are just the diagonal entries themselves:

Both stay on their own line — one unmoved, one flipped — which is exactly what makes them eigenvectors of this particular transformation.

Finding Eigenvalues — The Characteristic Equation

For a matrix without such an obviously convenient diagonal shape, eigenvalues need to be solved for directly. Starting from A v = λ v, rewrite as A v − λ v = 0, then (A − λI) v = 0. For a nonzero v to satisfy this, (A − λI) can't have an inverse — per Chapter 7, that means its determinant must be exactly zero:

For a 2×2 matrix A = [[a, b], [c, d]], this expands to a quadratic in λ:

Worked Example

Let A = [[2, 1], [1, 2]]. Trace = 4, determinant = 2(2) − 1(1) = 3. The characteristic equation is:

To find each eigenvector, substitute the eigenvalue back into (A − λI) v = 0 and solve — this is exactly a homogeneous version of Chapter 6's system-solving, and it's guaranteed to have a whole line of solutions (not just v = 0), since (A − λI) is singular by construction.

Checking both: A[1,-1] = [2−1, 1−2] = [1,-1] = 1×[1,-1] ✓, and A[1,1] = [2+1, 1+2] = [3,3] = 3×[1,1] ✓.

Repeated Application & the Dominant Eigenvalue

Applying A over and over to almost any starting vector reveals something striking: the result increasingly lines up with the eigenvector belonging to the largest-magnitude eigenvalue. Starting from v₀ = [1, 0] (not an eigenvector) and repeatedly applying A = [[2,1],[1,2]]:

The direction is visibly converging toward [0.707, 0.707] — the normalized form of the λ = 3 eigenvector, [1, 1], which is the larger of the two eigenvalues. And the magnitude roughly triples at each step for large n, matching that same dominant eigenvalue.

Real Relevance

Eigenvalues & Eigenvectors in Code

Hands-On Exercises

One continuous worked project, touching every chapter of this course in the order a real engineer would actually reach for each idea: building a small 2D sprite transform pipeline for a game, then using the exact same mathematics to analyze where players click on the resulting screen.

Part 1 — A Sprite Transform Pipeline

Part 2 — Feeding Into Player-Click Analysis

This is, in essence, exactly what a real 2D game engine's rendering pipeline and its analytics tooling both look like under the hood — every step traceable to a specific chapter of this course, none of it abstract math floating free of the actual code.

What This Course Doesn't Cover

In the interest of an honest accounting: calculus-based optimization (how gradient descent's update rule is actually derived), formal vector space theory (axiomatic proofs over abstract fields), and numerical stability at scale were all named in Chapter 1 as deliberately out of scope, each reserved for its own future course under this same Maths for Programmers subject — Calculus & Optimization, and Numerical Methods & Floating-Point Computation respectively. This course is the direct, concrete foundation those subjects will each build on, not a substitute for studying them when their own turn comes.

This Course's Throughline, Restated

Where This Course Connects

Reusing Chapter 1's own five connections: this course is the direct foundation under Machine Learning Fundamentals and Neural Networks & Deep Learning's use of vectors and matrices, Blender Fundamentals and Vector Graphics/Figma's own transform tooling, and Data Science Fundamentals' use of PCA for dimensionality reduction — all of which this capstone touched directly. Within this subject's own future courses, Calculus & Optimization will build directly on Chapter 5's transformations and Chapter 9's eigenvalues (gradient descent is, at its core, repeatedly applying a transformation), and Numerical Methods & Floating-Point Computation will build directly on Chapter 6's elimination and Chapter 7's inverse computations, where floating-point rounding actually starts to matter at scale.

Hands-On Exercises