Vectors: Operations & Geometric Intuition

Linear Algebra Fundamentals

Chapter 2 · Vectors: Operations & Geometric Intuition

Chapter 1 introduced a vector as "an ordered list of numbers, often a point or a direction." This chapter makes that concrete: the handful of operations vectors support, what each one actually does to the arrow it represents, and — the operation that does the most real work in practice — the dot product, which turns out to be the mathematical basis for "how similar are these two things."

Vector Addition — Tip to Tail

Adding two vectors is done component by component: a + b = [a₁+b₁, a₂+b₂]. Geometrically, this is the "tip to tail" picture — place the second vector's tail at the first vector's tip, and the sum is the arrow from the very start to the very end.

a = [3, 4]
b = [1, 2]

# component-wise addition
a_plus_b = [a[i] + b[i] for i in range(len(a))]
print(a_plus_b)  # [4, 6]

A worked example used throughout this chapter: a = [3, 4] and b = [1, 2]. So a + b = [4, 6] — verified above.

Scalar Multiplication — Stretching, Shrinking & Reversing

Multiplying a vector by a plain number (a scalar) scales every component by that number: k·a = [k·a₁, k·a₂]. Geometrically, a scalar greater than 1 stretches the arrow, a scalar between 0 and 1 shrinks it, and a negative scalar reverses its direction entirely while still scaling its length.

Scalar	2 · a (a = [3, 4])	Geometric effect
k = 2	[6, 8]	Same direction, twice as long
k = 0.5	[1.5, 2]	Same direction, half as long
k = -1	[-3, -4]	Exactly opposite direction, same length

Magnitude — How Long Is the Arrow

The magnitude (or "norm," written |a|) of a vector is its length, computed with the Pythagorean theorem generalized to as many dimensions as the vector has:

# |a| = sqrt(a1^2 + a2^2 + ... + an^2)
import math

def magnitude(v):
    return math.sqrt(sum(x**2 for x in v))

print(magnitude([3, 4]))  # 5.0

For a = [3, 4]: |a| = √(3² + 4²) = √(9 + 16) = √25 = 5 — the classic 3-4-5 right triangle, just relabelled as a vector.

Normalization — Keeping Only the Direction

Dividing a vector by its own magnitude produces a unit vector — a vector of length exactly 1 that points in the same direction as the original, with the "how far" information stripped away and only "which way" left behind. This matters constantly in graphics and game code, where a direction is often needed independent of any particular distance.

Normalizing a = [3, 4] (magnitude 5): a / |a| = [3/5, 4/5] = [0.6, 0.8]. Checking the result is genuinely a unit vector: √(0.6² + 0.8²) = √(0.36 + 0.64) = √1 = 1 ✓.

A real bug: normalizing the zero vector

The zero vector [0, 0] has magnitude 0, and dividing by 0 either crashes or produces NaN depending on the language. Any code that normalizes a vector coming from user input or a computed difference (e.g. "direction from A to B" when A and B happen to be the same point) needs to guard against this case explicitly.

The Dot Product — Measuring "How Aligned"

The dot product of two vectors is computed algebraically by multiplying corresponding components and summing the results: a · b = a₁b₁ + a₂b₂ + ... + aₙbₙ. It produces a single number, not a vector — and that number turns out to have a precise geometric meaning.

def dot(a, b):
    return sum(a[i] * b[i] for i in range(len(a)))

a = [3, 4]
b = [1, 2]
print(dot(a, b))  # 3*1 + 4*2 = 3 + 8 = 11

The geometric meaning connects the dot product directly to the angle θ between the two vectors:

The dot product / angle formula

a · b = |a| |b| cos(θ) — rearranged, cos(θ) = (a · b) / (|a| |b|). The dot product is, in effect, "how much of b points in the same direction as a," scaled by both lengths.

Using a = [3, 4] and b = [1, 2]: a · b = 11, |a| = 5, |b| = √5 ≈ 2.236, so cos(θ) = 11 / (5 × 2.236) ≈ 0.984, giving θ ≈ 10.3° — a and b point in nearly, but not exactly, the same direction.

Sign of a · b	What it means about the angle
Positive	Angle is less than 90° — vectors point in roughly the same general direction
Zero	Angle is exactly 90° — the vectors are orthogonal (perpendicular)
Negative	Angle is more than 90° — vectors point in roughly opposite directions

Why this is "similarity" in machine learning

A feature vector is just a vector with more than two or three components. The dot product (usually normalized into what's called cosine similarity) is exactly how tools compare two feature vectors — two documents, two user preference profiles, two embeddings — for "how alike are these," with no geometry required in the visual sense at all. The same formula, unchanged.

Vectors in Code — By Hand vs. NumPy

Every operation above was written from scratch using plain Python lists, which is worth doing once to see exactly what's happening. In real code, especially anything performance-sensitive or higher-dimensional, the NumPy library provides all of these operations directly and considerably faster:

import numpy as np

a = np.array([3, 4])
b = np.array([1, 2])

print(a + b)                      # [4 6]
print(2 * a)                        # [6 8]
print(np.linalg.norm(a))           # 5.0  (magnitude)
print(a / np.linalg.norm(a))       # [0.6 0.8]  (normalized)
print(np.dot(a, b))                # 11  (dot product)

Hands-On Exercises

Exercise 1

Given c = [6, 8] and d = [-2, 1], compute by hand: (a) c + d, (b) 3 · d, (c) the magnitude of c, (d) the normalized (unit) vector for c. Show your working for each.

📄 View solution

Exercise 2

Compute the dot product of c = [6, 8] and d = [-2, 1], then use it to find the angle between them in degrees. Based only on the sign of the dot product (before finishing the full angle calculation), predict whether the angle should be less than, equal to, or greater than 90° — then confirm your prediction with the final answer.

📄 View solution

Exercise 3

A recommendation system represents two users as feature vectors: user_x = [5, 1, 0] and user_y = [4, 2, 1] (three genres, rated 0–5). Compute the dot product of the two vectors, and explain in plain terms what a high dot product suggests about these two users' tastes — and why a raw dot product alone can be misleading if one user rates everything much higher than the other (hint: think about what normalizing each vector first would fix).

📄 View solution

Chapter 2 Quick Reference

Addition: component-wise, geometrically "tip to tail" — a + b = [a₁+b₁, a₂+b₂, ...]
Scalar multiplication: scales every component; a negative scalar reverses direction — k·a = [k·a₁, k·a₂, ...]
Magnitude: |a| = √(a₁² + a₂² + ... + aₙ²) — the vector's length
Normalization: a / |a| gives a unit vector (length 1) pointing in the same direction — guard against dividing by a zero-length vector
Dot product: a · b = a₁b₁ + a₂b₂ + ... ; geometrically a · b = |a||b|cos(θ)
Sign of the dot product tells you the angle category: positive → <90°, zero → exactly 90° (orthogonal), negative → >90°
The dot product is the mathematical basis of "similarity" between feature vectors in machine learning
Next chapter: The cross product and working specifically in 3D