Orthogonality & Projections

How do you decompose a vector into a part that points along a direction and a part perpendicular to it? That decomposition is projection -- and it's everywhere.

You've been working with dot products, transformations, and subspaces. Now we bring them together. Given a vector and a direction, you can split the vector into exactly two pieces: one that lies along the direction, and one that's perpendicular to it. The piece along the direction is the projection. The perpendicular piece is the rejection (or the perpendicular component).

This isn't abstract housekeeping. Projection is how shadows work. It's how you compute the component of velocity along a slope. It's how least-squares regression finds the best fit line. Every time you ask "how much of this vector goes in that direction?", you're doing projection.

Orthogonal projection onto a vector

Start with the simplest case. You have a vector $\vec{a}$ and a direction $\vec{b}$. You want to find the "shadow" of $\vec{a}$ onto the line defined by $\vec{b}$ -- the component of $\vec{a}$ that points along $\vec{b}$.

The projection drops a perpendicular from the tip of $\vec{a}$ down onto the line through $\vec{b}$. The foot of that perpendicular is the projection. The vector from there back up to the tip of $\vec{a}$ is the perpendicular component.

Let $\vec{a} = (3, 2)$ and $\vec{b} = (4, 1)$. The projection of $\vec{a}$ onto $\vec{b}$ is:

$$\text{proj}_{\vec{b}}\vec{a} = \frac{\vec{a}\cdot\vec{b}}{\vec{b}\cdot\vec{b}}\vec{b} = \frac{12 + 2}{16 + 1}(4, 1) = \frac{14}{17}(4, 1) \approx (3.29, 0.82)$$

And the perpendicular component is $\vec{a} - \text{proj}_{\vec{b}}\vec{a} \approx (-0.29, 1.18)$.

The orange arrow is the projection -- the part of $\vec{a}$ that goes along $\vec{b}$. The purple arrow is the perpendicular component -- the part that's "left over." Notice the right angle where they meet. Together, orange + purple = blue.

The projection answers a precise question: if you shine a light straight down onto the line through $\vec{b}$, where does the shadow of $\vec{a}$'s tip land? The perpendicular component is how far above the line $\vec{a}$ reaches. The right angle is the whole point -- that's what makes this an orthogonal projection.

Projection onto a subspace

The same idea extends beyond a single direction. Instead of projecting onto a line, you can project onto a plane or any subspace. The projection is the closest point in the subspace to the original vector.

In 2D, we can visualize this as projecting onto a line (a 1D subspace). Take a vector $\vec{v} = (1, 3)$ and project it onto the subspace spanned by $\vec{u} = (3, 1)$. The projection lands on the line, and it's the point on that line closest to $\vec{v}$.

$$\text{proj}_{\vec{u}}\vec{v} = \frac{\vec{v}\cdot\vec{u}}{\vec{u}\cdot\vec{u}}\vec{u} = \frac{3 + 3}{9 + 1}(3, 1) = \frac{6}{10}(3, 1) = (1.8, 0.6)$$

The distance from $\vec{v}$ to this projection point is the shortest distance from $\vec{v}$ to the subspace. No other point on the line is closer.

The projection (orange dot) is the unique point on the gold subspace line that is closest to $\vec{v}$. The dashed gray lines show other points on the subspace -- they're all farther away. The perpendicular (purple) connecting $\vec{v}$ to its projection is the shortest path.

This "closest point" property is what makes projection so powerful. In high dimensions, when you project a data point onto a subspace, you're finding the best approximation of that point within the subspace. That's exactly what least-squares regression does: it finds the projection of the data onto the column space of the model matrix.

Orthogonal decomposition

Here's the fundamental fact: every vector can be split into exactly two perpendicular pieces relative to any subspace. One piece lives in the subspace, the other is perpendicular to it. This is the orthogonal decomposition.

$$\vec{v} = \text{proj}_{\vec{b}}\vec{v} + \vec{v}_\perp$$

The two components are perpendicular, and they add up to the original vector. No part of $\vec{v}$ is unaccounted for. No part is double-counted.

Let's see this with $\vec{v} = (2, 3)$ and $\vec{b} = (3, 0)$ -- projecting onto the x-axis direction. The projection is simply $(2, 0)$ and the perpendicular component is $(0, 3)$.

The blue vector splits cleanly into two perpendicular pieces. The orange piece (projection) lies along the x-axis. The purple piece (perpendicular component) points straight up. The right angle marker confirms they're perpendicular. Together, they reconstruct the original vector exactly.

This decomposition is unique. For a given subspace, there's only one way to split any vector into a parallel part and a perpendicular part. That uniqueness comes from the right angle constraint -- if the two pieces weren't perpendicular, you could find a shorter perpendicular path, and the decomposition wouldn't be the same.

Think of it like separating a force into horizontal and vertical components. The horizontal part tells you how much push goes sideways. The vertical part tells you how much goes up. They're independent, they don't interfere, and together they account for the entire force.

Orthogonal complement

If projection decomposes a single vector, the orthogonal complement decomposes the entire space.

Given a subspace $W$, its orthogonal complement $W^\perp$ is the set of all vectors that are perpendicular to every vector in $W$. Together, $W$ and $W^\perp$ fill up the entire space. Every vector in the space can be uniquely written as a sum of one vector from $W$ and one from $W^\perp$.

In 2D, if $W$ is a line, then $W^\perp$ is the perpendicular line. In 3D, if $W$ is a plane, then $W^\perp$ is the line perpendicular to that plane (the normal direction). If $W$ is a line in 3D, then $W^\perp$ is the plane perpendicular to it.

The green line is the subspace $W$ (direction $(2, 1)$). The purple line is its orthogonal complement $W^\perp$ (direction $(-1, 2)$). Every green vector is perpendicular to every purple vector -- their dot product is always zero. Together, these two lines span all of 2D space.

Let's verify: $(2, 1) \cdot (-1, 2) = -2 + 2 = 0$. Indeed perpendicular. And $(4, 2) \cdot (-1, 2) = -4 + 4 = 0$. Every vector in $W$ is perpendicular to every vector in $W^\perp$. That's the definition.

The orthogonal complement is deeply connected to the null space from Chapter 9. The null space of a matrix $A$ is the orthogonal complement of the row space of $A$. And the left null space is the orthogonal complement of the column space. These four subspaces -- column space, null space, row space, left null space -- form the four fundamental subspaces of a matrix, and orthogonal complements pair them up perfectly.

The formal bit

The orthogonal projection of vector $\vec{a}$ onto vector $\vec{b}$ is:

$$\text{proj}_{\vec{b}}\vec{a} = \frac{\vec{a}\cdot\vec{b}}{\vec{b}\cdot\vec{b}}\vec{b}$$

The scalar $\frac{\vec{a}\cdot\vec{b}}{\vec{b}\cdot\vec{b}}$ tells you how many copies of $\vec{b}$ make up the projection. Multiplying that scalar by $\vec{b}$ gives you the actual projection vector.

The perpendicular component is what's left over:

$$\vec{a}_\perp = \vec{a} - \text{proj}_{\vec{b}}\vec{a}$$

This perpendicular component is orthogonal to $\vec{b}$ -- you can verify by computing the dot product $\vec{a}_\perp \cdot \vec{b} = 0$.

An orthogonal set of vectors is one where every pair has zero dot product: $\vec{v}_i \cdot \vec{v}_j = 0$ for all $i \neq j$. If each vector also has length 1, the set is orthonormal. Orthogonal sets are maximally "independent" -- not just linearly independent, but pointing in completely unrelated directions.

The orthogonal complement of a subspace $W$ is:

$$W^\perp = \{\vec{v} \mid \vec{v} \cdot \vec{w} = 0 \text{ for all } \vec{w} \in W\}$$

Key properties:

• $W$ and $W^\perp$ together span the whole space: $\mathbb{R}^n = W \oplus W^\perp$
• Their dimensions add up: $\dim(W) + \dim(W^\perp) = n$
• The only vector in both $W$ and $W^\perp$ is the zero vector

Worked example: shadow casting in 3D rendering

Projection shows up literally whenever you compute shadows. Here's how.

You have a point light at position $P$ casting light in direction $\vec{L} = (1, -2, 1)$ (pointing down and to the right). You have a flat ground plane whose normal vector is $\vec{n} = (0, 1, 0)$ (pointing straight up). An object is floating above the ground at point $Q$. Where does its shadow fall?

The shadow is where the line from $Q$ in direction $\vec{L}$ intersects the ground plane. But the core of the computation is projection. You need to figure out how much of the object's position is "above" the ground and how much is "along" the ground.

Decompose $\vec{L}$ relative to $\vec{n}$:

$$\text{proj}_{\vec{n}}\vec{L} = \frac{\vec{L}\cdot\vec{n}}{\vec{n}\cdot\vec{n}}\vec{n} = \frac{-2}{1}(0, 1, 0) = (0, -2, 0)$$

That's the vertical component of the light direction -- how fast the light descends toward the ground.

$$\vec{L}_\perp = \vec{L} - \text{proj}_{\vec{n}}\vec{L} = (1, -2, 1) - (0, -2, 0) = (1, 0, 1)$$

That's the horizontal component -- the direction the shadow gets "stretched" along the ground.

Now suppose the object is at height $h = 3$ above the ground. The shadow gets displaced horizontally by:

$$\text{shadow offset} = h \cdot \frac{\vec{L}_\perp}{\lvert\text{proj}_{\vec{n}}\vec{L}\rvert} = 3 \cdot \frac{(1, 0, 1)}{2} = (1.5, 0, 1.5)$$

The shadow lands 1.5 units in $x$ and 1.5 units in $z$ away from the point directly below the object. The steeper the light angle (larger vertical component relative to horizontal), the shorter the shadow. A light shining straight down ($\vec{L} = (0, -1, 0)$) would have zero horizontal component -- the shadow would be directly beneath the object with no offset at all.

This is a simplified version of what real-time shadow mapping does. Projection matrices in graphics pipelines are quite literally named after this operation. When you set up a "projection matrix" in OpenGL or Vulkan, you're computing where 3D points project onto your 2D screen -- and the math is the same principle of orthogonal decomposition, extended to handle perspective.

What's next

Projections rely on perpendicularity. But what if your vectors aren't perpendicular yet? Gram-Schmidt gives you a recipe to make any set of vectors orthogonal -- systematically projecting out the parts that overlap, one vector at a time, until every pair is perpendicular. That's the tool that makes orthogonality practical.