Projections

Projections

Let's project b onto a.

e is the error. We know that p is some multiple x of a. we'd like to find x. The key is that a is perpendicular to e. In other words that:

(because Ax = b -> AT * Ax = b * At, then factorize and make equation = 0).

If we simplify we get x:

The projection is, if we take p=ax

(when you multiply a vector b by a matrix, you always land in the column space). Therefore:

because P (capital P) is a matrix.

is a matrix (it's a column times a row which this is the case.)... We also know that P T = T so it's symetric.

Also, if I project twice, its the same result as projecting once, so P^2 = P.

Why project? Because Ax=b may have no solutions. So i'll try to solve the closest problem that I can solve. So therefore, I change b, and i chose the closest vector in the column space. Which means i'll solve Ax-hat=p instead, where p is the proejction of b in C(A) and it's the closest possible vector to b. The x-hat is there because it's not our initial x.

In 3 dimensions, this is the problem:

We have a matrix A, which has cols a1 and a2 (independent) and b, which is probably not in the plane. If it were we'd just take b. But we want the projection of b onto a, where e = b - p is perpendicular to the plane.

So we're looking for x-hat where p: which means that the error vector is perp to the plane. (In 2 dimensions p= xa, but in 3 dimensions, there is a1, and a2).

We have 2 equations because 2 x-hats and 2 a's.

Now let's put this into matrix form:

(b - A x-hat) = e. We're learning here that e is in the nullspace of A because the equation = 0 so they are orthogonal.

Our final equation to solve is:

Let's simplify and remember the definition of p, then combine both equations

So actually, P is the identify matrix IF and only IF A is square and invertible. If that is the case then the column space of A = R^n, so the projection of b onto the column space is the identify matrix, because b is already in the column space.

P is supposed to project that vector b to the nearest point in the column space.

If b is in the column space of A, then Pb = b because b we apply the identify matrix to b, which is b. If b is in the col space, then b = Ax. If we replace b by Ax in the second formula above, everything cancels, we are left with Ax, which = b.

Pb = 0 if b is perpendicular to the column space. if b is perpendicular to the column space, we also know it belongs to the nullspace of A transpose.

To sum up:

We have our col space and nullspace of A transpose. The projection of b onto the col space = p, and e is the error, both of them for b : b=p+e. This is what our matrix does.

For other A's, we can't simplify this formula and take the inverse, cuz it doesn't exist. Now let's use the propertises we know about P we defined earlier.

Is it symetrical? Yes, P T = P. How about P^2 = P? Yep it works out.

Now let's look at the application with least squares.