Inner Product Space

$\spadesuit$Piecewise Continuous Function $\spadesuit$

In this section, we introduce a vector space which is very different from spce vector. First let $C(a, b)$ be the set of continous functions on $(a,b)$. Then let $PC(a,b)$ be the set of piecewise continuous function). Now for the function $f(t)$ to be piecewise continuous on $I$, if the followings are satisfied.

1. $f(t)$ is continuous on $I$ except finitely many points.
2. There exist a left-hand limit and a right-hand limit at the discontinuity $t_{0}$ of $f(t)$.

We now define an addition and a scalar multiplication on $C(a, b)$ and $PC(a,b)$ .
For $f,g$ in $C(a, b)$ or $PC(a,b)$, 1. $f+g$ is defined as a function which has the value $f(x)+g(x)$ at $x$.
2. $\alpha f$ is defined as a function which has the value $\alpha f(x)$ at $x$.

Example 1..3  

For $f(x) = x , g(x) = x^2$, find $f+g, \frac{1}{2}f, 2g$ .

Answer $(f+g)(x) = f(x) + g(x) = x + x^2 ,$
$(\frac{1}{2}f)(x) = \frac{1}{2}f(x) = \frac{x}{2} ,$
$(2g)(x) = 2g(x) = 2x^2 .$ $\blacksquare$

With this addition and scalar multiplication,the properties 1 thru 9 to be a vector space is satisfied in $C(a, b)$ and $PC(a,b)$.

Theorem 1..2  

Show that $C(a, b)$ and $PC(a,b)$ are vector space.

From now on we can call $f(x)$ and $g(x)$ vectors. It looks very different form a geomtric vectors. But it is a vector. You might notice these vectors might not have a magnitude or direction. For any vector space we can define a magnitude is called normed vector space. In this section, we consider a normed vector space which admits a dot product. $\spadesuit$Inner Product $\spadesuit$

For geometric vectors, addition , scalar multiplicaiton ,and inner product are basic.

We have already studied an addition and a scalar multiplication of geometric vectors. So, here we study an inner product.

Consider nonzero vector A and B and the angle $\theta(0 \leq \theta \leq \pi)$ between A and B. Then the dot product of A and B is defined as $\Vert{\bf A}\Vert\Vert{\bf B}\Vert\cos{\theta}$. In other words,

$${\bf A} \cdot {\bf B} = \Vert{\bf A}\Vert\Vert{\bf B}\Vert\cos{\theta}$$

If either A or B is 0, then we define ${\bf A} \cdot {\bf B} = 0$.

Up to now, we have tried to generalize the addition and the scalar multiplication. Now we try to generalize an inner product.

Definition 1..1  

For two vectors ${\bf v}_{1} and {\bf v}_{2}$, A real number ${\bf v}_{1}\cdot{\bf v}_{2}$ is defined and satisfies the following properties, ${\bf v}_{1}\cdot{\bf v}_{2}$ is called an inner product of vectors ${\bf v}_{1}$ and ${\bf v}_{2}$

For any vectors ${\bf v}, {\bf v}_{1},{\bf v}_{2},{\bf v}_{3}$ and real numbers $\alpha, \beta$, the followings are hold.

1. $(\alpha \vec{v}_{1} + \beta \vec{v}_{2}) \cdot (\vec{v}_{3}) = \alpha (\vec{v}_{1} \cdot \vec{v}_{3}) + \beta (\vec{v}_{2} \cdot \vec{v}_{3})$ (linearlity)
2. $\vec{v}_{1} \cdot \vec{v}_{2} = \vec{v}_{2} \cdot \vec{v}_{1}$ symmetric
3. $\vec{v} \cdot \vec{v} \geq 0, \vec{v} \cdot \vec{v} = 0$ and $\vec{v} = \vec{0}$ are equivalent . (positive definite)

Example 1..4  

For geometric vectors A and B, The inner product A $\cdot$ B can be thought of

$${\bf A} \cdot {\bf B} = \vert{\bf A}\vert \vert{\bf B}\vert \cos{\theta} .$$

Here, $\Vert{\bf A}\Vert$ and $\vert{\bf B}\vert$ are the magnitude of A and B. $\theta$ represents a smaller angle between A and B.

Figure 1.2: Scalar multiple

Example 1..5  

${\bf A} \cdot {\bf B} = \vert{\bf A}\vert \vert{\bf B}\vert \cos{\theta}$ satisfies the definition of inner product.

Answer Let $\theta$ be the angle between A and B. Let $\gamma$ be the angle between A and A+B. Let $\omega$ be the angle between A and C
Then
1.

$$\vert{\bf A+B}\vert\cos{\gamma} = \vert{\bf A}\vert + \vert{\bf B}\vert\cos{\theta},$$

$$\vert{\bf A+B}\vert\sin{\gamma} = \vert{\bf B}\vert\sin{\theta}.$$

Thus,

$$({\bf A +B})\cdot {\bf C} = \vert{\bf A}+{\bf B}\vert \vert{\bf C}\vert \cos({\omega} - {\gamma})$$

$$= \vert{\bf A}+{\bf B}\vert \vert{\bf C}\vert(\cos{\omega}\cos{\gamma} + \sin{\omega}\sin{\gamma})$$

$$= \vert{\bf A}\vert \vert{\bf C}\vert \cos{\omega} + \vert{\bf B}\vert \vert{\bf C}\vert \cos({\theta} - {\omega})$$

$$= {\bf A} \cdot {\bf C} + {\bf B} \cdot {\bf C}.$$

Also, for $\alpha \neq 0$, we have

$$\alpha({\bf A}\cdot{\bf B}) = \alpha\vert{\bf A}\vert \vert{\bf B}\vert \cos{\theta} = (\alpha{\bf A})\cdot {\bf B}.$$

We are able to show linearlity.
2.

$${\bf A}\cdot{\bf B} = \vert{\bf A}\vert \vert{\bf B}\vert \cos{\theta} = \vert{\bf B}\vert \vert{\bf A}\vert \cos{\theta} = {\bf B}\cdot{\bf A}.$$

3. ${\bf A}\cdot{\bf A} = \vert{\bf A}\vert^2$, Thus, ${\bf A}\cdot{\bf A} = 0$ and A = 0 are equivalent. $\blacksquare$

Example 1..6  

Define an inner product on ${ R}^3$.

Answer Let $(a_{1},a_{2},a_{3}), (b_{1},b_{2},b_{3})$ be elements of ${ R}^3$. Then

$$(a_{1},a_{2},a_{3}) \cdot (b_{1},b_{2},b_{3}) = a_{1}b_{1} + a_{2}b_{2} + a_{3}b_{3}$$

is an inner product because if ${\bf v}_{1} = (a_{1},a_{2},a_{3}), {\bf v}_{2} = (b_{1},b_{2},b_{3}), {\bf v}_{3} = (c_{1},c_{2},c_{3}) \in { R}^{3}, \alpha, \beta \in R$, then we have 1.

$$(\alpha {\bf v}_{1} + \beta {\bf v}_{2})\cdot {\bf v}_{3} = (\alpha a_{1} + \beta b_{1}, \alpha a_{2} + \beta b_{2}, \alpha a_{3} + \beta b_{3}) \cdot (c_{1}, c_{2}, c_{3})$$

$$= (\alpha a_{1} + \beta b_{1})c_{1} + (\alpha a_{2} + \beta b_{2})c_{2} + (\alpha a_{3} + \beta b_{3})c_{3}$$

$$= \alpha(a_{1}c_{1} + a_{2}c_{2} + a_{3}c_{3}) + \beta(b_{1}c_{1} + b_{2}c_{2} + b_{3}c_{3})$$

$$= \alpha(a_{1},a_{2},a_{3}) \cdot (c_{1}.c_{2},c_{3}) + \beta(b_{1},b_{2},b_{3}) \cdot (c_{1},c_{2},c_{3})$$

$$= \alpha {\bf v}_{1} \cdot {\bf v}_{3} + \beta {\bf v}_{2} \cdot {\bf v}_{3}.$$

2.

$${\bf v}_{1} \cdot {\bf v}_{2} = (a_{1},a_{2},a_{3}) \cdot (b_{1},b_{2},b_{3})$$

$$= a_{1}b_{1} + a_{2}b_{2} + a_{3}b_{3}$$

$$= b_{1}a_{1} + b_{2}a_{2} + b_{3}a_{3}$$

$$= (b_{1},b_{2},b_{3}) \cdot (a_{1},a_{2},a_{3})$$

$$= {\bf v}_{2} \cdot {\bf v}_{1}.$$

3.

$${\bf v}_{1} \cdot {\bf v}_{1} = a_{1}^2 + a_{2}^2 + a_{3}^2 \geq 0$$

$${\bf v}_{1} \cdot {\bf v}_{1} = 0 \Leftrightarrow a_{1}^2 + a_{2}^2 + a_{3}^2 = 0$$

$$\Leftrightarrow a_{1} = a_{2} = a_{3} = 0. \ensuremath{ \blacksquare}$$

Definition 1..2  

Suppose that $f(x),g(x)$ are elements of $PC[a,b]$. Then the inner product $(f,g)$ is given by the followings:.

$$(f,g) = \int_{a}^{b}f(x)g(x)dx .$$

(notice) This satisfies the properties of the inner product $1, 2, 3$.

Example 1..7  

For $f(x) = x , g(x) = x^2$ are in $PC[0,2]$, find $(f,g)$.

Answer

$$(f,g) = \int_{0}^{2}(x)(x^2)dx = \frac{x^4}{4}\mid_{0}^{2} = 4. \ensuremath{ \blacksquare}$$

If an inner product is defined on a vector space, then we can define a norm.

Definition 1..3  

The norm of a vector v is written as $\Vert{\bf v}\Vert _{2}$ and represented by the followings:

$$\Vert{\bf v}\Vert _{2} = \sqrt{{\bf v}\cdot{\bf v}}.$$

Then by the properties of inner product,$l_{2}$ norm has the following properties.

For any vectors ${\mathbf x}, {\mathbf y} \in {\cal R}^{n}$ and any real number $\alpha$, we have

1. $\Vert{\mathbf x}\Vert _{2} \geq 0, \Vert{\mathbf x}\Vert _{2} = 0$ and ${\mathbf x} = 0$ are equivalent.
2. $\Vert \alpha {\mathbf x}\Vert _{2} = \vert\alpha\vert\Vert{\mathbf x}\Vert _{2}$
3. $\Vert{\mathbf x} + {\mathbf y}\Vert _{2} \leq \Vert{\mathbf x}\Vert _{2} + \Vert{\mathbf y}\Vert _{2}$

For example in the geometric vector space, $\Vert{\bf A}\Vert _{2} = \sqrt{{\bf A}\cdot{\bf A}} = \vert{\bf A}\vert$. Thus it is the same as the length of A. For the space vectors,

$$\Vert(a_{1},a_{2},a_{3})\Vert _{2} = \sqrt{a_{1}^2+a_{2}^2+a_{3}^2}$$

which can be considered as the shortest distance to $(a_{1},a_{2},a_{3})$ . For the function space $PC[a,b]$, we have

$$\Vert f\Vert _{2} = \{\int_{a}^{b}[f(x)]^2dx\}^{1/2}.$$

Other than $l_{2}$ norm, it is often used in ${\cal R}^{n}$, we have $l_{\infty}$ norm.

Definition 1..4  

For a vector ${\mathbf x} = (x_{1},x_{2},\ldots,x_{n})$, $l_{\infty}$ norm(norm) is written as $\Vert{\mathbf x}\Vert _{\infty}$ and defined as follows:.

$$\Vert{\mathbf x}\Vert _{\infty} = \max_{1 \leq i \leq n}\vert x_{i}\vert.$$

Example 1..8  

Find a $l_{2}$ norm and $l_{\infty}$ norm of ${\mathbf x} = (-1,1,-2)$.

Answer

$$\Vert{\mathbf x}\Vert _{2} = \sqrt{(-1,1,-2) \cdot (-1,1,-2)} = \sqrt{1 + 1 + 4} = \sqrt{6}$$

$$\Vert{\mathbf x}\Vert _{\infty} = \max_{1 \leq i \leq 3}\vert x_{i}\vert = \max\{1, 1, 2\} = 2$$

$l_{\infty}$norm and $l_{2}$ norm have the following properties.

For any vectors ${\mathbf x}, {\mathbf y} \in {\cal R}^{n}$ and any real number $\alpha$, we have

1. $\Vert{\mathbf x}\Vert \geq 0, \Vert{\mathbf x}\Vert = 0$ and ${\mathbf x} = 0$ equivalent.
2. $\Vert\alpha {\mathbf x}\Vert = \vert\alpha\vert\Vert{\mathbf x}\Vert$
3. $\Vert{\mathbf x} + {\mathbf y}\Vert \leq \Vert{\mathbf x}\Vert + \Vert{\mathbf y}\Vert$

$\spadesuit$Orthogonal $\spadesuit$

If any two geometric vectors are orthogonal, then、 ${\bf A} \cdot {\bf B} = \vert{\bf A}\vert \vert{\bf B}\vert \cos{\theta}$で $\cos{\theta} = 0$. Thus, the inner product is 0、On the other hand, if ${\bf A},{\bf B}$ are not zero vectors and the inner product is 0, then $\cos{\theta} = 0$ and ${\bf A}$ and ${\bf B}$ are orthogonal.

In this way, when the inner product is defined in a vector space, not only norm but the concept of orthogonal can be introduced.

Definition 1..5  

Two vectors ${\bf v}_{1}$ and ${\bf v}_{2}$ are said to be orthogonal if ${\bf v}_{1} \cdot {\bf v}_{2} = 0$. A set of vectors whose elements are orthogonal to each other is called orthogonal system.

In the geometric vector space by 1.2 , an inner product of directed lines ${\bf A}\cdot{\bf B}$ is given by ${\bf A} \cdot {\bf B} = \vert{\bf A}\vert \vert{\bf B}\vert \cos{\theta}$. For if nonzero directed lines A and B, ${\bf A} \cdot {\bf B} = 0$, then $\theta = 90^{\circ}$. Thus, A and B are orthogonal.

In the 3D vector space, the inner product of $(a_{1},a_{2},a_{3})$ and $(b_{1},b_{2},b_{3})$ is given by $a_{1}b_{1}+a_{2}b_{2}+a_{3}b_{3}$ according to 1.2. If we think of 3D vector as a geometric vector, then

$$(a_{1}, a_{2}, a_{3}) \cdot (b_{1}, b_{2}, b_{3}) = \Vert(a_{1}, a_{2}, a_{3})\Vert \Vert(b_{1}, b_{2}, b_{3})\Vert \cos{\theta}.$$

Then, we have

$$\cos{\theta} = \frac{a_{1}b_{1} + a_{2}b_{2} + a_{3}b_{3}}{\Vert(a_{1},a_{2},a_{3})\Vert \Vert(b_{1},b_{2},b_{3})\Vert}$$

Thus, we can find the angle between two 3D vectors

Example 1..9  

Find the angle between ${\bf A} = (1,1,2)$ and ${\bf B} = (-1,2,1)$.

Answer

$$\cos{\theta} = \frac{a_{1}b_{1} + a_{2}b_{2} + a_{3}b_{3}}{\Vert(... ...Vert(b_{1},b_{2},b_{3})\Vert} = \frac{-1+2+2}{\sqrt{6}\sqrt{6}} = \frac{1}{2}.$$

To find the $\theta$ in the interval $80,\frac{\pi}{2}]$ satisfying the above, we have $${\theta = \frac{\pi}{3}}$$. $\blacksquare$

$\spadesuit$Equation of Plane $\spadesuit$

Figure 1.3: Equation of Plane

Consider a coordinate axis in the space and imagine a plane. A normal vector is a vector orthogonal to any tangent vector. If we use a inner product, we can find an equation of this plane. Put a point $(a_{1},a_{2},a_{3})$ on the plane. Let N = $(A,B,C)$ be the normal vector to the plane. Let ${\bf r}_{0}$ be the position vector connecting the origin $(0,0,0)$ and $(a_{1}, a_{2},$ $a_{3})$. Now let r be a vector connecting the origin $(0,0,0)$ and the point different from $(a_{1},a_{2},a_{3})$. Then the vector ${\bf r} - {\bf r}_{0}$ is on the plane and the angle between ${\bf r} - {\bf r}_{0}$ and N is $90^{\circ}$. Thus, we have $({\bf r} - {\bf r}_{0}) \cdot {\bf N} = 0$. This is the equation of the plane.

Example 1..10  

Find the equation of the plane going thru the point $(3,-1,4)$ and the normal vector is $(-1,1,2)$.

Answer Let the position vector r be ($x, y, z$). Then the equation of the plane is

$$(x-3, y+1, z-4) \cdot (-1,1,2) = 0.$$

Thus, we have

$$-(x-3) + (y+1) + 2(z-4) = 0$$

or

$$-x + y + 2z = 4. \ensuremath{ \blacksquare}$$

Example 1..11  

Show that a function $\sin{x}$ and a function $\cos{x}$ is orthogonal on $[-\pi,\pi]$ . But nor on $${[0,\frac{\pi}{4}]}$$.

Answer $[-\pi,\pi]$ で

$$(\sin{x},\cos{x}) = \int_{-\pi}^{\pi}\sin{x}\cos{x}dx = \frac{\sin^{2}{x}}{2}\mid_{-\pi}^{\pi} = \frac{\sin^{2}{\pi} - \sin^{2}{(-\pi})}{2} = 0 .$$

$${[0,\frac{\pi}{4}]}$$ で

$$(\sin{x},\cos{x}) = \int_{0}^{\frac{\pi}{4}}\sin{x}\cos{x}dx = \f... ...\frac{1}{2}(\frac{\sqrt{2}}{2})^{2} = \frac{1}{4}. \ensuremath{ \blacksquare}$$

In function space, an orthogonal doen not mean perpendicular.

A unit vector ${\bf u}$ is a vector whose magnitude is 1. Given nonzero vector A, we can find a unit vector with the same direction as A. To do so, simply divide A by its magnitude $\Vert{\bf A}\Vert$. For a general vector, we might try the same thing, that is, divide the vector v by its norm $\Vert{\bf v}\Vert$. To find a unit vector by dividing its norm is called normalization. A set of vectors whose elements are all normalized is called orthonormal system. As an example of orthonormal system, we have seen {i,j,k}.

Example 1..12  

Show that the function system ${\{\cos{mx}\}}_{m=0}^{\infty}$ is an otthogonal system on $[0,\pi]$. Then find a correspondin orthonormal system.

Answer For $m \neq n$, we have

$$(\cos{mx},\cos{nx}) = \int_{0}^{\pi}\cos{mx}\cos{nx}dx$$

$$= \frac{1}{2}\int_{0}^{\pi}[\cos{(m+n)x} + \cos{(m-n)x}]dx$$

$$= \frac{1}{2}[\frac{\sin{(m+n)x}}{(m+n)} + \frac{\sin{(m-n)x}}{(m-n)}] \mid_{0}^{\pi} = 0.$$

For $m = 0$, $\cos{mx} = \cos{0} = 1$ implies that

$$\Vert 1 \Vert = [ \int_{0}^{\pi}(1)^2 dx]^{1/2} = \sqrt{\pi}.$$

For $m \neq 0$, we have

$$\Vert\cos{mx}\Vert = [ \int_{0}^{\pi}\cos^2{mx} dx]^{1/2} = \sqrt{\frac{\pi}{2}}.$$

Thus the required orthonomal system is as follows:

$$\{\frac{1}{\sqrt{\pi}}, \frac{\sqrt{2}}{\sqrt{\pi}}\cos{x}, \cdots , \frac{\sqrt{2}}{\sqrt{\pi}}\cos{mx},\cdots \}. \ensuremath{ \blacksquare}$$

$\spadesuit$Fourier Series $\spadesuit$

Using the orthnormal system mentioned above, we can represent the piecewise continuous function $f(x) \in [0,L]$ as follows:

$$f(x) \sim \frac{a_{0}}{2} + \sum_{n=1}^{\infty}a_{n}\cos{\frac{n\pi x}{L}}$$

Here,

$$a_{n} = \frac{2}{L}\int_{0}^{\infty}f(x)\cos{\frac{n\pi x}{L}}dx, n = 0,1,2,\ldots,$$

The right hand side of the first expression is called Fourier series of $f(x)$. $a_{n}$ is called Fourier coefficient.

Example 1..13  

Show the norm on ${\bf v}_{1}$ and ${\bf v}_{2}$ satisfies the triangle inequality

$$\Vert{\bf v}_{1} + {\bf v}_{2}\Vert \leq \Vert{\bf v}_{1} \Vert + \Vert {\bf v}_{2}\Vert$$

Answer

$$\Vert{\bf v}_{1} + {\bf v}_{2}\Vert^{2} = ({\bf v}_{1} + {\bf v}_{2})\cdot ({\bf v}_{1} + {\bf v}_{2})$$

$$\Vert{\bf v}_{1}\Vert^{2} + 2 {\bf v}_{1} \cdot {\bf v}_{2} + \Vert{\bf v}_{2}\Vert^{2}$$

Here we use Cauchy-Schwarz inequality (Exercise1.2). Then

$$2 {\bf v}_{1} \cdot {\bf v}_{2} \leq 2\Vert{\bf v}_{1}\Vert \Vert{\bf v}_{2}\Vert$$

Thus,

$$\Vert{\bf v}_{1} + {\bf v}_{2}\Vert \leq \Vert{\bf v}_{1} \Vert + \Vert {\bf v}_{2}\Vert \ensuremath{ \blacksquare}$$

From this, distance satifies the followings:.

$$\Vert{\bf v}_{1} - {\bf v}_{2}\Vert \leq \Vert{\bf v}_{1} - {\bf v}_{3}\Vert + \Vert{\bf v}_{3} - {\bf v}_{2}\Vert .$$

Thus, $l_{2}$ norm just behaves like the distance we use.

Example 1..14  

Find the distance between $x$ and $x^2$ on $PC[0,2]$.

Answer

$$\Vert x - x^2\Vert = [\int_{0}^{2}(x -x^2)^2 dx]^{1/2} = \frac{4}{\sqrt{15}} . \ensuremath{ \blacksquare}$$

Exercise1-4

1. For vectors ${\bf A} = (-1,3,1)$ and ${\bf B} = (2,4,-3)$, find the followings: (a) $\vert{\bf B}\vert$ (b) ${\bf A}\cdot{\bf B}$ (c) Angle between A and B (d) Unit vector in the dierction of A

2. Determine which system is orthogonal. If it is orthogonal, find the orthonormal system.

(a) $\{(1,3),(6,-2)\}$

(b) $\{(1,2,2),(-2,2,-1),(2,1,-2)\}$

(c) $\{{\bf i} - 2{\bf j} + 3{\bf k}, 2{\bf i} - \frac{1}{2}{\bf j} - \frac{1}{3}{\bf k}, 3{\bf i} + 3{\bf j} + {\bf k}\}$

3. Find an equation of plane going thru a point $(5,-1,3)$ and normal vector is 2i + j - k.

4. Let A,B be space vectors. Then prove the following inequality:

$$\vert{\bf A}\cdot{\bf B}\vert \leq \Vert{\bf A}\Vert \Vert{\bf B}\Vert .$$

This result is called Cauchy-Schwarz inequality.

5. LetA, B, C be space vectors. Then show the following inequality:

$$\Vert{\bf A} - {\bf B}\Vert \leq \Vert{\bf A} - {\bf C}\Vert + \Vert{\bf C} - {\bf B}\Vert .$$

6. Let $f(x),g(x)$ be a function vector in $PC[a,b]$. Show the following:

$$\vert(f,g)\vert \leq \Vert f\Vert \Vert g\Vert .$$

This result is called Schwarz inequality.

7. For $PC[0,2]$, Find the norm of the followings:.

(a) $f(x) = x$ (b) $f(x) = \sin{\pi x}$ (c) $f(x) = \cos{\pi x}.$

8. Next three polynomials are called Legendre polynomial.

$$P_{0}(x) = 1, P_{1}(x) = x, P_{2}(x) = \frac{3x^2 - 1}{2}$$

Show that these polynomials are orthogonal system in $PC[-1,1]$.