Mathematical Foundations

Taylor Series

Functions don’t always play nice with other functions. Nested non-linear functions can be particularly difficult to work with. Derivatives and polynomials are the Switzerland and Lichtenstein (respectively) of functions. They get along with pretty much all the other functions, and this explains the remarkable success of the Taylor expansion. The Taylor expansion is a way of approximating a finite part of any differentiable function as a really big polynomial. Let’s take the function $f(x)$, and try to approximate it at a value of $x=x_0+\epsilon$, where $x_0$ is a constant and $\epsilon$ is a variable. We might logically start by saying

$$ f(x_0+\epsilon) \approx f(x_0), $$

but that really only works for very small values of $\epsilon$. As you get further away from $x_0$, you’ll need to somehow correct for how $f(x)$ changes as $x$ changes...enter, the derivative! The first derivative describes the slope of a function at any given point, so we could add a line to our approximation that matches that slope at that point:

$$ f(x_0+\epsilon) \approx f(x_0) + f'(x_0)\epsilon. $$

Now our approximation matches the target function’s rate of change immediately around that point. That’s great, but rates of change don’t stay the same forever...enter, the second derivative! Now it might be tempting to continue this pattern with $f''(x_0)\epsilon^2$, but we need to be careful—the second derivative of that would be $2f''(x_0)$. The second-order term should therefore be $\frac{1}{2}f''(x_0)\epsilon^2$ to adjust for that. Our approximation is now

$$ f(x_0+\epsilon) \approx f(x_0) + f'(x_0)\epsilon + \frac{1}{2}f''(x_0)\epsilon^2. $$

Let’s add a few more terms in the same way and watch the pattern emerge.

$$ \begin{aligned} f(x_0+\epsilon) &\approx f(x_0) + f'(x_0)\epsilon+ \frac{1}{2}f''(x_0)\epsilon^2+ \frac{1}{6}f'''(x_0)\epsilon^3 + \frac{1}{24}f^{(4)}(x_0)\epsilon^4 \\ &\approx \frac{f(x_0)\epsilon^0}{0!} + \frac{f'(x_0)\epsilon^1}{1!} + \frac{f''(x_0)\epsilon^2}{2} + \frac{f'''(x_0)\epsilon^3}{3!} + \frac{f^{(4)}(x_0)\epsilon^4}{4!} +\cdots + \frac{f^{(n)}(x_0)\epsilon^n}{n!} \end{aligned} $$

This approximation gets better and better with more terms, and as $n$ goes to infinity, it will actually become an exact representation of the original function as long as the derivatives are well defined. Now here’s where the magic happens. A factorial in the denominator will quickly dominate a monomial in the numerator, which means that the higher order terms will tend to be less and less important (especially if $\epsilon$ is small). Most functions can be approximated with just one or two non-zero terms! For a more detailed explanation of the Taylor expansion, as well as the expansions of some common functions, visit the Wikipedia article.

Sine waves

A lot of what we do depends heavily on a few key properties of sine waves (or sinusoids).

Frequency and Wavenumber

The angular frequency of a sinusoid is defined as $\omega=2\pi/T$, where $T$ is the period of oscillation. Likewise, the wavenumber (spatial frequency) is defined as $k=2\pi/\lambda$, where $\lambda$ is the wavelength. In this paper, I’ll use $k$ and $x$ for all of my sine waves, but you could also use $k$ with any other spatial variable ($y$, $z$, $r$, etc.), or you could use $\omega$ and $t$.

Amplitude and Phase

All sine waves are of the form:

$$ A\cos(kx+\phi) $$

Yes, I’ve just called a cosine a sine wave on purpose. That’s because the sine and cosine functions are actually just special cases of sine waves.^[1] Any sine wave can be represented with a sine or cosine function with a proper amplitude and phase.

Phase describes where something is in a cyclic pattern (for example, the phases of the moon). However, because cyclic patterns don’t typically have an inherent starting point, it’s meaningless to talk about "absolute" phase. Phase only has meaning as a relative quantity.

Superposition

Any sine wave can be represented as a superposition of one sine function and one cosine function with proper amplitudes. This can be expressed as

$$ A\cos(kx+\phi) = B\cos(kx) + C\sin(kx). $$

For any values of $A$ and $\phi$ , there is a corresponding set of values $B$ and $C$ that will make this equation valid. One way to think of this is that any sinusoid at a given frequency has two degrees of freedom. These may be represented as amplitude and phase or as sine amplitude and cosine amplitude.

Notes

1. For the traditional "sine" function, just plug in $\phi=\pi/2$.

Complex Exponentials

The complex number $\tilde{z}=a+bi$ has a real part $a$ and an imaginary part $b$, corresponding to its projection on the real or imaginary number lines. The first step in understanding complex exponentials is to think of a complex number as a vector. You could think of it as a rectangular vector $\textbf{z}=(a,b)$. You could also think of it as a polar^[1] vector $\textbf{z}=(r,\theta)$, where $r=\sqrt{a^2 + b^2}$ and $\theta=\arctan\frac{b}{a}$. From here, you can apply Euler’s formula:

$$ e^{i\theta}=\cos\theta + i\sin\theta $$

Some trigonometry will tell you that this exponential represents a complex unit vector (i.e. a vector with magnitude 1) in the $\theta$ direction. You can therefore represent any complex vector in polar space as:

$$ \textbf{z} = r e^{i\theta} $$

This notation is called a complex exponential, and it’s very useful for what we’re about to do. You see, complex exponentials behave almost exactly like sine waves. The magnitude $r$ of the complex exponential corresponds to amplitude, and the angle $\theta$ of the complex exponential corresponds to phase.

I’ll give you one example of the usefulness of complex exponentials. Let’s say you want to add or multiply some sine waves with different amplitudes and phases. Combining trig functions can get really messy. The wavenumber and the phase are stuck together inside the cosine function, and the amplitude is completely separate. So let’s use Euler’s formula to put it into complex exponential form:

$$ A\cos(kx+\phi) = Ae^{i(kx+\phi)} $$

Yes, I’m playing fast-and-loose with my equals signs—technically, the cosine is only the real part of the complex exponential. But it turns out that you can actually hold off on taking the real part until the very very end,^[2] so it’s easier in practice to just say that the wave is the same as the complex exponential. In this form, we can easily separate the phase from the rest of the wave argument:

$$ Ae^{i(kx+\phi)} = Ae^{i\phi}e^{ikx} $$

The $e^{ikx}$ term will change with $x$, bu $Ae^{i\phi}$ is a constant, so let’s lump them together into a complex vector defined as:

$$ \tilde{A} \equiv Ae^{i\phi} = (A,\phi) $$

We now have a complex amplitude that contains both the amplitude and the phase information. These complex amplitudes are much easier to work with, particularly for computers.

For a much more in-depth explanation of complex exponentials, the University of Wisconsin-Madison has published this document, which includes some homework-style practice problems as well.

Notes

1. Fun fact: The first time you ever learned about vectors, you were probably told something to the effect that it’s a quantity with both a magnitude and a direction. In other words, you first learned about vectors in polar space.
2. That’s due to the orthogonality of real and imaginary numbers as well as that of sine and cosine, if you’re interested. It’s actually a really fascinating topic.