Differential Calculus: From Practice to Theory

We will use Newton’s Method to “re–invent” the Babylonian square root algorithm and, more importantly, to generalize it. Here’s the essential idea: We know how to find the root of a linear equation like \(ax+b=0\text{.}\) We want to use this knowledge to approximate the root of a nonlinear equation. As with the rest of Calculus this means that the Principle of Local Linearity is the postulate underlying Newton’s Method.

For our purposes this means that we can approximate the coordinates of points on a curve with the coordinates of points on the line tangent to the curve when the the point of tangency is nearby. Moreover this approximation gets better as we get closer to the point of tangency.

Before we return to the square root problem we consider the problem of approximation more broadly.

The approximation we found in Problem 7.2.2 is better than our original guess of \(r_1=4\text{,}\) but it is still not great since \(f(4.526) \approx 2.6\text{,}\) whereas if we’d found the actual root, \(r\text{,}\) we’d have \(f(r)=0.\)

But we don’t have to stop there, and the pattern should be clear. If we wanted a more accurate approximation we could use \(r_2\) to generate an \(r_3,\) and so on. We stop when our approximation is accurate enough for our purposes.

When computing approximations our goal is to compute good approximations efficiently, not to do a lot of arithmetic. We’d actually like to avoid doing arithmetic as much as possible. So it will be worth the effort needed to organize our computations as much as we can. Fortunately this is not hard to do.

We summarize the foregoing succinctly as follows:

One reason for writing down Newton’s Method in this compact form is that this makes it easier to program into a computer.

As intimidating as the general formula might first appear, remember that ultimately we are simply constructing and solving a single linear equation. This is clear from the derivation of the formula. The apparent complexity appears only because we have arranged to construct and solve the equation in a single step.

Newton’s Method works because the property of being the root of a function is a local, not a global property. If our approximation, \(r_n,\) is quite close to the root we seek then we can safely assume that the curve and its tangent line are practically the same (at least, most of the time). This is, of course, the Principle of Local Linearity again.

At the end of step \(2\) we used the phrase “until the desired level of accuracy is obtained.” Clearly we are being cagey. How can we tell when we have an estimate which is accurate to any level of precision, let alone the desired level? This is a harder question than it probably appears to be. For that reason we will not be looking into it very deeply. For our purposes it will be good enough to assume that if the first four digits to the right of the decimal have not changed from one iteration to the next then our estimate is accurate to four decimal places.

If the initial guess is reasonably close to the root Newton’s Method finds a very accurate approximation to the root of a function in just a few iterations most of the time. This made it extremely useful in the \(17\)th century when such computations were done by hand. Indeed, it computes the square root of a number pretty quickly even if the initial guess is very bad. For example in Figure 7.2.13 the blue curve is the graph of \(f(x)=x^2-2.\) We start with an initial guess of \(r_1=5\) (obviously a terrible guess) and find the \(x\) intercept of the red tangent line to find the next guess, \(r_2=2.7\) which is better but still terrible. Repeating we generate the green tangent line at \((r_2,f(r_2))\) which crosses the \(x\)–axis at \(r_3= 1.72.\) Finally we generate the orange tangent line at \((r_3,f(r_3))\) which crosses the \(x\)–axis at \(r_4=1.44\) which is correct to one decimal. If we continue one more iteration (not shown) we get \(r_5=1.4141\) which is correct to three decimals.

But this paper–and–pencil procedure is not as simple as handing the problem off to your favorite computational software so a natural question to ask is, “Why bother? why should we learn this?”

The answer is that whatever software you end up using will be performing either the computations above or something very like them. And these Numerical Methods are not 100% reliable.

However if you are familiar with the limitations of the algorithm(s) being used you will also be aware of the limitations of your software. Perhaps more importantly, you are less likely to believe a wrong answer when you get it. Depending on what you are computing an incorrect approximation could mean anything from a minor annoyance (if you are calculating \(\sqrt{2}\) just for fun) to a deadly disaster (if you are designing a control procedure for a self–driving car).

So, how do you know if you should trust the approximation you get back from your favorite software? First, be aware of what a reasonable answer should be. In Example 7.2.1 it is clear that the root must be positive. If software tries to tell you that it is equal to \(-2.5\) you know it is lying. Second be aware of the ways that an approximation method might fail.

For example, Newton’s Method is not perfect. It can fail in two distinct ways that we’ll call Spectacular Failure and Subtle Failure. Let’s take a look at them in turn.

Newton’s Method can fail spectacularly in two distinct, but related, ways. The first is very easy to spot.

Because Newton’s Method requires that we divide by \(y^\prime(r_n)\) we will get an error if it happens that \(y^\prime(r_n) =0\text{.}\) We call this a spectacular failure because it is easy to spot whether we do the computation with a computer or with paper and pencil. Indeed, modern computer software is very adept at alerting us to division by zero errors.

The second kind of spectacular failure is exemplified by the following problem.

Obviously we don’t need to use Newton’s Method to compute the root of \(f(x)=x^{1/3}.\) The point of Problem 7.2.16 is that Newton’s Method will not find the root no matter how close our initial guess is. Instead it will continue to generate alternately positive and negative “approximations” to zero, that are farther and farther from zero.

The other way Newton’s Method fails can be quite subtle. The best way to demonstrate this is with an example.

The problem of converging to a wrong answer is particularly acute when a function has two roots which are very close together, for example if \(f(x) = (10x^2-21x+11)(x-0.05)(x^3+7).\)

From the previous examples it should be clear that it is important that the initial guess be close enough to the root we seek to guarantee that the iterations will converge to that root. This, of course, begs the question, “How close should our initial guess be?” This is actually a hard question. Since we don’t yet possess all of the knowledge and skills needed we can’t answer it in any definite way. So we’ll just say that in general your initial guess should be as close as you can make it.

But take care. As we saw in Problem 7.2.16 sometimes even having a very accurate first guess is not sufficient.