Floating-point computation
Why it matters
Scientific software stores a finite approximation to real numbers; understanding that representation prevents silent computational error.
Prerequisites
Courses 2.2 and 2.4; earlier canonical classes where applicable.
Concept and explanation
A floating-point value has finite precision. Most decimal fractions are not represented exactly, so arithmetic is rounded after operations. Accuracy concerns closeness to a reference; precision concerns resolution or repeatability.
Key terms
floating point; significand; exponent; machine precision; overflow; underflow; cancellation; tolerance
Notation / representation
Use ≈ for approximation. A comparison |a−b|≤atol+rtol|b| combines absolute and relative tolerance.
Numerical method / algorithm
Represent → compute → inspect scale → compare with tolerance → reformulate if cancellation or range threatens the result.
Mathematical development
For x=1 and a small δ, subtracting nearly equal rounded quantities can erase significant digits. Relative error |x−x̂|/|x| is scale-aware when x≠0.
Learning objectives
- Explain finite binary representation, rounding, overflow and underflow.
- Use tolerances and stable reformulations instead of inappropriate exact equality.
Worked examples
Worked example 1
Mathematical problem · given · goal: Explain why 0.1+0.2 may not equal 0.3 exactly in binary floating point.
Method · assumptions · algorithm · computation: The inputs are rounded binary approximations; compare the computed sum with 0.3 using a declared tolerance.
Result · error/residual · validation · interpretation · limitations: The tiny discrepancy is representation error, not evidence that real-number addition changed.
Worked example 2
Mathematical problem · given · goal: Evaluate (sqrt(1+x)−1)/x for very small positive x.
Method · assumptions · algorithm · computation: Rationalize to 1/(sqrt(1+x)+1), avoiding subtraction of nearly equal numbers.
Result · error/residual · validation · interpretation · limitations: Both formulas are mathematically equal, but the rationalized form is numerically more stable and approaches 1/2.
Common mistake and counterexample
Incorrect: more printed digits guarantee accuracy. Printed precision cannot repair model error, poor data, cancellation or an ill-conditioned problem.
Guided practice
- Rebuild Worked Example 1 and record every approximation, parameter, error or residual.
- Change one input, predict the effect, recompute and validate independently.
A/B/C/D practice
A · Foundation
Classify roundoff, overflow, underflow and cancellation examples.
B · Application
Design a scale-aware equality test.
C · Reasoning and error detection
Diagnose a cancellation-prone formula.
D · Challenge / transfer
Reformulate and validate a computation across several scales.
Knowledge check and summary
- State whether the result is exact or approximate.
- Record the method, parameters and stopping criterion.
- Estimate error or residual and interpret it.
- Validate with an independent method or known case.
Mastery criterion: 4/4 correct, including exact-versus-approximate distinction, error/residual and independent validation. Correct each miss and complete a fresh equivalent check.
Related laboratory
Next class: Numerical error