Worked example
Example: Calibration data. The inputs, in the units selected above:
| Input | Value |
|---|---|
| Number of points used | 5.0000 - |
| Point 1: x | 20.000 - |
| Point 1: y | 41.500 - |
| Point 2: x | 40.000 - |
| Point 2: y | 61.000 - |
| Point 3: x | 60.000 - |
| Point 3: y | 82.300 - |
| Point 4: x | 80.000 - |
| Point 4: y | 99.800 - |
| Point 5: x | 100.00 - |
| Point 5: y | 121.00 - |
| Point 6: x | 0 - |
| Point 6: y | 0 - |
| Point 7: x | 0 - |
| Point 7: y | 0 - |
| Point 8: x | 0 - |
| Point 8: y | 0 - |
| x for prediction | 70.000 - |
Applying the formula below gives:
| Result | Value |
|---|---|
| Slope | 0.98900 |
| Intercept | 21.780 |
| Coefficient of determination | 99.928 % |
| Predicted y at x₀ | 91.010 |
Open this example in the calculator and change any input.
Formula
\(b = \frac{\sum(x - \bar x)(y - \bar y)}{\sum(x - \bar x)^2}, \quad a = \bar y - b\bar x, \quad R^2 = \frac{S_{xy}^2}{S_{xx}S_{yy}}\)
Variables
| Symbol | Meaning | SI unit |
|---|---|---|
| \(n\) | Number of points used | - |
| \(x_{1}\) | Point 1: x | - |
| \(y_{1}\) | Point 1: y | - |
| \(x_{2}\) | Point 2: x | - |
| \(y_{2}\) | Point 2: y | - |
| \(x_{3}\) | Point 3: x | - |
| \(y_{3}\) | Point 3: y | - |
| \(x_{4}\) | Point 4: x | - |
| \(y_{4}\) | Point 4: y | - |
| \(x_{5}\) | Point 5: x | - |
| \(y_{5}\) | Point 5: y | - |
| \(x_{6}\) | Point 6: x | - |
| \(y_{6}\) | Point 6: y | - |
| \(x_{7}\) | Point 7: x | - |
| \(y_{7}\) | Point 7: y | - |
| \(x_{8}\) | Point 8: x | - |
| \(y_{8}\) | Point 8: y | - |
| \(x_0\) | x for prediction | - |
| \(b\) | Slope | - |
| \(a\) | Intercept | - |
| \(R^2\) | Coefficient of determination | - |
| \(\hat y\) | Predicted y at x₀ | - |
Method and standards
Least squares.
Where it is used
- Instrument calibration.
- Willans line and performance curves.
Common mistakes
- Extrapolating.
- Fitting a line to curved data.
Assumptions and limits
- Errors in y only. Extrapolation beyond the data and the trivial two-point fit are flagged.
References
- Ordinary least squares y = a + bx
Questions and answers
What does the default data fit?
y ≈ 21.78 + 0.9890 x, R² 0.9993.
Calibrations?
Fit instrument readings against standards to find gain and offset.
Extrapolation?
Risky beyond the measured range.
Method and article reviewed by FireCAD engineering on 12 November 2026. Every result is checked against an independent calculation on each release: how we verify our calculators.