Calibration Science, Part V: Spotting Outliers

In the first four installments of this calibration science series, we covered sources of error, signal-to-noise ratio (SNR), precision, accuracy, reducing noise, how to plot a calibration line, and how to measure calibration quality (1-4). In this, the fifth installment, we will continue the calibration quality control theme. Sometimes in a data set used to generate a calibration, one or more data points will stand out as being not like the others and these data points are called outliers. How to spot outliers and potential ways to correct them will be discussed here.

Spotting Outliers

Back in the day on the American children’s show Sesame Street (5) they played the game “Which of These Things is Not Like the Others”. For example, they might show pictures of a duck, a hen, a pigeon, and a dog. In this case, the thing not like the others was the dog since it is a mammal and not a bird, making the dog an outlier. In calibration science spotting outliers is like this, spotting data points that are not like the others.

For our purposes we will limit ourselves to a discussion of outliers in the types of analyses performed in a cannabis analysis lab. Probably the most important thing measured in these labs is total tetrahydrocannabinol (THC) or potency. Two of the most common potency methods are high pressure liquid chromatography (HPLC) with ultraviolet-visible (UV-Vis) detection (6) and mid-infrared spectroscopy (7). In both these cases, spectral absorbance readings and concentration data are used to plot calibration lines. Thus, we will restrict our discussion to spotting concentration and spectral outliers.

Concentration Outliers

A good way to spot concentration outliers is to first calculate a set of concentration residuals. A concentration residual is the difference between the known and predicted concentration for a standard used in a calibration. This quantity is calculated using Equation 1:

where R_c is the concentration residual; C is the actual concentration; and C’ is the predicted concentration.

The concentration residual measures how well the concentration in a specific standard was predicted.

The next step in spotting concentration outliers is to plot the residuals versus sample number as seen in Figure 1. The data in this case are for the isopropyl alcohol (IPA) calibration discussed in a previous column (4), thus the units of the residuals are volume percent IPA.

Under normal circumstances this plot should be a random scatter of points with no apparent structure and with the magnitude of all the residuals approximately the same. Note in the plot that for samples 1,3,4, and 5 the residuals are all right around 1% IPA, but that for sample 2 the residual is -15.9% IPA; this thing is certainly not like the others! This residual is 15 times bigger than the others.

Once we spot an outlier, the question we have to ask is whether to include that standard’s data in a calibration or not. Ideally, we need to find the assignable cause as to why something is an outlier. This term is statistics speak for the reason why something is an outlier. If we understand why something is an outlier, we can fix the problem or justify not using the data in a calibration.

In the case of sample 2 in Figure 1 the assignable cause was easy to find: I typed the wrong concentration value for sample 2 into the spreadsheet used to generate the calibration line. Correcting this typo gave the concentration residual plot seen in Figure 2.

With the typo corrected the concentration residual for sample 2 was reduced from -15.9% IPA to 0.3% IPA, illustrating the power of concentration residual plots to improve calibrations.

Concentration Residual Assignable Causes

There are a number of reasons why concentration outliers happen including:

1. Transcription Errors: In other words, typos. Often times the absorbance and concentration data for a calibration have to be typed into a spreadsheet or some other computer program. It is very easy to mess up the transcription of these numbers, i.e. to commit a typo. This is exactly what we saw with Figures 1 and 2 above.

2. Poorly made standards: In calibrations that use concentration data, the error in the known concentrations in the standards is often times the biggest source of error in the analysis, and lack of care and poor technique when making standards can condemn you to always having poor calibrations. Hence,anything you can do to improve the quality of known concentrations in your standards will pay dividends in calibration quality.

3. Sample Problems: If the concentration of the analyte in your standard changes between when you make the standard and when you measure its absorbance that is a problem. Things that can cause this to happen include reactions taking place within the sample, evaporation of the sample, or analyte precipitating out of the sample. A good way to check this problem is to monitor your standard samples over time. For instance, once you make a fresh standard measure its absorbance every few minutes, then hours, to see if it is stable or not.

Spectral Outliers

Remember our calibration consists of concentration data and spectral absorbance data. Spotting concentration outliers involved calculating concentration residuals, similarly, spotting an outlier in the spectral absorbance data is done by calculating a spectral residual, the formula for which is seen in Equation 2:

where R_s is the spectral residual; A is the actual absorbance; and A’ is the predicted absorbance and where the actual and predicted absorbances may be peak heights or peak areas.

For the calculation of concentration residuals we had the predicted concentration, C’, already in our hands since we needed it to calculate calibration metrics. Where do we obtain the predicted absorbance? From the calibration line equation itself. The calibration line for our IPA method is seen in Figure 3 and Equation 3:

We discussed calibration lines in previous articles (1-4). Briefly, 0.165 is the slope of the calibration line, and -0.6834 is the y-intercept. Since peak area (spectral absorbance) is on the y-axis, and %IPA is on the x-axis, if we know the concentration of any given IPA sample (X), we can calculate its corresponding peak area by plugging the concentration value into Equation 3 and then solving.

Now, to obtain the predicted peak area as needed for the spectral residual calculation we would use the predicted concentration as the X in Equation 3, and solving for Y would give us the predicted absorbance. The predicted absorbance represents the peak area the calibration says would correspond to a given predicted concentration.

For example, in Figure 2 the concentration residual for sample 1 is -1.4% IPA. The known concentration for sample 1 is 9% (4). If we rearrange Equation 1 we can solve for the predicted concentration as seen in Equation 4:

where R_c is the concentration residual; C is the actual concentration; and C’ is the predicted concentration.

In our example then C = 9%,
R = -1.4%, and the predicted concentration C’ is then 10.4%. To predict the peak area that corresponds to this concentration we set X = 10.4 in Equation 3 and to obtain Equation 5:

to see that the predicted peak area that corresponds to a concentration of 10.4% IPA is 1.03. The right-hand column in Table 1 shows the measured peak areas for each IPA standard.

To calculate the spectral residual for the 5 standards used to determine the calibration line seen in Figure 3 one would repeat the exercise seen above for sample 1 for all 5 samples. This will yield a set of five spectral residuals. Like with concentration residuals, it is useful to plot spectral residuals versus sample number to look for outliers. Such a plot is seen in Figure 4.

Note in Figure 4, that all the residuals are small, about the same magnitude, and do not have a discernible pattern. This is unlike Figure 1 where the sample with the concentration residual of -15.9% was clearly an outlier.

Spectral Residual Assignable Causes

1. Transcription errors: Same as for concentration outliers, typos happen. This is the first thing to check when any outlier of any type is spotted.
2. Improperly measured absorbances: The peak height or area to be used in the calibration must be measured properly. More on that in a future column.
3. Noise: Recall (1,2) that all measurements have noise/error, and this is true of spectra as well. Calibrations are always a garbage in/garbage out process, and noisy spectra will give inaccurate predicted concentrations. How to improve the SNR of spectra was discussed previously (1-4).
4. Instrument malfunction: Analytical instruments are finicky beasts and don’t always work properly leading to incorrectly measured data. Before using any analytical instrument for quantitative work make sure it has been calibrated, passed its quality control checks, and that you are following the method standard operating procedure (SOP) properly.
5. Impurities/interferents: If a molecule other than the analyte absorbs light at the same wavelength where we are measuring the peak height or area, this will interfere with the correct determination of the concentration of the analyte, hence the term interferents. Traditionally we dealt with interferents by making sure only the analyte contributed to the peak being used in the calibration. More recently, algorithmic methods known as chemometrics can get around the problem of interferents (8). However, this discussion is beyond the scope of this column series. Thus, for our purposes, it is important to choose a calibration peak that is known to be free of interferents.
6. Operator error: As always humans are fallible and make mistakes. Make sure the person using the instrument has been properly trained, the method has an SOP, and that it is followed properly.

What if There is No Assignable Cause for an Outlier?

This gets tricky. A data point may be an outlier because there is truly something wrong with the measurement, or it might contain information important for the calibration model that is not consistent with the other data points. Statisticians have spilled much ink as to what to do in this case, here is my rule of thumb based on my understanding of the literature. The first thing you should do is calculate the average of the residuals by using Equation 6:

where R is the average residual; i is the sample index; |R_i| is the absolute value of an individual residual; and N is the number of residuals.

The purpose of using the absolute value is to keep positive and negative residual values from cancelling each other. For the five concentration residuals seen in Figure 2 their absolute values are 1.4, 0.3, 1.0, 0.2, and 0.9 % IPA, and their average is 0.76%.

In general, outliers whose residual is 10x that of the average can be discarded without an assignable cause, which would have been the case with the -15.6% concentration outlier, although we did find an assignable cause for it. Outliers of 3x or less should almost always be included in the calibration. The gray area is where the residual is >3x but <10x the residual average. In the absence of an assignable cause some judgement for these is needed. What I tend to do is remake the standard, measure its peak height or area, calculate its residual, and see if the residual value is reproducible. If so, I tend to keep this data point.

Conclusions

Outliers are data points in a calibration not like the others. They can be spotted by calculating residuals and plotting them versus sample number. Calculating the average residual and comparing it against individual residuals can be used to judge whether an outlier should be included or excluded from a calibration. Assignable causes of concentration and spectral outliers were discussed.

References

1. Smith, B., Calibration Science, Part I: Precision, Accuracy, and Random Error, Cannabis Science and Technology, 2023, 6(9), 6-9.
2. Smith, B., Calibration Science, Part II: Systematic Error, Signal-to-Noise Ratios, and How to Reduce Random Error, Cannabis Science and Technology, 2024, 7(1), 8-11.
3. Smith, B., Calibration Science, Part III: Calibration Lines and Correlation Coefficients, Cannabis Science and Technology, 2024, 7(2), 6-11.
4. Smith, B., Calibration Science, Part IV: Calibration Metrics, Cannabis Science and Technology, 2024, 7(3), 8-11.
5. Sesame Street https://en.wikipedia.org/wiki/Sesame_Street.
6. Giese, M. E; Lewis, M. A.; Giese, L.; and Smith, K. M.; Journal of AOAC International, 2015, 98(6) 1503.
7. Smith, B., A Proposed Representative Sampling Plan for Hemp Grows, Cannabis Science and Technology, 2020, 3(6), 10-13.
8. Brian C. Smith, Quantitative Spectroscopy: Theory and Practice, 2002.

How to Cite this Article

Smith, B., Calibration Science, Part V: Spotting Outliers, Cannabis Science and Technology, 2024, 7(4), 6-9.

About the Columnist

Brian C. Smith, PhD, is Founder, CEO, and Chief Technical Officer of Big Sur Scientific. He is the inventor of the BSS series of patented mid-infrared based cannabis analyzers. Dr. Smith has done pioneering research and published numerous peer-reviewed papers on the application of mid-infrared spectroscopy to cannabis analysis, and sits on the editorial board of Cannabis Science and Technology. He has worked as a laboratory director for a cannabis extractor, as an analytical chemist for Waters Associates and PerkinElmer, and as an analytical instrument salesperson. He has more than 30 years of experience in chemical analysis and has written three books on the subject. Dr. Smith earned his PhD on physical chemistry from Dartmouth College.
Direct correspondence to: brian@bigsurscientific.com