The Pet Lab Syndrome

The well documented inter-laboratory error problem in cannabis analysis means results from different laboratories cannot be compared. As a result, there arise disputes based on these varying results where both sides stubbornly defend their laboratory’s results, the Pet Lab Syndrome. The problem can make it difficult to set a fair price for biomass or oil transactions. Suggested solutions will be offered.

Here is a scenario I have heard about far too often in this industry. Imagine a hemp extractor wants to buy biomass from a hemp grower. The farmer shows up with a certificate of analysis (COA) from his laboratory claiming 14% total cannabidiol (CBD) content. The extractor sends a sample of this hemp to his laboratory to be tested, and it comes back 9% total CBD. The two then argue about whose laboratory is right. Claims of a laboratory’s quality, accreditations, previous experience, the brilliance of the lab director, and so forth are cited. This insistence that “my laboratory is always right” and “your laboratory is always wrong” is what I call the Pet Lab Syndrome—an overreliance or dogmatic insistence on the quality of one laboratory’s results over another.

The inter-laboratory error problem, where different cannabis testing laboratories report radically different results on the same samples, has been well documented (1–7). It is the cause of the Pet Lab Syndrome. The causes of inter-laboratory error include variations in sample preparation, analytical methods, lack of standard reference materials, and poor training (5). Inter-laboratory error means that results from different cannabis laboratories cannot be compared, and that accuracy in cannabis analysis is a myth. The inter-laboratory error problem means that there are times, such as the purchase situation above, where different parties will have COAs with different results on the same sample. Who do we trust in these situations? I have offered my thoughts on solutions to the inter-laboratory error problem (6).

The inter-laboratory error problem has a corrosive effect on our industry. It sows seeds of doubt in the public’s mind about the safety and efficacy of cannabis medicines (3,4), it makes it difficult for cannabis businesses to have the data they need to manage their businesses properly to make safe and effective medicines, and makes it difficult for transactions to take place if no one can agree on the composition of the thing being sold. Despite exposure of the problem and pleas for improvement (1–7), the inter-laboratory error problem is not going to be solved until sufficient regulation and oversight comes to the cannabis industry. In the meantime, I offer some thoughts on how we can deal with this problem.

How Do You Find a Laboratory You Can Trust?

Every cannabis business needs analytical chemical data to run properly. In my own business, where we make and sell cannabis analyzers (1,8–12), we need reliable reference data to calibrate and validate our instruments. Any analyzer’s results are only as good as the reference data with which it is calibrated.

However, the inter-laboratory error problem is so bad that when we build calibration models using data from multiple laboratories we get poor results. However, when we build models from a single laboratory results improve. The variance here then is in the 3rd party laboratory’s data, not within the instrument itself. In another example of the Pet Lab Syndrome, I have had customers insist there unit be re-calibrated to match their Pet Lab’s numbers, despite overwhelming evidence that their unit matches the results from our ISO certified, state licensed laboratory just fine.

To find a reliable laboratory, I have literally roamed the country visiting dozens of cannabis analysis laboratories. I meet the lab director, observe analyses, and ask to see written standard operating procedures. In some cases, I will analyze the potency on a set of samples by mid-infrared spectroscopy (13), and have the laboratory run the same samples using their chromatographic method. Here the spectroscopy is used as an independent, orthogonal test of a specific laboratory’s chromatography. Assuming the mid-infrared method is unbiased, the correlation of the chromatographic results and spectroscopy results is a measure of the quality of the chromatography. The results have been that some laboratories match the spectrometer’s numbers well, some match poorly, and some not at all. Since the variable here is the laboratories, we must conclude that some laboratories are better than others. Only a small fraction of the laboratories I have visited have passed muster.

Does every cannabis business have to go to such lengths to find a laboratory they trust? No, but you need to do your homework before picking a laboratory. Certainly, you should visit a laboratory’s facility, interview the lab director, and find out what degree level and relevant experience they possess. Ask how the technicians performing the analyses are trained. Watch analyses being performed. And most importantly, ask for a copy of their method. If the method does not exist on paper it should be a red flag. If they refuse to give you a copy claiming it is “confidential information” that is another red flag. Slowly but surely standard methods are being promulgated in this industry. If your laboratory’s method is not in conformance with any of the standard methods out there, that is a red flag too.

What about state licensing and International Standards Organization (ISO) certifications? These are necessary but not sufficient conditions to trust a laboratory. Certainly, if a laboratory is not state licensed and ISO certified it should be avoided. However, laboratories with these accreditations still suffer from the inter-laboratory error problem. One paper I coauthored in this journal (1), where the same cannabis distillate samples were sent to five different state licensed, ISO certified laboratories for potency and pesticide analysis, found troubling differences in results across the five laboratories. Thus, these certifications are no guarantee that a specific laboratory will do a good job on a specific sample.

Given all this uncertainty, what is a cannabis business to do? Accuracy may be a myth in this industry, but precision is not. Precision measures how reproducible results are for a given set of samples (14). Our work has clearly shown that different cannabis laboratories have different precisions (1). The precision of different laboratories is easily quantified. To determine this, put together a set of samples that you are confident are homogeneous. Let’s say five samples. Split each sample into three aliquots, and ask your laboratory to analyze each sample. When the results come back you will have a set of three readings on each of the five samples. Calculate the average and standard deviation for each sample and analyte. Do this for multiple laboratories. The laboratory that has the best overall precision is who you should do business with. The point here is that in the absence of true accuracy, a laboratory that provides consistent results means you can compare samples to each other, compare results over time, and use this laboratory’s data to run your business going forward.

What About In-House Analyzers?

In practically every manufacturing industry, in-house testing is a common feature. In-house testing gives data on-the-spot about manufacturing processes and products, insuring that safe and effective medicines are made with high quality. Amongst the things that cannabis products may be required to be analyzed for are potency, pesticides, and heavy metals (various tests for mold and pathogens may be required as well, but since this type of testing is not my area of expertise, I won't comment further). Performing in-house testing for pesticides is impractical because the required instrumentation can cost hundreds of thousands of dollars. The expense of heavy metals testers are probably beyond the financial means of most cannabis businesses as well. However, there exist several types of potency analyzers that are accurate, easy to use, and cost a fraction of what a pesticide or metals analyzer may cost. Potency is the one thing that is practical for cannabis businesses to test for in-house.

How Do I Pick an In-House Potency Analyzer?

First, you should choose technology that best suits your needs. In previous columns I have discussed that to choose a testing method, four criteria should be used: accuracy, speed, cost, and the ability to perform representative sampling (15). These comprise what I call the “Golden Rectangle of Chemical Analysis” as seen in Figure 1.

In an ideal world a given analytical technology would be accurate, fast, low cost, and allow for representative sampling. The latter is particularly important for testing cannabis plant material because these samples are inevitably inhomogeneous and require significant amounts of testing to overcome sampling error (16). Because of this, any chosen method needs to be inexpensive and fast to allow the sample load required by representative sampling to be accomplished in a practical fashion. In a previous column I compared and contrasted chromatography and spectroscopy for potency analysis (15). Using the Golden Rectangle of Chemical Analysis, I concluded that chromatography was accurate, but suffered from being slow and expensive and hence did not lend itself well to representative sampling. On the other hand I concluded that infrared spectroscopy, while not necessarily as accurate as chromatography because it is a secondary method, enjoys advantages of speed, low cost, and the ability to handle a large sample load—meaning it is well suited to representative sampling programs.

How Do I Pick an In-House Analyzer Company?

After choosing an appropriate potency testing technology, you need to choose a vendor to do business with. You should ask the following two questions of your potential vendor.

1. Has this technology ever been used before in any application besides cannabis?

When I went to publish my first peer-reviewed paper on the use of mid-infrared spectroscopy for cannabis analysis, the editor, a very smart person, asked me the above question. The reason she had to do this is that there exist analyzers claiming they can analyze cannabis for potency, but use technologies that have never been used before in the history of science to determine concentrations of analytes in samples. I am particularly alarmed by analyzers that use cell phones and the “cloud” to determine potency measurements. The need for an internet connection means your company’s proprietary data is being sent over insecure links, to a mysterious location, to a server that may be easy to hack into. The details of how the results are determined on the cloud are typically kept secret, and the results magically appear on your phone’s screen with no information whatsoever on how the device was calibrated or validated. Analyzers like this are to be avoided.

2. Where is the peer-reviewed data supporting your accuracy claims?

Peer review is how science knows what it knows. For example, when I want to publish a paper in this journal, a copy of the manuscript is sent to my peers. They make anonymous comments on the quality of my work and my paper. The verdict may be to publish the paper, re-write the paper, go back to the laboratory and do more work, or reject the paper. For any potency analyzer company to claim they have scientific proof of their accuracy data, that data must be published in a peer-reviewed journal like this one. Here are some examples of cannabis analyzers that have published peer reviewed accuracy data (1,8–12). Any claims made without peer-review are just that, claims. Application notes and white papers put out by instrument companies that are not peer-reviewed are not necessarily scientifically correct either. You should only consider doing business with a company that can back up their claims with peer-reviewed science.

What’s a Cannabis Business to Do?

How do you run a cannabis business given the inter-laboratory error problem and Pet Lab Syndrome? In any situation where cannabis products are being bought or sold based on COAs, the buyer and seller must agree ahead of time on which laboratory’s results will be used to set the purchase price. This is the only way to avoid the Pet Lab Syndrome. For potency analyses, there exist portable analyzers that can provide readings on the spot. The advantage of this is that data is obtained in minutes rather than days so transactions can take place quickly and easily. Also, the low cost per analysis of these tests means that multiple aliquots of the material being purchased can be analyzed to minimize sampling error. Of course, the buyer and seller need to agree ahead of time to abide by the results of whatever testing instrument is being used.

Conclusions

The inter-laboratory error problem continues to plague the cannabis industry. This has led to the Pet Lab Syndrome, where arguments of which laboratory is right inevitably occur. Cannabis businesses need to vet any laboratory they intend to do business with, paying particular attention to precision. Another solution to this problem is for cannabis businesses to do their own testing. Potency testing is the only in-house testing that is practical. How to choose an in-house potency tester was discussed. To alleviate the Pet Lab Syndrome problem, whenever cannabis products are bought and sold, the buyer and seller must agree on a laboratory or analyzer whose results will be used to set a purchase price.

References

1. B.C. Smith, P. Lessard, and R. Pearson, Cannabis Science and Technology 2(1), 48 (2019).
2. M.O. Bonn-Miller, M.J.E. Loflin, B.F. Thomas, J.P. Marcu, T. Hyke, and R. Vandrey, Journal of the American Medical Association, 318, (2017) 1708 (2017).
3. B. Young, The Seattle Times, January 5, 2016. https://www.seattletimes.com/seattle-news/marijuana/some-pot-labs-in-state-failed-no-pot-at-all-says-scientist/ .
4. https://www.nbcbayarea.com/investigations/Industry-Insiders-Warn-of-Fraud-at-Marijuana-Testing-Labs-458125743.html?_osource=SocialFlowFB_BAYBrand .
5. B.C. Smith, Cannabis Science and Technology 2(2), 12-17 (2019).
6. B.C. Smith, Cannabis Science and Technology 2(3), 10-14 (2019).
7. B.C. Smith, Cannabis Science and Technology 3(2), 10–15 (2020).
8. B. C. Smith, Cannabis Science and Technology, 2(6), 28-33 (2019).
9. B.C. Smith, Terpenes and Testing, Jan.-Feb. 2018., Pg. 32.
10. B.C. Smith, Terpenes and Testing, Nov.-Dec. 2017, pg. 48.
11. B.C. Smith and C. Fucetola, Cannabis Science and Technology 3(6), 24–38 (2020).
12. www.bigsurscientific.com.
13. B.C. Smith, Fundamentals of Fourier Transform Infrared Spectroscopy, 2nd Ed. (CRC Press, Boca Raton, Florida, 2011).
14. B.C. Smith, Cannabis Science and Technology 1(4), 12–16 (2018).
15. B.C. Smith, Cannabis Science and Technology 2(6), 10–14 (2019).
16. B.C. Smith, Cannabis Science and Technology 3(6), 10–13 (2020).

ABOUT THE COLUMNIST

Brian C. SMITH, PHD, is Founder, CEO, and Chief Technical Officer of Big Sur Scientific. He is the inventor of the BSS series of patented mid-infrared based cannabis analyzers. Dr. Smith has done pioneering research and published numerous peer-reviewed papers on the application of mid-infrared spectroscopy to cannabis analysis, and sits on the editorial board of Cannabis Science and Technology. He has worked as a lab director for a cannabis extractor, as an analytical chemist for Waters Associates and PerkinElmer, and as an analytical instrument salesperson. He has more than 30 years of experience in chemical analysis and has written three books on the subject. Dr. Smith earned his PhD on physical chemistry from Dartmouth College.

How to Cite this Article

B. Smith, Cannabis Science and Technology 3(7), 10-12 (2020).