Discussions of laboratory innovations are oftentimes focused on automation and the other advanced technologies that optimize and accelerate the work of scientists. Controls and reagents, however, play an equally important role – if not more so – in the modern clinical lab. Every researcher will tell you a story of the time that “one bad apple ruined the barrel.” While the so-called bad apples can take many forms (practices, procedures and so on), failing controls and reagents are sometimes the hardest and costliest to identify wasting time and resources for products that should arrive with implicit trust in quality.1
Researchers have well characterized the impacts of inferior materials when used in development activities for biopharmaceuticals, which at this point are well understood processes.2 Now imagine the impact of poor-quality controls or reagents when used in post-development activities such as those found in the clinical laboratory setting, especially those involving new or emerging technologies like multi-quadrupole ICP-MS (Inductively coupled Mass Spectrometry). If you don’t know what a good result looks like, how do you know when you have a bad result? Here, the importance of being able to trust your controls and reagents is amplified significantly.
Controlling the controls and rating the reagents
Finding a trusted controls provider may be less intimidating than finding a reagents provider, as there are several established brands that have become the de-facto standard for most applications. However, the search for full reagent kits becomes more complicated as there are vendors that offer products for sale with unknown quality, untraceable source materials. and potentially suspect manufacturing processes. Even within regulated markets, we cannot always be sure of ongoing quality: a quick search of the U.S. Food and Drug Administration’s (FDA) Medical Device Recalls database shows multiple reagents recalls every year. In these regulated environments, how do we ensure that we are acquiring the highest quality products possible? Gibson (2019) points out that sourcing reagents from Good Manufacturing Practice (GMP) facilities “ensures a standard that minimizes variation during assay or drug development and, ultimately, an introduction into the clinical setting. (GMP) covers all aspects of production, including the location, starting resources, equipment and staff training.”3
Now consider the largely unregulated world of research use only (RUO) reagents. These RUO reagents are often used in scenarios where labs choose to develop and validate-for-purpose their own internally lab developed tests (LDTs). According to the FDA (2018), “While the uses of an LDT are often the same as the uses of FDA-cleared or approved in vitro diagnostic tests, some labs may choose to offer their own test.” The FDA also notes that accurate in vitro diagnostic tests ensure that “patients and healthcare providers do not seek unnecessary treatments, delay needed treatments, or become exposed to inappropriate therapies.”4
As we move to new technologies and new approaches to clinical testing, such as mass spectrometry, LDTs may become more common,5 causing the impact that reagent quality has on test results to become more pronounced.
However, the regulations surrounding LDTs don’t mirror those of FDA-approved tests, even though they may be used in similar situations: “As LDTs have grown in use and complexity, FDA regulations have not kept pace. Instead, LDTs are principally regulated by CMS (the Centers for Medicare & Medicaid Services) under separate regulations known as Clinical Laboratory Improvement Amendments (CLIA). CLIA oversight focuses on laboratory operations and staff training but does not assess the validity of individual tests in a lab”5 Inaccuracies in LDTs may be caused by invalid/incomplete methodologies but can equally be caused by poor-quality reagents or controls.6
As is the case with regulated reagents, manufacturing facilities that operate under GMP and carry a minimum of ISO:9001 certifications are important thresholds for maintaining quality for reagents used in LDTs. Because standards such as ISO:9001 require continuous review and both internal and external audits, confidence in suppliers holding such certifications is increased.7
As we move to clinical testing in which LTDs may become more common — and explore the use of more and more sensitive analytical techniques and instrumentation, such as multi-quadrupole ICP-MS systems that offer PPT levels of sensitivity— it becomes increasingly important that we source our reagents from sources that treat RUO reagents with a high level of care.
Ensuring quality over time
How do we establish the quality of sourced reagents? Do we head to Google and put our faith in the first reagent manufacturer that appears in the search results? Do we phone a friend, or perhaps rely on past experiences? Compounding this conundrum is the knowledge that as we move forward with new and emerging technologies, the list of references and experiences can shorten significantly. The first step: establishing initial trust in a vendor. To do so, labs should implement both proficiency testing (PT) and external quality assessment (EQA) programs, which provide assurance internally and externally that testing (and by extension, the reagents) taking place at your laboratory provides accurate and reliable results.
Managing lot-to-lot variation
Unfortunately, it has been shown that even after a reagent lot has passed initial qualifications, there can still be significant lot-to-lot variation for “qualified” reagents.8 As Algeciras-Schimnich (2014) explains, “Multiple factors can affect performance of a new reagent lot, including changes in a critical reagent material or in stability of the reagents, reagent damage during transportation or storage, or incorrect calibration” When used in the clinical setting, these variations have potential to affect the results of previously validated assays.6
The generally accepted way to deal with this lot-to-lot variability is to constantly evaluate assay performance as new lots of reagents are received. However, this continual re-evaluation process is time-consuming and comes with its own challenges and risks. For example, there are no mandated metrics for ensuring lot-to-lot quality, leading to wide variability in the number of sample tests run at any given lab and inconsistent testing methodologies. In addition, the decision to test with real patient samples or contrived controls is left to the individual lab.
Using quality management system
CLSI has provided a guideline for user evaluation of between-lot reagent variation (CLSI, 2013) that should aid in evaluating lot-to-lot performance. Rigorous performance of these evaluations can help to establish a prediction of performance, creating trust between the clinical labs and their suppliers.9
This is not to say that once we have sourced a trusted supplier and have assured ourselves of quality over time, we have completely absolved ourselves of responsibility. In fact, sourcing is just the beginning. We must all follow our own downstream quality processes including (but not limited to) incoming inspection, process control and chain-of-custody documentation. After all, there’s no point in establishing supplier quality if you can’t later tell which lot was used in a particular clinical test.
Supplier quality goes hand-in-hand with our own quality efforts and really is the only way to ensure quality over time. Where organizational size and bandwidth allows, the establishment of a quality management System (QMS) should be considered, as the QMS will allow labs to efficiently focus on both internal and external indicators of quality.
CLSI has provided several guidelines and standards to help laboratories develop quality management systems.10
Sometimes living under a QMS can seem like a burden. We trust ourselves; why should we have put up with the burden of QMS overhead to prove it to others? The answer is simple: we are suppliers to others, be-it elsewhere in the lab or to physicians and to the patients they care for. If we expect our own suppliers to adhere to a certain quality standard, so too should we hold ourselves accountable to at least the same standard, especially as we push forward into new territories like multi-quadrupole ICP-MS. As sensitivity increases, so too does the magnification of errors. Those depending on us must trust us, as we trust those that we depend on.
Trust is the key
As with any relationship, there must be trust. When we purchase our controls and reagents, we must be able to trust that our vendor is giving us products of high quality. GMP and ISO certifications are excellent indicators of this quality. So too is the track record of the manufacturer. We must also trust ourselves. We must establish the practices, processes and protocols that make these quality-supplied products meaningful and worthwhile investments, lest that be our bad apple be inside the barrel.
References
1. Lee M, Ji Q., Protein analysis using mass spectrometry: accelerating protein biotherapeutics from lab to patient. Hoboken: Wiley. 2017.
2. Mayer A, Fraley K. 2021. The importance of quality critical reagents for the entire development lifecycle of a bioparmaceutical: a pharmacokinetic case study. Bioanalysis. 2021; 13(10): 817-827. doi: 10.4155/bio-2020-0253.
3. Gibson J. Does reagent quality play a role in reproducibility of experimental data? Medical Laboratory Observer. https://www.mlo-online.com/management/qa-qc/article/21089049/does-reagent-quality-play-a-role-in-reproducibility-of-experimental-data. Published July 23, 2019. Accessed October 13, 2021.
4. U.S. Food & Drug Administration, 2018. Laboratory developed tests. https://www.fda.gov/medical-devices/in-vitro-diagnostics/laboratory-developed-tests. Updated September 27, 2018. Accessed 28 October 28, 2021.
5. The role of lab-developed tests in the in vitro diagnostics market. Pew Charitable Trusts https://www.pewtrusts.org/en/research-and-analysis/reports/2021/10/the-role-of-lab-developed-tests-in-the-in-vitro-diagnostics-market. Published October 22, 2021. Accessed October 28, 2021.
6. Algeciras-Schimnich A., 2014. Tackling reagent lot-to-lot verification in the clinical laboratory. Clinical Laboratory News. https://www.aacc.org/cln/articles/2014/july/bench-matters. Published July 1, 2014. Accessed October 28, 2021.
7. Keen R. 2019. ISO 9001 Checklist. ISO 9000 vs ISO 9001: What’s the difference. https://www.iso-9001-checklist.co.uk/ISO-9000-vs-ISO-9001-whats-the-difference.htm. Published July 27, 2019. Accessed October 18, 2021].
8. Staveline A, Riksheim B, Christensen N, Sandberg S. The importance of reagent lot registration in external quality assurance/proficiency testing schemes. Clin Chem. 2016 May;62(5):708-15. doi: 10.1373/clinchem.2015.247585.
9. User evaluation of between-reagent lot variation, 1st Edition. CLSI. https://clsi.org/standards/products/method-evaluation/documents/ep26/. Published September 13, 2013. Accessed October 28, 2021.
10.CLSI’s Quality Management System Resources. CLSI. https://clsi.org/standards/products/quality-management-systems/. Accessed November 5, 2021.