Broad sharing of laboratory data is a major undertaking with huge potential payoffs in patient care and outcomes, said experts who spoke at the 2021 AACC Annual Scientific Meeting in Atlanta. In the session “Data Aggregation and Integration in Laboratory Medicine: How to Build Prediction Models and Learn from Multi-Institutional Data,” speakers explained how interoperability of electronic health records (EHRs) and systems can drive important research findings, how prediction models can be transferred across practices, and what labs can do to improve the quality of clinical prediction models.
Connecting data from healthcare organization and payer systems will drive more generalizable research findings and better outcomes, said Thomas Durant, MD, assistant professor and medical director of chemical pathology and laboratory medicine at Yale University, and a speaker at the session.
But challenges abound. They include data’s typically complicated journey between lab instruments, lab information systems (LISs), and EHRs, as well as siloing at multiple sites of patient care and payor organizations. “Data is living in all these boxes. We need to figure out a way to connect these boxes and share data seamlessly,” Durant said.
These experts also shared their vision for the potential benefits of broad sharing of laboratory data, its reliance on community standards and specifications, challenges to learning and deploying prediction models, and how laboratorian decisions affect secondary use of data.
Far From Seamless
In laboratory medicine, the transfer of data among many institutions is far from smooth, Durant noted. Problems include variably implemented semantic standards and a long path of interoperability building blocks. As a result, determining the integrity of external test results may be difficult. For example, a researcher working on SARS-CoV-2 assays must consider the different targets, antibodies, and sensitivities of commercial tests and lab-developed ones when deciding to use an external lab or to repeat tests in-house.
Getting high quality data not only means deriving meaning from and trusting a result, but also sending it along an appropriate path. Also, this process involves multiple steps, using messaging standards to successfully transmit the information—such as HL-7 and HL7-FHIR (Fast Healthcare Interoperability Resources), as well as semantic interoperability standards that ensure that data from the sender means the same thing to the receiver. Such systems of terminology include LOINC or SNOMED.
Harmonizing Data in the National COVID Cohort Collaborative
A recent multi-institutional data collection effort shows both the promise and pitfalls for sharing laboratory data. Formed in response to the COVID-19 pandemic, the National COVID Cohort Collaborative (N3C) has aimed to create a large data repository and allow data sharing across dozens of states. Its governance structure and data partnerships set the stage for data sharing at scale, according to Patrick Mathias, MD, PhD, a speaker during the AACC Annual Scientific Meeting session. N3C set up a centralized institutional review board framework, transfer and institutional data use agreements, and data structure and use processes that ensure researchers could submit requests quickly. Participating institutions’ use of common data models jump-started N3C, Mathias said. Using semantic standards in local LIS configurations ensured linkage to the common data model and helped N3C ensure security of data and make institutions comfortable with participation.
Mathias, who is an assistant professor, vice chair of clinical operations, and associate medical director of informatics at the University of Washington Medicine, described how N3C annotates, shares, and aggregates clinical data. Compared to studies of data from single institutions and research groups, NC3 processes produce data that reflect the broader community, lead to more generalizable insights, and ease validation of findings, he said.
A recent study found that N3C data and machine learning models accurately predicted clinical severity by using commonly collected clinical data about hospital admissions’ first 24 hours (JAMA Netw Open 2021; doi: 10.1001/jamanetworkopen.2021.16901). Using data from 65 sites on 8 million patients, 2.6 million COVID-19 cases, clinical observations, 4.3 billion lab results, and 1.3 billion medication records, the researchers showed which patient characteristics were associated with higher clinical severity. Analysis of BNP, CRP, creatinine, D-dimer, ferritin, lactate, and whole blood cell count showed greater increases in inflammatory markers among patients who died, Mathias noted, with the caveat that creatinine and ferritin increases were very similar in patients who died and in those who survived.
While the study shows exploratory data analysis can find trends over time, the N3C model reveals challenges for data sharing among labs, Mathias said. For example, universal device de-identifiers may be useful for unambiguously identifying Food and Drug Administration-cleared tests but were not incorporated into the N3C data sharing infrastructure. Meanwhile, LOINC doesn’t provide standardized, exact mapping between a test method and a code.
“We should learn from N3C, set an agenda by determining important research questions and how to answer them, and decide how to augment existing models for transporting existing metadata we care about,” Mathias said.
He encouraged labs to get involved in their institutions’ mapping projects by understanding the relevant data sharing agreements and studies. “Help by being a check on what goes out the door,” he said.
Prediction Models in Clinical Practice
Another speaker in the session, Daniel Herman, MD, PhD, explained how clinical prediction models can use laboratory test data to identify which patients might benefit from a specific intervention. He described some of the current problems with using prediction models across clinical practices for decision support, including access to patient data, which needs semantic interoperability and annotation so it can be mapped.
Before they can produce usable insights, models need to account for differences in data at various sites. Those differences include variability in clinical practice, documentation, and assay methods, said Herman, who is an assistant professor of pathology and laboratory medicine at the Hospital of the University of Pennsylvania and director of the endocrine laboratory at the Perelman School of Medicine in Philadelphia.
In one example, a model takes concentrations of alpha-fetoprotein (AFP) and other protein biomarkers, plus patient age, to determine Down syndrome risk. Women with unaffected pregnancies usually have lower AFP concentrations. The tests involved in the model require each lab to adjust for local factors in part by figuring out a median concentration across its population and dividing an individual test result by that median. Models and adjustments are complicated by institutions’ use of different AFP measurement methods that report varying concentrations in different units. Additionally, models must recognize multivariate distribution of clinical factors and associations between protein concentration and race, weight, status for smoking and diabetes, and singleton versus twin pregnancy.
Designing such a prediction model with data aggregated across multiple sites requires data interoperability and LOINC codes. However, LOINC codes cannot specify whether AFP concentrations are for tumor markers or prenatal screening. Aggregated data labeled by LOINC and unique device identifiers for AFP could help show differences between assays or account for the known differences, Herman said. In addition, LISs and EHRs don’t use existing device identifiers, which are not part of standard messages for test results.
A Sepsis Case Study
Current sepsis models, such as one from EPIC’s EHR, need local verification, Herman noted. He pointed to a recent study that aimed to verify the EPIC sepsis prediction model and found it lacking. In a study population of 27,697 patients undergoing 38,455 hospitalizations, sepsis occurred in 7% (JAMA Intern Med 2021; doi:10.1001/jamainternmed.2021.2626). The EPIC model predicted the onset of sepsis with an area under the curve of 0.63, substantially worse than the performance reported by its developer.
Meanwhile, sepsis models that consider procalcitonin concentrations are likely beset by differences in methods and variability across sites, and controversy surrounds some models that use estimated glomerular filtration rate, Herman added.
When considering use of sepsis models and their performance, laboratorians should determine how complicated the relevant clinical concept and model are, the quality of annotation from the model, and the difficulty of getting outcomes to calibrate the model locally, Herman advised.
What Laboratories Can Do
Laboratorians are equipped to take the lead in building and using prediction models. Herman suggested that labs continue improving analytic standardization and assay harmonization, consider how well they currently code tests with existing annotation standards, and use LOINC and SNOMED. Labs can also consider how to make systems more interoperable and help develop and implement standards for nomenclature, result annotation, and result transmission.
“We need rich, precise information in our LISs and EHRs. We need these data to be transmitted between these systems in largely automated ways so we’re not copying and pasting,” Herman said.
He pointed to one group working on these problems. The Systemic Harmonization Interoperability Electronic Laboratory Data (SHIELD) group aims to develop strategies to improve annotation and interoperability enough to make clinical prediction models possible. Led by FDA, SHIELD includes a broad set of stakeholders such as AACC, government agencies, manufacturers, laboratorians, and researchers. Herman is a member of SHIELD’s Strategic Planning Implementation Committee.
“We can all help by improving how we annotate our clinical data, partnering with regulatory agencies, IVD manufacturers, and EHR/LIS vendors to improve interoperability,” Herman said.
Deborah Levenson is a freelance writer in College Park, Maryland. +Email: [email protected]