Winner Announcement

The FairLabs Challenge closed with seven outstanding entries. These were carefully reviewed by judges representing ADLM’s Health Equity & Access Division, Informatics Division, and Data Analytics Steering Committee. All of the entries were very impressive, and the host committees thank everyone for participating.

The FairLabs Challenge was an especially difficult ADLM data analytics competition. FairLabs asked participants to go beyond simple data analysis to create a sharable tool that effectively presents fairness metrics and actionable information to drive meaningful improvement in health equity. Competitors were also incentivized to engage with local experts and datasets to use their tools to further health equity at their own institutions.

Congratulations go to the University of Washington Department of Laboratory Medicine & Pathology Team as the winner of the FairLabs Challenge. (See their entry.) The judges also recognized the Tricore Clinical Innovations Team as first runners-up and the Oregon Health & Science University team as second runners-up.

Be sure to attend the Health Equity & Access Division breakfast meeting during ADLM 2024 to learn more about applying data science to health equity and about the winning entry.

University of Washington Department of Laboratory Medicine & Pathology FairLabs Team:

Nathan Breit, Analytics Lead
Jing Zhang, QI Data Analyst
Joyce Liao, Assistant Professor
Kate Crawford, Clinical Pathology Resident

About the Competition

The third annual ADLM data analytics competition is cohosted by ADLM's Data Analytics Steering Committee, Health Equity & Access Division, and Informatics Division, and by the Washington University Section of Pathology Informatics. A representative of the winning team will be invited to speak at /strong>ADLM 2024 with travel support and free meeting registration included.

The Challenge: Develop a Tool for Analyzing Fairness in Laboratory Testing

This year's competition aims to develop novel tools for monitoring fairness in laboratory testing. Competitors will be provided with a real de-identified clinical dataset documenting peripartum urine drug screening in the labor and delivery department of a single institution. The challenge is to create a dashboard that analyzes and visualizes fairness using these data. Solutions will be judged based upon functionality and adherence to best practices (see Scoring, below). Extra points will be awarded to teams that go beyond the dataset provided by engaging local stakeholders and applying their tool to analyze fairness in their own laboratory practice.

Context

Fairness in Laboratory Testing

Laboratory testing carries benefits and unintended harms that are distributed unequally among patients. Patients belonging to vulnerable subgroups often bear a disproportionate burden of harms while enjoying fewer benefits. Inequities in laboratory medicine may be underappreciated due to limited real time monitoring in practice.

Peripartum Urine Drug Screening

Peripartum Urine Drug Screening (UDS) is an example of laboratory testing with benefits and unintended harms.

Benefits: Detection of specific compounds can be highly informative and guide immediate management decisions, as in the case of opiate positivity leading to treatment of neonatal abstinence syndrome. Additional benefits include the opportunity to provide counseling and support cessation efforts. In other cases the clinical benefit is unclear, as in the detection of isolated cannabis positivity.

Harms: Peripartum UDS testing can undermine the mother's trust in the clinical team and lead to reporting to Child Protective Services (CPS), which can be traumatic and harmful to the new family.

A recent retrospective study performed at Washington University School of Medicine identified history of isolated marijuana use (IMU) as the most frequently cited indication for peripartum UDS1. The study demonstrated that 99% of the positive UDS ordered for this indication were positive for cannabis only. The vast majority (89.8%) of mothers with a positive UDS ordered for IMU were reported to CPS. Of those mothers reported to CPS, 20.9% were White and 79.1% were Black despite only about 50% of mothers identifying as Black and positivity for a nonprescribed substance other than marijuana being 3 times more likely among White mothers.

To try to address these inequities in UDS harms, a quality improvement initiative was developed. This initiative included revising the labor and delivery policies to prohibit ordering UDS for an indication of IMU. To promote accountability to these policy changes, an electronic question was added to the peripartum UDS order requiring the clinician select an appropriate testing indication.

Your task is to create a dashboard that can evaluate the impact of this quality improvement initiative by analyzing and visualizing the fairness of peripartum UDS testing over time.

Scoring

Competitors will be evaluated based on the functionality of their tool, their adherence to best practices for application development, and their ability to deploy their tool to address fairness in their local laboratory practice. Scoring will be performed by expert reviewers using a standardized scoring rubric and blinded to the competitor's identity.

Functionality (45 points)

There are many ways to conceptualize and quantify fairness2. Points will be awarded for each concept that is correctly implemented. Additional functionality points will be awarded for quality of visualization and user experience.

Implementation of Fairness Concepts (20 points)

  • Demographic parity (4 points)
  • Equalized odds (4 points)
  • Predictive parity (4 points)
  • Equal outcomes (4 points)
  • Other (4 points)

Quality of Visualizations (10 points)

  • Impactful visuals (2 points)
  • Clean and appealing visualizations (2 points)
  • Axes and plots labeled (2 points)
  • Appropriate use of colors (2 points)
  • Appropriately sized fonts (2 points)

Statistics (5 points)

  • Confidence intervals provided for point estimates (2 points)
  • Formal hypothesis testing (3 points)

User Experience (10 points)

  • Intuitive navigation (5 points)
  • Interactivity (5 points)

Local Activation (20 points)

Competitors are highly encouraged to identify local stakeholders that would utilize a fairness monitor in practice. Points will be awarded for running the analytical tool on a local dataset. Please provide a screenshot of your local analysis (without PHI) and a short letter from the end user documenting their engagement. Please summarize key findings from local activation.

  • Local testing data analyzed for fairness (10 points)
  • End-user engages with fairness dashboard running on local data (5 points)
  • Describe insight provided by local analysis in 500 words or fewer (5 points)

Best Practices (12 points)

Best practices in software development help to optimize quality, maintainability, and reusability. Note that source code and commit histories are required to be eligible for points in this section.

Readability (4 points)

  • Code is clearly and cleanly commented (1 point)
  • Code is simple and not bloated (1 point)
  • Code utilizes a clear and consistent naming convention (1 point)
  • Code is organized into hierarchy of modular functional units (1 point)

Reusability (2 points)

  • Code utilizes functionalized or object oriented programming (1 point)
  • Code can be configured to new dataset without modification of source code (1 point)

Version Control (2 points)

  • Version control system used to track development (1 point)
  • Commits are modular, logical, and appropriately scoped (1 point)

Documentation and Deployment (4 points)

  • Usage notes provided (1 point)
  • Dependencies are defined (1 point)
  • Virtualized or containerized environment used (1 point)
  • Tool accessible through web hosting (1 point)

Significance

The tools developed in this competition could be used more broadly to help laboratories to identify unfairness in laboratory testing and monitor quality improvement projects aimed to promote equity in laboratory practice.

Timeline

January 15, 2024: Competition begins

May 15, 2024: Competition ends

June 15, 2024: Announcement of winning team

July 29, 2024 (anticipated): Presentation from winning team at ADLM 2024

How to Participate

Sign Up for a GitHub Account

  1. Navigate to https://github.com
  2. Click "Sign up"
  3. Follow the prompts to create your personal account

Fork the Competition Repository

  1. Navigate to https://github.com/WUSM-LGM-Informatics-Section/adlm-fairlabs-competition
  2. Click "Fork"
  3. Select "Create a new fork"
  4. Set your GitHub account as the owner (default)
  5. Click "Create fork"

Build Your Solution

  1. Clone the forked repository with the command
    git clone https://github.com/myGitHubUsername/adlm-fairlabs-competition.git
    Note: Replace myGitHubUsername with your GitHub handle
  2. Make a folder in the cloned repository with your team name
  3. Build your solution within your team folder

Submit Your Solution Via a Pull Request

  1. Navigate to https://github.com/myGitHubUsername/adlm-fairlabs-competition
    Note: Replace myGitHubUsername with your GitHub handle
  2. Click "Contribute"
  3. Select "Open pull request"
  4. Click "Create pull request"
    Note: we will review your pull request to ensure that it contains everything needed to score your submission
    Important: You must submit your complete solution by May 15, 2024, to be eligible to win the competition!

Need Help?

If you are unfamiliar with GitHub, need help getting starting, or have other questions, please email [email protected] for assistance.

References

  1. 1.Rubin A, Zhong L, Nacke L, Woolfolk C, Raghuraman N, Carter E, Kelly J. Urine drug screening for isolated marijuana use in labor and delivery units. Obstet Gynecol 2022;140:607–9. https://doi.org/10.1097/AOG.0000000000004930
  2. 2.Azimi V, Zaydman MA. Optimizing equity: working towards fair machine learning algorithms in laboratory medicine. J Appl Lab Med 2023;8:113–28. https://doi.org/10.1093/jalm/jfac085