Tag / artificial-intelligence

by Sydney Lash

IT Strategization for Streamlining Lung Cancer Registry Processes In NC

Problem Statement

Lung cancer is one of the most significant chronic disease burdens in North Carolina, both in terms of incidence and mortality. According to CDC data from 2023, the state experienced 34.9 deaths per 100,000 residents from lung cancer and 58.4 new cases per 100,000 residents, both higher than the national averages.(1)These elevated rates place lung cancer among the most pressing cancer-related public health challenges in North Carolina.

Disparities are evident among racial and ethnic minorities across timeliness of diagnosis, treatment access, and five-year survival rates. (2) Risk factors also contribute to North Carolina’s elevated burden; adult smoking prevalence alongside of youth e-cigarette use are both higher than the national average.(3)

These statistics underscore the urgency of strengthening surveillance systems to provide timely, representative data that can inform prevention, screening, and equitable treatment strategies for lung cancer in North Carolina.

Current State of Surveillance

The North Carolina Central Cancer Registry (NCCCR) is the state’s legally mandated system for tracking cancer incidence, treatment, and outcomes. As part of the CDC’s National Program of Cancer Registries (NPCR) and the North American Association of Central Cancer Registries (NAACCR), it provides a comprehensive record of cancer cases across hospitals, clinics, and laboratories. NCCCR data are essential for understanding cancer trends, guiding public health policy, and supporting national cancer control efforts.

Despite its strengths, the NCCCR faces several limitations that reduce its effectiveness in addressing urgent public health concerns, such as lung cancer. Timeliness, integration, and equity monitoring are the main challenges. The registry data is often delayed up to 2 years behind real-world diagnoses, NCCCR is often not linked to behavioral data, and equity-based variables may be incomplete.

For a condition such as lung cancer, these limitations hinder North Carolina’s ability to respond quickly and equitably. Modernizing NCCCR to improve timeliness and representativeness is therefore critical to reducing the state’s disproportionate lung cancer burden.

Modernization Strategy

To address these gaps, North Carolina should modernize its cancer surveillance system by aligning the NCCCR with CSTE Objective 2.1: “Improve traditional surveillance systems to provide timely and representative chronic disease insights.” (4) The goal is to reduce reporting lag, strengthen representativeness, and create actionable insights for prevention and treatment of lung cancer.

A unique opportunity exists through the Cancer Identification and Precision Oncology Center (CIPOC) at UNC-Chapel Hill, which was recently awarded ARPA-H funding to aggregate and analyze cancer data from diverse sources—including electronic health records, pathology and radiology images, claims, and geographic information utilizing large language models.(5) CIPOC is designed to support real-time cancer case identification and equitable care delivery. Integrating NCCCR modernization with CIPOC’s infrastructure would allow the registry to improve timeliness, enhance data linkage, and support equity-focused initiatives

By grounding modernization in CSTE’s national strategy while leveraging CIPOC’s cutting-edge infrastructure, North Carolina can create a best-practice model for other states. This integrated approach would demonstrate how traditional registries and advanced AI-enabled systems can work together to provide high-quality data while leveraging the improved efficiency that AI brings.

Summary

North Carolina faces an urgent burden from lung cancer, with incidence and mortality rates above the national average and significant disparities across racial and geographic groups.

Modernizing the NCCCR to improve timeliness, completeness, and representativeness is critical to addressing this challenge. By aligning with CSTE Objective 2.1 and leveraging the AI-enabled infrastructure of CIPOC, the state can reduce delays in reporting, link surveillance data to risk factors and screening uptake, and generate equity-focused insights for targeted interventions.

This integrated approach demonstrates how traditional registries can evolve into rapid, representative systems and provides a best-practice model that other states and chronic conditions can adopt.

The model has clear implications beyond lung cancer. The same framework can be applied to other cancers, as well as non-cancer conditions like COPD or cardiovascular disease. Importantly, the CIPOC project’s use of retrieval-augmented generation and advanced prompting strategies to extract and synthesize multi-modal data provides an adaptable toolkit for modern surveillance. By applying the most effective AI methods refined within CIPOC, North Carolina can not only strengthen its lung cancer registry but also inform future AI applications in healthcare surveillance more broadly. This positions the state as a leader in operationalizing CSTE’s strategic plan while demonstrating how cutting-edge AI methods can scale across diseases and conditions.

by Sydney Lash

Using Machine Learning to Understand Treatment Delays in Breast Cancer Care

Introduction

Cancer treatment delays and modifications can significantly impact patient survival and quality of life. Research has consistently shown that marginalized populations, including Black, Hispanic, Asian, and American Indian/Alaska Native (AIAN) patients, experience higher rates of late-stage cancer diagnoses and lower rates of timely treatment.

For my project, I used machine learning models to examine how race, socioeconomic status, tumor characteristics, cancer stage, grade, subtype, age, clinical trial participation, and time to treatment initiation predict treatment interruptions in breast cancer patients.

This project highlights both the potential and limitations of using machine learning for predicting healthcare disparities in cancer treatment.

Dataset:

This project utilized Simulacrum v2.1.0, a synthetic dataset derived from the National Disease Registration Service (NDRS) Cancer Analysis System at NHS England. While this dataset mimics real-world cancer data, it ensures patient anonymity.

After data cleaning and removing incomplete observations, the dataset contained 69,367 patients, all diagnosed with breast cancer.

Demographics Overview

  • Gender Distribution:
    • Women: 98.7%
    • Men: 1.3% (Male breast cancer cases were retained for analysis.)
  • Age Distribution:
    • Mean Age: 61.06 years
    • Median Age: 61 years
  • Racial Distribution:
    • White: 85.9%
    • Asian: 3.7%
    • Black: 1.9%
    • Other: 1.7%
    • Mixed Race: 0.6%
    • Unknown: 6.1% (Reweighting was applied to mitigate algorithmic bias.)

  • Neighborhood Deprivation (Socioeconomic Status):
    • Scored from 1 (most deprived) to 5 (least deprived).
    • The dataset was fairly balanced across deprivation levels.

Average Time to Treatment Initiation (by Race)

  • Overall: 61 days
  • Other Racial Groups: 71 days
  • Black Patients: 63 days
  • White Patients: 62 days
  • Asian Patients: 54 days
  • Mixed Race Patients: 45 days
  • Unknown Race: 57 days

Clinical Trial Participation

  • 84.4% of patients were enrolled in a clinical trial.

Data Processing & Feature Engineering

1. Cleaning & Standardizing Data

  • Removed inconsistent staging classifications.
  • Imputed missing values for key variables such as comorbidity scores and time to treatment initiation.

2. Encoding Variables

  • One-Hot Encoding:
    • Race, tumor subtype, estrogen receptor (ER), progesterone receptor (PR), and HER2 status.
  • Ordinal Encoding:
    • Tumor stage, node stage, metastasis stage, overall stage, grade, and deprivation index.

3. Feature Engineering

  • Created a new feature,ANY_REGIMEN_MOD
    • Combined dose reduction, time delay, and early termination variables into one binary target variable.
  • Grouped tumor biomarkers into cancer subtypes:
    • Luminal A: ER+, PR+, HER2- (Least aggressive)
    • Luminal B: ER+, PR+, HER2+ (Slightly more aggressive)
    • HER2-Enriched: ER-, PR-, HER2+
    • Triple Negative (Basal-like): ER-, PR-, HER2- (Most aggressive)

Bias Mitigation Strategies

1. Gender Bias

  • While 98% of patients were female, male breast cancer cases were retained for rare case analysis.
  • Applied weighting techniques to balance gender representation during model training.

2. Racial Bias

  • Since 85% of patients in the dataset were White, inverse weighting was applied to ensure fair contributions across racial groups.

Machine Learning Models & Performance

I trained Random Forest and XGBoost models to predict treatment modifications.

1. Initial Model Performance (Random Forest & XGBoost)

Results were poor. The models performed only slightly better than random guessing.

2. Hyperparameter Tuning & Feature Selection

  • Used GridSearchCV to optimize parameters.
  • Dropped the least important features and retrained the models.
  • Results did not improve, and performance worsened in some cases.

3. Model Performance Issues

  • Models failed to generalize to testing data.
  • ROC AUC scores hovered around 0.5, meaning models were barely better than random guessing.
  • Models overfitted the ‘no treatment modification’ group, inaccurately predicting treatment delays.

4. Alternative Models Attempted

To address these issues, I tested additional models:

  • Support Vector Machines (SVM)
  • Neural Networks
  • Logistic Regression
  • K-Nearest Neighbors (KNN)
  • Naïve Bayes

Findings from Alternative Models:

  • SVM and Logistic Regression performed best (Accuracy: 0.65), but logistic regression completely failed at predicting treatment modifications.
  • Neural Networks, KNN, and Naïve Bayes were slightly better balanced but still had low accuracy (~0.60–0.62).

5. Last Attempt – Oversampling with SMOTE

  • Used Synthetic Minority Oversampling (SMOTE) to balance the dataset.
  • SVM improved slightly, but performance gains were minimal.

Uncovering Insights Through Clustering

Since predictive modeling was unsuccessful, I conducted clustering analysis to find patterns in the data.

Three Distinct Patient Clusters Emerged:

Cluster 0: Moderate-Stage Cancer (35.4% Treatment Modifications)

  • Average Age: 60.8
  • Tumor Stage: T2, N0.7, M0.03 (Localized)
  • Time to Treatment: 32.6 days

Cluster 1: Early-Stage Cancer with Delayed Treatment (36.7% Treatment Modifications)

  • Average Age: 61.0
  • Tumor Stage: T1, N0.03, M0.0 (Localized, very early-stage)
  • Time to Treatment: 46.8 days (longest)

Cluster 2: Older Patients with Shortest Time to Treatment (36.8% Treatment Modifications)

  • Average Age: 69.4 years
  • Higher Comorbidities (Charlson Score = 0.63)
  • Time to Treatment: 14.8 days (shortest)

Key Takeaways from Clustering:

  • Cluster 1 had the longest time to treatment despite being early-stage. Further investigation needed.
  • Older patients (Cluster 2) received faster treatment but had more comorbidities.

Conclusion & Next Steps

Machine learning models failed to predict treatment modifications accurately.
Clustering analysis revealed patterns in treatment delays and patient subgroups.

Future Steps:

  1. Explore more sophisticated models (Deep Learning, Bayesian Networks).
  2. Use real-world data instead of synthetic datasets.
  3. Investigate non-quantifiable factors, like patient-provider interactions and healthcare policies.

Final Thoughts

This project highlighted the complexity of predicting cancer treatment interruptions and the importance of interdisciplinary approaches in health equity research.

If you’re interested in machine learning for healthcare, data-driven health equity research, or predictive modeling, let’s connect!

by Sydney Lash

Addressing Healthcare Disparities in North Carolina

Through my Healthcare Data Visualization course I, alongside two other classmates, were tasked with creating a dashboard using a data set and platform of our choice. Using data from the 2018 Health Professional Shortage Area (HPSA) dataset provided by the U.S. Department of Health & Human Services, our analysis reveals critical insights into the challenges and opportunities for improving healthcare access statewide.

Key Findings from the Analysis

  1. Healthcare Shortages Are Severe in Underserved Areas
    The average provider-to-population ratio in HPSA-designated areas is 2.11 clinicians per 10,000 residents, significantly below the recommended 6.67 clinicians. This stark disparity highlights the strain on underserved communities, especially in rural regions.
  2. Poverty Rates Compound Access Issues
    North Carolina’s poverty rate—measured as residents below 200% of the federal poverty line—is 27%, slightly above the national average of 26.9%. This economic disadvantage exacerbates barriers to healthcare, disproportionately affecting rural counties.
  3. Medically Underserved Areas (MUAs) Require Immediate Attention
    MUA scores, which account for clinician ratios, infant mortality, poverty levels, and the percentage of elderly populations, show an alarming average of 51.76 across NC—well below the national threshold of 62. Henderson and Transylvania counties, with MUA scores of 0, represent the most critically underserved areas in the state.
  4. Rural Hospitals Closures and Policy Impacts
    Historical trends show peaks in shortages in 2002 and 2015-2018, correlating with changes in HPSA methodology and rural hospital closures. These events further stress the importance of sustained policy interventions.
  5. Prioritization of Resources by HPSA Scores
    The HPSA scoring system, used by the National Health Service Corps (NHSC), prioritizes counties for clinician assignments. Mecklenburg County has the highest HPSA score due to its large population, indicating where current resources are concentrated. However, smaller rural counties with lower scores risk being overlooked despite their critical needs.

Access to quality healthcare is a fundamental need, yet many counties across North Carolina face significant shortages in healthcare providers, particularly in rural and economically disadvantaged areas. Our findings underscore the urgent need for targeted interventions. Allocating resources to counties with the highest ratios of underserved populations, addressing the economic and geographic barriers to care, and replicating successful policies in declining shortage areas can help mitigate these disparities. For policymakers, healthcare providers, and community leaders, this analysis serves as a roadmap for reducing inequities and ensuring better access to healthcare for all North Carolinians.

Sources:

https://data.hrsa.gov/tools/shortage-area/hpsa-find

https://www.bls.gov/opub/reports/working-poor/2020/#:~:text=In%202020%2C%2037.2%20million%20people%2C%20or%2011.4,notes%20section%20for%20examples%20of%20poverty%20levels.)

https://healthycommunitiesnc.org/

ciceroinstitute.org/research/north-carolina-physician-shortage-facts/

by Sydney Lash

Clustering Individuals Based on Health and Socioeconomic Indicators Using the CDC’s BRFSS Data

Project Overview: I analyzed the CDC’s Behavioral Risk Factor Surveillance System (BRFSS) 2015 dataset using K-means clustering to identify groups based on reported health and life satisfaction patterns. By combining health indicators with socioeconomic factors (household income and education) I aimed to understand how these social determinants relate to individual health outcomes. The initial dataset contained over 400,000 observations, which I reduced to 15,032 by cleaning out incomplete data.

Variables Used:

  • General Health (GENHLTH): Measures perceived overall health (higher values indicate poorer health).
  • Mental Health (MENTHLTH): Number of days mental health negatively impacted daily life (higher values indicate more frequent struggles).
  • Physical Health (PHYSHLTH): Number of days physical health was poor.
  • Life Satisfaction (LSATISFY): Reflects self-reported quality of life (higher values mean lower satisfaction).

Clustering Analysis and Findings:

Analysis 1: Health and Income Using the elbow method, I determined an optimal cluster count of 6. Here’s what I found:

  • Cluster 0 (18%): Highest income group (>$75,000) with excellent health and high life satisfaction.
  • Cluster 1 (9%): Lowest income group ($10,000-$15,000) with significant health challenges but moderate life satisfaction.
  • Cluster 2 (6%): Middle-income earners ($35,000-$50,000) with the poorest health indicators but moderate satisfaction.
  • Cluster 3 (14%): Middle-income group ($35,000-$50,000) with good health and high life satisfaction.
  • Cluster 4 (35%): Largest group with highest income levels (>$75,000), showing good health and high life satisfaction.
  • Cluster 5 (18%): Upper middle-income earners ($50,000-$75,000) with similar good health and high satisfaction.

This analysis highlights that higher income is associated with better health outcomes and life satisfaction, reinforcing existing evidence on the impact of socioeconomic factors.

This image displays the relationship between income (INCOME2, y-axis) and physical health (PHYSHLTH, x-axis). Clusters 0, 3, and 5 are skewed to the left, indicating that these groups experience fewer days where physical health negatively impacts their daily life. These clusters also belong to the highest income categories, suggesting that higher income groups tend to have better physical health outcomes. In contrast, Cluster 2 is skewed to the right, showing a higher number of days of poor physical health, with income levels spread throughout the range. Clusters 1 and 2 have a denser concentration of observations on the right side, reflecting the groups with the poorest health outcomes, regardless of their income distribution.

Analysis 2: Health and Education For this analysis, I used 9 clusters based on the elbow method. Key findings include:

  • Cluster 0 (19%): Highly educated (college graduates) with very good health and high life satisfaction.
  • Cluster 2 (4%): Individuals with some college education but severe physical and mental health challenges.
  • Cluster 6 (8%): High school graduates with moderate health challenges yet high life satisfaction, suggesting resilience.
  • Other clusters demonstrated how different levels of educational attainment impact health outcomes and satisfaction levels.
In general, the majority of the data set reported high or moderate life satisfaction. Clusters 0, 1, and 4 show high concentrations toward the left side of the plot, indicating high life satisfaction. These clusters also represent individuals with the highest levels of educational attainment (primarily college graduates). In contrast, Cluster 2 displays the widest spread in life satisfaction levels and consists mostly of individuals with high school education or lower.

Key Takeaways:

  • The majority of the dataset reported moderate to high life satisfaction. Clusters with the highest educational levels (college graduates) were concentrated in groups with higher satisfaction and better health outcomes.
  • Cluster 2 showed the widest spread of life satisfaction and predominantly consisted of individuals with high school education or lower, indicating the need for a more in-depth understanding of what contributes to variability in well-being among this group.

Critical Reflections and Future Directions:

  1. Dataset Limitations: The dataset is predominantly composed of white and highly educated individuals, limiting the generalizability of these findings. To make public health insights more inclusive, future analyses should use more diverse datasets.
  2. Adding More Variables: Incorporating factors like healthcare access, chronic disease indicators, and racial identity could provide a more comprehensive understanding of health disparities and social determinants.
  3. Methodological Improvements: While K-means clustering in Weka is effective for straightforward analysis, it has limitations with non-linear relationships and imbalanced datasets. Future projects will explore more advanced clustering techniques like DBSCAN or hierarchical clustering using Python for deeper insights.
  4. Actionable Steps: I plan to expand future analyses by integrating more demographic variables and advanced techniques to provide a fuller picture of factors influencing health and life satisfaction in the U.S. population.

By continually refining my approach, I aim to produce more meaningful and comprehensive public health insights. This project served as a valuable practice in understanding how socioeconomic factors impact health outcomes.

Full lab write up

Lab 06 (1)Download
by Sydney Lash

Exploring Social Data with Principal Component Analysis (PCA)

During my summer internship at the NIH, I was introduced to Principal Component Analysis (PCA) through a colleague. Intrigued by PCA’s potential, I wanted to apply this technique to social data, particularly from the National Longitudinal Study of Adolescent to Adult Health (Add Health), which contains a rich dataset of 10,237 variables.

Objective

My goal was to identify underlying patterns in social factors like academic performance, self-esteem, relationships with parents, and substance use. I narrowed down the vast dataset to 50 key variables to uncover trends and relationships.

Approach

I began by learning PCA through various resources, including Kaggle tutorials and DataCamp courses. I also revisited linear algebra fundamentals to ensure a solid mathematical understanding.

For the analysis:

  1. Data Cleaning: Initially, I filled missing values with -1, but realized this approach needed refinement based on the scale of responses.
  2. PCA Implementation: I used the prcomp function in R to perform PCA. Focusing on the first two principal components, which explained 27.3% of the variance, allowed me to manage the complexity.
  3. Visualization: I created a biplot to visualize the results. Due to the large number of variables, I filtered for the most influential ones, revealing that alcohol usage significantly impacts dataset variability.

Findings

  • Principal Component 1: Associated with lower self-esteem, moderate alcohol use, and less satisfaction in parent relationships.
  • Principal Component 2: Linked to positive school behavior, higher grades, less loneliness, and lower alcohol consumption.

Using K-means clustering, I identified two groups:

  • Cluster 1 (Red): Higher on PC1, indicating lower self-esteem and weaker parental bonds.
  • Cluster 2 (Blue): Higher on PC2, suggesting better academic performance and less loneliness.

The analysis highlighted how alcohol usage and social factors contribute to overall data variability. I plan to refine my approach with a smaller dataset for better interpretation.

Resources Used