Diagnostic Performance of Machine Learning-Derived OSA Prediction Tools in Large Clinical and Community-Based Samples

Steven J. Holfinger; M. Melanie Lyons; Brendan T. Keenan; Diego R. Mazzotti; Jesse Mindel; Greg Maislin; Peter A. Cistulli; Kate Sutherland; Nigel McArdle; Bhajan Singh; Ning Hung Chen; Þórarinn Gíslason; Thomas Penzel; Fang Han; Qing Yun Li; Richard Schwab; Allan I. Pack; Ulysses J. Magalang

doi:10.1016/j.chest.2021.10.023

Diagnostic Performance of Machine Learning-Derived OSA Prediction Tools in Large Clinical and Community-Based Samples

Steven J. Holfinger, M. Melanie Lyons, Brendan T. Keenan, Diego R. Mazzotti, Jesse Mindel, Greg Maislin, Peter A. Cistulli, Kate Sutherland, Nigel McArdle, Bhajan Singh, Ning Hung Chen, Þórarinn Gíslason, Thomas Penzel, Fang Han, Qing Yun Li, Richard Schwab, Allan I. Pack, Ulysses J. Magalang

Faculty of Medicine

University of Iceland

Research output: Contribution to journal › Article › peer-review

Abstract

Background: Prediction tools without patient-reported symptoms could facilitate widespread identification of OSA. Research Question: What is the diagnostic performance of OSA prediction tools derived from machine learning using readily available data without patient responses to questionnaires? Also, how do they compare with STOP-BANG, an OSA prediction tool, in clinical and community-based samples? Study Design and Methods: Logistic regression and machine learning techniques, including artificial neural network (ANN), random forests (RF), and kernel support vector machine, were used to determine the ability of age, sex, BMI, and race to predict OSA status. A retrospective cohort of 17,448 subjects from sleep clinics within the international Sleep Apnea Global Interdisciplinary Consortium (SAGIC) were randomly split into training (n = 10,469) and validation (n = 6,979) sets. Model comparisons were performed by using the area under the receiver-operating curve (AUC). Trained models were compared with the STOP-BANG questionnaire in two prospective testing datasets: an independent clinic-based sample from SAGIC (n = 1,613) and a community-based sample from the Sleep Heart Health Study (n = 5,599). Results: The AUCs (95% CI) of the machine learning models were significantly higher than logistic regression (0.61 [0.60-0.62]) in both the training and validation datasets (ANN, 0.68 [0.66-0.69]; RF, 0.68 [0.67-0.70]; and kernel support vector machine, 0.66 [0.65-0.67]). In the SAGIC testing sample, the ANN (0.70 [0.68-0.72]) and RF (0.70 [0.68-0.73]) models had AUCs similar to those of the STOP-BANG (0.71 [0.68-0.72]). In the Sleep Heart Health Study testing sample, the ANN (0.72 [0.71-0.74]) had AUCs similar to those of STOP-BANG (0.72 [0.70-0.73]). Interpretation: OSA prediction tools using machine learning without patient-reported symptoms provide better diagnostic performance than logistic regression. In clinical and community-based samples, the symptomless ANN tool has diagnostic performance similar to that of a widely used prediction tool that includes patient symptoms. Machine learning-derived algorithms may have utility for widespread identification of OSA.

Original language	English
Pages (from-to)	807-817
Number of pages	11
Journal	Chest
Volume	161
Issue number	3
DOIs	https://doi.org/10.1016/j.chest.2021.10.023
Publication status	Published - Mar 2022

Bibliographical note

Funding Information: Financial/nonfinancial disclosures: The authors have reported to CHEST the following: A. I. P. is the John Miclot Professor of Medicine, Division of Sleep Medicine/Department of Medicine at the University of Pennsylvania Perelman School of Medicine. The Miclot chair was provided by funds from Respironics Foundation. P. A. C. has an appointment to an endowed academic Chair at the University of Sydney that was created from ResMed funding; he receives no personal fees, and this relationship is managed by an Oversight Committee of the University. P. A. C. has received research support from ResMed, SomnoMed, Zephyr Sleep Technologies, and Bayer; is a consultant/adviser to Zephyr Sleep Technologies, ResMed, SomnoMed, and Signifier Medical Technologies; and has a pecuniary interest in SomnoMed related to a previous role in research and development (2004). None declared (S. J. H., M. M. L., B. T. K., D. R. M., J. M., G. M., K. S., N. M., B. S., N.-H. C., T. G., T. P., F. H., Q. Y. L., R. S., U. J. M.). Funding Information: FUNDING/SUPPORT: A. I. P. was supported by a Program Project Grant from the National Institutes of Health [ P01 HL094307 ]. D. R. M. is funded by the American Academy of Sleep Medicine Foundation [ #194-SR-18 ]. The project was also supported by the National Center for Advancing Translational Sciences [Grant UL1TR001070 ]. The Sleep Heart Health Study was supported by Grants U01HL53916 , U01HL53931 , U01HL53934 , U01HL53937 , U01HL53938 , U01HL53940 , U01HL53941 , and U01HL64360 from the National Institutes of Health . The National Sleep Research Resource was supported by the National Heart, Lung, and Blood Institute [ R24 HL114473 and 75N92019R0022 ]. Publisher Copyright: © 2021 American College of Chest Physicians

Other keywords

OSA
artificial neural network
electronic medical record
kernel support vector machine
machine learning
prediction model
random forest

Access to Document

10.1016/j.chest.2021.10.023

Cite this

Holfinger, S. J., Lyons, M. M., Keenan, B. T., Mazzotti, D. R., Mindel, J., Maislin, G., Cistulli, P. A., Sutherland, K., McArdle, N., Singh, B., Chen, N. H., Gíslason, Þ., Penzel, T., Han, F., Li, Q. Y., Schwab, R., Pack, A. I., & Magalang, U. J. (2022). Diagnostic Performance of Machine Learning-Derived OSA Prediction Tools in Large Clinical and Community-Based Samples. Chest, 161(3), 807-817. https://doi.org/10.1016/j.chest.2021.10.023

@article{5c378eafe7a64ed7ad46c2e18d629209,

title = "Diagnostic Performance of Machine Learning-Derived OSA Prediction Tools in Large Clinical and Community-Based Samples",

abstract = "Background: Prediction tools without patient-reported symptoms could facilitate widespread identification of OSA. Research Question: What is the diagnostic performance of OSA prediction tools derived from machine learning using readily available data without patient responses to questionnaires? Also, how do they compare with STOP-BANG, an OSA prediction tool, in clinical and community-based samples? Study Design and Methods: Logistic regression and machine learning techniques, including artificial neural network (ANN), random forests (RF), and kernel support vector machine, were used to determine the ability of age, sex, BMI, and race to predict OSA status. A retrospective cohort of 17,448 subjects from sleep clinics within the international Sleep Apnea Global Interdisciplinary Consortium (SAGIC) were randomly split into training (n = 10,469) and validation (n = 6,979) sets. Model comparisons were performed by using the area under the receiver-operating curve (AUC). Trained models were compared with the STOP-BANG questionnaire in two prospective testing datasets: an independent clinic-based sample from SAGIC (n = 1,613) and a community-based sample from the Sleep Heart Health Study (n = 5,599). Results: The AUCs (95\% CI) of the machine learning models were significantly higher than logistic regression (0.61 [0.60-0.62]) in both the training and validation datasets (ANN, 0.68 [0.66-0.69]; RF, 0.68 [0.67-0.70]; and kernel support vector machine, 0.66 [0.65-0.67]). In the SAGIC testing sample, the ANN (0.70 [0.68-0.72]) and RF (0.70 [0.68-0.73]) models had AUCs similar to those of the STOP-BANG (0.71 [0.68-0.72]). In the Sleep Heart Health Study testing sample, the ANN (0.72 [0.71-0.74]) had AUCs similar to those of STOP-BANG (0.72 [0.70-0.73]). Interpretation: OSA prediction tools using machine learning without patient-reported symptoms provide better diagnostic performance than logistic regression. In clinical and community-based samples, the symptomless ANN tool has diagnostic performance similar to that of a widely used prediction tool that includes patient symptoms. Machine learning-derived algorithms may have utility for widespread identification of OSA.",

keywords = "OSA, artificial neural network, electronic medical record, kernel support vector machine, machine learning, prediction model, random forest",

author = "Holfinger, \{Steven J.\} and Lyons, \{M. Melanie\} and Keenan, \{Brendan T.\} and Mazzotti, \{Diego R.\} and Jesse Mindel and Greg Maislin and Cistulli, \{Peter A.\} and Kate Sutherland and Nigel McArdle and Bhajan Singh and Chen, \{Ning Hung\} and {\TH}{\'o}rarinn G{\'i}slason and Thomas Penzel and Fang Han and Li, \{Qing Yun\} and Richard Schwab and Pack, \{Allan I.\} and Magalang, \{Ulysses J.\}",

note = "Funding Information: Financial/nonfinancial disclosures: The authors have reported to CHEST the following: A. I. P. is the John Miclot Professor of Medicine, Division of Sleep Medicine/Department of Medicine at the University of Pennsylvania Perelman School of Medicine. The Miclot chair was provided by funds from Respironics Foundation. P. A. C. has an appointment to an endowed academic Chair at the University of Sydney that was created from ResMed funding; he receives no personal fees, and this relationship is managed by an Oversight Committee of the University. P. A. C. has received research support from ResMed, SomnoMed, Zephyr Sleep Technologies, and Bayer; is a consultant/adviser to Zephyr Sleep Technologies, ResMed, SomnoMed, and Signifier Medical Technologies; and has a pecuniary interest in SomnoMed related to a previous role in research and development (2004). None declared (S. J. H., M. M. L., B. T. K., D. R. M., J. M., G. M., K. S., N. M., B. S., N.-H. C., T. G., T. P., F. H., Q. Y. L., R. S., U. J. M.). Funding Information: FUNDING/SUPPORT: A. I. P. was supported by a Program Project Grant from the National Institutes of Health [ P01 HL094307 ]. D. R. M. is funded by the American Academy of Sleep Medicine Foundation [ \#194-SR-18 ]. The project was also supported by the National Center for Advancing Translational Sciences [Grant UL1TR001070 ]. The Sleep Heart Health Study was supported by Grants U01HL53916 , U01HL53931 , U01HL53934 , U01HL53937 , U01HL53938 , U01HL53940 , U01HL53941 , and U01HL64360 from the National Institutes of Health . The National Sleep Research Resource was supported by the National Heart, Lung, and Blood Institute [ R24 HL114473 and 75N92019R0022 ]. Publisher Copyright: {\textcopyright} 2021 American College of Chest Physicians",

year = "2022",

month = mar,

doi = "10.1016/j.chest.2021.10.023",

language = "English",

volume = "161",

pages = "807--817",

journal = "Chest",

issn = "0012-3692",

publisher = "Elsevier Inc.",

number = "3",

}

Holfinger, SJ, Lyons, MM, Keenan, BT, Mazzotti, DR, Mindel, J, Maislin, G, Cistulli, PA, Sutherland, K, McArdle, N, Singh, B, Chen, NH, Gíslason, Þ, Penzel, T, Han, F, Li, QY, Schwab, R, Pack, AI & Magalang, UJ 2022, 'Diagnostic Performance of Machine Learning-Derived OSA Prediction Tools in Large Clinical and Community-Based Samples', Chest, vol. 161, no. 3, pp. 807-817. https://doi.org/10.1016/j.chest.2021.10.023

TY - JOUR

T1 - Diagnostic Performance of Machine Learning-Derived OSA Prediction Tools in Large Clinical and Community-Based Samples

AU - Holfinger, Steven J.

AU - Lyons, M. Melanie

AU - Keenan, Brendan T.

AU - Mazzotti, Diego R.

AU - Mindel, Jesse

AU - Maislin, Greg

AU - Cistulli, Peter A.

AU - Sutherland, Kate

AU - McArdle, Nigel

AU - Singh, Bhajan

AU - Chen, Ning Hung

AU - Gíslason, Þórarinn

AU - Penzel, Thomas

AU - Han, Fang

AU - Li, Qing Yun

AU - Schwab, Richard

AU - Pack, Allan I.

AU - Magalang, Ulysses J.

N1 - Funding Information: Financial/nonfinancial disclosures: The authors have reported to CHEST the following: A. I. P. is the John Miclot Professor of Medicine, Division of Sleep Medicine/Department of Medicine at the University of Pennsylvania Perelman School of Medicine. The Miclot chair was provided by funds from Respironics Foundation. P. A. C. has an appointment to an endowed academic Chair at the University of Sydney that was created from ResMed funding; he receives no personal fees, and this relationship is managed by an Oversight Committee of the University. P. A. C. has received research support from ResMed, SomnoMed, Zephyr Sleep Technologies, and Bayer; is a consultant/adviser to Zephyr Sleep Technologies, ResMed, SomnoMed, and Signifier Medical Technologies; and has a pecuniary interest in SomnoMed related to a previous role in research and development (2004). None declared (S. J. H., M. M. L., B. T. K., D. R. M., J. M., G. M., K. S., N. M., B. S., N.-H. C., T. G., T. P., F. H., Q. Y. L., R. S., U. J. M.). Funding Information: FUNDING/SUPPORT: A. I. P. was supported by a Program Project Grant from the National Institutes of Health [ P01 HL094307 ]. D. R. M. is funded by the American Academy of Sleep Medicine Foundation [ #194-SR-18 ]. The project was also supported by the National Center for Advancing Translational Sciences [Grant UL1TR001070 ]. The Sleep Heart Health Study was supported by Grants U01HL53916 , U01HL53931 , U01HL53934 , U01HL53937 , U01HL53938 , U01HL53940 , U01HL53941 , and U01HL64360 from the National Institutes of Health . The National Sleep Research Resource was supported by the National Heart, Lung, and Blood Institute [ R24 HL114473 and 75N92019R0022 ]. Publisher Copyright: © 2021 American College of Chest Physicians

PY - 2022/3

Y1 - 2022/3

N2 - Background: Prediction tools without patient-reported symptoms could facilitate widespread identification of OSA. Research Question: What is the diagnostic performance of OSA prediction tools derived from machine learning using readily available data without patient responses to questionnaires? Also, how do they compare with STOP-BANG, an OSA prediction tool, in clinical and community-based samples? Study Design and Methods: Logistic regression and machine learning techniques, including artificial neural network (ANN), random forests (RF), and kernel support vector machine, were used to determine the ability of age, sex, BMI, and race to predict OSA status. A retrospective cohort of 17,448 subjects from sleep clinics within the international Sleep Apnea Global Interdisciplinary Consortium (SAGIC) were randomly split into training (n = 10,469) and validation (n = 6,979) sets. Model comparisons were performed by using the area under the receiver-operating curve (AUC). Trained models were compared with the STOP-BANG questionnaire in two prospective testing datasets: an independent clinic-based sample from SAGIC (n = 1,613) and a community-based sample from the Sleep Heart Health Study (n = 5,599). Results: The AUCs (95% CI) of the machine learning models were significantly higher than logistic regression (0.61 [0.60-0.62]) in both the training and validation datasets (ANN, 0.68 [0.66-0.69]; RF, 0.68 [0.67-0.70]; and kernel support vector machine, 0.66 [0.65-0.67]). In the SAGIC testing sample, the ANN (0.70 [0.68-0.72]) and RF (0.70 [0.68-0.73]) models had AUCs similar to those of the STOP-BANG (0.71 [0.68-0.72]). In the Sleep Heart Health Study testing sample, the ANN (0.72 [0.71-0.74]) had AUCs similar to those of STOP-BANG (0.72 [0.70-0.73]). Interpretation: OSA prediction tools using machine learning without patient-reported symptoms provide better diagnostic performance than logistic regression. In clinical and community-based samples, the symptomless ANN tool has diagnostic performance similar to that of a widely used prediction tool that includes patient symptoms. Machine learning-derived algorithms may have utility for widespread identification of OSA.

AB - Background: Prediction tools without patient-reported symptoms could facilitate widespread identification of OSA. Research Question: What is the diagnostic performance of OSA prediction tools derived from machine learning using readily available data without patient responses to questionnaires? Also, how do they compare with STOP-BANG, an OSA prediction tool, in clinical and community-based samples? Study Design and Methods: Logistic regression and machine learning techniques, including artificial neural network (ANN), random forests (RF), and kernel support vector machine, were used to determine the ability of age, sex, BMI, and race to predict OSA status. A retrospective cohort of 17,448 subjects from sleep clinics within the international Sleep Apnea Global Interdisciplinary Consortium (SAGIC) were randomly split into training (n = 10,469) and validation (n = 6,979) sets. Model comparisons were performed by using the area under the receiver-operating curve (AUC). Trained models were compared with the STOP-BANG questionnaire in two prospective testing datasets: an independent clinic-based sample from SAGIC (n = 1,613) and a community-based sample from the Sleep Heart Health Study (n = 5,599). Results: The AUCs (95% CI) of the machine learning models were significantly higher than logistic regression (0.61 [0.60-0.62]) in both the training and validation datasets (ANN, 0.68 [0.66-0.69]; RF, 0.68 [0.67-0.70]; and kernel support vector machine, 0.66 [0.65-0.67]). In the SAGIC testing sample, the ANN (0.70 [0.68-0.72]) and RF (0.70 [0.68-0.73]) models had AUCs similar to those of the STOP-BANG (0.71 [0.68-0.72]). In the Sleep Heart Health Study testing sample, the ANN (0.72 [0.71-0.74]) had AUCs similar to those of STOP-BANG (0.72 [0.70-0.73]). Interpretation: OSA prediction tools using machine learning without patient-reported symptoms provide better diagnostic performance than logistic regression. In clinical and community-based samples, the symptomless ANN tool has diagnostic performance similar to that of a widely used prediction tool that includes patient symptoms. Machine learning-derived algorithms may have utility for widespread identification of OSA.

KW - OSA

KW - artificial neural network

KW - electronic medical record

KW - kernel support vector machine

KW - machine learning

KW - prediction model

KW - random forest

UR - https://www.scopus.com/pages/publications/85124398725

U2 - 10.1016/j.chest.2021.10.023

DO - 10.1016/j.chest.2021.10.023

M3 - Article

C2 - 34717928

SN - 0012-3692

VL - 161

SP - 807

EP - 817

JO - Chest

JF - Chest

IS - 3

ER -

Diagnostic Performance of Machine Learning-Derived OSA Prediction Tools in Large Clinical and Community-Based Samples

Abstract

Bibliographical note

Other keywords

Access to Document

Fingerprint

Cite this