Download PDF
Methods Inf Med 2022; 61(03/04): 099-110
DOI: 10.1055/s-0042-1756649
Original Article

Automated Cognitive Health Assessment Using Partially Complete Time Series Sensor Data

Brian L. Thomas
1   School of Electrical Engineering and Computer Science, Washington State University, Pullman, Washington, United States
,
Lawrence B. Holder
1   School of Electrical Engineering and Computer Science, Washington State University, Pullman, Washington, United States
,
Diane J. Cook
1   School of Electrical Engineering and Computer Science, Washington State University, Pullman, Washington, United States
› Author Affiliations Funding This work is supported in part by National Institutes of Health grant R41EB029774.
› Further Information
    Permissions and Reprints

Abstract

Background Behavior and health are inextricably linked. As a result, continuous wearable sensor data offer the potential to predict clinical measures. However, interruptions in the data collection occur, which create a need for strategic data imputation.

Objective The objective of this work is to adapt a data generation algorithm to impute multivariate time series data. This will allow us to create digital behavior markers that can predict clinical health measures.

Methods We created a bidirectional time series generative adversarial network to impute missing sensor readings. Values are imputed based on relationships between multiple fields and multiple points in time, for single time points or larger time gaps. From the complete data, digital behavior markers are extracted and are mapped to predicted clinical measures.

Results We validate our approach using continuous smartwatch data for n = 14 participants. When reconstructing omitted data, we observe an average normalized mean absolute error of 0.0197. We then create machine learning models to predict clinical measures from the reconstructed, complete data with correlations ranging from r = 0.1230 to r = 0.7623. This work indicates that wearable sensor data collected in the wild can be used to offer insights on a person's health in natural settings.

Keywords

time series - data imputation - health assessment


Publication History

Received: 20 January 2022

Accepted: 06 July 2022

Article published online:
11 October 2022

© 2022. Thieme. All rights reserved.

Georg Thieme Verlag KG
Rüdigerstraße 14, 70469 Stuttgart, Germany

 

  • References

  • 1 Elflein J. Amount of time U.S. primary care physicians spent with each patient as of 2018. Statista. Published 2019. Accessed June 13, 2020 at: https://www.statista.com/statistics/250219/us-physicians-opinion-about-their-compensation/
    PubMedGoogle Scholar
  • 2 Iriondo J, Jordan J. Older People Projected to Outnumber Children for First Time in U.S. History; 2018. Accessed April 1, 2022 at: https://www.census.gov/newsroom/press-releases/2018/cb18-41-population-projections.html
    PubMed
  • 3 Center for Medicare and Medicaid Services. NHE Fact Sheet. Center for Medicare and Medicaid Services. Published online 2018. Accessed April 1, 2022 at: https://www.cms.gov/Research-Statistics-Data-and-Systems/Statistics-Trends-and-Reports/NationalHealthExpendData/NHE-Fact-Sheet
    PubMedGoogle Scholar
  • 4 Administration On Aging. Aging statistics. ACL. Published online 2018. Accessed April 1, 2022 at: https://acl.gov/aging-and-disability-in-america/data-and-research/profile-older-americans
    PubMedGoogle Scholar
  • 5 Office of The Assistant Secretary for Planning and Evaluation. National Plan to Address Alzheimer's Disease: 2018. ASPE. Published 2019. Accessed April 1, 2022 at: https://aspe.hhs.gov/national-plans-address-alzheimers-disease
    PubMedGoogle Scholar
  • 6 Fowler NR, Head KJ, Perkins AJ. et al. Examining the benefits and harms of Alzheimer's disease screening for family members of older adults: study protocol for a randomized controlled trial. Trials 2020; 21 (01) 202
    CrossrefPubMedGoogle Scholar
  • 7 Herman WH, Ye W, Griffin SJ. et al. Early detection and treatment of Type 2 diabetes reduce cardiovascular morbidity and mortality: a simulation of the results of the Anglo-Danish-Dutch study of intensive treatment in people with screen-detected diabetes in primary care. Diabetes Care 2015; 38 (08) 1449-1455
    CrossrefPubMedGoogle Scholar
  • 8 Akl A, Snoek J, Mihailidis A. Unobtrusive detection of mild cognitive impairment in older adults through home monitoring. IEEE J Biomed Health Inform 2017; 21 (02) 339-348
    CrossrefPubMedGoogle Scholar
  • 9 Spruijt-Metz D. Etiology, treatment and prevention of obesity in childhood and adolescence: a decade in review. J Res Adolesc 2011; 21 (01) 129-152
    CrossrefPubMedGoogle Scholar
  • 10 Lee MK, Oh J. Health-related quality of life in older adults: Its association with health literacy, self-efficacy, social support, and health-promoting behavior. Healthcare (Basel) 2020; 8 (04) 407
    CrossrefPubMedGoogle Scholar
  • 11 Nelson BW, Pettitt A, Flannery JE, Allen NB. Rapid assessment of psychological and epidemiological correlates of COVID-19 concern, financial strain, and health-related behavior change in a large online sample. PLoS One 2020; 15 (11) e0241990
    CrossrefPubMedGoogle Scholar
  • 12 Betsinger TK, DeWitte SN. Toward a bioarchaeology of urbanization: demography, health, and behavior in cities in the past. Am J Phys Anthropol 2021; 175 (Suppl. 72) 79-118
    CrossrefPubMedGoogle Scholar
  • 13 Gorman J. “Ome,” the sound of the scientific universe expanding. The New York Times. Accessed April 1, 2022 at: https://www.nytimes.com/2012/05/04/science/it-started-with-genome-omes-proliferate-in-science.html?_r=1 2012
    PubMedGoogle Scholar
  • 14 Asim Y, Azam MA. Context-aware human activity recognition (CAHAR) in-the-wild using smartphone accelerometer. IEEE Sensors 2020; 20 (08) 4361-4371
    CrossrefPubMedGoogle Scholar
  • 15 Vaizman Y, Ellis K, Lanckriet G. Recognizing detailed human context in the wild from smartphones and smartwatches. IEEE Pervasive Comput 2017; 16 (04) 62-74
    CrossrefPubMedGoogle Scholar
  • 16 Tran DH. Automated change detection and reactive clustering in multivariate streaming data. IEEE-RIVF International Conference on Computing and Communication Technologies.; 2019: 1-6
    PubMedGoogle Scholar
  • 17 Sterne JAC, White IR, Carlin JB. et al. Multiple imputation for missing data in epidemiological and clinical research: potential and pitfalls. BMJ 2009; 338: b2393
    CrossrefPubMedGoogle Scholar
  • 18 Austin PC, White IR, Lee DS, van Buuren S. Missing data in clinical research: a tutorial on multiple imputation. Can J Cardiol 2021; 37 (09) 1322-1331
    CrossrefPubMedGoogle Scholar
  • 19 Lachin JM. Fallacies of last observation carried forward analyses. Clin Trials 2016; 13 (02) 161-168
    CrossrefPubMedGoogle Scholar
  • 20 Bokde N, Martínez Álvarez F, Beck MW, Kulat K. A novel imputation methodology for time series based on pattern sequence forecasting. Pattern Recognit Lett 2018; 116: 88-96
    CrossrefPubMedGoogle Scholar
  • 22 Fang C, Wang C. Time series data imputation: a survey on deep learning approaches. arXiv 2020; 2011.11347
    PubMedGoogle Scholar
  • 22 Cao W, Wang D, Li J, Zhou H, Li L, Li Y. BRITS: Bidirectional recurrent imputation for time series. In: Neural Information Processing Systems; 2018: 6776-6786
    Google Scholar
  • 23 Wu X, Mattingly S, Mirjafari S, Huang C, Chawla NV. Personalized imputation on wearable sensory time series via knowledge transfer. ACM International Conference on Information and Knowledge Management; 2020: 1625-1634
    PubMedGoogle Scholar
  • 24 Yoon S, Sull S. GAMIN: Generative adversarial multiple imputation network for highly missing data. IEEE/CVF Conference on Computer Vision and Pattern Recognition; 2020: 8456-8464
    PubMedGoogle Scholar
  • 25 Yoon J, Jordon J, van der Schaar M. GAIN: Missing data imputation using generative adversarial networks. International Conference on Machine Learning; 2018
    PubMedGoogle Scholar
  • 26 Luo Y, Zhang Y, Cai X, Yuan X. E2gan: end-to-end generative adversarial network for multivariate time series imputation. International Joint Conference on Artificial Intelligence; 2019: 3094-3100
    PubMedGoogle Scholar
  • 27 Guo Z, Wan Y, Ye H. A data imputation method for multivariate time series based on generative adversarial network. Neurocomputing 2019; 360: 185-197
    CrossrefPubMedGoogle Scholar
  • 28 Chen K, Zhang D, Yao L, Guo B, Yu Z, Liu Y. Deep learning for sensor-based human activity recognition: overview, challenges and opportunities. J Assoc Comput Mach 2020; 37 (04) 111
    PubMedGoogle Scholar
  • 29 Bulling A, Blanke U, Schiele B. A tutorial on human activity recognition using body-worn inertial sensors. ACM Comput Surv 2014; 46 (03) 107-140
    CrossrefPubMedGoogle Scholar
  • 30 Tian Y, Zhang J, Chen L, Geng Y, Wang X. Selective ensemble based on extreme learning machine for sensor-based human activity recognition. Sensors (Basel) 2019; 19 (16) 3468
    CrossrefPubMedGoogle Scholar
  • 31 Nazabal A, Garcia-Moreno P, Artes-Rodriguez A, Ghahramani Z. Human activity recognition by combining a small number of classifiers. IEEE J Biomed Health Inform 2016; 20 (05) 1342-1351
    CrossrefPubMedGoogle Scholar
  • 32 Wang J, Chen Y, Hao S, Peng X, Hu L. Deep learning for sensor-based activity recognition: a survey. Pattern Recognit Lett 2019; 119: 3-11
    CrossrefPubMedGoogle Scholar
  • 33 Hammerla NY, Halloran S, Ploetz T. Deep, convolutional, and recurrent models for human activity recognition using wearables. International Joint Conference on Artificial Intelligence; 2016
    PubMedGoogle Scholar
  • 34 Ploetz T, Guan Y. Deep learning for human activity recognition in mobile computing. Computer 2018; 51 (05) 50-59
    CrossrefPubMedGoogle Scholar
  • 35 Guan Y, Ploetz T. Ensembles of deep LSTM leaners for activity recognition using wearables. ACM on Interactive, Mobile, Wearable and Ubiquitous Technol 2017; 1 (02) 11
    PubMedGoogle Scholar
  • 36 Culman C, Aminikhanghahi S, Cook DJ. Easing power consumption of wearable activity monitoring with change point detection. Sensors 2020; 20 (01) 310
    CrossrefPubMedGoogle Scholar
  • 37 Kankanhalli A, Saxena M, Wadhwa B. Combined interventions for physical activity, sleep, and diet using smartphone apps: a scoping literature review. Int J Med Inform 2019; 123: 54-67
    CrossrefPubMedGoogle Scholar
  • 38 Sprint G, Cook DJ, Schmitter-Edgecombe M. Unsupervised detection and analysis of changes in everyday physical activity data. J Biomed Inform 2016; 63: 54-65
    CrossrefPubMedGoogle Scholar
  • 39 de Zambotti M, Cellini N, Goldstone A, Colrain IM, Baker FC. Wearable sleep technology in clinical and research settings. Med Sci Sports Exerc 2019; 51 (07) 1538-1557
    CrossrefPubMedGoogle Scholar
  • 40 Wang R, Wang W, DaSilva A. et al. Tracking depression dynamics in college students using mobile phone and wearable sensing. Proc ACM Interact Mob Wearable Ubiquitous Technol 2018; 2 (01) 1-26
    CrossrefPubMedGoogle Scholar
  • 41 Dhana K, Evans DA, Rajan KB, Bennett DA, Morris MC. Healthy lifestyle and the risk of Alzheimer dementia: findings from 2 longitudinal studies. Neurology 2020; 95 (04) e374-e383
    CrossrefPubMedGoogle Scholar
  • 42 Alberdi A, Weakley A, Schmitter-Edgecombe M. et al. Smart home-based prediction of multi-domain symptoms related to Alzheimer's disease. IEEE J Biomed Health Inform 2018; 22 (06) 1720-1731
    CrossrefPubMedGoogle Scholar
  • 43 Li J, Rong Y, Meng H, Lu Z, Kwok T, Cheng H. TATC: Predicting Alzheimer's disease with actigraphy data. ACM SIGKDD International Conference on Knowledge Discovery and Data Mining; 2018: 509-518
    PubMedGoogle Scholar
  • 44 Yoon J, Jarrett D, van der Schaar M. Time-series generative adversarial networks. Conference on Neural Information Processing Systems; 2019
    PubMedGoogle Scholar
  • 45 Menendez ML, Pardo JA, Pardo L, Pardo MC. The Jensen-Shannon divergence. J Franklin Inst 1997; 334 (02) 307-318
    CrossrefPubMedGoogle Scholar
  • 46 Kachuee M, Karkkainen K, Goldstein O, Darabi S, Sarrafzadeh M. Generative imputation and stochastic prediction. IEEE Trans Pattern Anal Mach Intell 2022; 44 (03) 1278-1288
    CrossrefPubMedGoogle Scholar
  • 47 van Erven T, Harremos P. Renyi divergence and Kullback-Leibler divergence. IEEE Trans Inf Theory 2014; 60 (07) 3797-3820
    CrossrefPubMedGoogle Scholar
  • 48 Goodfellow IJ, Pouget-Abadie J, Mirza M. et al. Generative adversarial networks. Adv Neural Inf Process Syst 2014; 27: 1-9
    PubMedGoogle Scholar
  • 49 Marteau TM, Hollands GJ, Fletcher PC. Changing human behavior to prevent disease: the importance of targeting automatic processes. Science 2012; 337 (6101): 1492-1495
    CrossrefPubMedGoogle Scholar
  • 50 U.S. Department of Health and Human Services. Healthy People 2020; 2015
    PubMed
  • 51 Tourangeau R, Rips LJ, Rasinski K. The Psychology of Survey Response. Cambridge University Press; 2000
    CrossrefGoogle Scholar
  • 52 Palmer MG, Johnson CM. Experimenter presence in human behavior analytic laboratory studies: confound it?. Behavior Analysis: Research and Practice 2019; 19 (04) 303-314
    PubMedGoogle Scholar
  • 53 Li H, Abowd GD, Ploetz T. On specialized window lengths and detector based human activity recognition. ACM International Symposium on Wearable Computers; 2018: 67-71
    PubMedGoogle Scholar
  • 54 Aminikhanghahi S, Cook DJ. Enhancing activity recognition using CPD-based activity segmentation. Pervasive Mobile Comput 2019; 53: 75-89
    CrossrefPubMedGoogle Scholar
  • 55 Wan J, Li M, O'Grady M, Gu X, Alawlaqi M, O'Hare G. Time-bounded activity recognition for ambient assisted living. IEEE Trans Emerg Top Comput 2022; 10 (02) 1130-1141
    PubMedGoogle Scholar
  • 56 Du Y, Lim Y, Tan Y. A novel human activity recognition and prediction in smart home based on interaction. Sensors (Basel) 2019; 19 (20) 4474
    CrossrefPubMedGoogle Scholar
  • 57 Bharti P, De D, Chellappan S, Das SK. HuMAn: complex activity recognition with multi-modal multi-positional body sensing. IEEE Trans Mobile Comput 2019; 18 (04) 857-870
    CrossrefPubMedGoogle Scholar
  • 58 Kwon M-C, You H, Kim J, Choi S. Classification of various daily activities using convolution neural network and smartwatch. IEEE International Conference on Big Data; 2018
    PubMedGoogle Scholar
  • 59 Nweke HF, Teh YW, Al-Garadi MA, Alo UR. Deep learning algorithms for human activity recognition using mobile and wearable sensor networks: state of the art and research challenges. Expert Syst Appl 2018; 105: 233-261
    CrossrefPubMedGoogle Scholar
  • 60 Anguita D, Ghio A, Oneto L, Parra X, Reyes-Ortiz JL. A public domain dataset for human activity recognition using smartphones. European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning; 2013
    PubMedGoogle Scholar
  • 61 Stisen A, Blunck H, Bhattacharya S. et al. Smart devices are different: assessing and mitigating mobile sensing heterogeneities for activity recognition. ACM Conference on Embedded Networked Sensor Systems; 2015: 127-140
    PubMedGoogle Scholar
  • 62 Kwapisz JR, Weiss GM, Moore SA. Activity recognition using cell phone accelerometers. International Workshop on Knowledge Discovery from Sensor Data; 2010
    PubMedGoogle Scholar
  • 63 Lockhart JW, Weiss GM, Xue JC, Gallagher ST, Grosner AB, Pulickal TT. Design considerations for the Wisdm smart phone-based sensor mining architecture. International Workshop on Knowledge Discovery from Sensor Data; 2011: 25-33
    PubMedGoogle Scholar
  • 64 Cook DJ, Schmitter-Edgecombe M, Jonsson L, Morant AV. Technology-enabled assessment of functional health. IEEE Rev Biomed Eng 2019; 12: 319-332
    CrossrefPubMedGoogle Scholar
  • 65 Dodge HH, Mattek NC, Austin D, Hayes TL, Kaye JA. In-home walking speeds and variability trajectories associated with mild cognitive impairment. Neurology 2012; 78 (24) 1946-1952
    CrossrefPubMedGoogle Scholar
  • 66 Kaye J, Mattek N, Dodge HH. et al. Unobtrusive measurement of daily computer use to detect mild cognitive impairment. Alzheimers Dement 2014; 10 (01) 10-17
    CrossrefPubMedGoogle Scholar
  • 67 Petersen J, Austin D, Mattek N, Kaye J. Time out-of-home and cognitive, physical, and emotional wellbeing of older adults: a longitudinal mixed effects model. PLoS One 2015; 10 (10) e0139643
    CrossrefPubMedGoogle Scholar
  • 68 Petersen J, Larimer N, Kaye JA, Pavel M, Hayes TL. SVM to detect the presence of visitors in a smart home environment. International Conference of the IEEE Engineering in Medicine and Biology Society; 2012: 5850-5853
    PubMedGoogle Scholar
  • 69 Cook DJ. Sensors in support of aging-in-place: The good, the bad, and the opportunities. National Academies Workshop on Mobile Technology for Adaptive Aging; 2019
    PubMedGoogle Scholar
  • 70 Cook DJ, Schmitter-Edgecombe M. Fusing ambient and mobile sensor features into a behaviorome for predicting clinical health scores. IEEE Access 2021; 9: 65033-65043
    CrossrefPubMedGoogle Scholar
  • 71 Schmitter-Edgecombe M, Sumida C, Cook DJ. Bridging the gap between performance-based assessment and self-reported everyday functioning: an ecological momentary assessment approach. Clin Neuropsychol 2020; 34 (04) 678-699
    CrossrefPubMedGoogle Scholar
  • 72 WSU CASAS. Tools. Published 2021. Accessed April 1, 2022 at: http://casas.wsu.edu/tools/
    PubMed
  • 73 Fong TG, Fearing MA, Jones RN. et al. Telephone interview for cognitive status: creating a crosswalk with the mini-mental state examination. Alzheimers Dement 2009; 5 (06) 492-497
    CrossrefPubMedGoogle Scholar
  • 74 Peaker A, Stewart LE. Rey's auditory verbal learning test – a review. Crawford JR, Parker DM, eds. Developments in Clinical and Experimental Neuropsychology. Plenum Press; 1989
    Google Scholar
  • 75 Wilson BA, Alderman N, Burgess PW, Emslie H, Evans JJ. Behavioural Assessment of the Dysexecutive Syndrome. Thames Valley Test Company; 1996
    Google Scholar
  • 76 Sprint G, Cook D, Weeks D. Towards automating clinical assessments: a survey of the timed up and go (TUG). Biomedical Engineering, IEEE Reviews in. 2015; 8: 64-77
    CrossrefPubMedGoogle Scholar
  • 77 Burckhardt CS, Anderson KL. The quality of life scale (QOLS): reliability, validity, and utilization. Health Qual Life Outcomes 2003; 1: 60
    CrossrefPubMedGoogle Scholar
  • 78 Huo T, Guo Y, Shenkman E, Muller K. Assessing the reliability of the short form 12 (SF-12) health survey in adults with mental health conditions: a report from the wellness incentive and navigation (WIN) study. Health Qual Life Outcomes 2018; 16 (01) 34
    CrossrefPubMedGoogle Scholar
  • 79 Ware JE, Koskinski M, Keller SD. SF-12: How to Score the SF-12 Physical and Mental Health Summary Scores; 1995
    PubMed
  • 80 Smith G, Della Sala S, Logie RH, Maylor EA. Prospective and retrospective memory in normal ageing and dementia: a questionnaire study. Memory 2000; 8 (05) 311-321
    CrossrefPubMedGoogle Scholar
  • 81 Sheikh JI, Yesavage JA. Geriatric Depression Scale (GDS): recent evidence and development of a shorter version. Clin Gerontol 1986; 5: 165-173
    CrossrefPubMedGoogle Scholar
  • 82 Spitzer RL, Kroenke K, Williams JB, Löwe B. A brief measure for assessing generalized anxiety disorder: the GAD-7. Arch Intern Med 2006; 166 (10) 1092-1097
    CrossrefPubMedGoogle Scholar
  • 83 Gerstorf D, Siedlecki KL, Tucker-Drob EM, Salthouse TA. Executive dysfunctions across adulthood: measurement properties and correlates of the DEX self-report questionnaire. Neuropsychol Dev Cogn B Aging Neuropsychol Cogn 2008; 15 (04) 424-445
    CrossrefPubMedGoogle Scholar
  • 84 Schmitter-Edgecombe M, Parsey C, Lamb R. Development and psychometric properties of the instrumental activities of daily living: compensation scale. Arch Clin Neuropsychol 2014; 29 (08) 776-792
    PubMedGoogle Scholar
  • 85 Yu S, Zhang K, Xiao C, Huang JZ, Li MJ, Onizuka M. HSGAN: reducing mode collapse in GANs by the latent code distance of homogeneous samples. Comput Vis Image Underst 2022; 214: 103314
    CrossrefPubMedGoogle Scholar
  • 86 Zuo Z, Zhao L, Li A. et al. Dual distribution matching GAN. Neurocomputing 2022; 478: 37-48
    CrossrefPubMedGoogle Scholar
  • 87 Williams JA, Cook DJ. Forecasting behavior in smart homes based on sleep and wake patterns. Technol Health Care 2017; 25 (01) 89-110
    CrossrefPubMedGoogle Scholar
  • 88 Wang W, Harari GM, Wang R. et al. Sensing behavioral change over time: using within-person variability features from mobile sensing to predict personality traits. Proc ACM Interact Mob Wearable Ubiquitous Technol 2018; 2 (03) 141
    CrossrefPubMedGoogle Scholar
  • 89 Schmitter-Edgecombe M, Parsey CM, Lamb R. Development and psychometric properties of the instrumental activities of daily living – compensation scale (IADL-C). Arch Clin Neuropsychol 2014; Dec; 29 (08) 776-792
    CrossrefPubMedGoogle Scholar