A Method to Extract Feature Variables Contributed in Nonlinear Machine Learning Prediction

Abstract

Background Although advances in prediction accuracy have been made with new machine learning methods, such as support vector machines and deep neural networks, these methods make nonlinear machine learning models and thus lack the ability to explain the basis of their predictions. Improving their explanatory capabilities would increase the reliability of their predictions.

Objective Our objective was to develop a factor analysis technique that enables the presentation of the feature variables used in making predictions, even in nonlinear machine learning models.

Methods A factor analysis technique was consisted of two techniques: backward analysis technique and factor extraction technique. We developed a factor extraction technique extracted feature variables that was obtained from the posterior probability distribution of a machine learning model which was calculated by backward analysis technique.

Results In evaluation, using gene expression data from prostate tumor patients and healthy subjects, the prediction accuracy of a model of deep neural networks was approximately 5% better than that of a model of support vector machines. Then the rate of concordance between the feature variables extracted in an earlier report using Jensen–Shannon divergence and the ones extracted in this report using backward elimination using Hilbert–Schmidt independence criteria was 40% for the top five variables, 40% for the top 10, and 49% for the top 100.

Conclusion The results showed that models can be evaluated from different viewpoints by using different factor extraction techniques. In the future, we hope to use this technique to verify the characteristics of features extracted by factor extraction technique, and to perform clinical studies using the genes, we extracted in this experiment.

Keywords

machine learning - deep learning - factor analysis - genomics

Authors' Contributions

All persons who meet authorship criteria are listed as authors, and all authors certify that they have participated sufficiently in the work to take public responsibility for the content, including participation in the concept, design, analysis, writing, or revision of the manuscript. M.S. and T. S., in particular, contributed equally to this work.

Publication History

Received: 07 May 2019

Accepted: 18 December 2019

Article published online:
07 May 2020

• References

• 1 Klambauer G, Unterthiner T, Mayr A, Hochreiter S. Self-Normalizing Neural Networks. Neural Information Processing Systems (NIPS). Available at: https://papers.nips.cc/paper/6698-self-normalizing-neural-networks.pdf . Accessed January 8, 2019
• 2 Ribeiro MT, Singh S, Guestrin C. “Why Should I Trust You?”: Explaining the Predictions of Any Classifier. Published at: KDD '16: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 2016: 1135-1144 . Doi: https://doi.org/10.1145/2939672.2939778
• 3 Caruana R, Lou Y, Gehrke J, Koch P, Sturm M, Elhadad N. Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission. Available at: http://people.dbmi.columbia.edu/noemie/papers/15kdd.pdf . Accessed January 8, 2019
• 4 Suzuki M, Shibahara T, Muragaki Y. Factor Analysis Technique to Extract Feature Contributed to Prediction by Machine Learning. The 37th Joint Conference on Medical Informatics (JCMI). 2017 November;37:854–856. Japan
• 5 Lin J. Divergence measures based on the Shannon entropy. IEEE Trans Inf Theory 1991; 37 (01) 145-151
• 6 Hukushima K, Nemoto K. Exchange monte carlo method and application to spin glass simulations. J Phys Soc Jpn 1996; 65 (06) 1604-1608
• 7 Gelman A. . John B. Carlin, Hal S. Stern, Donald B. Rubin, Bayesian data analysis. 3rd ed.Vol. 2. Boca Raton, FL: Taylor & Francis; 2014
• 8 Gretton A, Bousquet O, Smola A, Scholkopf B. Measuring statistical dependence with Hilbert-Schmidt norms. International Conference on Algorithmic Learning Theory. Published at: ALT'05: Proceedings of the 16th international conference on Algorithmic Learning Theory. 2005: 63-77 ; Doi: https://doi.org/10.1007/11564089_7
• 9 Le Song JB, Borgwardt KM, Gretton A, Smola A. The BAHSIC family of gene selection algorithms. Available at: https://pdfs.semanticscholar.org/7def/cd626cbd7a1bbe662b50583262e0f323ccd0.pdf . Accessed January 8, 2020
• 10 Singh D, Febbo PG, Ross K. , et al. Gene expression correlates of clinical prostate cancer behavior. Cancer Cell 2002; 1 (02) 203-209
• 11 Bergstra J, Bengio Y. Random search for hyper-parameter optimization. J Mach Learn Res 2012; 13 (01) 281-305
• 12 Hinton GE, Srivastava N, Krizhevsky A, Sutskever I, Salakhutdinov RR. Improving neural networks by preventing co-adaptation of feature detectors. Available at: https://arxiv.org/pdf/1207.0580.pdf . Accessed Janurary 17, 2019
• 13 Song L, Bedo J, Borgwardt KM, Gretton A, Smola A. Gene selection via the BAHSIC family of algorithms. Bioinformatics 2007; 23 (13) i490-i498
• 14 De Pinto V, Messina A, Lane DJ, Lawen A. Voltage-dependent anion-selective channel (VDAC) in the plasma membrane. FEBS Lett 2010; 584 (09) 1793-1799
• 15 Thinnes FP. Neuroendocrine differentiation of LNCaP cells suggests: VDAC in the cell membrane is involved in the extrinsic apoptotic pathway. Mol Genet Metab 2009; 97 (04) 241-243
• 16 The human protein atlas. Available at: https://www.proteinatlas.org/ . Accessed Janurary 17, 2019
• 17 De Semir D, Nosrati M, Bezrookove V. , et al. Pleckstrin homology domain-interacting protein (phip) as a marker and mediator of melanoma metastasis. Proc Natl Acad Sci U S A. 2012; 109 (18) 7067-7072
• 18 Xie Y-F, Macdonald JF, Jackson MF. TRPM2, calcium and neurodegenerative diseases. Int J Physiol Pathophysiol Pharmacol 2010; 2 (02) 95-103
• 19 The human protein atlas: ANGEL2. Available at: https://www.proteinatlas.org/ENSG00000174606-ANGEL2/pathology . Accessed Janurary 17, 2019
• 20 Rimkus C, Friederichs J, Boulesteix AL. , et al. Microarray-based prediction of tumor response to neoadjuvant radiochemotherapy of patients with locally advanced rectal cancer. Clin Gastroenterol Hepatol 2008; 6 (01) 53-61
• 21 Cardoso WP, Denardin OVP, Rapoport A, Araújo VC, Carvalho MB. Proliferating cell nuclear antigen expression in mucoepidermoid carcinoma of salivary glands. Sao Paulo Med J 2000; 118 (03) 69-74