CC-BY-NC-ND 4.0 · J Card Crit Care 2017; 01(01): 024-029
DOI: 10.1055/s-0037-1604202
Original Article
Official Publication of The Simulation Society (TSS), accredited by International Society of Cardiovascular Ultrasound (ISCU)

Contactless and Hassle Free Real Time Heart Rate Measurement with Facial Video

Shubham Pratap Singh
1  Department of Electronics and Communications, Maharaja Surajmal Institute of Technology, New Delhi, India
,
Neeru Rathee
1  Department of Electronics and Communications, Maharaja Surajmal Institute of Technology, New Delhi, India
,
Harsh Gupta
1  Department of Electronics and Communications, Maharaja Surajmal Institute of Technology, New Delhi, India
,
Paolo Zamboni
2  Vascular Disease Center, University of Ferrara, Italy
,
Ajay Vikram Singh
3  Physical Intelligence Department, Max Planck Institute for Intelligent Systems, Stuttgart, Germany
› Author Affiliations
Further Information

Address for correspondence

Ajay Vikram Singh, PhD
Physical Intelligence Department
Max Planck Institute for Intelligent Systems
Stuttgart 70569
Germany   

Publication History

Publication Date:
29 September 2017 (online)

 

Abstract

In the era of automation, an automatic and noninvasive tool is required for day-to-day health monitoring. Cardiac pulse being one of the vital physiological parameters has drawn attention of researchers for its measurement. In this article, the authors present an approach for automatic cardiac pulse measurement, using a typical webcam. In the presented approach, face region is detected and RGB traces are retrieved for the face region. FFT is applied on the extracted traces to compute power spectrum of the respective traces. The peaks of the power spectrum limited in the band region (0.75–4 Hz) are investigated to provide cardiac pulse measurement. The presented approach is hassle free in nature, and hence, can be adopted even in home environment. Experimental results represent the efficacy of the proposed approach.


#

Introduction

In the past decade, the usage of user-friendly home health monitoring systems, such as health watches and health wrist bands, etc., has seen eminent increase due to the growing consciousness of people about their health. Talking about health, the recent hike in the catastrophic cardiovascular disease has been a major concern. These cardiovascular measurements highlight not only the presence of heart-related problems but also heart rate variability, which plays a key role in giving information about the physiological state of the person. In this article, the authors propose a noncontact and noninvasive means of estimation of a person's heartbeat in bpm (beats per minute) by observing color variations occurring in the skin during blood flow based on the great work of Poh and colleagues.[1] Cardiac pulse measurement helps diagnose heart diseases such as bradycardia and tachycardia,[2] and thereby assists in the diagnosis and tracking of medical conditions of individual. It can also be used by athletes who are interested in monitoring their heart rate and improve their performance. Increased heartbeat may be due to physical exercises, illness, and/or mental stress. Until recent developments in technology, the sole methods used for cardiac pulse measurement is via electrocardiogram (ECG) in which users have to strap on bulky sensors (electrodes) on the chest, wrist, etc., or manual measurement by pressing gently with the index and middle fingers at any point on the body where the artery's pulsation is transmitted to the surface. To sum up, attempts so far are nonautomated, difficult to implement in ambient home environment, typically expensive, can increase risk of infection in patients with weak immune system such as neonatal infants, and inappropriate for persons with skin allergies. Using the proposed technique, users can experience remote health monitoring on a regular basis with minimal effort, without visiting a physician, whenever and wherever they want.[3] In addition, with the advent of latest technology and growing application of digital image processing, it becomes easier to implement it.

This implementation is based on photoplethysmography (PPG), which is volumetric measurement of an organ by using a pulse oximeter that illuminates the skin and measures changes in absorption of light. Similarly, every time heart beats, the blood flow in the veins increases slightly. Hemoglobin present in blood absorbs light, and so this increase absorbs more light, thereby decreasing the amount of light being reflected by the body (skin). These subtle variations occurring in skin color may not be visible with naked eyes but eventually get embedded in the pixels of image/video recorded by any camera. It is these color variations that may help the doctors calculate the pulse (heartbeat) of the person. Because face is the most expressive and interacting part of the body, the authors used video recording of the face to determine pulses.


#

Methods

For implementation of this article, the authors recorded the video of the person sitting at a distance of nearly 30 to 50 cm from the laptop. The video was recorded at 15 frames per second in true color (24-bit RGB [red, green, and blue] with three channels) through built-in webcam of the laptop. Because the entire implementation of the project is done in Matlab, the authors also created a user-friendly graphic user interface (GUI) to make the process more interactive. The recorded video of the person is divided into frames, and on each frame, KLT algorithm[4] for face detection is applied. The region of interest (ROI) (70% of the face) from the detected face is selected. The ROI is further subdivided into three RGB channels. These channels are the normalized, and then independent component analysis (ICA) was performed on the channels to remove the surrounding noise. On one of the components obtained after ICA, fast Fourier transform (FFT) was applied. Later ideal band pass filtering was applied to confine the frequency in desired range (0.75–4 Hz) and peak detection algorithm to obtain the desired peak frequency. To convert the peak frequency into bpm, it was multiplied by 60.


#

Theory

Face Detection

Postprocessing and analysis of the video and physiological recordings were done using custom software written in Matlab (The MathWorks, Inc.). An overview of the general steps in this approach to recovering the blood volume pulse is illustrated in the following section. The CV face detection algorithm is based on work by Viola and Jones, as well as Lienhart and Maydt. To track the face over time, this example uses the Kanade-Lucas-Tomasi (KLT) algorithm.[4]


#

Kanade-Lucas-Tomasi Algorithm

Kalman-Lucas-Tomasi (KTL) face detection algorithm is used for detection of the face from the first frame of the video and marking certain feature points in the detected face region. Subsequently, these points are reliably tracked in the further video frames, and it results in faster processing of the video and also provides freedom of enough head movements to the user. Alternatively cascade object detector could have been used on every frame, but it is not computationally feasible. It may also fail to detect the face, when the patient turns or tilts the head. This limitation is overcome in case of KLT algorithm ([Fig. 1]).

Zoom Image
Fig. 1 (a) Detected face from the video frame, (b) KLT features used for face tracking during motion.

#

Region of Interest Extraction

The region of interest (ROI) is selected as a rectangle containing nearly 60% the face region at the first frame of video recording. As shown in [Fig. 1], the detected face is captured by webcam. The ROI contains three primary color RGB channels.[1]


#

Red, Green, and Blue Trace Separation

The frames obtained from the video are further separated into constituent RGB (red, green, and blue) traces. The ROI is then separated into the three RGB channels and spatially averaged overall pixels in the ROI to yield a red, blue, and green measurement point for each frame and form the raw traces, respectively.


#

Normalization

The raw RGB traces are normalized as follows:

Zoom Image

for each i = 1, 2, 3 where μ and σ are the mean and standard deviation of x t respectively. The normalization results in zero-mean and has unit variance of the raw traces.[1]


#

Independent Component Analysis

The traces after normalization are then decomposed into its three independent source signals using independent component analysis (ICA). ICA separates the source signals from linear combination of these signals. In this implementation, the authors used RADICAL (Robust, Accurate, Direct Independent Components Analysis algorithm).[5] This method is better than previous ICA algorithms, such as JADE, FastICA, etc. Thus, the RADICAL algorithm is more robust, more accurate, and easier to implement.


#

Fast Fourier Transform and Bandpass Limit

Finally, the authors applied the fast Fourier transform (FFT) on the selected source signal to obtain the power spectrum. An FFT is a discrete Fourier transform (DFT) but is much faster for calculations. The whole point of using FFT is its high speed in calculating a DFT. The 8-input butterfly diagram of FFT has 12 2-input butterflies and thus 12 × 2 = 24 multiplies. A straight DFT has N × N multiplies or 8 × 8 = 64 multiplies. Thus, there is enough computation efficiency in implementation of FFT. The pulse frequency was designated as the frequency that corresponded to the highest power of the spectrum within an operational frequency band. For these experiments, the authors set the operational range to (0.75–Hz) (corresponding to [45–240 bpm]) to provide a wide range of heart rate measurements.


#

Graphic User Interface

The graphic user interface (GUI) for the MedCam project was developed in the GUI Layout Editor of the MATLAB 2013a.[6] The following components have been used to make the GUI as user friendly as possible ([Fig. 2]).

Zoom Image
Fig. 2 The GUI for MedCam.

The GUI is interactive and interesting. It is provided with various push buttons, static text box, and graphical axes to display the results generated during the implementation of the process. Various push buttons have been used in the GUI to serve different purposes as the name suggests correspondingly for each of the push button ([Table 1]).

Table 1

List of push buttons in the GUI

S. No.

Push button

Function

1

REC

To start the video recording

2

STOP

To stop the video recording

3

Get Your Pulse Rate

To start the pulse measurement

4

Detected Face

To display the detected face

5

Detected Face Features

To display the KLT face features

6

Red Trace

To display the red trace

7

Green Trace

To display the green trace

8

Blue Trace

To display the blue trace

9

Normalized Red Trace

To display the normalized red trace

10

Normalized Green Trace

To display the normalized green trace

11

Normalized Blue Trace

To display the normalized blue trace

12

Power Spectrum

To display the power spectrum

13

Quit

To exit the GUI

Abbreviations: GUI, graphic user interface; KLT, Kanade-Lucas-Tomasi.



#
#

Results and Discussion

As shown in the [Table 2], the results obtained through MedCam are verified by one of the traditional methods for pulse measurement, that is, by using stethoscope. The readings were taken for three patients: two males and one female under various physical conditions and results were obtained to be 94 to 95% accurate ([Fig. 3]).

Zoom Image
Fig. 3 Flowchart. RGB, red-green-blue; ROI, region of interest.
Table 2

Results and cross-examination with the traditional stethoscope

S. No.

Patient's condition

MedCam result

Manual test result

1

Normal (resting)

72

75

2

After swimming

117

100

3

Normal (resting)

64

60

4

After rigorous exercise

103

100

5

Normal

63

60

6

Brisk walking/exercise

90

95

[Fig. 4] depicts the results obtained after the execution of the MATLAB code. As visible from the above, the importance of normalization process becomes quite evident, because the actual red and green traces are quite distinct from the blue trace, and after the implementation of the normalization function, the three (RGB) traces become quite coherent with each other. Thus, the process of normalization is very effective and important. The power spectrum is obtained after the application of FFT algorithm on the component, which is obtained after implementation of ICA, that is, RADICAL algorithm. The input to the radical algorithm is an array of the normalized RGB channels and the output obtained includes three independent source signal components. It is observed that the second component gives better results due to better signal-to-noise ratio than the other two signals; hence, FFT algorithm is applied on second component. The system can be further extended to send the daily health reports of the participant/patient to his/her health advisor.

Zoom Image
Fig. 4 (a) Detected ROI from face, (b) marked feature points (c) power spectrum after FFT and band limiting, (d) red trace, (e) green trace, (f) blue trace, (g) normalized red trace, (h) normalized green trace, (i) normalized blue trace.

Overview of the Process

The detailed overview of the process flow of how RGB traces are extracted for facial ROI, traces generated, and normalized and FFT-based power spectrum generated to predict the accurate cardiac pulse are shown in [Fig. 5].

Zoom Image
Fig. 5 Process overview.

#
#

Conclusion

In this implementation, the authors have demonstrated the semiautomated, advanced and noncontact means of measurement of cardiac pulse measurement, using the typical laptop/computer webcam. This is beneficial for babies or for people with sensitive skin or burn wounds. It will monitor one's health on a regular basis with minimal effort. The previous versions of this implementation were the android/iPhone applications or were intolerant to user's motion. However, this method cannot be used by multiple users at the same time and is also dependent on the lighting conditions; that is, more natural light (sunlight) must be present for more accurate results. Because this technology is the first of its kind (involves noncontact means for heart rate measurement), certain deviation in results can be expected. Further improvements can be made by implementing the code in hardware.


#
#

No conflict of interest has been declared by the author(s).


Address for correspondence

Ajay Vikram Singh, PhD
Physical Intelligence Department
Max Planck Institute for Intelligent Systems
Stuttgart 70569
Germany   


Zoom Image
Fig. 1 (a) Detected face from the video frame, (b) KLT features used for face tracking during motion.
Zoom Image
Zoom Image
Fig. 2 The GUI for MedCam.
Zoom Image
Fig. 3 Flowchart. RGB, red-green-blue; ROI, region of interest.
Zoom Image
Fig. 4 (a) Detected ROI from face, (b) marked feature points (c) power spectrum after FFT and band limiting, (d) red trace, (e) green trace, (f) blue trace, (g) normalized red trace, (h) normalized green trace, (i) normalized blue trace.
Zoom Image
Fig. 5 Process overview.