INTERNET OF THINGS BASED FACEMASK DETECTION AND BODY TEMPERATURE MEASURING OF HUMAN IN CROWDED SPACE

Abstract

With the COVID-19 pandemic, enforcing mask usage and temperature checks in crowded areas like workplaces has become essential. To minimize supervision costs and reduce human contact, this work presents a deep learning-based system for automated mask detection and temperature measurement at workplace entrances. Utilizing Python and image processing techniques, the system processes webcam images for face and object detection. The MobileNetV2 deep learning model is used to accurately identify mask-wearing status, while a non-contact MLX90614 thermal sensor integrated with Arduino measures body temperature. Results for both mask detection and temperature are displayed on a graphical user interface (GUI), with high temperature alerts sent to smartphones via IoT integration. Achieving around 98% accuracy, the system can differentiate between masks and hands but may incorrectly classify other objects covering the face. Nonetheless, it performs reliably compared to commercial alternatives, especially in scenarios where users attempt to simulate mask-wearing with their hands.

Specification

Description:FIELD OF INVENTION
This invention relates to an Internet of Things (IoT)-based system for real-time facemask detection and body temperature measurement in crowded spaces. Utilizing sensors and image recognition, it identifies individuals without masks and those with elevated temperatures, transmitting alerts to authorities for immediate response. The system enhances public health safety, particularly in environments vulnerable to airborne disease transmission.
BACKGROUND OF INVENTION
The COVID-19 pandemic underscored the importance of monitoring health measures, such as wearing facemasks and maintaining normal body temperature, especially in crowded spaces like public transportation hubs, shopping centers, and workplaces. High-density areas increase the risk of airborne disease transmission, making it crucial to ensure compliance with public health guidelines in real-time. Traditional monitoring methods often rely on manual inspection, which can be inefficient, labor-intensive, and prone to human error.
To address these limitations, advancements in the Internet of Things (IoT) offer innovative solutions for automated health monitoring. An IoT-based system for facemask detection and body temperature measurement provides a more efficient approach. By combining thermal sensors, cameras, and machine learning algorithms, such a system can automatically identify individuals not wearing masks and detect elevated body temperatures without requiring direct human intervention. This not only accelerates the process but also minimizes close contact, reducing the risk for monitoring personnel.
The IoT system can connect multiple devices through a centralized network, allowing data from various entry points to be gathered and analyzed in real-time. Alerts can be generated for immediate action when a potential health risk is detected, enhancing safety protocols and facilitating rapid responses. Furthermore, the data collected can assist in contact tracing and provide valuable insights for public health officials. This automated, scalable approach to health monitoring is essential in promoting safer environments and mitigating the spread of infectious diseases in crowded settings.
The patent application number 202121062261 discloses a system and method for unusual human activity detection in crowded scenes.
The patent application number 202147025750 discloses a system for measuring body temperature in poultry.
The patent application number 202111020581 discloses a smart temperature measuring system with minimal human intervention.
The patent application number 202111012112 discloses an advanced wearable face mask.
The patent application number 202047043438 discloses a breathing assistance face mask and method of its control.
SUMMARY
The invention is an IoT-based system designed for automated facemask detection and body temperature monitoring in crowded spaces, aiming to enhance public health safety and reduce the spread of infectious diseases. It integrates thermal imaging, facial recognition, and temperature sensors to accurately detect individuals who are not wearing masks and those with elevated body temperatures, both key indicators of non-compliance with health protocols. The system uses image processing and machine learning algorithms to identify facemasks and measure body temperatures quickly and efficiently, even in high-traffic environments.
Upon detecting individuals without facemasks or with abnormal temperatures, the system generates real-time alerts that can be sent to security or healthcare personnel for immediate response. Data from these detections are processed and stored on a secure, centralized IoT network, allowing for easy monitoring and tracking over time. This data can also contribute to broader health management systems for contact tracing and epidemiological analysis.
The IoT-based framework is designed for scalability, enabling multiple sensors and detection points to operate within large areas such as airports, shopping malls, or office buildings. By automating mask detection and temperature screening, this system minimizes the need for manual inspections, thereby reducing potential exposure risks for monitoring personnel. Additionally, it promotes adherence to health protocols efficiently, enhancing public safety in densely populated settings. This invention provides a proactive, technology-driven approach to disease prevention and can be adapted to various environments with minimal infrastructure adjustments.
DETAILED DESCRIPTION OF INVENTION
COVID-19 is a contagious disease that affects the respiratory system, caused by the SARS-CoV-2 virus. Since the pandemic began in December 2019, it has infected more than 500 million people worldwide and resulted in approximately 6 million deaths. According to the World Health Organization (WHO), wearing masks in public spaces is among the most effective measures to limit the spread of COVID-19. Governments globally, particularly in China, have mandated mask-wearing indoors, including workplaces. Additionally, early detection of the disease is crucial for controlling the spread of the pandemic. Elevated body temperature is a common symptom in COVID-19 cases, making temperature checks important for identifying potential infections. To ensure compliance with health guidelines, monitoring is necessary. However, manually supervising mask use and measuring body temperature at company entrances can be labor-intensive and costly. Maintaining a safe distance between monitoring staff and employees is also challenging, increasing the risk of virus transmission. Therefore, a graphical user interface (GUI)-based system was developed to automatically perform real-time mask detection and contactless temperature measurement for employees in workplaces. To integrate Internet of Things (IoT) capabilities, a notification feature was added to the system.

Figure 1: System Architecture for Mask and Temperature Monitoring
The framework of the proposed face mask detection and temperature measurement system is illustrated in Figure 1. For mask detection, a real-time video stream of human faces is captured through a webcam and fed into the system. Various image processing techniques are applied to each video frame. After preprocessing, the frame is sent to a face detector that identifies the Region of Interest (RoI), which is the human face in this context. The mask detection is conducted using a model based on the MobileNetV2 deep learning algorithm. This model analyzes each RoI to classify whether a mask is present or not, and a label displaying the probability of the classification result is shown on the GUI. For non-contact temperature measurement, an MLX90614 thermal sensor and Arduino UNO R3 board are used. To prevent interference with the continuous image data capture, multithreading is implemented, allowing temperature data to be processed on a separate thread via a serial port. The GUI displays the temperature measurement, and if it exceeds a defined threshold (e.g., 37°C), the system informs staff that entry is not permitted. Additionally, using an external application called IFTTT, a high-temperature notification alert is sent to an administrator's smartphone.

LITERATURE REVIEW
A. Object Detection
Face mask detection involves identifying whether a person is wearing a mask, starting with the localization of the face. This task falls under the broader category of object detection within computer vision, focusing on recognizing the type of object present. With advancements in deep learning, models have shown high robustness and feature extraction capabilities. Object detection models are generally categorized into one-stage and two-stage detectors.
One-stage detectors utilize a single neural network for object detection, requiring predefined anchor box ratios for object width and height. For instance, the YOLO algorithm divides an image into cells and assigns anchor boxes to objects within each cell. However, this approach tends to perform suboptimally for small objects. Multi-scale detection methods like Single Shot Multibox Detector (SSD) have been introduced to address this, as they detect objects across various feature maps, allowing for object detection at multiple scales.
In contrast, two-stage detectors first generate region proposals and then refine these proposals in the second stage. A Region-based Convolutional Neural Network (R-CNN) was introduced by R. Girshick and colleagues in 2014, followed by an improved "Fast R-CNN" in 2015, which resolved limitations of the original model by implementing a RoI pooling layer that processes all candidate regions simultaneously. In this project, a two-stage detector was employed for mask detection, performing classification on the Regions of Interest (RoIs).
B. Convolutional Neural Networks
Convolutional Neural Networks (CNNs) are foundational in pattern recognition, particularly image classification. However, adding excessive layers to the network can lead to accuracy degradation after reaching saturation. To mitigate this, K. He and colleagues introduced Residual Networks (ResNet), which use residual blocks to skip certain layers, allowing inputs to directly connect to the outputs of stacked layers. This residual structure simplifies the learning process and improves training efficiency for deep networks.
For object detection tasks on mobile or embedded devices, where computational resources are limited, the Mobile Network (MobileNet) was proposed. MobileNet employs depth-wise convolution instead of traditional convolution for feature extraction, significantly reducing computational cost. It also incorporates residual learning principles, similar to ResNet.
C. Related Work on Face Mask Detection
In traditional object detection, Deformable Part Model (DPM) has been applied to face mask detection, using the structural orientation of faces. A DPM-based mask detector, using around 30,000 faces, achieved a high accuracy of approximately 97%, although its computational demands are substantial. Since the emergence of COVID-19, extensive research has been conducted on face mask detection using pre-trained CNN models. One solution using transfer learning with InceptionNetV3 achieved an impressive accuracy of 99%.
Considering the need for mask detection on embedded devices, a model named SSDMNV2 was developed, combining SSD for face detection with MobileNetV2 for classification, and achieved a competitive accuracy of 93%. Another approach, called RetinaFaceMask, employed both MobileNet and ResNet as the backbone for mask detection. Drawing from this research, this project utilized the pre-trained MobileNetV2 CNN model as the backbone of the proposed mask detection system.






Methodology and Design


Figure 2: Mask Detection Process Flow
Figure 2 illustrates the step-by-step approach used to implement the real-time mask detection function. The procedure consists of four main stages: preparing the dataset for training and testing, preprocessing the input data, developing and training the proposed mask detector, and performing real-time mask detection on each frame of the webcam video stream. The detailed methodology is described below.
A. Dataset Preparation and Preprocessing
The dataset is composed of two types of images: those with masks and those without. The majority of the images come from an online open-source platform [18], while a smaller portion is generated through the Generative Adversarial Network (GAN) by searching "This person does not exist" on Google. To enable the face mask detector to recognize hand-covered faces as "without a mask," several images with hands covering the mouth were added to the "without a mask" category. Later in the project, a limitation was identified: the detector could misclassify other objects, such as books or mobile phones, as masks. To address this, an additional 60 images of similar items were incorporated to improve classification. In total, the dataset comprises 710 images in the "with mask" category and 770 in the "without a mask" category.
Image preprocessing is performed using TensorFlow/Keras, where each image is converted into arrays and processed using MobileNetV2's specific preprocessing technique. Image augmentation is also applied to enhance data variety without collecting new images, providing sufficient data for training, and ultimately improving model performance and accuracy.
B. Mask Detector Development and Training
After preprocessing, the dataset is split so that 80% is used for training and 20% for testing. The model construction begins by loading MobileNetV2 as the base model, omitting the head fully-connected layers to allow for later modifications. A new head with fully-connected layers is added and trained as the mask detector model, while the base layers remain untrainable. This setup follows the fine-tuning approach, where only the new head layers are updated during training.
The model is optimized using the Adam optimizer with a learning rate of 0.0001, commonly used in machine learning. Binary Cross-Entropy is selected as the loss function, ideal for binary classification tasks like mask detection. The model's accuracy and loss are tracked and presented in the results section. Once trained, the model is saved and ready for real-time detection tasks.
C. Real-time Implementation
For real-time detection, the system processes each frame of the video stream, completing processing for each frame before the next one begins. To maintain this pace, QTimer in PyQt is used to trigger processing for each new frame every 0.002 seconds. Each time this occurs, the system calls the Detect and Predict Mask function, consisting of two stages: the first uses FaceNet to extract the region of interest (ROI), specifically the human face; the second applies the trained mask detector model to identify the presence of a mask on each detected face.
Face detection is performed using OpenCV's SSD face detector. Once a face is located, the Detect and Predict Mask function returns the face coordinates and mask prediction. Depending on the prediction, a label ("mask" or "no mask") and a bounding box are overlaid on the output frame in the graphical interface.
D. Fine-tuning MobileNetV2
Several pre-trained CNN models, such as ResNet and MobileNet, offer strong performance due to training on extensive datasets like ImageNet. Using these pre-trained models allows for effective feature extraction across both shallow and deep network layers. Fine-tuning these models can reduce the need for large datasets and substantial computational power, as it avoids building a new model from scratch. Proper fine-tuning also helps tailor the model to specific tasks, optimizing parameters, enhancing generalization, improving accuracy, and reducing overfitting risks.
In this process, the base layers of MobileNetV2 are frozen to keep their weights unchanged during backpropagation, allowing only the new head layers to be trained. This approach directs the network to focus on learning task-specific features in the deeper layers, establishing a robust baseline model while conserving significant time and resources

Figure 3: Face Mask Classifier CNN Architecture
In this project, when integrating the pre-trained MobileNetV2 model for the mask detection task, the fully connected head layers were excluded initially to allow for later reconstruction. The newly designed head of the model incorporates max-pooling layers to reduce computational cost while retaining the most significant features. These high-level features are then flattened and passed through a fully connected layer for classification. To introduce non-linearity to the model, the "ReLU" activation function is applied, and 30% of neurons are randomly dropped to mitigate the risk of overfitting. The softmax function is used at the output layer to generate probability distributions for each class label, selecting the highest probability as the predicted output. Since this is a binary classification problem, the model is designed to output two classes: "mask" or "no mask." The complete CNN architecture of the proposed mask detector model is illustrated in Figure 3.
E. Multithreading
During the integration of the temperature measurement feature into the mask detection system, it was observed that the concurrent reading of temperature data and webcam images interfered with each other, causing the program to terminate unexpectedly. To resolve this, multithreading was implemented: a separate thread was allocated to handle the temperature readings from the MLX90614 thermal sensor connected to the Arduino UNO R3, while another thread continues to capture images from the webcam. This ensures that both processes run smoothly in parallel without interfering with each other.
F. IFTTT for IoT Notification
IFTTT (If This Then That) is an external service that facilitates automation by connecting different applications and services through a simple "if this happens, then do that" logic. In this project, IFTTT is used to trigger notifications when a high temperature is detected. Specifically, an HTTP POST request is sent to trigger an alert on an administrative staff member's smartphone whenever the temperature exceeds a predefined threshold.
G. Brightness Checking
While testing the system's performance, it was discovered that ambient lighting conditions could affect mask detection accuracy. Specifically, when the lighting is poor and a hand is used to cover the mouth to simulate wearing a mask, the system may falsely detect a mask. In contrast, under sufficient lighting, the system provides accurate detection results. To address this, a brightness-checking mechanism was implemented. Grayscale values in dark images typically fall within the range of 0-30 on the grayscale map (ranging from 0 to 255), while brighter images have higher values. This property allows brightness to be assessed by analyzing the distribution of grayscale values. Using OpenCV, the system crops each frame to focus solely on the face, converts the face image to grayscale, and evaluates the proportion of dark pixels to determine the brightness level. This ensures that poor lighting conditions do not lead to incorrect mask detection.

Figure 4: Training Loss and Accuracy of the Proposed System

Figure 5: Training Loss and Accuracy Using ResNet
SYSTEM EVALUATION AND RESULTS
A. Training Loss and Accuracy of the Mask Detector
The performance of the proposed mask detector model was evaluated based on the loss and accuracy metrics throughout the training process. Figure 4 illustrates the loss and accuracy at each epoch. The evaluation includes four key variables: training loss, training accuracy, validation loss, and validation accuracy. Since MobileNetV2 is built on the ResNet architecture using residual blocks, it was considered beneficial to incorporate ResNet as the backbone of the mask detection model. As a result, ResNet was tested as a baseline model. In this trial, the approach of fine-tuning the head layers and keeping hyperparameters consistent was applied. Figure 5 displays the loss and accuracy at each epoch for the ResNet-based model. Additionally, the comparison of the training speeds between MobileNetV2 and ResNet across epochs is presented in Figure 6.
B. System Performance Testing
The performance of the proposed system was tested under various conditions, as shown in Figure 7. The results indicate that the system can accurately detect whether a hand is being used to simulate a mask. Moreover, the system allows entry when the body temperature is normal. For convenience, the temperature is measured from the wrist, where the thermal sensor is positioned at least 1 cm above the wrist. While the wrist may show a lower temperature (below 36°C) compared to other body parts, it is still suitable for temperature measurements. To simulate an abnormal temperature scenario (e.g., above 37°C), the system detects this when the user touches a hot object, like a cup of hot water, and prevents entry. When an abnormal temperature is detected, the system triggers an immediate notification. An example of such a notification is displayed on the mobile phone screen: "Alarm! Someone's temperature is very high! 41.90 degrees Celsius!"
Additionally, the system includes a warning for insufficient lighting conditions. If no face is detected after the mask detection and temperature measurement processes are completed, the system waits for the next user and resets as necessary, as shown in the final sub-figure.

Figure 6: Comparison of Training Speed: MobileNetV2 vs. ResNet
Results and Discussion
This section further explores the results of the proposed system, highlighting its robust functionality and potential for practical applications. It also addresses some limitations and outlines possible directions for future enhancements.
A. Result Reliability
As shown in Figure 4, the mask detector, which uses MobileNetV2 as the base model, achieves an accuracy of approximately 98%. The model demonstrates effective convergence and good fitting, as indicated by the decreasing training loss, which suggests that the predictions become more accurate as the training progresses. Additionally, the validation accuracy surpasses the training accuracy, likely due to the smaller validation dataset (20% of the total data) compared to the training dataset (80%). Moreover, validation data typically doesn't undergo data augmentation, which is applied to the training set to increase diversity and introduce more learning challenges. Another factor contributing to this observation is the dropout technique, which is only applied during training, potentially resulting in a higher loss in the training dataset.
In contrast, the mask detector based on ResNet, as depicted in Figure 5, experiences slower convergence, despite reaching a promising 94% accuracy. As shown in Figure 6, the training speed for the ResNet model is significantly slower compared to MobileNetV2. Specifically, MobileNetV2 requires about 35 seconds per epoch, whereas ResNet takes approximately 125 seconds per epoch. After several hyperparameter adjustments, the proposed MobileNetV2-based mask detection model was fine-tuned to deliver optimal performance.

Figure 7: Performance Evaluation in Various Scenarios


B. Limitations
During the project, it was discovered that the system would misclassify objects other than masks that cover the mouth as a mask. To address this, additional confusing data were added to the dataset, including images of items that might resemble masks, such as books and mobile phones. However, even after training with about 60 such images, the system still failed to identify some non-mask objects as not being a mask. This limitation is not unique to the proposed system; discussions with technical staff from various commercial mask detection systems revealed that this issue is common in the industry.
There are two potential reasons for this limitation. First, the features of masks may not be sufficiently distinct, allowing other objects with similar characteristics to be mistaken for masks. Second, the system may rely on detecting skin versus non-skin areas, which could lead to non-skin objects being interpreted as masks. To improve detection accuracy, it is suggested to incorporate at least 200 additional confusing data samples into the training process.
C. Future Work
The system's capability to detect multiple faces simultaneously positions it for future expansion. In the future, the system could benefit from higher-quality images by using cameras with better resolution. Additionally, while the current system is a software-based solution running on a laptop, future development could include integrating it into dedicated hardware. For example, the mask detection algorithm could be embedded in a standalone device, potentially combined with temperature measurement features, making it more convenient for real-world use.
Moreover, there are significant opportunities to enhance the system's functionality within the Internet of Things (IoT) ecosystem. The data from the device could be transmitted via Wi-Fi or Bluetooth to a central server for storage and analysis. To increase the system's competitiveness and applicability across various scenarios, integrating face recognition with mask detection could further broaden the system's potential.
In conclusion, this work presents a system capable of automatic real-time mask detection and body temperature measurement. The mask detection is powered by a two-stage neural network algorithm, developed by fine-tuning the MobileNetV2, a pre-trained convolutional neural network model. Temperature measurement is carried out using a contactless thermal sensor integrated with an Arduino. The results of both functions are displayed via a user-friendly graphical interface (GUI).
Additionally, the project incorporates an IoT-based feature that sends a high-temperature alert to the mobile phone of the administration staff when an elevated temperature is detected. This feature is implemented using an external application called IFTTT. Recognizing that facial brightness may affect mask detection accuracy, the system is also equipped with the ability to assess image brightness and issue a warning if the lighting is insufficient.
The test results indicate that the mask classification model achieves an accuracy of approximately 98%. However, there are some limitations, such as the system occasionally misidentifying objects other than masks. To address this, the inclusion of more confusing data is recommended to improve object detection accuracy. Future improvements should also include the use of higher-resolution cameras and more precise thermal sensors to enhance system performance. To increase its market potential, transitioning the system to dedicated hardware modules would be essential for commercial viability.

DETAILED DESCRIPTION OF DIAGRAM
Figure 1: System Architecture for Mask and Temperature Monitoring
Figure 2: Mask Detection Process Flow
Figure 3: Face Mask Classifier CNN Architecture
Figure 4: Training Loss and Accuracy of the Proposed System
Figure 5: Training Loss and Accuracy Using ResNet
Figure 6: Comparison of Training Speed: MobileNetV2 vs. ResNet
Figure 7: Performance Evaluation in Various Scenarios , Claims:1. Internet of things based facemask detection and body temperature measuring of human in crowded space claims that the system provides real-time monitoring of individuals in crowded spaces for facemask compliance and body temperature measurements.
2. The system uses non-contact infrared sensors to measure body temperature without physical contact, ensuring safety and comfort.
3. By detecting individuals without masks or those with elevated body temperatures, the system helps manage the crowd effectively and ensure public health safety.
4. It automatically sends alerts to security personnel or administrators if any individual fails to wear a facemask or if their temperature exceeds a predefined threshold.
5. Administrators can monitor the entire system remotely through a central dashboard, enhancing surveillance efficiency and response time.
6. The system leverages IoT technology to transmit real-time data to cloud platforms, where it can be analyzed and stored for further reference.
7. The system ensures that personal data is not compromised, focusing solely on health-related information like body temperature and facemask compliance.
8. The IoT-based system is scalable and can be deployed in various public spaces such as airports, shopping malls, and offices.
9. The system helps authorities respond quickly to potential health risks, such as fever or mask non-compliance, preventing outbreaks in crowded environments.
10. The system is designed to be energy-efficient, using low-power IoT devices that require minimal maintenance, making it suitable for long-term deployment.

Documents