KSII Transactions on Internet and Information Systems (TIIS)

Korean Society for Internet Information (한국인터넷정보학회)
권호 표지
DOI QR코드

DOI QR Code

Boosting the Face Recognition Performance of Ensemble Based LDA for Pose, Non-uniform Illuminations, and Low-Resolution Images

  • Received : 2018.10.12
  • Accepted : 2018.12.27
  • Published : 2019.06.30
Download PDF
⟨ Previous Next ⟩

Abstract

Face recognition systems have several potential applications, such as security and biometric access control. Ongoing research is focused to develop a robust face recognition algorithm that can mimic the human vision system. Face pose, non-uniform illuminations, and low-resolution are main factors that influence the performance of face recognition algorithms. This paper proposes a novel method to handle the aforementioned aspects. Proposed face recognition algorithm initially uses 68 points to locate a face in the input image and later partially uses the PCA to extract mean image. Meanwhile, the AdaBoost and the LDA are used to extract face features. In final stage, classic nearest centre classifier is used for face classification. Proposed method outperforms recent state-of-the-art face recognition algorithms by producing high recognition rate and yields much lower error rate for a very challenging situation, such as when only frontal ($0^{\circ}$) face sample is available in gallery and seven poses ($0^{\circ}$, ${\pm}30^{\circ}$, ${\pm}35^{\circ}$, and ${\pm}45^{\circ}$) as a probe on the LFW and the CMU Multi-PIE databases.

Keywords

1. Introduction

 Face Recognition (FR) is a natural and nonintrusive way of biometric measure because it requires no special arrangements and disturbances to the person being identified. The FR is the process of comparing face features in two different images to determine whether they are of same person or not [1]. The FR is generally performed in two distinct ways, which are Verification (1:1 match) [2] and Identification (1: N match) [3]. Many recent state-of-the-art FR algorithms successfully recognize faces when they are investigated under constrained conditions [4]-[5]. However, their performance significantly degrades when face images are obtained under diverse conditions, such as variable illuminations, emotions, facial expressions, poses, occlusion, or low-resolutions [6]-[7]. To test the accuracy of FR algorithms, many databases have been created against each of the aforementioned conditions [8]. Each database is designed to test a specific task, such as expression, occlusion, pose, or illumination. For instance, Legal Faces in Wild (LFW) and Multi-Pose Illumination Expression (PIE) are two popular databases widely used in the FR research. This paper presents a state-of-the-art face recognition algorithm that primarily focuses to improve the recognition accuracy on crucial factors, such as variation in face pose, non-uniform illuminations, and low-resolution. In next Section, we briefly describe recent advances in face recognition domain.

2. Related Work

 During the past decade, many algorithms, for instance, LDA [9], Local Binary Pattern [10], Convolution Neural Network (CNN) [11], and Eigen faces [12] have been developed to improve the robustness and accuracy of FR algorithms. Simonyan et al. [13], use Fisher vectors on densely sampled SIFT features to identify faces. The overall accuracy of the algorithm is 87.7% for the standard Labelled Faces in the Wild (LFW) dataset [14]. A Wavelets Transform (WT) based facial recognition algorithm has been proposed in [15]. The WT based FR algorithm provides insensitivity towards illumination changes. The algorithm was tested on the FERET facial database and reports encouraging results for pose variations. The overall False Rejection Ratio (FRR) is 12%. However, the authors did not provide the information about the occlusion and the maximum angle for pose variation investigation in their paper. In [16], authors describe a Kernelized Locality-sensitive Group Sparsity Representation (KLS-GSRC) method for face recognition. Extensive experiments were conducted on advanced databases, such as the LFW, the ORL, Extended Yale B, and the AR dataset. Published results show much higher recognition accuracy than few of sparse representation-based techniques compared therein. Taigman et al. [17], propose Deep CNN (DCNN) based FR algorithm, which outclasses few well-known methods in terms of accuracy. Authors claim to obtain much higher accuracy even in large pose variation and illumination conditions. Moreover, the algorithm was trained and tested on the LFW, the SFC, and YouTube Face dataset (YTF). The average accuracy achieved for these datasets is 97.35%. In [18], authors present an efficient approach that uses Radial Basis Function (RBF) based approach for face recognition to reduce the computational burden and avoid overfitting. In the proposed approach, initially, low level face features are extracted using Principal Component Analysis (PCA) method. Later, the resulting features are processed by the Fisher’s Linear Discriminant (FLD) technique to acquire lower-dimensional discriminant patterns. The system achieves excellent performance on the ORL dataset. However, the computation time of this algorithm is quite high since it integrates multiple classifiers during recognition phase.

 In [19], authors use a novel methodology and introduce the 3D face models as an input to Deep Neural Network (DNN). The 3D approach achieves robust recognition for pose variation and occlusion. For the LFW dataset, the proposed algorithm achieves an accuracy of 99.63%, while on the YTF database it achieves 95.12%. However, the computation time of this algorithm is also high since it estimates face poses using 3D face model and also applies neural network for recognition. In [20], a supervision signal center loss-based face recognition algorithm has been proposed to train deep model. In the proposed approach, a robust CNN is supervised by joint combination of center loss and SoftMax loss. This algorithm was tested on the LFW and the YTF dataset and reports an average accuracy of 99.28% and 94.9%, respectively. Algorithm requires little training dataset and was tested on 0.7 Million images. In [21], Multiple Classifier System (MCS) has been developed by an intelligent combination of the PCA, the Linear Discriminant Analysis (LDA), and the Independent Component Analysis (ICA). The proposed algorithm was tested on the ORL, the Yale, the AR, and authors’ own designed dataset. Published results show over 90% average accuracy. Authors in [22] presented a comparative analysis of the PCA, the Local Binary Patterns (LBP) and the Adaptive Boosting (AdaBoost) with LDA as a weak learner on the PIE database. Study was performed using images with illumination, variation in pose and size of images. Authors claim that PCA is robust in classifying pose variations while AdaBoost is most accurate in identifying low-resolution images having accuracy 100% up to 10×10 and 5×5 pixels images. The LBP does not classify low resolution face images of size 20×20 pixels and below. In [31], researchers propose a Discriminative Deep Metric Learning (DDML) method to train a DNN to learn a set of hierarchical nonlinear transformations. Moreover, they project face pairs into the latent feature space to obtain the reduced distance of each positive pair and maximum distance of negative pairs. In [32], authors develop a Sparse-Deep Simultaneous Recurrent Network (S-DSRN) to achieve a robust recognition. The feature sparsity is obtained by adopting dropout learning in the DSRN as opposed to usual handcrafting of additional penalty terms for the sparse representation of data. In [33], published work reports an interesting face sketch recognition using a 3D morphable model to synthesise a set of images and allowing the network to prevent over-fitting and learn better features. In final stage, fusion of the proposed method is done with a state-of-the-art algorithm boost performance with minimum error rate. In [34], proposed face recognition algorithm is based on deep learned features and a novel Set-to-Set (S2S) distance measure to find the similarity between two sets. The proposed S2S distance adopts the kNN-average pooling for the similarity scores computed on all the media in two sets, which makes the identification far less susceptible to the outliers than traditional feature-average pooling. Recently, researchers in [28], [29], [30], [35], and [36] have proposed efficient feature extraction schemes that can be further investigated to develop a face recognition algorithm.

 Aforementioned are nice efforts in the domain of face recognition. However, a perfect face recognizer that can reliably handle the important issues, for instance face pose variation, non-uniform illuminations, low-resolutions, or occlusion is yet to be developed. In this paper, we present a new face recognition method, which partially uses the Principal Component Analysis (PCA) and is mainly inspired by the application of the Adaptive Boosting (AdaBoost) integrated with the Linear Discriminant Analysis (LDA), which we refer as Principal component analysis, Adaptive Boosting, and LDA (PAL) based face recognition algorithm in this manuscript. The proposed PAL face recognition algorithm initially uses 68 points to locate a face in the input image followed by the partial application of the PCA. Meanwhile, the AdaBoost algorithm integrated with the LDA is used to extract face features and finally classic nearest centre classifier is used for classification. The proposed PAL face recognition algorithm uses a single training image per person to reduce memory. The proposed PAL method is tested on seven different face poses from frontal (0º) to ±45º view with different lighting conditions and face image resolution. Our main contributions in this work are highlighted below.

  • The proposed PAL face recognition algorithm is composed of partial PCA, the AdaBoost, and the LDA style learners, which makes it extremely reliable for a challenging situation when only one training image per subject/class is available. The aforesaid algorithms were chosen to develop the proposed algorithm, because our earlier study [8] revealed that many recent state-of-the-art face recognition algorithms have been developed from these algorithms.
  • The proposed PAL algorithm efficiently handles following three aspects which are:

 (i) pose variation: the proposed PAL face recognition algorithm successfully recognizes seven different pose variations, which are: (0° , ±30° , ±35° , ±45° ).

 (ii) image resolution: the proposed PAL face recognition algorithm is suitable on image resolutions that start from 640×480, 231×251, 160×160, 80×80, 40×40, and down to 20×20 pixels.

 (iii) non-uniform illuminations: the proposed PAL face recognition yields much higher accuracy in diverse lighting conditions. The proposed PAL method is validated on the LFW and CMU-Multi-PIE databases.

The rest of this paper is organized as follows: Section 3 describes the proposed PAL face recognition algorithm. Detailed simulation results are given in Section 4. Finally, Section 5 concludes and hints towards possible future work. For each Section, Table 1 shows the nomenclature used in this paper.

Table 1. Nomenclature

3. Proposed PAL Face Recognition Algorithm

 In this Section, we describe our developed face recognition algorithm, which we refer as PAL in this paper. Proposed PAL face recognition algorithm comprises two main steps, (i) face localization and (ii) face recognition. Below, we briefly explain each step.

3.1. Face Localization:

 While developing the PAL face recognition algorithm, one of our goals is to extract and use the key points and features that lie on a human face. For facial images, these are important points around the eyes, the bridge of the nose, and the track of the lips on the mouth. Therefore, we extract two types of features from the face. Face localization step involves features and statistical parameters calculation as described below.

3.1.1. Low-Level Features:

 In our work, low-level features are edges and corners in the face image. Calculating the lowlevel features helps us to find high-level features in the subsequent stages. The low-level features are extracted by normalizing the face image. A face region can be considered by {x, y, w, h}, where (x, y) are the co-ordinates of the center of the face region and (w, h) depict the width and height of the face region, respectively. Later, each visible point is moved with respect to the region center (x, y), and normalized by (w, h) as given by Eq. (1)

\(\begin{equation}\left(a_{i}, b_{i}\right)=\left[\frac{x_{i}-x}{w}, \frac{y_{i}-y}{h}\right]\end{equation}\)       (1)

Where (x, y) represents the given ground truth fiducial coordinates and (𝑎𝑎𝑖𝑖, 𝑏𝑏𝑖𝑖) are treated as labels for the edges or corners on human face [23]. Once the low-level features are obtained, in next step, we find the high-level features as described in the next section.

3.1.2. High-Level Features:

 In our work, high-level features are the main features that lie on the human face, such as eyebrows, eye, length of nose, lips, and cheeks. After these features are calculated, the first step in any face recognition system is to locate a face. Recently, many algorithms on face detection have appeared in the literature [27]. Each of the algorithms has its own strengths and limitations. For current study, we have selected the method proposed in [23], in which authors’ discuss a Face Landmark Estimation (FLE) method to detect the human face. The basic idea of the FLE method is that on every face, specific points exist around the outer edge of both eyes, the top of the chin, the eyebrows, and the inner edge. In [23], different points are shown on face to locate a human face. One of our aims is to identify a pose variation. Therefore, we locate 68 specific points using the FLE method as elaborated in Fig. 1(a) and Table 2. It is important to state here that the FLE method is quite robust to handle variation in pose in real-time.

3.1.3. Mean and Standard Deviation Calculation:

 The mean and standard deviation of the face images of each subject are calculated and updated according to relation given in Eq. (2), which reduces the errors due to lighting conditions. Meanwhile, by updating the mean and standard deviation, all the intensities values of images are shifted onto a specific intensities range as shown in Fig. 1(b).

\(\begin{equation}I_{n}=\frac{\left(I_{{cropped}}-\overline{X}\right) \times \sigma_{ {def}}}{\sigma_{i}+\overline{X}_{{def}}}\end{equation}\)       (2)

 Where \(\overline{X}\) and \(\sigma_{i}\) represents the mean and standard deviation of each image, while \(\bar{X}_{d e f}\) and \(\sigma_{\text {def}}\) is predefined mean and standard deviation, respectively. In the FLE, only one training image (mean image) per subject/class is considered. For training, mean images are obtained by taking the mean of several images of each subject as shown in Fig 1(c) and under the 2nd bullet of Table 2. This particular step is partially inspired by the application of the PCA to obtain the mean image \(\left(I_{m}^{S}\right)\), which is calculated using Eq. (3):

\(\begin{equation}I_{m}^{S}=\frac{\sum_{j=1}^{J} I_{n j}^{S}}{J}\end{equation}\)       (3)

Where \(I_{n j}^{S}\) 𝑠𝑠 is the jth training image (normalized) of subject (s) and J represents the total number of training images of (s) subjects. Both probe and gallery images are first estimated using the FLE method and then normalized using Eq. (1). These normalized images are fed to the Adaptive Boosting (AdaBoost) algorithm that is combined with the LDA for recognition as described in next Section.

Fig. 1. Face Localization Procedure

Table 2. The Proposed PAL Face Recognition Algorithm

3.2. Face Recognition:

 To perform the task of face recognition on the detected face is now processed with following three algorithms.

 (i). The AdaBoost [25]: it works as a weak learner while performing the face recognition task. The AdaBoost embedded in the PAL is shown in bullets (3) to (8) of Table 2.

 (ii). The LDA [24]: it works as feature selection as shown in the for loop in bullet (4) of Table 2.

 (iii). The CNCC [22]: Classic Nearest Centre Classifier (CNCC) is used for final face classification as shown in bullet (7) of Table 2. Complete Pseudocode of the proposed PAL is shown in Table 2. The CNCC is based on normalized Euclidian distance. The AdaBoost when integrated with the LDA takes advantage of boosting technique and leads to ensembled based learning [25]. Ensembles of several LDA solutions are then used to obtain the final PAL classifier as shown in 5th bullet of Table 2. For the task of learning, AdaBoost with the LDA based the proposed PAL face recognition algorithm is formulated as: A training set, Z = {Z1, Z2, Z3, .....Zc} that has classes (C) with each class Zi= {(zij, yij)}. The zij is the jth sample of ith class and yij represent their labels. Let Z be the total sample space: zijϵ Z, and Y= {1, 2, ……., C} be the label set i.e: yij= iϵY. Now taking an unseen face samples (z), such that the objective of the learning is to estimate a function or classifier h(z): Z→Y. The AdaBoost repeatedly applies a given weak learner (ht) to the training set with weighted version in several rounds (T) and finally linearly combines all weak classifiers (ht) constructed in each round into a single and strong classifier ℎ𝑓(𝑧). The final strong PAL classifier is:

\(\begin{equation}h_{f}(z)=\stackrel{argmax}{y \in Y} \sum_{t=1}^{z}\left(\left(\log \frac{1}{\beta_{t}}\right)\left(h_{t}(z, y)\right)\right)\end{equation}\)       (3)

 

Fig. 2. Databases description (a) the LFW database and (b) the CMU Multi-PIE database.

 The ℎ𝑓(𝑧𝑧) obtained in bullet (5) of Table 2 is an ensemble of numerous LDA results and is more precise. A score value of probe image with all classes is obtained and the maximum score achieved with the class is considered as recognized face image with that class.

4. Simulation Results:

 This section describes detailed simulation results. Experiments have been conducted using Dell Precision Tower 7810, Dual Intel Xeon Processor E5-2699 V3 machine with on board 192 GB of RAM. Aforesaid is a high performance workstation and reliably handles execution of complex object detection and recognition algorithms in a real-time. Simulations are executed using OpenCV. The proposed PAL face recognition algorithm mainly focuses to evaluate accuracy on pose variation, light variation, and low-resolution face images on following two widely used face databases.

 The LFW database [14]: this database contains 13,233 images of 5,749 subjects of various resolutions, such as 150×150 pixels up to 580×440 pixels. Images in this database exhibit rich intra-personal variations of pose, illumination, and expression. It is a popular database to analyse recognition ability of recently developed algorithms in unconstrained environments. Face images in the LFW are organized into two Views. View 1 is for model selection and parameter tuning while View 2 is for performance reporting. We follow the official protocol of LFW and report the mean recognition accuracy by the 10-fold cross-validation scheme on the View 2 data. Few of the sample images of the LFW database are shown in Fig. 2(a). 

 The CMU Multi-PIE database [26]: this database has more than 750000 facial images of 337 subjects collected in five months with several poses and viewpoint displaying a range of expressions. This database contains high face image resolutions of 580×440 pixels. However, for current experiments, sizes of the face images have been cropped to 231×251 pixels. Few of the sample images of the CMU Multi-PIE database are shown in Fig. 2(b).

 Proposed PAL face recognition algorithm was tested and compared with four recent state-of-the-art algorithms, which are, (i) Local Binary Pattern (LBP) [10], (ii) AdaBoostLDA [31], (iii) EigenFaces [12], and (iv) Multi Modal Deep Face Recognition (MM-DFR) algorithms [39]. Experiments have been performed in three phases. In phase-1, effects of variations in face pose are investigated. In phase-II, effects of non-uniform illuminations are analyzed, while in phase-III, the effects of image resolution are explored to test the accuracy of the PAL face recognition algorithm. It is important to state here that the proposed PAL face recognition algorithm was evaluated for a very challenging task when only frontal gallery image is available for training and four different poses as shown in Fig. 2(b) as a probe/testing. In subsequent sections, we present the detailed simulation results for the aforesaid scenarios.

4.1. Pose Analysis

 Pose of a person can significantly change the accuracy of any face recognition algorithm [8]. In our simulations, we test and compare the PAL face recognition algorithm on seven different poses, which are: ±450 , ±350 , ±300 , and 00 (frontal). The ±450 has been chosen as a maximum face pose, because this is one of the most investigated poses in practice, such as searching images of licensed drivers, missing peoples, immigrants, and person verification at entry ports [8]. Fig. 3(a)-(b) lists the recognition accuracy on the LFW and CMU Multi-PIE databases, respectively on 2600 subjects out of which, 1300 subjects are taken from each database. Out of the seven faces poses as shown in Fig. 3(a)-(c), 100 frontal (00) facial images while, 200 images of each other pose are analysed in the experiments. Fig. 3(c) lists the overall recognition accuracy of 4000 subjects. From Fig. 3, few important observations are listed below.

  • As shown in Fig. 3(a)-(b), the MM-DFR, the PAL and Eigen faces based face recognition algorithm yield 100% recognition accuracy for frontal (00) facial images on the LFW and Multi-PIE databases. While, the LBP yields the least accuracy of 30% on LFW database for frontal facial images. The accuracy of the AdaBoost for the aforesaid pose is 90% on the Multi-PIE database.
  • For ±450 of pose variations on the LFW and Multi-PIE databases, the MM-DFR algorithms slightly outperforms the proposed PAL face recognition algorithm and yields the highest recognition accuracy of 71.19% and 74.27%, respectively on the LFW and Multi-PIE databases. For the aforementioned pose, the proposed PAL face recognition algorithm yields 70% recognition accuracy, while the LBP is unable to recognize the ±450 poses.
  • For +300 of pose variations on the LFW base, we observe the Eigen faces bases face recognition algorithm to slightly outperform the PAL recognition algorithm. However, for the aforesaid pose, both the MM-DR and PAL yields the highest recognition accuracy on the Multi-PIE database.
  • For −300 of pose variations on the LFW database and +300 and −350 on the Multi-PIE databases, the proposed PAL recognition algorithm and the MM-DFR yield equal recognition rate of 82.9%, 89.76%, and 82.63%, respectively.
  • For opposite orientations and the same pose, such as ±450 recognition accuracy of the five compared algorithms is not same. This might be due to the factors, for example, moles, scars, or tattoos on the face. Non-uniform illuminations might be the other factor to yield different outputs.
  •  As shown in Fig. 3(c), for frontal facial images, only the LBP algorithm yields an average of 41.23% recognition accuracy on both the databases, while all other compared algorithms produce 100% recognition accuracy. For ±300 , ±350 , and ±450 of pose variations, the MM-DFR outperforms companion algorithms on both databases followed by the proposed PAL recognition algorithm. In particular, to handle the large pose variations reliably is the main characteristics of the MM-DFR and the PAL recognition algorithms. We believe this particular finding will be useful in several security and augmented reality applications.

Fig. 3. Recognition accuracy comparison on the LFW and CMU Multi-PIE databases

Fig. 4. Recognition accuracy under non-uniform illuminations on the LFW database

4.2. Non-Uniform Illuminations Analysis

 Since many of the images captured in the LFW database have been collected under diverse lighting conditions as shown in Fig. 2(a), while images in the CMU Multi-PIE database have been collected under controlled and constraint environment. Therefore, we find that to test the accuracy of face recognition algorithms under non-uniform illuminations, the LFW database is the best available dataset. It is important to state here that to test the accuracy of the face recognition algorithms under non-uniform illuminations; we manually select 1600 images from the LFW database across the seven poses investigated in this study. Moreover, the non-uniform illuminations have been analyzed for standard facial images of size 231×251 pixels and down to 150×150 pixels. Fig. 4 summarizes the results with following important observations.

  • As shown in Fig. 4, the MM-DFR, the PAL, the Eigen Faces, and AdaBoost based face recognition algorithms yield 100% recognition accuracy for frontal facial images across the entire range of 231×251 pixels and down to 150×150 pixels.
  • For −450 of pose variation on the LFW database and under non uniform illuminations, both the PAL and the MM-DFR recognition algorithms yield an equal recognition rate of 74.19%.
  • The proposed PAL face recognition algorithm yields an average of 86.910% recognition accuracy across seven poses followed by the Eigen Faces, which has about 70% recognition accuracy. As shown in Fig. 4, the MM-DFR outperforms other face recognition algorithms from frontal to ±450 of pose variation by producing 87.012% accuracy.
  • We observe the LBP face recognition algorithm to be least effective under non-uniform illuminations by yielding the maximum accuracy of 35% for frontal facial images. For ±450 of face pose, the LBP barely yields any recognition results.
  • For ±300, ±350, and +450 of pose variations, the MM-DFR outperforms companion algorithms on the LFW database under non-uniform illuminations followed by the proposed PAL recognition algorithm. To handle the large pose variations under nonuniform illuminations is one key characteristics of the MM-DFR recognition algorithm. We believe this particular finding will be useful in several security and augmented reality applications. For −450 of pose variation on the LFW database an equal recognition rate of the MM-DFR and the PAL indicates the reliable recognition ability of the proposed PAL face recognition algorithm.

Fig. 5. Face Image Resolutions of: (a) 80×80, (b) 50×50, (c) 30×30, and (d) 20×20 pixels.

4.3. Low-Resolution Image Analysis

 Image resolution plays an important role in determining the face recognition algorithm ability. Many recent face recognition algorithms yield very low recognition accuracy while the resolution is changed [8]. Fig. 5 presents different face images of sizes 80×80, 50×50, 30×30, and 20×20 pixels. Simulation results shown in Fig. 3 were validated across entire range of 231×251 down to 50×50 pixels. However, as the face image resolution was decreased from 30×30 pixels and below, we observed change in recognition rate. Tables 3(a)-(b) show the detailed recognition rates on low-resolution face images of size 30×30 and 20×20 pixels. From Table 3(a)-(b), few important observations are in order:

  • For low-resolution images, such as 30×30 pixels and below, the proposed PAL face recognition algorithm, the AdaBoost-LDA, the MM-DFR, and Eigen Faces based yield 100% recognition accuracy for frontal facial images on both the LFW and Multi-PIE databases.
  • For +350 and −300 of face pose on both the databases, three base line algorithms are unable to identify a face. On aforesaid poses, the proposed PAL is robust by outperforming the MM-DFR and yields up to 90.07% recognition accuracy.
  • On extremely challenging LFW database and on low-resolution face images up to 20×20 pixels and below, the proposed PAL face recognition algorithm yields over 70% accuracy.
  • On the CMU Multi-PIE database and on low-resolution face images up to 20×20 pixels and below, the proposed PAL face recognition algorithm outperforms the compared algorithms and yields over 80% recognition accuracy across six poses.
  • As shown in Table 3(a)-(b), for a very large pose variation, for instance −450 of orientation, only the PAL face recognition algorithm and MM-DFR work and yield up to 60.09% classification accuracy on both the LFW and the Multi-PIE databases.
  • For +450 of face orientation and on 30×30 and 20×20 pixels face image, the proposed PAL face recognition algorithm yields 70% recognition rate, which is slightly higher than the algorithms compared therein. On the aforesaid pose, the MM-DFR produces 69.89% classification accuracy.
  • Therefore, one of interesting finding is that the proposed PAL face recognition algorithm outperforms the MM-DFR on low-resolution face images across seven different face poses investigated in this study.

Table 3(a). Recognition accuracy on low-resolution face images on the LFW database

Table 3(b). Recognition accuracy on low-resolution face images on the CMU Multi-PIE database

4.4. Error Comparison

 For the LFW database, we observe several obstacles, such as occluded face, occlusion, or multiple faces in back ground as shown in Table 4. Fig. 6 shows the error comparison, which was calculated when a face was not recognized under conditions as illustrated in Table 4. From Fig. 6, few of the observations are in order:

  • The LBP face recognition algorithm is most prone to errors while the PAL and the MMDFR are the least. Moreover, we empirically observed that from frontal (00) to ±300 of pose variations, the Eigen faces based face recognition algorithm and MM-DFR performed nearly equal to the proposed PAL recognition algorithm. Specifically, for frontal pose, equal error rate was observed for the PAL, the MM-DFR, and Eigen faces algorithm.
  • For the AdaBoost-LDA and LBP face recognition algorithms, an increase in error rate was observed for an increase in towards negative orientation. Moreover, the decrease in error rate was observed as the pose was varied from positive orientation towards the frontal pose.
  • For same pose and opposite orientations, nearly equal error rate was observed for Eigen Faces based face recognition algorithm. This particular fact is observed for only the aforesaid face recognition algorithm. In future, it will be a base line for researchers’ to improve this recognition algorithm.

Table 4. Error images with reason of error output

Fig. 6. Error Comparison of different face recognition algorithms

4.5. Computational Complexity

 Time complexity is an important factor for real time face recognition algorithms. The computational complexity is evaluated in terms of the time consumed by each algorithm to recognize a probe face image. The execution time of algorithm of testing an image for different image sizes are shown in Fig 7 with following key points.

  • We observe the MM-DFR algorithm to be most computationally expensive and for lowresolution image of size 40×40 pixels and below it consumes more than a half second. For high image resolution of 231×251 pixels, the MM-DFR algorithm consumes more than three seconds to identify a human face.
  • We observe that both the LBP and the Eigen faces based face FR algorithms are  near real-time and consume less than one second to recognize a probe image of size 231×251 pixels. For face image resolution of 20×20 pixels, the aforesaid algorithms consume about 0.6 seconds to recognize a face.
  • The AdaBoost-LDA face recognition algorithm consumes about 2.3019 to 0.6041 seconds to recognize a face image of various resolutions shown in Fig. 7. The number of iterations was set to 30 for this algorithm to achieve the face recognition task. We observed that the recognition time for this algorithm increases up to 4.9197 seconds for 50 numbers of iterations.
  • Proposed PAL face recognition algorithm, when trained with only a single frontal image and tested with seven different face poses, then for 80×80 pixels face image consumes about 0.8571 seconds. For extremely low-resolution of 20×20 pixels, the proposed PAL face recognition algorithm consumes about 0.4129 seconds to identify a face image. This particular finding is very useful for situations in which low-resolution images are obtained. For instance, images captured through CCTV camera or from a crime scene. It is also quite evident from Fig. 7 that the proposed PAL face recognition algorithm consumes much less time than the MM-DFR recognition algorithm for all image resolutions.
  • We observe that each of the compared algorithms on the average consume at most three seconds to recognize a probe face for all image resolutions of 231×251 pixels and below. Therefore, we conclude that the compared algorithms are near real-time.

Fig. 7. Time cost comparison

4.6. Discussion:

 Although the aforementioned simulation results shed detailed light on the performance of the proposed PAL face recognition algorithm. Points below give further insight on the developed face recognition algorithm.

  • Although we presented detailed simulation results of the proposed PAL face recognition and compared with four recently developed algorithms. However, the potential of computer vision algorithms to efficiently, reliably, and accurately identify human faces in naturally occurring circumstances still remains elusive. Recent trends in deep learning are taking us closer to human-level recognition [39]-[40]. In this study, we empirically observed that deep learning based face recognition algorithm [39], on high image resolutions, such as 80×80 pixels and above outperforms compared algorithms on the LFW and Multi-PIE databases for pose variation and under non-uniform illuminations. Proposed PAL face recognition algorithm showed superior performance on lowresolution face images of 30×30 pixels and below and outperforms the MM-DFR.
  • Although the error performance in Fig. 6 showed the superiority of the proposed PAL face recognition algorithm. However, we empirically observe when lighting conditions become too much dim or when the images captured at the night time are analyzed through the PAL, its performance significantly drops down. In such cases, an efficient color image enhancement scheme [38] can be used as a pre-processing module to obtain high recognition accuracy. In such situations nearly zero percent face recognition rate was observed for all of the compared algorithms.
  • While comparing the performance of the proposed PAL face recognition algorithm on large face poses, it should not be mixed with the face expression, such as happy, cry, or yawn. As on those situations, it does not yield reliable results.
  • For face image resolution of 231×251 and down to 20×20 pixels, the proposed PAL face recognition algorithm and MM-DFR can be reliably used for frontal facial images only. However, for large face pose variation, such as ±450, performance of both the algorithm significantly drops down below 50%.

5. Conclusions and Future Work:

 This paper proposed a novel face recognition method to recognize faces with high accuracy for three important aspects, which are (i) face pose, (ii) non-uniform illuminations, and (iii) low-resolution face images. Proposed face recognition algorithm initially uses 68 points to locate a face followed by the AdaBoost integrated with the LDA, which is used for face feature extraction. In final stage, classic nearest centre classifier is used for face classification. Proposed method out performs recent state-of-the-art face recognition algorithms by producing high accuracy on the aforementioned issues at the cost of higher computations on the LFW and the CMU Multi-PIE databases. Based on the developed face recognition algorithm, we conclude the following.

 − For same pose and opposite orientation, even state-of-the-art face recognition algorithms yield low recognition accuracy. The proposed PAL face recognition algorithm outperforms the recently developed algorithms by producing the high recognition rate across seven different face poses on low-resolution face images as described in Section 4. Moreover, deep learning based recognition algorithm [38] struggles on low-resolution face images and yields lower recognition accuracy on pose variation and under nonuniform lighting illuminations for 30×30 pixels and below.

 − Although, face recognition for frontal face images and under constraint conditions has matured. However, non-uniform illuminations severely degrade the recognition accuracy of recent algorithms. Similarly, low-image resolution seriously challenges the accuracy of deep learning based face recognition algorithms. The proposed PAL face recognition algorithm reliably handles low-resolution images.

 − Human face appearance surprisingly changes under artifacts, such as make up, tattoos, scar or mole on face, or beard/moustaches. In such situations the developed PAL recognition algorithm is not recommended to use.

 In the future, we aim to optimize the proposed algorithm to be executed in real-time. Moreover, we also aim to investigate a large pose variation, such as from frontal (00) to profile view (±900) in the input images. Furthermore, we intend to develop a parallel version of the proposed face recognition algorithm to be reliably used in real-time applications. For instance, security, surveillance, and face verification at any entry ports.

References

  1. G.I. Davida, Y. Frankel, and B.J. Matt., "On enabling secure applications through off-line biometric identification," in Proc. of IEEE Symposium on Privacy and Security, 1998.
  2. J. Lu, G. Wang, W. Deng, and K. Jia, “Reconstruction-based metric learning for unconstrained face verification,” IEEE Transactions on Information Forensics and Security, Vol. 10, No. 1, pp. 79-89, Jan 2015. https://doi.org/10.1109/TIFS.2014.2363792
  3. N. Hezil, A. Boukrouche, "Multimodal biometric recognition using human ear and palmprint," IET Biometrics, vol. 6, no. 5, pp. 351-359, Sep 2017. https://doi.org/10.1049/iet-bmt.2016.0072
  4. S.A. Nazeer, N. Omar and M. Khalid, "Face Recognition System using Artificial Neural Networks Approach," IEEE-ICSCN, pp. 420-425, February 2007.
  5. Y. Su, "Robust Video Face Recognition Under Pose Variation," Neural Processing Letters, pp. 1-15, 2017.
  6. O. Aldrian and W. A. Smith, "Inverse rendering of faces with a 3D morphable model," IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 35, no. 5, pp. 1080-1093, 2013. https://doi.org/10.1109/TPAMI.2012.206
  7. H. Han, S. Shan, X. Chen, W. Gao, "A comparative study on illumination preprocessing in face recognition," Pattern Recognition, Vol. 46, No. 6, 2013, pp. 1691-1699. https://doi.org/10.1016/j.patcog.2012.11.022
  8. Z. Mahmood, N. Muhammad, N. Bibi, and T. Ali, "A review on state-of-the-art face recognition approaches," Fractals, Vol. 25, No. 02, pp 1750025, 2017. https://doi.org/10.1142/S0218348X17500256
  9. J. Lu, K. N. Plataniotis, and A. N. Venetsanopoulos, “Face recognition using LDA-based algorithms,” IEEE Transactions on Neural Networks, Vol. 14, No. 1, pp. 195-200, January 2003. https://doi.org/10.1109/TNN.2002.806647
  10. T. Ahonen, A. Hadid, and M.Pietikainen, “Face Description with Local Binary Patterns: Application to Face Recognition,” IEEE Trans. Pattern Analysis and Machine Intelligence, Vol. 28, No. 12, pp. 2037-2041, 2006. https://doi.org/10.1109/TPAMI.2006.244
  11. S.U. Rehman, S.Tu, Y. Huang, and Z. Yang, "Face recognition: A novel un-supervised convolutional neural network method," in Proc. of In Online Analysis and Computing Science (ICOACS), IEEE International Conference of, pp. 139-144, 2016.
  12. M. Turk and A. Pentland, “Eigenfaces for Recognition,” Journal of cognitive neuroscience, Vol. 3, No. 1, pp. 71-86, 1991. https://doi.org/10.1162/jocn.1991.3.1.71
  13. K.Simonyan, O.M. Parkhi, A. Vedaldi, A. Zisserman, "Fisher Vector Faces in the Wild," in Proc. of In BMVC, Vol. 2, No. 3, pp. 8.1-8.12, 2013.
  14. E. Learned-Miller, G.B. Huang, A. RoyChowdhury, "Labeled faces in the wild: A survey," In Advances in face detection and facial image analysis, pp. 189-248, Springer, Cham, 2016
  15. S. Kakarwal, R. Deshmukh, "Wavelet transform based feature extraction for face recognition," IJCSA, Issue-I, pp. 9740767, June 2010.
  16. S Tan, X Sun, W Chan, L Qu, "Robust face recognition with kernelized locality-sensitive group sparsity representation," IEEE Transactions on Image Processing, Vol. 26, No. 10, pp 4661-4668, 2017. https://doi.org/10.1109/TIP.2017.2716180
  17. Y. Taigman, M. Yang, M. Ranzato, and L. Wolf, "DeepFace: Closing the gap to human-level performance in face verification," in Proc. of the IEEE conference on computer vision and pattern recognition, pp. 1701-1708, 2014.
  18. M. J. Er, S. Wu, J. Lu, and H. L. Toh, "Face recognition with radial basis function (RBF) neural networks," IEEE Trans. Neural Network, Vol. 13, No. 3, 2002, pp. 697-710. https://doi.org/10.1109/TNN.2002.1000134
  19. F. Schroff, D. Kalenichenko, and J. Philbin, "Facenet: A unified embedding for face recognition and clustering," in Proc. of the IEEE conference on computer vision and pattern recognition, pp. 815-823, 2015.
  20. Y Wen, K Zhang, Z Li, Y Qiao, "A discriminative feature learning approach for deep face recognition," in Proc. of In European Conference on Computer Vision, pp. 499-515, 2016.
  21. L. Lu, Y. Mao, L. Wenyin, H.J. Zhang, "Audio restoration by constrained audio texture synthesis," in Proc. of ICASSP 2003, Vol. 3, pp. 405-408, July 2003.
  22. Z. Mahmood, T. Ali, and S. U. Khan, “Effects of pose and image resolution on automatic face recognition,” IET biometrics, Vol. 5, No. 2, pp. 111-119, 2016. https://doi.org/10.1049/iet-bmt.2015.0008
  23. R. Ranjan, V.M. Patel,. Chellappa, "Hyperface: A deep multi-task learning framework for face detection, landmark localization, pose estimation, and gender recognition," IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 41, No 1, pp. 121-135, 2019. https://doi.org/10.1109/TPAMI.2017.2781233
  24. Z. Mahmood, T. Ali, and S. Khattak, "Automatic player detection and recognition in images using AdaBoost," in Proc. of Applied Sciences and Technology (IBCAST), 9th International Bhurban Conference on, IEEE, pp. 64-69, 2012.
  25. J. Lu, K. Plataniotis, A. Venetsanopoulos, and S. Li, “Ensemblebased discriminant learning with boosting for face recognition,” IEEE Trans. Neural Networks, Vol. 17, No. 1, pp. 166-178, Jan 2006. https://doi.org/10.1109/TNN.2005.860853
  26. Gross, R., Matthews, I., Cohn, J., Kanade, T., Baker, S., "Multi-pie," Image and Vision Computing, Vol. 28, No 5, pp. 807-813, 2010. https://doi.org/10.1016/j.imavis.2009.08.002
  27. Z. Mahmood, O. Haneef, N. Muhammad, and S. Khattak, “Towards a fully automated car parking system,” IET Intelligent Transport Systems, Vol. 13, No. 2, pp. 293, 2019. https://doi.org/10.1049/iet-its.2018.5021
  28. M. Wan, G. Yang, S. Gai, and Z. Yang, “Two-dimensional Discriminant Locality Preserving Projections (2DDLPP) and Its Application to Feature Extraction via Fuzzy Set,” Multimedia Tools and Applications, Vol. 76, No. 1, pp. 355-371, 2017. https://doi.org/10.1007/s11042-015-3057-8
  29. M. Wan, M. Li, G. Yang, and Z. Jin, "Feature extraction using two-dimensional maximum embedding difference," Information Sciences, Vol. 274, pp. 55-69, 2014. https://doi.org/10.1016/j.ins.2014.02.145
  30. M. Wan, Z. Lai, G. Yang, et el, "Local graph embedding based on maximum margin criterion via fuzzy set," Fuzzy Sets and Systems, Vol. 318, pp. 120-131, 2017. https://doi.org/10.1016/j.fss.2016.06.001
  31. J. Lu, J. Hu, and Y. P. Tan, “Discriminative deep metric learning for face and kinship verification,” IEEE Transactions on Image Processing, Vol. 26, No. 9, pp. 4269-4282, 2017. https://doi.org/10.1109/TIP.2017.2717505
  32. M. Alam, L. S. Vidyaratne, and I. M. Khan, “Sparse simultaneous recurrent deep learning for robust facial expression recognition,” IEEE Transactions on Neural Networks and Learning Systems, Vol. 29, No. 10, pp. 4905-4916, 2018. https://doi.org/10.1109/TNNLS.2017.2776248
  33. C. Galea and R. A. Farrugia, “Forensic face photo-sketch recognition using a deep learning-based architecture,” IEEE Signal Processing Letters, Vol. 24, No. 11, pp. 1586-1590, 2017. https://doi.org/10.1109/LSP.2017.2749266
  34. J. Zhao, J. Han, and L. Shao, “Unconstrained face recognition using a set-to-set distance measure on deep learned features,” IEEE Transactions on Circuits and Systems for Video Technology, Vol. 28, No. 10, pp. 2679-2689, 2018. https://doi.org/10.1109/TCSVT.2017.2710120
  35. L.M. Dang et al, "UAV based wilt detection system via convolutional neural networks," Sustainable Computing: Informatics and Systems, 2018,
  36. D.L. Minh et al, "Deep learning approach for short-term stock trends prediction based on two-stream gated recurrent unit network," IEEE Access, Vol. 6, pp. 55392-55404, 2018. https://doi.org/10.1109/ACCESS.2018.2868970
  37. Z. Mahmood, T. Ali, N. Muhammad, N. Bibi, I. Shahzad, and S. Azmat, “EAR: Enhanced augmented reality system for sports entertainment applications,” KSII Transactions on Internet and Information Systems, Vol. 11, No. 12, pp. 6069-6091, 2017. https://doi.org/10.3837/tiis.2017.12.021
  38. Z. Mahmood, N. Muhammad, N. Bibi, Y. Malik, and N. Ahmed, "Visual enhancement of human observatory system using multi-scale retinex," Informatics in Medicine Unlocked, Vol. 13, 2018.
  39. C. Ding and D. Tao, “Robust face recognition via multimodal deep face representation,” IEEE Transactions on Multimedia, Vol. 17, No. 11, pp. 2049-2058, 2015. https://doi.org/10.1109/TMM.2015.2477042
  40. S. Munasinghe C. Fookes, and S. Sridharan, “Human-level face verification with intra-personal factor analysis and deep face representation,” IET Biometrics, Vol. 7, No. 5, pp. 467-473, 2018. https://doi.org/10.1049/iet-bmt.2017.0050
  41. Z. Mahmood, T. Ali, S. Khattak, L. Hassan, and S. U. Khan "Automatic player detection and identification for sports entertainment applications," Pattern Analysis and Applications, Vol. 18, No. 4, pp. 971-982, 2015. https://doi.org/10.1007/s10044-014-0416-4

Cited by

  1. Implementation of a face recognition system as experimental practices in an artificial intelligence and pattern recognition course vol.28, pp.3, 2019, https://doi.org/10.1002/cae.22218
  2. Generation of an EDS Key Based on a Graphic Image of a Subject’s Face Using the RC4 Algorithm vol.12, pp.1, 2019, https://doi.org/10.3390/info12010019
  3. Phrase-level sentence patterns for estimating positive and negative emotions using Neuro-fuzzy model for information retrieval applications vol.80, pp.13, 2021, https://doi.org/10.1007/s11042-020-10422-6