2D-MELPP: A two dimensional matrix exponential based extension of locality preserving projections for dimensional reduction

Abstract

Two dimensional locality preserving projections (2D-LPP) is an improved algorithm of 2D image to solve the small sample size (SSS) problems which locality preserving projections (LPP) meets. It's able to find the low dimension manifold mapping that not only preserves local information but also detects manifold embedded in original data spaces. However, 2D-LPP is simple and elegant. So, inspired by the comparison experiments between two dimensional linear discriminant analysis (2D-LDA) and linear discriminant analysis (LDA) which indicated that matrix based methods don't always perform better even when training samples are limited, we surmise 2D-LPP may meet the same limitation as 2D-LDA and propose a novel matrix exponential method to enhance the performance of 2D-LPP. 2D-MELPP is equivalent to employing distance diffusion mapping to transform original images into a new space, and margins between labels are broadened, which is beneficial for solving classification problems. Nonetheless, the computational time complexity of 2D-MELPP is extremely high. In this paper, we replace some of matrix multiplications with multiple multiplications to save the memory cost and provide an efficient way for solving 2D-MELPP. We test it on public databases: random 3D data set, ORL, AR face database and Polyu Palmprint database and compare it with other 2D methods like 2D-LDA, 2D-LPP and 1D methods like LPP and exponential locality preserving projections (ELPP), finding it outperforms than others in recognition accuracy. We also compare different dimensions of projection vector and record the cost time on the ORL, AR face database and Polyu Palmprint database. The experiment results above proves that our advanced algorithm has a better performance on 3 independent public databases.

1. Introduction

In pattern recognition, massive high dimensional image inputs are computationally challenging to analysis[1-6]. We hope to retain as much information as possible while reducing data dimensions to improve the accuracy and efficiency of later processing such as inference [7-10]. Generally, dimensionality reduction methods can be classified into two classes: feature extraction and feature selection. The former is focused on creating new features and the later is focused on selecting most relevant subsets. And there are two feature extractions classifications: (1)Nonlinear methods and (2)Linear methods. Nonlinear ones like locality linear embedding(LLE)[11], Laplacian egienmaps(LE)[12] pay attention to preserving local structure, while other methods like Isomap[13] is focused on global structure. It also tries to maintain geodesic distances between different samples. Linear methods, on the other hand, has become extremely important for itssimple and computationally efficient classification strategies. Principal component analysis(PCA)[14-15] as unsupervised method and linear discriminant analysis(LDA)[16-17] as supervised method are two representative linear methos dimensionality reduction. locality preserving projections(LPP)[18], which can be seen as linearization of LE, is an alternative to PCA. LPP is obtained during the process of searching for the optimized approximate value of eigenfunctions of Laplace Beltrami operator on the manifold when there is a low dimensional manifold embedded in relatively high dimensional spaces. It needs to point out that LPP possesses two non linear advantages while former linear algorithms don’t:

1. New minimal criterion preserves neighborhood structure, considering that Euclidean distances are meaningful only when they are local. It makes sense particularly when data lies on nonlinear manifold embedded in the ambient space.

2. For the ‘neighborhood structure preserved’ advantage above, LPP is also suitable for information retrieval applications

It is also very important that LPP is defined on more than just training data points and thus can be simply applied to new data points compared with traditional non-linear methods.

Two dimensional principal component analysis(2D-PCA)[19] saves the trouble of transforming images into one dimensional vectors and it inspires subsequent matrix based algorithm. And two dimensional linear discriminant(2D-LDA)[20] analysis extracts proper features directly from images based on Fisher’s linear discriminant analysis. In the process especially like image recognition, the data is usually insufficient and having high dimensions causing singularity problems which make algorithms (e.x. LPP) unable to be applied directly. Two dimensional locality preserving projections (2D-LPP) is therefore proposed. It works on 2D images directly. Two dimensional locality preserving projections(2D-LPP)[21-23]perfectly solves the singularity that traditional LPP faced and is proved to perform better on face recognition and palmprint recognition , it turns out to be faster than LPP and achieves higher recognition rates.

Matrix based exponential method is also used to solve such small sample sized problem (SSS problems). ELDE[24] is based on local discriminant embedding (LDE)[25], which was put forward to overcome the limits that global linear discriminant analysis method meets. Matrix based exponential method also helps to improve the flaw of LPP’s sensitivity to neighborhood sized k[26] and fix singularity problems.

According to[27], 2D methods don’t necessarily perform better than 1D methods. It implies the limitations that 2D method meets and the direction for feature work. 2D-LPP is simple and efficient, but it don’t perform well when training samples are limited and it seems to meet limitations in some public databases. We apply matrix exponential methods on 2D-LPP to improve its behavior and we get its enhanced version--2D-MELPP. However, the computational complexity of 2D-MELPP is extremely high, so we replace some of matrix multiplications with multiple multiplications to get a more efficient way for solving 2D-MELPP. We also use a more efficient way to calculate matrix exponential to save time. The new 2D-MELPP has high memory costs, so we replace some matrix multiplications with multiple multiplications and we investigate the enhanced version on a random 3D data set, ORL, AR Face Database and Polyu Palprint Database, 2D-PCA,2D-LDA and 2D-LPP are also tested.

The remainder of paper is arranged as follows: The mainly content is about reference to related works in Section 2. Section 3 is the retrospect of innovative work in this paper. Section 4 is composed of 4 experiments designed to show the efficiency and accuracy of our new algorithm. Section 5 is separated into two parts, respectively describing conclusions of experiments in section 4 and the direction of our feature work.

2. Related Work

Our work is based on 2D-LPP and matrix exponential methods, which both help to solve SSS problems and improve recognition accuracy.

2.1 2D-LPP method

2D-LPP applies the projection for each data as follows:

x_i = A_iw(i = 1, 2, 3...n)       (1)

x ∈ R^r×d A_j ∈ R^r×c w ∈ R^c×l

where r, c are row and column dimension of A_i respectively. w is the target projection matrix a, while A_i is the ith source image for testing, and n is the number of images in the datasets. It chooses[21] the criterion as :

\(\begin{aligned}\min \sum_{i, j} S_{i j}\left\|X_{i}-X_{j}\right\|^{2}(i, j=1,2,3 \ldots)\end{aligned}\)       (2)

is generated as : S_ij

(1). First, we have to construct adjacency matrix G ∈ R^mxn. For every G_ij both k-nearest neighbors and ε-nearest neighbors are appropriate. When doing the experiment in this paper, we choose the former and add label Information;

(2). For each G_ij not equal to zero, we suppose ith and jth image are “close”;

(3). For all “close” ones, we apply the Heat Kernel method to build the S matrix.

Looking back to the origin criterion, it can be written as:

\(\begin{aligned}\min \sum_{i, j}\left\|A_{i} W-A_{j} W\right\|^{2}(i, j=1,2,3 \ldots)\end{aligned}\)       (3)

where the target function can be simplified[23] as

minw^T A^T(L⊗I_n)Aw       (4)

where A is in the form like [A₁^T, A₂^T ...A_n^T]^T, D_ii is column sum of S, I_n is a n×n identity matrix and ⊗ is kronecker product.

To eliminate an arbitrary scaling element of the process, 2D-LPP dispose it as followed :

\(\begin{aligned}\sum_{i} D_{i j} X_{i}^{T} X_{i}=1\end{aligned}\)        (5)

which can also be presented as:

W^T A^T(D⊗I_n)Aw = 1       (6)

And the whole minimization problem is transformed into a classical generalized eigenvalue problem as follows:

A^T(L⊗I_n)Aw = λA^T(D⊗I_n)Aw       (7)

So we can get final target projection matrix w.

2.2. Matrix exponential Process

In the sub-section, we will briefly look into the definition and some basic properties of matrix exponential process for the prerequisite of 2D-MELPP:

The Matrix exponential for M(M ∈ R^mxn) is defined as follows:

\(\begin{aligned}\exp (M)=1+M+\frac{M^{2}}{2 !}+\ldots+\frac{M^{m}}{m !}+\ldots\end{aligned}\)        (8)

where 1 can be treated as an n × n identity matrix, the matrix exponential has properties as followed:

(1). exp(M) is the sum of a sequence of matrixes with finite numbers;

(2). exp(M) is a full rank matrix;

(3). For arbitrary square matrix M, there exists the inverse of its matrix exponential;

(4). Supposing that R is a nonsingular matrix, we can have:

exp(R^-1MR ) = R^-1exp(M)R

(5). For every eigenvectors of M(v₁, v₂...v_n) corresponding to α₁, α₂..., α_n there exists e^α₁, e^α₂..., e^α_n as eigenvalues of exp(M) having the same eigenvectors just like M.

3. 2D-Matrix exponential based discriminate locality preserving projection（2D-MELPP）

3.1. Notation used in 2D-MELPP

3.2. 2D-MELPP without further improvements

In 2D-MELPP, the objective function is:

minw^T A^T exp(L⊗I_n)Aw       (9)

To eliminate an arbitrary scaling element of the process, 2D-LPP dispose it as followed :

w^T A^T exp(D⊗I_n)Aw =1       (10)

With Matrix based exponential D⊗L operator included accordingly.

Then the whole optimal question is equal to the eigenvalue problem as follows :

exp(A^T)(L⊗I_n)A)w = λexp(A^T(D⊗I_n)A)w       (11)

Referring to the properties (5) in section B, it is mathematically similar and can solve SSS problem in classical eigenvalue questions for (4) in section B.

According to[26], it can also emphasize the geometry features by the same time.

3.3. Efficient procedure to solve 2D-MELPP

The computational complexities of 2D-MELPP is computed exp(A^T(L⊗I_n)A) and O(r³) respectively. A paper[28] meets the same problem as working with one dimension framework as follows:

w = argmin(w^T exp(XDX^T)w)^-1(w^T exp(XLX^T)w)       (12)

To solve this, we can apply the similar strategy.

As what it’s shown in the part followed, an efficient procedure formula is build to solve 2D-MELPP.

First, we assume the SVD decomposition of A is

A = PΣV^T       (13)

Let S_a = P^T exp(A^T(L⊗I_m)A)P, S_b = P^T exp(A^T(D⊗I_m)A)P, suppose that the eigenvector is u and λ is the eigenvalue.

Then we obtain:

exp(A^T(D⊗I_n)A)Pu = λexp(A^T(L⊗I_n)A)Pu       (14)

According to (4) the computational complexity is reduced. However, the computational complexity of computing exp(A(D⊗I_n)A) and exp(A^T(L⊗I_n)A) is still O(d³). Then, out target has changed into compute exp(A^T(D⊗I_n)A) and exp(A^T(L⊗I_n)A) efficiently.

The eigen-decomposition of H = D⊗I_n is supposed :

H = Q_HΣ_HQ_H^T       (15)

Let M_H = A^TQ_HΣ_H^0.5, and its economic decomposition is

M_H = Q_MΣ_MV_M^T       (16)

Then we obtain:

A^T HA = Q_MΣ_M²Q_M^T       (17)

Let column orthogonal matrix Q_M^τ be that [Q_M, Q_M^τ] is orthogonal.

According to the definition, we can have that:

exp(A^T HA) = [Q_M, Q_M^τ][Q_M, Q_M^τ]^T + ...       (18)

\(\begin{aligned}+\frac{\left[Q_{M}, Q_{M}^{\tau}\right]\left[\begin{array}{cc}\Sigma_{M}^{2} & O_{r_{M} \times\left(r-r_{M}\right)} \\ O_{\left(r-r_{M}\right) \times r_{M}} & O_{\left(r-r_{M}\right) \times\left(r-r_{M}\right)}\end{array}\right]}{m !}+\ldots\end{aligned}\)

\(\begin{aligned}=Q_{M} \exp \left(\Sigma_{M}^{\tau}\right) Q_{M}^{T}+1-Q_{M} Q_{M}^{T}\end{aligned}\)

where Q_c×d are zeros matrices, c,d are its row and column numbers respectively. And r_M is the dimension of Σ_M.

Then, we have:

P^T exp(A^T HA)P = P^Texp(Σ_H²)Q_M^TP + I - P^TQ_MQ_M^TP       (19)

Similarly, the eigen-decomposition of L = L⊗I_n is assumed:

L = Q_LΣ_LQ_L^T       (20)

Let X_L= V^TQ_LΣ_L^0.5, and its economic decomposition is:

X_L = Q_XΣ_XV_X^T       (21)

Then we obtain:

A^T LA=PQ_XΣ_X²Q_X^TP^T       (22)

Let column orthogonal matrix P_M^τ be that [PQ_X, P_X^τ] is orthogonal. Then we have:

\(\begin{aligned}\begin{array}{l}\exp \left(A^{T} L A\right)=\left[P Q_{X}, P Q_{X}^{\tau}\right]\left[P Q_{X}, P Q_{X}^{\tau}\right]^{T}+ \\ \ldots+\frac{\left[P Q_{X}, P Q_{X}^{\tau}\right]\left[\begin{array}{cc}\Sigma_{M}^{2 m} & O_{r_{x} \times\left(d-r_{x}\right)} \\ O_{\left(r-r_{x}\right) \times r_{x}} & O_{\left(r-r_{x}\right) \times\left(r-r_{x}\right)}\end{array}\right]}{m !} \\\end{array}\end{aligned}\)

\(=P Q_{X} \exp \left(\Sigma_{X}^{2}\right) Q_{X}^{T} P^{T}+P_{X}^{\tau}\left(P_{X}^{\tau}\right)^{T}\)       (23)

Then we have:

P^T exp(A^T LA)P = Q_X exp(Σ_X²)Q_X^T + I - Q_XQ_X^T       (24)

Now we have the following new process to solve 2D-MELPP:

Algorithm :2D-MELPP

Input: sample A

Output: The projected matrix of G

(1). Constructing the matrix A , H = exp(A^T DA), L = exp(A^T LA) just like normal 2D-LPP;

(2). Calculating the SVD of A as A = PΣV^T and the matrix H as H = Q_HΣ_HQ_H^T ;

(3). Compute the matrix M_H = A^TQ_HΣ_H^0.5and its economic SVD as M_H= Q_MΣ_MV_M^T ;

(4). Compute the eigen-decomposition of L as L = Q_LΣ_LQ_L^T ;

(5). Calculate the matrices X_L = V^TQ_LΣ_L^0.5 ,and its economic decomposition is X_L = Q_XΣ_XV_X^T ;

(6). Compute the matrix

J = P^T exp(A^T HA)P = P^T exp(Σ_H²)Q_M^TP^T + P^TQ_MQ_M^TP

U = P^T exp(A^T HA)P = Q_X exp(Σ_X²)Q_X^T+ I - Q_XQ_X^Trespectively;

(7). Solve the eigenvalue problem Ju_i = λ_iUu_i ;

(8). Let G = PU_k where U_k is the matrix combined of k eigenvectors corresponding to k largest eigenvalues, and then orthogonalize the projection matrix and get the G we wanted(k is a constant value) .

Then the time complexity is reduced to O(rn²).

4. Experimental results

In the experiment, we will evaluate the performance of 2D-MELPP with PCA+LPP, 2D-PCA, ELPP, 2D-LDA, and 2D-LPP on the random 3D data set and three different public image databases. We pick up KNN as the classifier for three image databases. Four experimental results are completed in the same experimental environment (CPU: 2.04 GHz, RAM: 3.9GB).

4.1 The Random 3D data set

In the first experiment, we will propose 2D-MELPP and other methods on a synthetic 3D matrix M. The data set is made up of 20 matrices, each of which is generated by a 20 by 20 matrix having 400 points subject to normal distribution. And the data set is divided into 2 class as follows(The data set is normalized) :

\(\begin{aligned}\operatorname{class}\left(A_{k i j}\right)=\left\{\begin{array}{ll}\text { class }-a & A_{k i j} \geq 0.999 \\ \text { class }-b & A_{k i j} \leq 0.001\end{array}\right.\end{aligned}\)       (25)

where A_kij is value of ith row and jth column of the kth matrix, (0 ≤ i, j, k ≤ 20), class(A_kij) is the class of A_kij and the class is either class-a or class-b. The distribution of two classes can be seen in Fig. 1.

Fig. 1. The class distribution of 3D data set. Each point is represented by(x_i, y_i, z_j) in Cartesian coordinates. Variable i ranges from 0 to 20. Two classes are marked by blue and red respectively. Each point is assigned with random values ranging from 0 to 1 and the rules to decide its class are shown in (25).

We test PCA+LPP, ELPP, 2D-PCA, 2D-LDA, 2D-LPP and 2D-MELPP on this random data set. And FIGURE 2 shows the result of dimension reduction with PCA+LPP, 2D-PCA, ELPP, 2D-LDA, 2D-LPP and 2D-MELPP (from left to right, up to down).

As can be seen from Fig. 2, 2D-MELPP provide projected data with excellent linear separability, while other methods fail to do so.

Fig. 2. The projected 2D data set with (a) PCA+LPP, (b) ELPP, (c) 2D-PCA, (d) 2D-LDA, (e) 2D-LPP and (f) 2D-MELPP (from left to right, up to down). The X axis is the reduced dimension, Y is the serial number of the picture.

4.2 The ORL face database

The ORL face image database consists of 400 face images with 40 classes, each having 10 samples within. The background were all set in homogeneous darkness, and the images are featured by different light intensities, facial expressions and facial details(glassed and not) as it can be shown in Fig. 3.

Fig. 3. The images of one person from ORL face database

We range dimensions of project vector from 1 to 20 in Table 1 and pick one with highest recognition accuracy each(from 2 to 6 training samples each class). Fig. 4 shows recognition accuracy varies under 6 samples with the changes of dimensions. In Table 2, we record the time cost on each algorithm under 6 samples on the ORL face database.

Table 1. highest mean value of rECOGNITION ACCURACY (IN PERCENT) ON ORL database. There is a one-to-one correspondence between colunms and fixed training numbers. values in each PARENTHESIS ranges between 1 and 20, they represent for corresponding projected matrix dimensions.

Fig. 4. The recognition accuracy with 6 samples per class on the ORL face database

Table 2. AVERAGE cost time(s) of PCA+LPP, ELPP, 2D-PCA, 2D-LDA, 2D-LPP and 2D-MELPP on The ON ORL FACE SET. EACH COLUMN CORRESPONDS TO A FIXED ALGORITHM.And THE 2D-MELPP IS THE IMPROVED VERSION.

4.3 The AR face database

AR face database is made up of face images of 120 people with 26 images of each. In this experiment we take 10 each which don’t have too many occlusion problems while testing algorithm on a lager scale than ORL in Fig. 5.

Fig. 5. 20 image samples of one person form AR database. Pictures in second line were token 2 weeks after the first line

We range dimensions of project vector from 1 to 20 in Table 3 and pick one with highest recognition accuracy each(from 2 to 6 training samples each class). Fig. 6 shows recognition accuracy varies under 6 samples per class with the changes of dimensions.

Fig. 6. The recognition accuracy with 6 samples per class on the AR face database

Table 3. highest mean values of rECOGNITION ACCURACY (IN PERCENT) ON Ar database. There is a one-to-one correspondence between colunms and fixed training numbers. values in each PARENTHESIS ranges between 1 and 20, they represent for corresponding projected matrix dimensions.

In Table 4, we record the time cost on each algorithm under 6 samples on the AR face database.

Table 4. AVERAGE cost time(s) of PCA+LPP, ELPP, 2D-PCA, 2D-LDA, 2D-LPP and 2D-MELPP on The ON ar FACE SET. EACH COLUMN CORRESPONDS TO A FIXED ALGORITHM.And THE 2D-MELPP IS THE IMPROVED VERSION.

4.4. The Polyu palmprint database

Polyu Palmprint Database consists of 600 palmprint image with 100 classes, each having 6 samples within. Fig. 7 shows the background were also set in homogeneous darkness and the images are featured by different light intensities. Half of the original image is taken 2 month later than the other half, and we set the training and testing division according to that.

Fig. 7. The palmprint samples from Polyu palmprint database

Fig. 8 shows recognition accuracy varies under 3 samples with the changes of dimensions ranging from 1 to 50, and Table 5 shows highest recognition accuracy of each algorithm, Table 6 records the time cost with 50 dimensions.

Table 5. highest mean value of rECOGNITION ACCURACY (IN PERCENT) ON polyu database. There is a one-to-one correspondence between colunms and methods.

Fig. 8. recognition accuracy with 3 samples per class on Polyu Palmprint database

Table 6. AVERAGE cost time(s) of PCA+LPP, ELPP, 2D-PCA, 2D-LDA, 2D-LPP and 2D-MELPP on PolyU palmprint database. EACH COLUMN CORRESPONDS TO A FIXED ALGORITHM.And THE 2D-MELPP IS THE IMPROVED VERSION.

From results of the experiment above, we can make observations:

(1). According to Fig.s 2, 4 and 6, provided with identical experimental conditions, the recognition accuracy of the advanced algorithm has better performances. The ability of the algorithm to preserve more discrimination than others may account for it.

(2). Generally, 2D-MELPP achieves higher recognition accuracy than other algorithms with small projection dimensions as shown in Table 1, 3, 5.

(3). 2D-MELPP takes less time than ELPP in tree databases, which provides supports for our development on computational complexities does reduce the it. It can be seen from Table 2, 4, 6.

5. Conclusion

5.1. Results and advanced nature of experiments

We are inspired by the idea of ELPP and comparison works between 2D-LDA and LDA to employee matrix exponential on 2D-LPP and work algebra procedure to promote its behavior both in theory and in test on random 3D data set, two public face databases and one palmprint database. We also develop a way to save memory cost for solving 2D-MELPP. Our advanced 2D-MELPP performs better in recognition accuracy and it cost less time than ELPP.

2D-MELPP is an efficient solution for the SSS problem elegantly since the matrix exponential of a symmetric matrix is positive definite all the time. Since 2D-MELPP is based directly on image matrix and the matrix exponential process has distances enlarged, it can preserve more information and achieve higher accuracy.

5.2. Spectrum of future work

Although it’s easy to apply linear methods for dimensionality reduction, it’s nonetheless hard to attain linear data in real world feature extraction application. In view of above-mentioned reason, improvements can be made with nolinear methods or struct based methods. We will verify them in future work.