Improved marine predators algorithm for feature selection and SVM optimization

Abstract

Owing to the rapid development of information science, data analysis based on machine learning has become an interdisciplinary and strategic area. Marine predators algorithm (MPA) is a novel metaheuristic algorithm inspired by the foraging strategies of marine organisms. Considering the randomness of these strategies, an improved algorithm called co-evolutionary cultural mechanism-based marine predators algorithm (CECMPA) is proposed. Through this mechanism, search agents in different spaces can share knowledge and experience to improve the performance of the native algorithm. More specifically, CECMPA has a higher probability of avoiding local optimum and can search the global optimum quickly. In this paper, it is the first to use CECMPA to perform feature subset selection and optimize hyperparameters in support vector machine (SVM) simultaneously. For performance evaluation the proposed method, it is tested on twelve datasets from the university of California Irvine (UCI) repository. Moreover, the coronavirus disease 2019 (COVID-19) can be a real-world application and is spreading in many countries. CECMPA is also applied to a COVID-19 dataset. The experimental results and statistical analysis demonstrate that CECMPA is superior to other compared methods in the literature in terms of several evaluation metrics. The proposed method has strong competitive abilities and promising prospects.

1. Introduction

Machine learning (ML) is an inevitable outcome of the development of artificial intelligence research. In the past two decades, the ability of humans to collect, store, transmit, and process data has been rapidly improved. ML attracts much attention since it can effectively analyze and utilize massive data [1]. According to whether the training data has labels, learning algorithms are divided into unsupervised learning and supervised learning [2]. Clustering [3] is representative of the former, while regression and classification are representative of the latter. The most common supervised learning algorithms include decision tree [4], naïve Bayes [5], k-Nearest Neighbor [6], neural networks [7], and support vector machine [8]. Among them, SVM was formally published in 1995 [9]. SVM is based on the statistical learning theorem and structural risk minimization principle to design the learning machine, which can not only tackle linear problems, but also tackle nonlinear problems using kernel methods. So far, SVM has provided powerful technical support for many areas due to excellent performance, including face recognition [10], intrusion detection [11], sound recognition [12], text classification [13], etc. Although SVM has many advantages, it also has some limitations. For instance, it is sensitive to the initial values of hyperparameters. These hyperparameters can affect the complexity and generalization ability of the model. Besides, the serious obstacle that all ML algorithms face is the curse of dimensionality [14]. If a dataset contains n features, the number of available solutions increases exponentially with the number of features, resulting in 2ⁿ solutions being generated and evaluated. It increases the computational cost and affects the performance of the classier.

Feature selection (FS) is a significant data preprocessing process. FS can reduce the difficulty of learning tasks by eliminating irrelevant or redundant features. If the subsequent learning process only constructs the model on these selected features, the curse of dimensionality can be significantly alleviated. Traditional subset search algorithms such as complete search and sequential search cannot achieve better output for the high computational cost. Many researchers have demonstrated the effectiveness of using metaheuristic algorithms (MAs) to solve complex optimization problems [15]. In 2020, Faramarzi et al. proposed the marine predators algorithm, which simulated the predation behavior of marine organisms based on Brownian motion and Lévy flight [16]. MPA has the features of few parameters, simple implementation, and flexibility in improving the version. Al-qaness et al. [17] presented an improved version of the adaptive neuro-fuzzy inference system (ANFIS) model to forecast the number of people infected by COVID-19 in four countries. In this work, MPA was used to optimize the ANFIS weights that lie between Layers 4 and 5. Over all the experiments, MPA-ANFIS outperformed all compared models on several measures. Zhong et al. [18] proposed a teaching-learning-based marine predator algorithm (TLMPA). This algorithm made full use of the good global search capability of teaching-learning-based optimization (TLBO) and the fast convergence capability of MPA. In addition, the cross mutation strategy of the differential evolution algorithm was also added to increase the diversity of the population. The comprehensive evaluation on IEEE CEC-2017 functions and engineering problems verified the outstanding performance of TLMPA. Elaziz et al. [19] hybridized MPA and moth flame optimization (MFO) based on their features. The proposed algorithm was called MPAMFO. This algorithm is first applied to segment medical images, such as COVID-19 CT images. Evaluation outcomes showed that the MPAMFO outperformed other algorithms in terms of structural similarity index (SSIM), peak signal-to-noise ratio (PSNR), and fitness value.

Due to the more flexible and complex application areas, researchers have higher requirements for the native algorithm. No Free Lunch also turns out that the world is dominated by real-world problems without a known provably efficient algorithm [20]. Hence, researchers either continue to study new algorithms or improve the native ones [21]. In this paper, a co-evolutionary cultural mechanism-based marine predators algorithm is proposed. This mechanism divides the population into different sub-populations. The sub-populations evolve in parallel in their respective spaces based on MPA. The designed operators can coordinate the knowledge and experience among sub-populations. This improved version CECMPA increases the diversity of the population and the probability of avoiding trapping into the local optimum. For performance evaluation, CECMPA is used as a subset search algorithm while optimizing the hyperparameters of SVM. The simultaneous implementation of the two phases not only reduces the computational cost, but also improves the performance of the classifier. Specifically, the evaluation on the COVID-19 dataset is also of real great value.

The remainder of this paper is organized as follows: Section 2 presents some related works. Some theoretical background is introduced in Section 3. Section 4 highlights the proposed method. Experiments are implemented and results are analyzed in Section 5. Section 6 applies the proposed method to a COVID-19 dataset. Finally, conclusions and future researches are given in Section 7.

2. Related Work

Throughout the years, MAs have outstanding performance in various real-world or academic problems. A research extends the application of MAs to data classification and feature selection.

GA is one of the most classical evolutionary algorithms that simulates Darwinian natural selection. Gauthama Raman et al. [22] introduced the hypergraph into GA and proposed HG- GA for FS and parameter optimization in SVM. The hyper clique property was used to generate the initial population to prevent GA from trapping into the local optimum. The performance on intrusion datasets showed that HG-GA SVM performed better than comparators. Huang et al. [23] proposed a method called PSO-SVM to achieve the FS and kernel parameter setting of SVM. The proposed method was implemented with a distributed parallel system to reduce the training time. Baliarsingh et al. [24] proposed a method known as memetic algorithm-based SVM (M-SVM), which was inspired by embedding social engineering optimizer (SEO) in emperor penguin optimizer (EPO). SEO was considered a local search strategy, and EPO was used as a global optimization framework. The experiment was analyzed from two aspects, including binary-class classification and multi-class classification. The numerical results showed that M-SVM outperformed other comparison methods regarding gene selection and accuracy for microarray datasets. In 2018, a hybrid method based on the grasshopper optimization algorithm (GOA) was presented by Aljarah et al. [25] to find the optimal features and appropriate hyperparameters in SVM. The proposed method outperformed traditional grid search and seven well-regarded methods. The designed experiments used only one dataset with 100 features. For large-scale classification problems, the proposed method has not been further tested in this paper.

In [26], simulated annealing (SA) was hybridized into whale optimization algorithm (WOA) to enhance the global search ability of WOA. The two versions are called WOASAT-1 and WOASAT-2, respectively. Through validation, WOASAT-2 achieved better results regarding classification accuracy and feature reducts. In [27], the hybrid of seagull optimization algorithm (SOA) and thermal exchange optimization (TEO) was applied to function optimization and FS. The authors proposed three different levels of hybrid versions. Through the comparison on UCI datasets, the third version SOA-TEO₃ could better train the classifier to distinguish classes. MAs applied in this area also include salp swarm algorithm (SSA) [28], TLBO [29], harris hawks optimization (HHO) [30], etc.

3. Theoretical Background

3.1 Support vector machine

Given the dataset D = { ( x ₁ , y ₁ ), ( x ₂ , y ₂ ), ..., ( x _m , y _m ) } , y _i ∈ { − 1, + 1 } , the basic idea of classification learning is to find a hyperplane and divide the samples into different classes. The optimal hyperplane can be described by the following linear equation:

h( x ) = ω^Tx + b = 0 (1)

where ω determines the direction of the hyperplane, and b determines the distance between

the hyperplane and the origin. If y _i = +1, then h( x ) ≥ + 1 ; if y _i= −1 , then h( x ) ≤ −1  . The sample1 point where h( x ) = ± 1 is true is called the support vector. The sum of the distances of two different classes of support vectors to the hyperplane is called the margin. Fig. 1 shows an example of a two-class SVM classification model using an optimal hyperplane.

Fig. 1. Linear SVM.

Further, maximizing margin is equal to minimizing ω. Introducing the slack variable ξ, ξ> 0 represents that there are a small number of outliers. The penalty factor c is a hyperparameter that represents the tolerance to outliers. The basic model of SVM is as following:

   (2)

For the nonlinear case, sample x can be mapped from the original space to the high- dimensional feature space. The mapped feature vector is φ(x)^T. Because it is hard to calculate the inner product φ ( x_i ) φ ( x_j ) , the kernel function k ( x_i , x_j ) is defined. Specifically, the inner product of x_i and x _j in the feature space is equal to what they calculated in the original space by the kernel function. Then, the model of SVM can be transformed into the following dual problem:

   (3)

where _iα represents the Lagrange multiplier.

In this paper, we adopt a widely useful radial basis function (RBF) kernel. Its expression is: 

   (4)

where γ is a kernel parameter which represents the width of the kernel function.

The penalty factor c and kernel parameter γ can affect the complexity and generalization ability and of SVM. This paper will use the proposed improved algorithm to optimize the two hyperparameters.

3.2 Marine predators algorithm

MPA is a novel swarm-inspired MA, which simulates the phenomenon of marine organisms looking for food. While the predator in the ocean is searching for prey, the prey is also searching for its food. According to the velocity ratio of predator and prey, the search process is defined as three phases. Here, the predator with the most robust foraging ability is called the elite. Like other swarm intelligence optimization algorithms, the final aim of MPA is to obtain the optimal solution (elite) according to the designed position update methods. The specific implementation is as follows:

   (5)

where P=0.5 , Elite represents the position of the elite, Prey represents the position of the prey, n represents the size of the population, R_B represents a vector based on Brownian motion, and R represents a random number in [0,1].

Phase 2: When the velocity of the prey matches that of the predator, the prey follows Brownian motion, and the elite follows Lévy flight. This process occurs in the middle of iterations and is in the exploration and exploitation phases of MPA. Half of the population is responsible for exploration using (6). The other half is responsible for exploitation using (7). 

   (6)

   (7)

where R_L represents a vector based on Lévy flight, and CF =( 1 − t / t_max )^(2·t/t_max).

Phase 3: In the last third of iterations, the velocity of the predator is higher than that of the prey. The exploitation phase of the algorithm is further extended.

   (8)

In addition to the above three phases, MPA also considers the Fish Aggregating Devices (FADs) effects. Equation (9) is used to simulate the prey jumping from one environment to another with a certain probability. It can effectively prevent the algorithm from trapping at the local optimum.

   (9)

where FADs= 0.2 , x _min and x_max represent the lower and upper boundaries, r 1 and r 2 represent two random indexes of the prey. If the random number in [0, 1] is less than FADs, then U= 0; otherwise, U= 1.

3.3 Co-evolutionary cultural mechanism

The co-evolutionary cultural mechanism, as shown in Fig. 2, divides the population into different spaces. Each subpopulation evolves respectively in its own space through knowledge sources. The spaces also share accumulated experience with the lapse of iteration. The mechanism can be described as two basic operators: cultural operator and co-evolutionary operator. Firstly, the population of size n is divided into four subpopulations using (10). There are n₁₁ agents in the belief space B1, n₂₁ agents in the belief space B2, n₁₂ agents in the population space P1, and n₂₂ agents in the population space P2. The accept behaviors are implemented from the population space to the belief space according to the accept probability Acp. If rand≤Acp, the best agent in P1 replaces the worst agent in B1; the best agent in P2 replaces the worst agent in B2. The affect behaviors are also implemented from the belief space to the population space according to the affect probability Afp. If rand≤Afp, the Afn excellent agents in P1 replaces the Afn bad agents in B1; the Afn excellent agents in P2 replaces the Afn bad agents in B2 [31].

   (10)

Fig. 2. Flowchart of the co-evolutionary cultural mechanism.

Secondly, the shared global belief (SGB) space is established based on (11). Then, the average fitness of agents in SGB space is calculated. The agents whose fitness is larger than the average fitness are reserved, and the agents whose fitness is less than the average fitness are reinitialized. The affect behaviors are implemented from the SGB space to the population space according to the affect probability SAfp. If rand≤SAfp, the SAfn₁ excellent agents in SGB replaces the SAfn₁ bad agents in P1, the SAfn₂ reinitialized agents in SGB replaces the SAfn₂ bad agents in P1; the SAfn₁ excellent agents in SGB replaces the SAfn₁ bad agents in P2, the SAfn₂ reinitialized agents in SGB replaces the SAfn₂ bad agents in P2. After the above affect behaviors, the experience exchange behaviors are implemented between P1 and P2 according to the exchange probability EEp. If rand≤EEp, the EEn excellent agents in P1 replaces the EEn bad agents in P2; the EEn excellent agents in P2 replaces the EEn bad agents in P1 [32].

n₀ = n ₁₁ + n ₂₁ (11)

In [33], the parameter guideline is given through a series of orthogonal experiments. (0.7, 0.2, 4, 0.7, 10, 10, 0.5, 10) is an appropriate combination of (Acp, Afp, Afn, SAfp, SAfn₁, SAfn₂, EEp, EEn).

4. Proposed Method

4.1 Improved marine predators algorithm

MPA has such features of few parameters, simple implementation, and flexible improvement. However, it can be observed from (5)-(8) that the movement of prey is mainly based on Brownian motion and Lévy flight. These strategies have a certain degree of blindness. The solution space in real-word problems is usually more complex, which increases the probability of the native algorithm trapping at the local optimum. The co-evolutionary cultural mechanism can enhance the diversity of the population through "accept operators" and "affect operators", and provide the exchange of experience and knowledge for the exploitation of the native algorithm. Under this framework, the improved algorithm CECMPA can robustly avoid local optimum and improve the accuracy of the final solution. Besides, the solution of each space evolves respectively based on MPA. CECMPA does not damage the exploration of the native MPA. Algorithm 1 provides the pseudo-code of CECMPA.

Algorithm 1 Pseudo-code of CECMPA.

4.2 CECMPA for simultaneous FS and SVM optimization

No matter what kind of swarm-inspired MAs, the phenomenon simulated by the algorithm is to solve how to generate a new population with the lapse of iteration. The updating method of the population has been presented in the above section. Then, the two core issues of applying the optimization algorithm to the real-world area are the representation of the solution and the definition of the objective function. Once the two issues are solved, the application can be transformed into an optimization algorithm to solve the objective function.

In this paper, each solution consists of the penalty factor c, the kernel parameter γ, and features in the dataset as shown in (12). When designing a SVM model based on the RBF kernel, the penalty factor c controls the tolerance for errors of the model, and the kernel parameter γ determines the number of support vectors. Therefore, the hyperparameters combination ( c, γ) can significantly affect the classification performance and generalization ability of the model. In FS tasks, all features ( f ₁ , f ₂ , ... , f_D  ) are encoded [0, 1]. If the value of the element is within [0.5, 1], the feature is reserved; if not, the feature is excluded. Most traditional methods only perform FS or tune hyperparameters. But in fact, both of them affect each other in improving the performance of the classifier. Simultaneous FS and hyperparameters optimization can not only reduce the experimental cost, but also fully explore the potential of the classifier.

The objective function can evaluate the quality of the solution. FS aims to minimize the number of selected features and maximize the classification accuracy. The objective function in (13) that combines these two factors is used in this paper.

   (12)

   (13)

where f represents the feature in the dataset, E_R is the error rate of SVM, |R| is the number of the selected features, D is the total number of the features of the dataset, and α= 0.99 [34]. The proposed method first preprocesses the dataset. After that, the dataset is divided into the training set and the testing set based on 10-fold cross validation. CECMPA executes the random vectors generated by (14). Then, SVM performs the training process on the training set. During this phase, the inner cross validation is carried out to produce unbiased classification results. CECMPA receives the fitness value at the end of the training process. All steps are repeated until the stop criterion is met. Finally, the proposed method reports the optimal solution. The final optimized hyperparameters and selected features are applied to the testing phase. The flowchart is shown in Fig. 3.

Fig. 3. Flowchart of the proposed method.

5. Experimental results and analysis

5.1 Dataset and parameters setting

Twelve UCI datasets [35] are used to evaluate the performance of the proposed method. Table 1 shows the number of features, instances, and classes in each dataset. Before executing the program, the dataset is preprocessed through two phases. Firstly, the logical features are converted into numeric form. And then, the min-max normalization scales the numeric features to [0,1]. In this way, the effect of numerical magnitude on feature weights can be avoided. LIBSVM is a well-known SVM toolbox [36]. 10-fold cross validation stratified resampling is performed to alleviate overfitting and underfitting in ML.

Table 1. Details of each UCI dataset.

The proposed method is compared with MPA and five other FS methods including PSO [23], GOA [25], SSA [28], WOASAT-2 [26], and SOA-TEO3 [27]. Their parameters are shown in Table 2. The agent size of P1 and P2 are all 30, and the agent size of B1 and B2 are all 15. For the eight parameters in CECMPA, we follow the guideline in [35], as given in subsection 3.3. The maximum number of iterations is 100. Each experiment is run 10 times and carried out on MATLAB R2016b.

Table 2. Parameter settings of each compared method.

5.2 Evaluation metrics

1. Classification accuracy: this metric evaluates the accurate of SVM in predicting the right class using selected features. And the mathematical equation is as follows:

   (14)

where Ⅱ( ⋅) is the indicator function. When the input is true, the output is 1, and when the input is false, the output is 0.

2. Feature size: this metric evaluates the size of the optimal feature subset obtained by the search algorithm. And the mathematical equation is as follows:

   (15)

3. Fitness value: this metric combines the above two factors as the objective function. Please refer to (13) for the mathematical expression.

4. Computational time: this metric reflects the memory cost and execution speed of the method.

5. P-value: this metric provides information about whether a statistical hypothesis test is significant or not, and it also indicates something about how significant the result is.

5.3 Comparison with other methods

Table 3 presents the average classification accuracy of each method on each dataset. The proposed method achieves the highest results on nine out of twelve cases. It is quite impressive that CECMPA obtains 100% classification accuracy on "Divorce Predictors" dataset. Taking "Sonar" dataset as an example, the classification accuracy of the improved algorithm is at least 6.57% higher than that of the native MPA. It demonstrates that the SVM model optimized by CECMPA can more accurately predict the classes of samples. In the case of the "Lung cancer" dataset, WOASAT-2 is better than CECMPA with a slightly different result. Table 4 presents the average feature size of each method on each dataset. The proposed method provides the smallest feature reduction on 66.67% of the datasets used. This shows that the proposed subset search algorithm CECMPA can more effectively eliminate irrelevant or redundant features. On the "Zoo" dataset, CECMPA provides 99.59% classification accuracy by using 4.8 features only, whereas SSA provides 94.05% classification accuracy and 5.6 features. Table 5 presents the average fitness value of each method on each dataset. The objective function in (13) is a minimization problem. Observing the results obtained, the proposed method contributes to the lowest fitness value on most datasets. It proves that CECMPA can deeply explore areas in the feature space to obtain the optimal solution. The introduction of the co-evolutionary cultural mechanism reduces the risk of the native MPA trapping into the local optimum. The stability of the solutions across the different runs is a very important issue when MAs are used. Table 6 presents the standard deviation of the fitness value. It can be seen that the optimal values (8 out for 12 cases) can be found by CECMPA. Observing the results of this comprehensive metric, the standard deviation of the solutions achieve a consistent ranking, indicating the stability of CECMPA. The average computational time of each method on each dataset is given in Table 7. In terms of the "CongressEW" dataset, the average computational time is sorted as follows: PSO>WOASAT-2>MPA>CECMPA>SSA>SOA-TEO₃>GOA. From Table 7, the computational time of CECMPA is ranked in the middle of the seven methods on nearly all datasets. Besides, the computational time of CECMPA is slightly lower than that of MPA. Due to parallel evolution in different spaces, the proposed method can search the global optimum quickly while balancing exploration and exploitation. The computational time is especially important for real-world problems. Therefore, the proposed method guarantees high performance in an acceptable time.

Table 3. Comparison the average classification accuracy.

Table 4. Comparison the average feature size.

Table 5. Comparison the average fitness value.

Table 6. Comparison the standard deviation of fitness value.

Table 7. Comparison the average computational time (s).

Finally, some nonparametric statistical tests are carried out. In the area of inferential statistics, the null hypothesis H₀ represents the distribution of no difference. The level of significance α decides the probability of H₀ being rejected. In this section, pairwise comparison such as Wilcoxon rank-sum test detects significant differences between the two methods. α is stipulated as 0.05 [37]. When p-value<0.05, the results are statistically different. The smaller p-values are stronger against H₀. The p-values are reported in Table 8. P-values obtained are less than 0.05 on most cases, seen from Table 8. It demonstrates that the superiority of our improvement is statistically significant. So the null hypothesis H₀ can be rejected. Considering the comprehensive results of this section, the evaluation of the proposed method in this paper on the UCI dataset is positive.

Table 8. P-value obtained based on Wilcoxon rank sum test.

6. Real-world application

The COVID-19 pandemic caused by severe acute respiratory syndrome coronavirus 2 (SARS- CoV-2) emerged in late 2019. Even over one year has passed since the COVID-19 onset, the confirmed cases and deaths still continue to increase significantly, and more than 90, 000, 000 people have been affected all over the world as of Jan. 2021. Besides, public health systems in most countries have been affected unprecedentedly. The clinical symptoms for individuals with confirmed COVID-19 include fatigue, cough, fever, and difficulty breathing within an incubation period of 2-14 days after exposure to the virus. At present, ML is the most common treatment for predicting large-scale pandemics. Through the possible prediction outcomes, the infected patients can be treated appropriately [38].

In this section, the proposed method uses the COVID-19 dataset to predict the severity of patients and is further compared and analyzed with other methods. The dataset used is public and available at [39]. To meet this study, the dataset has been pre-processed. It finally includes thirteen features and two targets as shown in Table 9.

Table 9. Details of the COVID-19 dataset.

Fig. 4 shows the classification accuracy and feature size for all the methods stated above. It reveals that CECMPA has the highest classification accuracy of 92.58%. We also discover that the proposed method achieves the prediction of the severity based on roughly four features. Besides, the results show that CECMPA does not selects the symptom 4, symptom 5, and symptom 6. Judging from the result, these features are unprofitable for patient health prediction. The purpose of this case study is to provide radiologists and other clinicians with a diagnostic aid. Under no circumstances is this pilot study a substitute for medical advice.

Fig. 4. Accuracy and feature size on a COVID-19 dataset.

7. Conclusion

In this paper, a co-evolutionary cultural mechanism-based marine predators algorithm is proposed for simultaneous FS and hyperparameters optimization in SVM. Considering the features of MPA, the co-evolutionary cultural mechanism is used to improve the performance of the native algorithm in tackling ML problems. Twelve UCI datasets are used to evaluate the performance of CECMPA. Our findings show that the proposed method achieves similar or even higher classification accuracy in 75% of datasets, and searches the smaller optimal feature subset in 66.67% of datasets. Wilcoxon rank-sum tests are conducted to validate the significancant improvement. Besides, the proposed method is also used to predict the COVID- 19 pandemic. According to the obtained results, roughly four factors mainly affect the death and recovery conditions of patients for CECMPA. Therefore, we achieve the real-world goals of proposing a novel ML method in this paper.

For further research, the framework can try other belief spaces based on any evolutionary or swarm intelligence algorithm to improve the computational performance and solutions quality of MPA. In the COVID-19 era, CECMPA can also be applied to medical image segmentation to extract regions containing clinical features.