An AutoML-driven Antenna Performance Prediction Model in the Autonomous Driving Radar Manufacturing Process

Abstract

This paper proposes an antenna performance prediction model in the autonomous driving radar manufacturing process. Our research work is based upon a challenge dataset, Driving Radar Manufacturing Process Dataset, and a typical AutoML machine learning workflow engine, Pycaret open-source Python library. Note that the dataset contains the total 70 data-items, out of which 54 used as input features and 16 used as output features, and the dataset is properly built into resolving the multi-output regression problem. During the data regression analysis and preprocessing phase, we identified several input features having similar correlations and so detached some of those input features, which may become a serious cause of the multicollinearity problem that affect the overall model performance. In the training phase, we train each of output-feature regression models by using the AutoML approach. Next, we selected the top 5 models showing the higher performances in the AutoML result reports and applied the ensemble method so as for the selected models' performances to be improved. In performing the experimental performance evaluation of the regression prediction model, we particularly used two metrics, MAE and RMSE, and the results of which were 0.6928 and 1.2065, respectively. Additionally, we carried out a series of experiments to verify the proposed model's performance by comparing with other existing models' performances. In conclusion, we enhance accuracy for safer autonomous vehicles, reduces manufacturing costs through AutoML-Pycaret and machine learning ensembled model, and prevents the production of faulty radar systems, conserving resources. Ultimately, the proposed model holds significant promise not only for antenna performance but also for improving manufacturing quality and advancing radar systems in autonomous vehicles.

1. Introduction

In recent years, the rapid advancement of technology and active research into artificial intelligence and the Internet of Things (IoT) has led to an increase in Research and development in the manufacturing industry [1][2]. These efforts aim to build smart factories and predict and improve manufacturing processes more efficiently and with higher quality, using machine learning and deep learning technologies. In the manufacturing process, lots of various data is generated, such as images, log data, and videos. Among them, research is active to improve efficiency through classification and regression prediction based on deep learning and machine learning using sensor-based process log data generated during the manufacturing process [3].

For instance, there have been several cases in which a machine learning-based method has been used to diagnose and classify defects in steel plate [4], and Research is being conducted to diagnose fabric defects or to improve the quality of textile processes efficiently using data generated during the textile process [5][6].

In particular, one significant area in which machine learning is being applied is the development of self-driving vehicles, including electric cars, robo-taxis, and self-driving robots [7]. In such vehicles, radar is an essential sensor used to measure critical factors such as distance to other vehicles, relative speed, and direction. Consequently, various studies are being conducted to improve the quality and performance of radar in the manufacturing process [8][9][10][11].

Therefore, in this paper, we propose an Antenna Performance Prediction Model of Autonomous Driving Radar Manufacturing Process using Automated Machine Learning (AutoML). Our study uses a Driving Radar Manufacturing Process Dataset that predicts antenna performance to improve the process quality of autonomous driving radar, with 56 input-features and 14 output-features, making it a multi-output regression problem. During the data regression analysis and preprocessing phase, we identified several input features having similar correlations and so detached some of those input features, which may become a serious cause of the multicollinearity problem that affect the overall model performance. In the training phase, we train each of output-feature regression models by using the AutoML-Pycaret approach which is an open-source machine learning library based on Python. To solve the multi-output regression problem, we selected the top 5 models showing the higher performances based upon RMSE in the AutoML-Pycaret result reports and applied the ensemble method so as for the selected models’ performances to be improved. In our experimental performance evaluation, we used MAE and RMSE as performance evaluation metrics for the regression prediction model, and achieved results of 0.6928 and 1.2065, respectively.

The next of this paper is organized as follows: The following section, we describe the conceptual background of the radar manufacturing process, which is the data domain used in this study, and AutoML, the method used in the proposed model. Subsequent sections explain the Analysis and Pre-processing of the Radar Manufacturing Process Dataset and the Antenna Performance Prediction Model of Autonomous Driving Radar Manufacturing Process we developed. Finally, we compare the experimental results of the model proposed in this paper with other deep learning and machine learning models such as XGBoost, Random Forest, and TabNet to show the performance excellence of the proposed model. The last section then presents a summary of conclusions and future work.

2. Related Works

2.1 Theoretical background

Autonomous Driving Radar: Radar is an electromagnetic wave-based sensor [12]. It emits electromagnetic waves and extracts information such as distance from nearby objects, relative speed, and direction based on the reflected signal. Radar was developed to acquire distance information from airborne objects to ground observation posts via long-range wireless antennas. Over time, as its applications expanded, radar technology advanced to obtain information on the speed, size, and even shape of moving objects. Radars equipped with antennas are capable of transmitting and receiving electromagnetic wave signals, and to obtain distance information on stationary or moving targets, they must receive the reflected waves from the emitted electronic board [13]. Recently, with the development of radar technology, radar not only measures short, medium, and long distances depending on frequency, but various products such as indoor and 4D imaging radars have appeared and are being applied to various ADAS technologies like Self-driving car, robotics, smart cruise control, and emergency automatic control device [14][15]. As these sensors are applied to various technologies, there are many problems that can occur if there is a defect in the data product. This is because, for example, a radar is a defective product and cannot be recognized during autonomous driving, which can lead to a safety accident, such as causing great loss of life.

Therefore, many companies go through the performance evaluation process to sort out defected products in the radar manufacturing process and strive to improve the quality of radar products and reduce product defects. Although the method and evaluation process for evaluating antenna performance may vary depending on the antenna manufacturing process, in general, the evaluation of antenna performance involves the detection of defects by assessing the RLC values of each Surface Mount Technology (SMT) component through the test points on the PCB. To ensure proper module operation, fundamental functions are verified by applying input power. Following the management of various components, an evaluation process is conducted to assess the impact on product performance by considering process and component characteristics.

Multicollinearity [16]: This means a strong correlation between independent variables. When performing regression analysis, since the precondition of regression analysis is that the explanatory variables are independent of each other, the assumption that the explanatory variables are independent of each other must be established to make the estimation of the regression coefficient more accurate. In the process of regressive analysis, if there is a correlation between explanatory variables and independence is doubtful, the presence or absence of multicollinearity should be determined. In order to solve this multicollinearity, you can remove independent variables with high correlation, transform variables, or used new observations. Alternatively, there is a method of selecting variables with high explanatory power using the principal component analysis method. Many Studies have been conducted to improve the performance of the model by resolving this multicollinearity problem [17][18].

Automated Machine Learning (AutoML) [19]: This is a rapidly growing field in artificial intelligence and machine learning that focuses on creating tools and techniques that can automate the process of building and deploying machine learning models. There are a few key components of an AutoML system. First, it must be able to automatically preprocess data, selecting and transforming the features of the dataset that are most relevant to the problem at hand. This includes handling missing data, encoding categorical features, and scaling continuous features. Next, an AutoML system can automatically select the appropriate algorithm or set of algorithms to use for the given problem. This may involve comparing the performance of multiple algorithms on a given dataset, or even combining multiple algorithms to create an ensemble model. Finally, AutoML can also improve the quality of machine learning models. By automating the process of model selection and hyperparameter tuning, it can potentially identify better performing models than a human expert would be able to find manually. The dataset used in this study is autonomous antenna manufacturing process data for antenna performance prediction. Since AutoML has the advantage of maximizing efficiency and productivity by automatically applying various model algorithms in the machine learning development process, there are many cases of using AutoML to solve various classification and regression prediction problems, as well as research and development. In this research, the dataset we used is structured tabular data and includes multiple input features and multiple output features. Therefore, AutoML functionalities that enable hyperparameter tuning and the application of various machine learning algorithms can be used to develop a high-performance antenna performance prediction model. Additionally, we choose AutoML to easily and quickly implement ensemble techniques which combine the predictions of multiple models to enhance predictive accuracy and model robustness as utilized in this paper. Also, there are various libraries such as AutoKeras, AutoGLon, H2O, and Pycaret to use AutoML, and this study uses the Pycaret library [20].

2.2 Previous study

As previously mentioned, the presence of defects in the autonomous driving radar system can render recognition during autonomous driving impossible. This can lead to safety accidents causing severe casualties, as well as significant time and financial losses of manufactoring process. Consequently, several studies have been conducted to evaluate and predict the performance of radar antennas [21][22][23]. In recent, research and development to predict and analyze product defects through correlation analysis between manufacturing process data and product performance using artificial intelligence technologies such as deep learning and machine learning and maximize yield to reduce product disposal costs due to defects. For instance, Boya Liu et al. [24] proposed Bayesian network diagnosis method based on multisource information fusion technology to solve the fault diagnosis problems during the radar manufacturing process. In addition, there is prior study to predict antenna performance defects using deep learning although it is not an antenna used in the autonomous driving radar manufacturing process. They proposed a method for predicting antenna performance and defect location using DNN, a basic deep neural network deep learning structure [25].

3. An Antenna Performance Prediction Model of Autonomous Driving Radar Manufacturing Process

In this section, we describe the overall procedure to predict the antenna performance, and the details of the proposed antenna performance prediction model in the autonomous driving radar manufacturing process. We utilized AutoML to implement the proposed model, specifically choosing the AutoML-PyCaret library from available AutoML libraries like AutoKeras, AutoGLon, H2O. Because Pycaret is not only easy to use, but also, we can use it to compare the performance of multiple models, as well as perform hyperparameter tuning and blending multiple models.

Our study is based upon a dataset generated from the autonomous driving radar manufacturing process, which comprises 56 input features and 14 output features, thus constituting a multi-output regression problem. To carry out this study, it generally consists of a data analysis and preprocessing phase, an antenna performance prediction model training phase, and a performance evaluation and comparison phase to verify the performance of the model.

First, during the data analysis and preprocessing phases, we identified input features with high correlations and consequently removed some of them to mitigate the potential multicollinearity issues that could adversely affect the overall model performance. Second, in the training phase, (1) we individually trained regression models for each of the 14 output features using AutoML-PyCaret. Upon completion of this training process, we obtained 14 distinct result reports. (2) From these reports, we selectively blended the top-performing models, which were determined based on the root mean square error (RMSE). This careful selection ensured that the resulting ensemble models consistently delivered the highest prediction accuracy for each output feature. (3) While predicting each output feature in isolation yielded impressive results, our ultimate objective was to develop a robust multioutput regression prediction model capable of simultaneously forecasting all 14 output features. To achieve this, we perform ensemble learning with the 5 models that frequently show the highest performance based on RMSE among each of the 14 model outputs. Later, we conduct an evaluation and comparison experiment of the proposed model through experiments to verify the performance of the model in section 4.

Fig. 1 is the overall development process for the antenna performance multi-output regression prediction model of autonomous driving radar manufacturing process.

Fig. 1. Overall model development framework

3.1 Driving Radar Manufacturing Process Dataset

The dataset used for this study is the Manufacturing Process Data [26] generated from the autonomous driving radar process provided by Dacon.io. Since the dataset we used in this study is a challenge e dataset provided by a company, data description such as the data's collection date and evaluation metrics are not disclosed, but it is a reliable, standardized dataset. The dataset contains the total 70 data-items, out of which 54 used as input features and 16 used as output features, and the dataset is properly built and well-fitted into resolving the multi-output regression problem. And the total number of the dataset is 39,607.

The 56 input-features are composed of various values such as pressing amount, weight and area of heat dissipation material, screw insertion length during the autonomous driving radar manufacturing process. In Table 1, shows the information on detailed input-features.

Table 1. Input-features which are independent variables used for learning.

The 14 output-features are the antenna performance values of the radar sensor in the autonomous driving radar manufacturing process. We intend to predict these 14 antenna performance values in this study. Table 2 shows detailed information about the 14 output-features we need to predict.

Table 2. Output-features which are dependent variables for predicting.

3.2 Data Analysis and Preprocessing

As a result of the analysis, there is a group of similarly correlated features, which can cause multicollinearity problems that can negatively affect performance due to high correlations between input features.

Fig. 2 shows the kernel density estimation (KDE) [27] between input-features with similar distributions. In order to data preprocessing, first, the input-feature values of X_04, X_23, X_47, and X_58, which indicate whether the test passed, all have a value of 1, so we removed the features. And since the values of X_03, X_10, and X_11 input-features, which means the weight of the heat dissipating material, mostly have a value of 0, they are measured incorrectly, so we removed the features as well. In addition, we remove some input-features that show high correlation between groups with similar distributions and proceed with training with a total of 34 preprocessed input-features.

Fig. 2. Data Analysis - kernel density estimation for input-feature

3.3 Antenna Performance Prediction Model using AutoML

In this study, we train the antenna performance prediction model using AutoML-Pycaret library, which automatically trains all machine learning models and drives optimized performance. We check AutoML-Pycaret result reports for Y_01 through Y_14. Then, we blend the models with the top 5 models that show the highest performance of AutoML-Pycaret based upon RMSE.

Table 3 indicates the results for Y_01 among the 14 AutoML-Pycaret result reports. Afterwards, in order to further improve the performance of the model and predict the 14 output-feature simultaneously, we conduct ensemble learning with the 5 models that frequently show the highest performance based on RMSE among each of the 14 model outputs. In other words, we developed the final ensemble model by performing hyperparameter tuning on random forest, LGBM, gradient boosting regressor, ridge, and lasso models, which most frequently showed high performance based upon RMSE in 14 AutoML-Pycaret result reports and performing ensemble learning using them.

Table 3. AutoML-Pycaret result report for output-feature Y_01

4. Experiments and Results

4.1 Experiment environments

The proposed antenna performance prediction model of autonomous driving radar manufacturing process is implemented in the experiment environment of operating system and hardware as shown in Table 4.

Table 4. Experiment environment

We use Ubuntu 20.04 LTS, AMD Ryzen 7 5800X 8-Core Processor, RAM 32GB, and NVIDIA GeForce RTX 3070. As for software, we use Scikit-learn, Pytorch, and Pycaret library for the proposed model implementation and comparison experiments. Scikit-learn Library is a Python library for machine learning. It provides a range of supervised and unsupervised learning algorithms for tasks such as classification, regression, clustering, and dimensionality reduction, as well as tools for model selection, preprocessing, and evaluation. We use Scikit-learn to implement the proposed model and machine learning models such as Decision Tree [28], Random Forest [29], LGBM [30], Linear Regression [31], XGBoost [32], and to evaluate and compare the model performances. Also, we use the Pytorch library to implement the TabNet [33] deep-learning model for comparison with our proposed model. We installed CUDA, including CUDA 11.2 and cuDNN 8.1, to accelerate GPU for fast learning of deep learning models. Lastly, the model we proposed uses AutoML to train machine learning models, and select 5 models with the best performance for ensemble learning. To implement this, the Pycaret library, which provides AutoML function, is used.

4.2 Evaluation Metrics

In this study, we implement the model that predicts antenna performance in Autonomous Driving Radar Manufacturing Process. The model training is based on learning from the performance values (y_i-output features) obtained through the evaluation process and the log values (x_i-input features) generated during antenna manufacturing. In other words, the model is a multiple regression problem that obtains 14 output features, such as the average and deviation of antenna gain and signal-to-noise ratio, which are factors in predicting antenna performance. Therefore, mean absolute error (MAE) and root mean square error (RMSE) performance indicators used in multiple regression analysis are used [34].

In a regression problem, MAE is a metric that measures the average absolute difference between the predicted values y_i and the actual values \(\begin{align}\hat{y}_{i}\end{align}\). It gives equal weight to all errors, regardless of their magnitude or direction. MAE is calculated by taking the absolute difference between each predicted and actual value, summing those differences, and dividing the sum by the total number of observations. The formula for MAE is presented (1) as below.

\(\begin{align}\mathrm{MAE}=\frac{1}{N} \sum_{i=1}^{N}\left|y_{i}-\hat{y}_{i}\right|\end{align}\)       (1)

Root Mean Square Error (RMSE) is similar to MAE in that it measures the difference between predicted values y_i and actual values \(\begin{align}\hat{y}_{i}\end{align}\), but RMSE takes into account the squared values of these differences which gives more wight to large errors. And since it is sensitive to both small and large errors, RMSE provides a comprehensive view of the model’s overall performance and can be useful metric for comparing different models. RMSE measures the square root of MSE metric. MSE quantifies the squared errors between predicted values y_i and actual values \(\begin{align}\hat{y}_{i}\end{align}\) for each data point. Since it squares the errors, it transforms them into positive numbers and is highly sensitive to larger errors. Equation (2) presents the formula of MSE and (3) shows the formula for RMSE which is calculated by taking the square root of the average of the squared differences between predicted and actual values.

\(\begin{align}\mathrm{MSE}=\frac{1}{N} \sum_{i=1}^{N}\left(y_{i}-\hat{y}_{i}\right)^{2}\end{align}\)       (2)

\(\begin{align}\text {RMSE}=\sqrt{M S E}=\sqrt{\frac{\sum\left(y_{i}-\hat{y}_{i}\right)^{2}}{N}}\end{align}\)       (3)

4.3 Experiment results

In this paper, we develop prediction models for each of the 14 output-features by boosting the model using AutoML-Pycaret. we then developed a final multi-output regression prediction model by ensemble learning with the five machine learning models that showed the highest performance based on RMSE metric from AutoML-Pycaret result reports.

We can see the MAE and RMSE performance results of the one-output regression model that predicts 14 output-features one by one using AutoML-Pycaret in Table 5 below. Regarding the output-feature models of Y_04, Y_05, Y_06, Y_10, and Y_11, both the MAE and RMSE results indicated high values in the range of 0.5 to 1.0, but the rest of the prediction models showed results of 0.5 to 0.7 or less.

Table 5. Results of MAE and RMSE performance for each one output-feature model using AutoML-Pycaret

However, when multi-output regression was finally performed to predict 14 output-features, the MAE result was 0.6928 and the RMSE result was 1.2065, which appeared to be relatively high. In order to prove the performance results of the proposed model in this study, we perform a performance comparison with Decision Tree, Random Forest, LGBM, Linear Regression, and XGBoost, which are machine learning models commonly used in multi-output regression models. In addition, we compare performance with the model proposed in this paper by using the TabNet deep learning model, which shows high excellence in tabular data. The models that were comparatively tested are shown in Table 6 below and show relatively high performance compared to the other existing models.

Table 6. Results of MAE and RMSE Performance comparison between conventional models and the proposed model

5. Conclusion and Future Research Works

This paper proposed an antenna performance prediction model in the autonomous driving radar manufacturing process. The proposed model has achieved the MAE of 0.69 and RMSE of 1.19, which are better than the baseline models. Our approach has demonstrated the effectiveness of using AutoML-Pycaret which is an open source typical AutoML machine learning workflow engine based on Python. We developed our final multi-output regression ensemble model by performing hyperparameter tuning on random forest, LGBM, gradient boosting regressor, ridge, and lasso models, which most frequently showed high performance based upon RMSE performance metric in AutoML-Pycaret results reports and performing ensemble learning using them.

In this work, the enhanced accuracy achieved through this research leads to improved safety and reliability for autonomous vehicles on the road. Furthermore, the utilization of AutoML-Pycaret and the optimization of a variety of machine learning models present a significant opportunity to reduce manufacturing costs. Precise prediction of antenna performance plays a pivotal role in preventing the production of faulty radar systems, thereby conserving resources, and minimizing waste in the manufacturing process. As the future works, we plan to explore additional machine learning models and deep learning model to further improve the performance of our prediction model. We will also investigate the impact of different data preprocessing techniques and domain knowledge to enhance the quality of our predictions. Our work has important implications for the automotive industry, as accurate prediction of antenna performance is critical for ensuring the safety and reliability of autonomous vehicles. Conclusively, we expect that the antenna performance prediction model using the AutoML approach and proposed in this paper ought to make a meaningful contribution in terms of not only predicting the antenna performance but also improving the manufacturing process quality in the manufacturing process automation industry.