Damage Detection and Damage Quantification of Temporary works Equipment based on Explainable Artificial Intelligence (XAI)

Abstract

This paper was studied abouta technology for detecting damage to temporary works equipment used in construction sites with explainable artificial intelligence (XAI). Temporary works equipment is mostly composed of steel or aluminum, and it is reused several times due to the characters of the materials in temporary works equipment. However, it sometimes causes accidents at construction sites by using low or decreased quality of temporary works equipment because the regulation and restriction of reuse in them is not strict. Currently, safety rules such as related government laws, standards, and regulations for quality control of temporary works equipment have not been established. Additionally, the inspection results were often different according to the inspector's level of training. To overcome these limitations, a method based with AI and image processing technology was developed. In addition, it was devised by applying explainableartificial intelligence (XAI) technology so that the inspector makes more exact decision with resultsin damage detect with image analysis by the XAI which is a developed AI model for analysis of temporary works equipment. In the experiments, temporary works equipment was photographed with a 4k-quality camera, and the learned artificial intelligence model was trained with 610 labelingdata, and the accuracy was tested by analyzing the image recording data of temporary works equipment. As a result, the accuracy of damage detect by the XAI was 95.0% for the training dataset, 92.0% for the validation dataset, and 90.0% for the test dataset. This was shown aboutthe reliability of the performance of the developed artificial intelligence. It was verified for usability of explainable artificial intelligence to detect damage in temporary works equipment by the experiments. However, to improve the level of commercial software, the XAI need to be trained more by real data set and the ability to detect damage has to be kept or increased when the real data set is applied.

1. Introduction

Temporary works equipment is used in all of construction sites, and it helps construction workers easily and comfortably to work their job. Even though temporary works equipment made of steel and aluminum is several times reused in construction sites, it does not establish a regulation and restriction when it used. From this problem, construction workers sometimes get injured and rarely threatened their life. Even though inspection of temporary works equipment was sometimes proceeded, the results are different depend on a level of trained inspector.

These weaknesses are very crucial, and in order to overcome them, artificial intelligence skill is adapted and developed. Using artificial intelligence (AI), it is possible to detect damaged temporary works equipment and evaluate temporary works equipment instantly and quickly. Moreover, explainable artificial intelligence (XAI) which is developed technique make inspector help evaluation of temporary works equipment with results by the XAI.

This study conducted a study to prove the accuracy of XAI (Explainable Artificial Intelligence), which detects defects in temporary equipment and verifies the condition of temporary equipment. The defect detection accuracy of artificial intelligence was shown using the F-1 score, and the accuracy was verified over 90%, and the verification method for explanation ability was verified using Heatmap detection accuracy and BLEU score, and performance close to 90% was verified. Confirmed. As a result, the performance of the XAI system developed for defect detection and damage quantification of temporary works equipment was demonstrated.

2. Related Research

2.1 Research on laws and regulations related to safety inspection of temporary works equipment

Temporary works equipment at construction sites is mainly composed of steel and is reused several times due to the feature of the material. In case of an accident which occur using temporary works equipment of poor quality, the safety and life of workers are threatened because a construction worker climbs and hangs on the temporary works equipment or works near the temporary works equipment. Therefore, strict quality management of temporary works equipment is required. According to the domestic rule, it has been changed from the performance test system to a stability certification system which is considered more a part of safety since 2009 in order to secure the safety and quality assurance for temporary works equipment. A system that allows the use of safety certification marks is in appropriate case of passed a comprehensive examination in the technical capability, production system, and product performance of the manufacturer based on safety certification of temporary works equipment stipulated in Article 83 of the Occupational Safety and Health Act.[1]

From the performance test system, the details of the conversion to the safety certification system are as shown in Table 1. It is set to be certified by safety certification system with testing and examining a kind of item among temporary works equipment which is installed in dangerous spot such as falls and collapse. However, tested in each kind of item among whole temporary works equipment, in other words sample test, is not verified safety in whole equipment. Moreover, the results are dependable by the inspector who has a level of training and skilled.

(Table 1) Standard changes of temporary works equipment in the South Korea[2]

In the other countries, such as the United States and Europe, have standards and systems for securing the stability of temporary works equipment and guaranteeing quality. ANSI's 'standards for testing and rating scaffold assemblies and components (2002)' [3] in the US, 'temporary works equipment' in the UK [4], and DIN's test standards in Germany to evaluate the performance of temporary works equipment. are doing [5]. However, as for the safety inspection of temporary works equipment, inspection based on visual inspection is in progress as in the South Korea, and it has the same weakness as the safety inspection conducted in the South Korea.

2.2 the Algorithm designed in the System, Mask-R-CNN

Mask-R-CNN [12], which evolved from Faster R-CNN, is an instance segmentation framework. The reason why the AI model was chosen is accuracy, stability, and capacity. This model was developed in 2017, after then it has been broadly used in the field of vision processing. This algorithm is basically divided into two steps.

The first stage is scanning the image and generatingsuggestions about the results by inferencing, and the second stage is classifyingthem and creating bounding boxes and masks among the suggestions. This is similar to the Fast-R-CNN and Faster-R-CNN, but the Mask-R-CNN bas difference because it was designed a branch, which is predicting segmentation masks on each ROI (Region of Interest), and the other branch, which is for classification and bounding box regression. According to the difference, the Mask-R-CNN model was shown the high performances in the accuracy, stability in a system, and time consumption.[12]

(Figure 1) Basic Structure of the System

Since the designed system detects various defect types and size information, the Mask-R-CNN model among several AI analysis algorithmswhich are segmentation detection models was applied, that is suitable for defect quantification, and it is capable of multi-object detection and has excellent performance.[11]

Even though this algorithm is more accurate that the FCN(Fully Convolution Network) and the U-Net which isable to generate segmentations, it requires more time to complete the procedure.[9]

2.2 the Algorithm designed in the System, Grad-Cam++

Grad-Cam++[13] is a technology that expresses explanations of the results of CNN - based models. This model was developed from Grad-Cam and complemented the limitations of the existing model. These include poor performance when localizing multiple occurrences and failure to completely capture the entire object in the case of single object images. The characteristics of Grad-Cam++ are as follows.

The Grad-Cam++ is the pixel-wise weighting of the output gradient applied to mark a specific spatial location in the final convolutional feature map. The important part is to recognize and measure the importance of the feature map in pixel units for the results by the algorithm based on CNN.[13] This method is able to express more advanced visualizations and achieve the same performance as the gradient-based method applied in previous models.

There are many ways to visualize the conclusions drawn by the algorithm based on CNN, but quality assessment is often done by humans or through metrics such aspositioning error for bounding boxes. In this part, the Grad-Cam++ algorithm was applied to evaluate the faithfulness of the explanation and its direct relevance to the visualization.[13]

It was applied to Grad-Cam++ using a method that users can trust more than the previous model (Grad-Cam) for the results by the algorithm based on CNN.

When delivering the results by the CNN regarding explanations, the part is able to be refined and delivered to the user as much as possible.[13]Captioning results generated in Grad-Cam++ show excellent results by maintaining the description methodology and improving performance by using a specific loss function.

Existing models were mainly limited to 2D image data, but Grad-Cam++ is an efficient model for image capture and 3D motion recognition. This can also be used for the visual description about the results by the CNN of 3D shapes.[13]

For these advantages, inthe case of visual inspection, which is one of the non-destructive inspection methods, the Grad-Cam++ model is able to be applied to the model developed for damage detect and quantify defects in temporary works equipment, evaluate the quality of all temporary works equipment tested, and obtain consistent results which make users understanding damage information which is inferenced by the designed AI algorithm model.

3. Damage Detection and Quantification System in Temporary Works Equipment based on XAI

3.1 Composition and function of the system

The basic system consists of input, damage detection and quantification with learned XAI, and output. The input system takes the form of data captured based on various images. In the damage detection and quantification system with the learned XAI, the trained artificial intelligence recognizes the image data and detects damage. The artificial intelligence model for defect detection is a combination of the Mask R-CNN model and the Grad CAM++-based XAI model. This model is an improved model based on the existing Faster R-CNN model and is a model thatimplements all functions of Bounding Box object detection, class classification, and object outline display (segmentation).[6] Detectable defects include rust, corrosion, deformation, dents, and flaws. Rust means iron oxide, and it means that it is generated by oxidation of iron, which is the main raw material of temporary works equipment. Corrosion is generally similar to rust, but with a clear concept, it refers to the electrochemical oxidation of a metal by reacting with an oxidant, and it means that the metal is peeled off by a chemical reaction. Deformation means that the structure or shape is changed temporarily or permanently by receiving an external force, and bending often occurs due to defects in temporary worksequipment, which was also considered as deformation. A dent refers to a phenomenon that is crushed by an external force or distorted by itself, and mainly refers to a concave surface. In the case of a defect, it was considered a defect that the object was damaged or part of it was torn off due to external force. The detected defects are classified by defect, the size and location of defects are quantified, the location and size of defects are visualized according to the absence of temporary works equipment, and the state evaluation for the safety inspection of the material is determined.[7] The output system functions to visualize the evaluation and results of the temporary works equipment that has been detected and quantified by XAI and analyzed so that the user can understand. It shows the location of defects and the contents of determining the type of defect of each temporary works equipment, and also shows all defects found in each equipment comprehensively and was developed to evaluate the condition accordingly.[8]

(Figure 2) Basic Structure of XAI Model

4. Experiment result

4.1 Dataset

For data, 5,500 images of temporary works equipment was used, and data collection tookusing the built-in camera of a UAV (Mavic GH5s) and a camera (S1R). In addition, initial weights were set using the coco dataset [10] which is opened in public. For the data augmentation, 54,444 data set were utilized.

There are three types of augmentation methods used. First, the image data generated by changing the left/right. Second, it was rotation by 90°/180°/270°. Last, the image data was changed by contrast. Through those method for the augmentation, the number of data set was 54,444.It was enough to proceeding the experiment and verification for the performance of the AI model which is designed for the damage detection and damage quantification.

4.2 Procedure

The procedure of experiment is shown below,

1. Temporary works equipment shooting (UAV + camera)

2. input into Deep Label+ (data labeling program) and labeling by experts

3. Saving and exporting labeling images

4. Resize the image (Unified to 1024x1024)

5. DLtools Filtering (Delete unlabeled images among resized images)

6. Split Image (data distribution, train/test/validation data)

7. XAI Learning

8. Check the initial results

9. Data augmentation (switch / rotate / contrast)

10. Damage detects by learned XAI 11. Check the results

The resulting value proved the effectiveness of AI damage detection using the f-1 score. When F-1 score is calculated as the harmonic average of precision and recall, it is mainly used when data imbalance between classification classes is severe.

4.3 Results

The value of the experimental result was calculated as the train data, verification data, and test data values according to the data distribution, and detection precision, recall, and F1-score were calculated for each type of damage (rust, corrosion, deformation, dent, damage). In addition, in order to check the overall performance of the AI damage detection system, the results of all damage detection are displayed as percentiles. The result value (f1-score) of system performance is train data: 95%, verification data: 92%, and test data: 90%, which shows the highest level of accuracy among developed techniques.

(Table 2) Results in Train Data Set

(Table 3) Results in Validated Data Set

(Table 4) Results in Test Data Set

(Figure 3) visualized damage detects by the XAI (1)

(Figure 4) visualized damage detects by the XAI (2)

In addition, in order to verify the explanatory ability of the XAI system, comprehensive performance tests such as the defect detection performance of the XAI model as well as the logical completeness of reasoning presented by the model were conducted. Basically, the part about the explanatory ability of the XAI model is divided into Captioning and Heatmap. First, Captioning explains the result of inferring to the user with simple sentences or phrases. For the part to be explained, the type of defect, the location of the defect, the criterion for determining the type of defect, and the cause of the defect are indicated. The method used to verify performance is the BLEU score, which measures translation performance by comparing how similar the results of machine translation and human translation are. The measurement criterion is based on n-gram, and although it is not a perfect verification method, it is currently widely used and intuitive verification shows the result value for users to understand easily.

The captioning result of the developed XAI model showed 89% Table 5 performance, and excellent results were derived. the performance results were calculated for the ratio of the diagonal length of the boundary box at the GT and the diagonal length of the boundary box by the inferencing Figure 5.

(Table 5) Results about Captioning by the XAI

(Figure 5) Captioning Result by the XAI

(Table 6) Example of Captioning by GT and the XAI

(Table 7) Comparison of Captioning by GT and the XAI

Second, the explanatory ability of XAI through Heatmap shows that the defect detection part, based on the inferencing result, is closer to red depending on the severity, and closer to blue when the severity is low. The method mainly used for performance verification is the ratio of the diagonal length of GT's Boundary Box and the distance between the center of GT's Boundary Box and the center of the inferencing Boundary Box.

The performance verification result of Heatmap of the developed XAI model is 90%, which is a very high result.

(Table 5) Results about Heatmap by the XAI

(Figure 5) Heatmap Result by the XAI

4.4 Future work

The data used in the experiment was self-photographed data, and the number was also limited, soit was difficult to commercialize defect detection artificial intelligence and learn to a level applicable to the field. Commercialization of defectdetection artificial intelligence can be achievedby learning artificial intelligence.

In addition, development and model learning to strengthen XAI's explanatory ability should be continuously conducted. Captioning performance verification in this experiment was a method of comparing congruency between words and phrases, but grammatical errors and errors in expressions often occurred in the entire captioning sentence. Therefore, it is necessary to improve the performance of Captioning through system development and model learning.

In addition, in order to be used in actual safety inspection sites for temporary equipment, web-based programs must be developed so that they can be used immediately at the place where safety inspections take place, and distributed learning and processing systems must be combined in preparation for large-scale program use.

5. Conclusion

In this research and development, in order to identify the weaknesses of safety inspection of temporary works equipment currently used in construction sites and to overcome them, image processing technology using the XAI with Mask-R-CNN is grafted to perform temporary equipment safety inspection based on artificial intelligence and images. The method was developed, and the conclusions are as follows.

1. Defect detection of temporary equipment based on explanatoryartificial intelligence is performed by an existing safety inspector, and consistent results can be derived compared to safety inspection based on visual inspection, and safety inspection of all temporary equipment is efficiently performed. can do.

2. As a result of confirming the defect detection ability of the artificial intelligence model through experiments, it was found that the detection performance was over 90%.

3. By using the XAI-based model, the user can understand the results more intuitively, so the artificial intelligence model can be trusted and the quality inspection of temporary equipment can be performed efficiently.

However, since the system is still in the research and development stage and the dataset for learning artificial intelligence is limited, it is necessary to collect large-scale datasets and advance the explainable artificial intelligence model in the future. In addition, as the data being larger, it is necessary to build a parallel deep learning and inferencing system because the Mask-R-CNNN has a weakness about processing time. If these are achieved, the system of damage detection and damage quantification at temporary works equipment by the XAI will be practical and enough to commercialized.