DOI QR코드

DOI QR Code

On Identifying Operational Risk Factors and Establishing ALARP-Based Mitigation Measures using the Systems Engineering Process for Parcel Storage Devices Utilizing Active Loading Technology

  • Mi Rye Kim (Department of Systems Engineering, Ajou University) ;
  • Young Min Kim (Department of Logistics System and SCM, Ajou University)
  • Received : 2023.09.05
  • Accepted : 2023.12.20
  • Published : 2023.12.31

Abstract

Due to the steady growth of the online shopping market and contact-free consumption, the volume of parcels in South Korea continues to increase. However, there is a lack of manpower for delivery workers to handle the growing parcel volume, leading to frequent accidents related to delivery work. As a result, the government and local authorities strive to enhance last-mile logistics efficiency. As one of these measures, unmanned parcel storage lockers are installed and utilized to handle last-mile deliveries. However, the existing parcel storage involves the inconvenience of couriers having to put each parcel in each locker, and this is somewhat insufficient to relieve the workload of delivery workers. In this study, we propose parcel storage devices that use active loading technology to minimize the workload of delivery workers, extract operation risk factors to apply this system to actual sites, and establish risk reduction methods based on the ALARP concept. Through this study, we have laid the groundwork for improving the safety of the system by identifying and proposing mitigation measures for the risk factors associated with the proposed parcel storage devices utilizing active loading technology. When applied in practical settings in the future, this foundation will contribute to the development of a more efficient and secure system. By applying the ALARP concept, a systems engineering technique used in this research, to the development and maintenance of storage devices leveraging active loading technology, it is thought to make the development process more systematic and structured. Furthermore, through the risk management of the proposed system, it is anticipated that a systematic approach to quality management can be employed to minimize defects and provide a stable system. This is expected to be more useful than the existing unmanned parcel storage devices.

Keywords

1. Introduction

Due to the explosive growth of the online shopping market starting from COVID-19, Korea's parcel volume in 2021 has recorded a growth of 158.2% since 2012. The volume of parcels also increased rapidly from 1,405.98 million in 2012 to 3,629.67 million in 2021. In addition, due to competition among online shopping companies for quick delivery, logistics services are being operated in a competitive structure to deliver large volumes of parcels faster by providing various logistics services such as 'same-day delivery', 'early morning delivery', 'morning star delivery', and 'guaranteed tomorrow's arrival'. However, as this last-mile operation system becomes a burden for delivery workers and is linked to overwork and accidents, efforts are being made to prepare measures for more efficient last-mile logistics processing. In the case of Korea, since 2015, it has been institutionalized to provide space for installing unmanned delivery boxes in apartments where more than 500 people live, and it has been institutionalized to operate a joint base-type delivery business that uses parking lots at community centers as space for installing unmanned delivery boxes.[1]

However, the installation and operation of unmanned parcel storage devices were expected to be used as an important component of last-mile logistics to reduce the labor intensity of delivery workers and shorten delivery time, but in reality, it is inconvenience to place each parcel in one parcel storage space. Due to inconvenient and complexity, the utilization of unmanned parcel storage devices is not high. Therefore, in this study, we propose parcel storage devices utilizing active loading technology to reduce the labor of delivery workers operate more efficient unmanned parcel storage devices and conduct safer and more efficient delivery of parcels through risk assessment when this system is applied to the field. To efficiently store and safely operate these cargo storage devices, it is necessary to derive and analyze system risk factors that may occur before the operational stage. Therefore, in this study, a clear concept definition based on systems engineering and interface considerations from a system perspective is needed to ensure the safe and efficient operation of Parcel Storage Devices Utilizing Active Loading Technology. Additionally, it is necessary to build a system that can derive efficient performance within the permitted range by analyzing the risk factors of the device in advance. To this end, this study first analyzed the input and output of the ISO/IEC-15288-based operating process suitable for confirming the overall operational process. ISO/IEC-15288, titled "System Life Cycle Process," describes the tasks to be performed during the entire system life cycle and is a new process standard that can be commonly applied to the hardware and software fields. Therefore, if applied to this study, it is believed that Parcel Storage Devices Utilizing Active Loading Technology can be operated more safely and efficiently.[2],[3]

This study is structured as follows. Chapter 2 reviewed existing studies on unmanned parcel storage devices through a literature review and analyzed the proposed system. In addition, we conducted theoretical research on ALARP (As Low As Reasonably Practicable), the risk assessment methodology to be used in this study. Chapter 3 conducted a comparative analysis study with the existing system on the system concept and operation scenario of the parcel storage box using the active loading technology proposed in this study. In Chapter 4, an operational risk factor analysis was performed to derive ALARP-based mitigation measures of active loading technology applied parcels storage devices. In the final chapter 5, we will present the implications of the research results and the future work of the study.

2. Literature Review

2.1 Unmanned Parcel Storage Device

Unmanned parcel storage devices are a technology that can alleviate delivery time and space constraints by allowing delivery workers and customers to store and receive parcels in a secure space without having to meet face-to-face. The unmanned parcel storage device is a system that includes a set of individual lockers of various sizes and is equipped with an individual security device by storage space, allowing customers to collect the parcel using the delivery password or mobile phone number registered in the delivery information.[1] When the unmanned parcel storage device was first introduced, it began to be widely used as a new solution for delivery and distribution logistics in that it allowed for non-face-to-face storage and receipt of parcels. However, in the case of existing unmanned parcel storage devices, it is inconvenient in that the delivery worker must personally determine whether there is an empty storage box, store many parcels in each delivery locker for each recipient, and then share the password.[4]

To resolve this inconveniences J. B. Seo and D. H. Kim conducted a study on the proposal of an application that can link the locking and unlocking of the unmanned parcel storage device through Bluetooth communication using a smartphone application.[4] This is a system proposal on how to improve the unmanned parcel storage devices locking and unlocking devices based on the Android operating system. Although it may be able to solve some of the inconvenience for delivery workers, it is judged that there is a limit to improving the overall storage procedure of the parcel storage devices. S. K. Lee researched the design of storage boxes using hall sensors, illuminance sensors, ultrasonic sensors, and FSR sensors.[5] The values collected from each sensor in the storage box are transmitted to the Raspberry Pi in real-time and stored in the DB on the server. Using the sensors collected in this way, a study was conducted to distinguish between locking and unlocking doors and to classify the states in which objects were stored and those in which objects were not stored by combining various sensor values. This study is expected to be useful in that delivery workers do not have to manually search for every empty space in the parcel storage devices, but like the above study, it is judged that there are limits to improving the overall storage procedure of the delivery item storage box.

C. H. Park, H. Y. Kang, C. S. Khang conducted research on the development of an unmanned parcel storage system based on the Internet of Things.[6] This paper studies the implementation of a safe delivery box based on the Internet of Things using weight sensors, shock sensors, Arduino, and Raspberry Pi. carried out. The developed system detects the weight of the unmanned parcel storage, recognizes the arrival of the product, and automatically transmits it to the recipient's smartphone. It is implemented as a system that enables the locking and unlocking function of the parcel storage system using the smartphone. This study can contribute to the implementation of a smart delivery system unlike existing studies in terms of detecting the weight of unmanned parcel storage and utilizing smartphones, but it is judged to have limitations in that each storage locker is still required for each parcel. Table 1 shows the analysis of the functions of each unmanned parcel storage box in the reviewed literature into ‘low’, ‘medium’, and ‘high’ levels.

<Table 1> Analysis of parcel storage device functions in the reviewed literature studies

HSSTBN_2023_v19n2_59_t0001.png 이미지

2.2 ALARP (As Low As Reasonably Practicable)

In general, the ALARP (As Low As Reasonably Practicable) principle is a fundamental aspect of risk management, primarily utilized in the UK and defined as a guiding principle. It posits that actionable risks should be minimized, and within the context of systems engineering, the principle indictates that risks should be reduced.[10] Risk management from a systems engineering standpoint plays a crucial role in defining the objectives of the system and identifying potential sources of risk to be avoided in the design and operation of the overall system. The identification of risk sources from a systems engineering perspective contributes to ensuring the safe operation of the system, playing a pivotal role in the process from the initial conceptualization of the system to the conclusion of its life cycle.[7]

Risk management is considered to be all activities and measures used to proactively manage and control risks, balance development, and prevent losses and accidents.[8] Risk management using ALARP is being applied in various industrial fields, and several studies have discussed how to evaluate the risk of the system and derive risk mitigation measures.

First, in the study by K. H. Park and J. O. Ryu, ALARP was used to verify whether water spray fire extinguishing equipment installed in road tunnels improves evacuation performance. Through research, an analysis study was conducted on the scope of ALARP without and with water spray fire extinguishing equipment, and it was found that water spray fire extinguishing equipment creates an environment favorable for evacuation through the cooling effect of high-temperature air currents and the cleaning effect of toxic gases in the event of a road tunnel fire.[11] In the study by K. H. Song and D. K. Lee, a study was conducted on the multi-criteria ALARP decision-making technique to add it as a cost-effective decision-making factor for safety-related decisions in the aviation safety management system. In the study, an exploratory factor analysis is performed on the selected benefit items based on the SIRA (Safety Issue Risk Assessment) model. Through this, the validity of the multi-criteria ALARP decision-making technique was presented by verifying the suitability of the established evaluation hierarchy and individual benefit item structure.[12] In a study by J. H. Na, C. W. Lee, S. J. Lee, J. H. Kim and U. H. Kim, a study was conducted to derive problem situations that may occur and propose solutions through functional analysis of water operation situations of wheeled combat vehicles.[13] This study derived the hazards that may arise from the water operation of combat vehicles and presented the results of proposing a solution to reduce the hazard level of the hazards derived according to the ALARP concept. Through literature research on ALARP, research has been conducted based on the ALARP concept in various industries and systems, and it has been confirmed that it is a useful methodology for suggesting system improvement measures.

3. Parcel Storage Devices Utilizing Active Loading Technology

3.1 System Configuration

Parcel storage device utilizing active loading technology is an unmanned delivery storage system with remote product information system control/monitoring technology (parcel Information System) applied, which facilitates storage and receipt of parcels in an untact environment, and stores delivery goods through a separate remote-control server. It refers to a system that utilizes a system platform that integrates and manages comprehensive information about. As shown in Figure 1, the internal and external components of the delivery goods storage device with active loading technology include internal elements such as the delivery goods storage box and control server, which are the main body, and external elements such as delivery workers and customers who utilize them.

HSSTBN_2023_v19n2_59_f0001.png 이미지

[Figure 1] Components of Parcel Storage Device Utilizing Active Loading Technology

First, the delivery goods storage device functions to receive, release, and store goods, and inputs related information. The control server receives this information and uses it for control and operation management to manage the delivery and delivery data. Courier workers perform storage work in the delivery goods storage device to transport and store the parcel, and customers receive SMS and receive the goods.

Depending on the function, the parcel storage devices has an active loading technology function and an ICT technology-based untact item loading and unloading function. The characteristics of each function are shown in Table 2.

<Table 2> Features of Each Function in the Proposed Unmanned Parcel Storage Device

HSSTBN_2023_v19n2_59_t0002.png 이미지

The parcel storage device utilizing active loading technology is largely composed of nine detailed systems, as shown in Figure 2, from the delivery of parcel to information recognition, processing, storage, and control. First, the display part performs the functions of device control through a touch screen and information display and manipulation for storing and receiving goods. The control part controls the operation of the device transmitted by the operation part and performs the function of maintaining and managing the status of each component. The output part performs a card payment function to process costs incurred due to product storage and a printing function for payment and product-related invoice information. The access part performs an access function to move items into or out of the device and operates and opens only when in use. The robot part plays the role of transporting shelves loaded with goods to a designated location for storage and movement of goods inside or outside the device. The power part receives and supplies the power required for the operation of the device from the outside and controls it in a constant state. The external part protects the internal device from external environmental factors and physical shock and provides a protective function to maintain the condition of the product.

HSSTBN_2023_v19n2_59_f0002.png 이미지

[Figure 2] Detailed System Components of the Parcel Storage Device

3.2 Comparative Analysis of System Operation Scenarios

In this chapter, we would like to compare and analyze the operating procedure of the normal unmanned parcel storage device and the suggested unmanned parcel storage device. First, in the existing unmanned parcel storage operation scenario, as shown in Figure 3, a delivery worker approaches the parcel storage and finds for an empty locker. Afterward, when you find an empty space in the storage box, set a password, insert the parcel, and activate the lock function. Afterward, the operating procedure is ended after sharing the password with the customer. Afterward, the operating procedure is ended after sharing the password with the customer. To overcome these shortcomings, this study proposed a system that can automatically load and store multiple parcels through one entrance.

HSSTBN_2023_v19n2_59_f0003.png 이미지

[Figure 3] Operating Procedures for Existing Unmanned Parcel Storage Device

The operating procedure of the proposed parcel storage device utilizing active loading technology is shown in Figure 4. First, the delivery workers sign up for the unmanned parcel storage system once, after that, the workers only need to log in to access it. The delivery worker approaches the proposed parcel storage device, puts several parcels in it, checks the display part on the parcel storage device and selects either Kiosk or Application as the login method. After logging in, pressing the ‘Keep’ button ends the delivery storage process. The advantages of the proposed parcel storage device operation method are that, firstly, there is no need to repeat the procedure shown in Figure 3 depending on the number of parcels, and it is possible to implement a system in which multiple parcels are automatically loaded into the parcel storage device at once. Second, there is no need to send a separate password to the customer, and the customer can know whether the parcel has arrived by linking the information system of the parcel storage device itself.

HSSTBN_2023_v19n2_59_f0004.png 이미지

[Figure 4] Operating Procedures for Existing Unmanned Parcel Storage Device

This newly proposed Parcel storage device utilizing active loading technology is more efficient than the existing parcel storage and can reduce the workload of delivery workers, so the expected effect on the function is high, but risk factors must be considered before the actual introduction of this system. There is a need to improve the reliability of operations through the process of deriving and proposing mitigation measures.

4. Risk Management of Suggested Parcel Storage devices based on ALARP Concept

4.1 ALARP function

ALARP is an abbreviation for ‘As Low As Reasonably Practicable’, which means ‘lower to a reasonably practicable level.[6] As shown in Figure 5, the ALARP principle increases in risk as it goes up, and is divided from the top into acceptable or unacceptable areas, acceptable ALARP areas, and acceptable areas.[14],[15] In demand or unacceptable areas, the risk is not justified except in special cases, and if reasonably practicable within the acceptable ALARP area, ways to reduce the risk as much as possible should be sought. It is believed that the ALARP principle can be applied as a representative safety principle when configuring a system as a guideline for what constitutes an acceptable risk and how much the risk should be lowered in various industrial fields.[16] In the case of the parcel storage device utilizing active loading technology proposed in this study, it is designed to overcome the shortcomings of existing delivery parcel storage and utilize ICT technology to increase the work efficiency of delivery workers and maximize user convenience. As a new system, for this system to be put into practice, it is necessary to conduct a risk assessment through the ALARP concept, a representative principle of industrial safety, to contribute to building a safer and more stable system.[17]

HSSTBN_2023_v19n2_59_f0005.png 이미지

[Figure 5] Types of Functions During Abnormal Operations

4.2 Derivation of Risk Factors in Parcel Storage Devices Utilizing Active Loading Technology

To derive risk factors for the parcel storage device utilizing active loading technology, abnormal situations that may occur during the operation of delivery goods storage boxes were subdivided into five types using a functional structure diagram, and possible problem situations and causes were described. Figure 6 shows the types of functions that may appear during abnormal operation of the parcel storage device proposed in this study. There is a total of four types: equipment performance error, errors caused by users, errors caused by external factors and disability caused by natural disasters. The ‘abnormal operation’ items were analyzed in detail.

HSSTBN_2023_v19n2_59_f0006.png 이미지

[Figure 6] Principle of ALARP

Table 3 analyzes the situations and causes of problems that may occur during ‘abnormal operation’of the parcel storage device utilizing active loading technology classified in Figure 6. The presented table lists each case in detail to analyze whether it is possible to overcome risk factors in each situation and establish ALARP-based mitigation measures through problem situation and cause analysis. According to Table 3, 50% of the causes of 'abnormal operation' problem situations in suggested parcel storage devices can be caused by equipment performance error, and because this is a system using unmanned automated robots such as ICT technology and active loading technology. Except for the equipment performance error, most problems occur due to the user's inexperience in using it. Among the problems caused by equipment performance error, if the equipment operates abnormally due to technical or operational defects, a situation may arise where delivery parcels cannot be stored or parcel information cannot be recognized. The challenges stemming from errors caused by users of the parcel storage device proposed in this study encompass situations where delivery workers lack experience in their tasks or are in a hurry. When storing parcels in the suggested parcel storage device, careless behaviors such as throwing or pushing objects, similar to those observed in existing parcel storage devices, could pose a risk factor in the operation of the proposed parcel storage. In the case of the errors caused by external factors, it is a risk factor that can be caused by the actions of passers-by or drunken person, and actions such as applying external force to the delivery storage box or arbitrarily cutting off the power supply may affect the operation of the parcel storage device. Lastly, concerning disability caused by natural disasters, the risk factors include situations where occurrences like heavy rain or snowfall could make power supply challenging or lead to malfunctions in display devices.

<Table 3> Analysis of Possible Problems and Causes During 'Unusual Operation'​​​​​​​

HSSTBN_2023_v19n2_59_t0003.png 이미지

4.3 Hazardous Elements Category Analysis

In this chapter, based on the four risk factors of the parcel storage device utilizing active loading technology derived above, a study was conducted to determine how much impact the factors would have on the system and establish measures to reduce them. Before establishing mitigation measures in this study, the Hazardous Elements Category classified from A to D presented by T. Weilkiens includes errors in equipment performance, equipment performance error, errors caused by users, errors caused by external factors and disability caused by natural disasters. We analyzed how much impact four risk factors, including disasters, would have on the system.

In this study, we applied T. Weilkiens' categorization of risk factor domains to systematically organize and manage safety and risk through a structured approach, to contribute to the minimization of risks. Conventional risk assessments for the systems can be classified into quantitative methods, which are based on the frequency of accidents or analyzed through past accident records, and qualitative methods, where the severity of risks is evaluated subjectively through judgment. This study aimed to categorize quantifiable aspects for risk analysis of the newly introduced parcel storage device and to explore a combined approach of quantitative and qualitative methods that allows for systematic risk management through T. Weilkiens' categorization of risk factor domains. The intention was to utilize a comprehensive method that combines both quantitative and qualitative assessments based on T. Weilkiens' categorization of risk factors to enhance the systematic analysis of risk management for the newly introduced parcel storage device.

The vertical axis of the Hazardous Elements Category can be evaluated by the frequency of risk elements, and the horizontal axis can be evaluated by severity, which can determine how much of a risk element can have a significant impact on the system. A represents the Intolerable risk level and B and C represent the ALARP region. Lastly, D represents Negligible risk level.[14] The four risk factors of delivery storage boxes with active loading technology can be classified as shown in Figure 7 in the Hazardous Elements Category.

HSSTBN_2023_v19n2_59_f0007.png 이미지

[Figure 7] Hazardous Elements Classification of Suggested Parcel Storage Device​​​​​​​

In the case of equipment performance error, it is not something that can be controlled by humans, and since unmanned automation technology is applied, it can be judged as a factor that can occur probable and critically. This is because equipment performance errors frequently occur not only in the proposed parcel storage device but also in existing parcel storage devices, it is necessary to prepare mitigation measures for this. In the case of the errors caused by users, it was determined that this could occur frequently enough and that it could often occur due to carelessness or inexperience due to excessive work by delivery workers. And although errors caused by external factors may occasionally occur, damage caused by external forces and the severity of system defects can cause more catastrophic risks than equipment performance error or errors caused by users.

The frequency of risk factors due to disability caused by natural disasters is the lowest, but the severity was deemed to be catastrophic high in comparison with the damage from natural disasters such as heavy rain that has been occurring recently.

4.4 Establishment of Mitigation Measures based on ALARP Concept

It can be seen that all four risk factors of the parcel storage device proposed in this study are related to system failure and damage to parcels, and are fully possible situations, which correspond to the ‘Intolerable Level’ when explained by the concept of ALARP. Therefore, it is necessary to reduce or eliminate the risk factors of the ‘Intolerance Level’, which is the purpose of ALARP, to the ‘Tolerance Level’, and we would like to prepare adjustment and mitigation measures for the level by presenting mitigation measures.[19],[20] First of all, as shown in Figure 8, measures that can be reduced at the existing level were proposed based on the four risk factors of the parcel storage device utilizing active loading technology.

HSSTBN_2023_v19n2_59_f0008.png 이미지

[Figure 8] Hazardous Elements Classification of Suggested Parcel Storage Device after the Establishment of Mitigation Measures​​​​​​​

As shown in Table 4, in the case of equipment performance error, it is believed that the frequency that may occur can be lowered by supplementing enhanced parts reliability and durability and Enhancement of S/W performance. In the case of errors caused by users, the frequency and severity can be reduced through mitigation measures to increase the user's proficiency and strengthen the durability of the storage box through the distribution of user manuals and guidance training for the new delivery storage box proposed in this study. It is deemed that in the case of errors caused by external factors, it is believed that the frequency can be adjusted by applying a warning notification when an external force occurs and a function to strengthen the durability of the delivery locker. Lastly, in the case of disability caused by natural disasters, measures were taken to reduce severity through the application of waterproofing function.

<Table 4> Analysis of Mitigation Measures based on ALARP Concept​​​​​​​

HSSTBN_2023_v19n2_59_t0004.png 이미지

5. Conclusion

This study proposes a new parcel storage device utilizing active loading technology that can compensate for the shortcomings of the existing parcel storage box, and suggests mitigation measures according to the concept of ALARP by deriving risk factors that may occur during the operation of the system. Four risk factors were identified: equipment performance error, errors caused by users, errors caused by external factors and disability caused by natural disasters and as a result, the unacceptable level was reduced to ALARP Region. In the case of equipment performance error, it is believed that the frequency that may occur can be lowered by supplementing enhanced parts reliability and durability and enhancement of S/W performance. In the case of errors caused by users, the frequency and severity can be reduced through mitigation measures to increase the user's proficiency and strengthen the durability of the storage box through distribution of user manuals and guidance training for the new delivery storage box proposed in this study. It is deemed that in the case of errors caused by external factors, it is believed that the frequency can be adjusted by applying a warning notification when an external force occurs and a function to strengthen the durability of the delivery locker. Lastly, in the case of disability caused by natural disasters, measures were taken to reduce severity through the application of a waterproofing function.

This paper utilized the system engineering-based ALARP concept to define the functions and analyze risk factors of parcel sorage devices utilizing active loading technology. Paarcel sorage devices utilizing active loading technology represent a newly developed system from a life-cycle perspective before actual operation. Given the necessity of analyzing operational risk sources, this study conducted an analysis by applying the ALARP concept, which performs risk assessment from a systems engineering perspective that encompasses the entire life cycle process. By applying the ALARP concept, a Systems Engineering methodology utilized in this paper, to the development and maintenance of storage devices utilizing active loading technology, it is deemed that the development process can be more systematic and structured. And, through the risk management of the proposed system, it is anticipated that a systematic approach to quality management can be employed to minimize defects and provide a stable system. In the future, we will conduct further risk analysis by applying functional analysis on-site after installing the proposed parcel storage box. We will also research ways to reduce accident causes. Additionally, we plan to perform a risk analysis of the target system's interface to improve its reliability and efficiency. This additional research is aimed at enhancing the safety and operational efficiency of the development system, resulting in an overall better system.

Acknowledgement

This work was supported by a grant from R&D program of the Korea Evaluation Institute of Industrial Technology.(Project Number: 20014664).

References

  1. Haejin Kim, & JIbok Chung, A Study on the Perceived Value and Intention to Use of Smart Locker, Journal of Management and Economics, p115-134, 2021.
  2. Han, Seok-Youn, Joo-Uk Kim, and Myung-Sung Choi. A Study on the Systems Engineering based Verification of a Systems Engineering Application Model for a LRT Project, Journal of the Korea Academia-Industrial cooperation Society, p425-433, 2016.
  3. International Standard ISO/IEC/IEEE 15288. System and software engineering System life cycle process. 2015.
  4. Jong-Beom Seo, & Dong-Hoi Kim, The Proposal and Implementation of New Private Parcel Locker with Automatic Transmission Function of Short Message Service by Smartphone Application, Journal of Digital Contents Society, p101-108, 2017.
  5. Sungkyu Lee, Seunghyun Hong, Won-Ho Jun, Woo-Chul Shin, Hyungjoo Kim, & Y.S. Hong, An Implementation of an Intelligent Unmanned Delivery Box Using Multi-Sensor Fusion, Annual Conference of IEIE, p989-991, 2021.
  6. Chan Hee Park, Hyun Tae Kang & Chang Soon Kang, Development of an IoT-based Unmanned Home-Delivery Box System, Korea Society of IT Services, p129-138, 2017.
  7. Walden, David D., Garry J. Roedler, and Kevin Forsberg. INCOSE systems engineering handbook version 4: updating the reference for practitioners, INCOSE International Symposium, 2015.
  8. Ki-Han Song, Dae-Kyum Lee, Multi-Criteria Evaluation Methodology Development on Aviation Safety Management System ALARP Decision-Making Process, Journal of Transport Research, p1-15, 2013.
  9. Langdalen, H., Abrahamsen, E. B., & Selvik, J. T, On the importance of systems thinking when using the ALARP principle for risk management, Reliability Engineering & System Safety, 204, 2020.
  10. Melchers, Robert E. On the ALARP approach to risk management, Reliability Engineering & System Safety, p201-208, 2001.
  11. Kyung Hwan Park & Ji-Oh Yoo, A study on use of quantitative risk analysis on life safety performance for the effect of fixed fire fighting systems at road tunnel fires, Journal of Korean Tunnelling and Underground Space Association, p1-22, 2012.
  12. Ki-Han Song, Dae-Kyum Lee, Validation of Multi-criteria ALARP Decision Method in Aviation SMS Using Exploratory Factor Analysis, Journal of Transport Research, p105-117, 2014.
  13. Jae Hyun Na, Cheol Woo Lee, Seung Jong Lee, Jong Hyun Kim & Eui Whan Kim, A Study on the Hazard Elements and Reduction for the Armored Wheeled Vehicle River Crossing Operation, Journal of the Korea Society of Systems Engineering, p15-20, 2014.
  14. Mackenzie. D, ICAO: A history of the international civil aviation organization, University of Toronto Press, 2010.
  15. Agustini, E., Kareng, Y., Victoria, O. A, The role of ICAO (international civil aviation organization) in implementing international flight safety standards, Kne Social Sciences, p100-114, 2021.
  16. Jin-U Jeong, Risk Evaluation, Occupational health, p73-79, 2013.
  17. Henrik Langdalen, Eirik Bjorheim Abrahamsen, Jon Tommeras Selvik, On the importance of systems thinking when using the ALARP principle for risk management, Reliability Engineering & System Safety, 2020.
  18. Engel, A, Verification, validation, and testing of engineered systems, John Wiley & Sons, p73, 2010.
  19. Joong-Hee Lee, Seong-Min Hwang & In-Sung Woo, A Study on the As Low As Reasonably Practicable (ALARP)-Concept Risk Assessment of Silane in Semiconductor and LCD Process, Journal of Korea Safety Management & Science, p93-98, 2010.
  20. Liu, Z., Li, Y., Zhang, Z., Yu, W, A new evacuation accessibility analysis approach based on spatial information, Reliability Engineering & System Safety, 222, 108395, 2022.