• Title/Summary/Keyword: missing data recovery

Search Result 23, Processing Time 0.024 seconds

Deep learning-based recovery method for missing structural temperature data using LSTM network

  • Liu, Hao;Ding, You-Liang;Zhao, Han-Wei;Wang, Man-Ya;Geng, Fang-Fang
    • Structural Monitoring and Maintenance
    • /
    • v.7 no.2
    • /
    • pp.109-124
    • /
    • 2020
  • Benefiting from the massive monitoring data collected by the Structural health monitoring (SHM) system, scholars can grasp the complex environmental effects and structural state during structure operation. However, the monitoring data is often missing due to sensor faults and other reasons. It is necessary to study the recovery method of missing monitoring data. Taking the structural temperature monitoring data of Nanjing Dashengguan Yangtze River Bridge as an example, the long short-term memory (LSTM) network-based recovery method for missing structural temperature data is proposed in this paper. Firstly, the prediction results of temperature data using LSTM network, support vector machine (SVM), and wavelet neural network (WNN) are compared to verify the accuracy advantage of LSTM network in predicting time series data (such as structural temperature). Secondly, the application of LSTM network in the recovery of missing structural temperature data is discussed in detail. The results show that: the LSTM network can effectively recover the missing structural temperature data; incorporating more intact sensor data as input will further improve the recovery effect of missing data; selecting the sensor data which has a higher correlation coefficient with the data we want to recover as the input can achieve higher accuracy.

Recovery the Missing Streamflow Data on River Basin Based on the Deep Neural Network Model

  • Le, Xuan-Hien;Lee, Giha
    • Proceedings of the Korea Water Resources Association Conference
    • /
    • 2019.05a
    • /
    • pp.156-156
    • /
    • 2019
  • In this study, a gated recurrent unit (GRU) network is constructed based on a deep neural network (DNN) with the aim of restoring the missing daily flow data in river basins. Lai Chau hydrological station is located upstream of the Da river basin (Vietnam) is selected as the target station for this study. Input data of the model are data on observed daily flow for 24 years from 1961 to 1984 (before Hoa Binh dam was built) at 5 hydrological stations, in which 4 gauge stations in the basin downstream and restoring - target station (Lai Chau). The total available data is divided into sections for different purposes. The data set of 23 years (1961-1983) was employed for training and validation purposes, with corresponding rates of 80% for training and 20% for validation respectively. Another data set of one year (1984) was used for the testing purpose to objectively verify the performance and accuracy of the model. Though only a modest amount of input data is required and furthermore the Lai Chau hydrological station is located upstream of the Da River, the calculated results based on the suggested model are in satisfactory agreement with observed data, the Nash - Sutcliffe efficiency (NSE) is higher than 95%. The finding of this study illustrated the outstanding performance of the GRU network model in recovering the missing flow data at Lai Chau station. As a result, DNN models, as well as GRU network models, have great potential for application within the field of hydrology and hydraulics.

  • PDF

Simulation combined transfer learning model for missing data recovery of nonstationary wind speed

  • Qiushuang Lin;Xuming Bao;Ying Lei;Chunxiang Li
    • Wind and Structures
    • /
    • v.37 no.5
    • /
    • pp.383-397
    • /
    • 2023
  • In the Structural Health Monitoring (SHM) system of civil engineering, data missing inevitably occurs during the data acquisition and transmission process, which brings great difficulties to data analysis and poses challenges to structural health monitoring. In this paper, Convolution Neural Network (CNN) is used to recover the nonstationary wind speed data missing randomly at sampling points. Given the technical constraints and financial implications, field monitoring data samples are often insufficient to train a deep learning model for the task at hand. Thus, simulation combined transfer learning strategy is proposed to address issues of overfitting and instability of the deep learning model caused by the paucity of training samples. According to a portion of target data samples, a substantial quantity of simulated data consistent with the characteristics of target data can be obtained by nonstationary wind-field simulation and are subsequently deployed for training an auxiliary CNN model. Afterwards, parameters of the pretrained auxiliary model are transferred to the target model as initial parameters, greatly enhancing training efficiency for the target task. Simulation synergy strategy effectively promotes the accuracy and stability of the target model to a great extent. Finally, the structural dynamic response analysis verifies the efficiency of the simulation synergy strategy.

A Study on the Formulation of High Resolution Range Profile and ISAR Image Using Sparse Recovery Algorithm (Sparse 복원 알고리즘을 이용한 HRRP 및 ISAR 영상 형성에 관한 연구)

  • Bae, Ji-Hoon;Kim, Kyung-Tae;Yang, Eun-Jung
    • The Journal of Korean Institute of Electromagnetic Engineering and Science
    • /
    • v.25 no.4
    • /
    • pp.467-475
    • /
    • 2014
  • In this paper, we introduce a sparse recovery algorithm applied to a radar signal model, based on the compressive sensing(CS), for the formulation of the radar signatures, such as high-resolution range profile(HRRP) and ISAR(Inverse Synthetic Aperture Radar) image. When there exits missing data in observed RCS data samples, we cannot obtain correct high-resolution radar signatures with the traditional IDFT(Inverse Discrete Fourier Transform) method. However, high-resolution radar signatures using the sparse recovery algorithm can be successfully recovered in the presence of data missing and qualities of the recovered radar signatures are nearly comparable to those of radar signatures using a complete RCS data without missing data. Therefore, the results show that the sparse recovery algorithm rather than the DFT method can be suitably applied for the reconstruction of high-resolution radar signatures, although we collect incomplete RCS data due to unwanted interferences or jamming signals.

Recovering missing data transmitted from a wireless sensor node for vibration-based bridge health monitoring

  • Kim, C.W.;Kawatani, M.;Ozaki, R.;Makihata, N.
    • Structural Engineering and Mechanics
    • /
    • v.38 no.4
    • /
    • pp.417-428
    • /
    • 2011
  • This paper presents recovering of missing vibration data of a bridge transmitted from wireless sensors. Kalman filter algorithm is adopted to reconstruct the missing data analytically. Validity of the analytical approach is examined through a field experiment of a bridge. Observations demonstrate that, even a part of recovered acceleration responses is underestimated in comparison with those responses taken from cabled sensors, dominant frequencies taken from the reconstructed data are comparable with those from cabled sensors.

A mathematical model to recover missing monitoring data of foundation pit

  • Liu, Jiangang;Zhou, Dongdong;Liu, Kewen
    • Geomechanics and Engineering
    • /
    • v.9 no.3
    • /
    • pp.275-286
    • /
    • 2015
  • A new method is presented to recover missing deformation data of lateral walls of foundation pit when the monitoring is interrupted; the method is called Dynamic Mathematical Model - Parameter Interpolation. The deformation of lateral walls of foundation pit is mainly affected by the type of supporting structure and the situation of constraints, therefore, this paper mainly studies the two different kinds of variation law of deep horizontal displacement when the lateral walls are constrained or not, proposes two dynamic curve models of normal distribution type and logarithmic type, deals with model parameters by interpolating and obtains the parameters of missing data, then missing monitoring data could be Figured out by these parameters. Compared with the result from the common average method which is used to recover missing data, in the upper 2/3 of the inclinometer tube, the result by using this method is closer to the actual monitoring data, in the lower 1/3 part of the inclinometer tube, the result from the common average method is closer to the actual monitoring data.

An Error Recovery Mechanism for Wireless Sensor Networks

  • Kim, Dong-Il
    • Journal of information and communication convergence engineering
    • /
    • v.10 no.3
    • /
    • pp.237-241
    • /
    • 2012
  • In wireless sensor networks, the importance of transporting data correctly with reliability is increasing gradually along with the need to support communications between the nodes and sink. Data flow from the sink to the nodes requires reliability for control or management that is very sensitive and intolerant of error; however, data flow from the nodes to the sink is relatively tolerant. In this paper, with emphasis on the data flow from the sink to the nodes, we propose a mechanism that considers accurate transport with reliability hop-by-hop. During the process of sending the data, if errors occur or data is missing, the proposed mechanism supports error recovery using a fixed window with selective acknowledgment. In addition, this mechanism supports congestion control depending on the buffer condition. Through the simulation, we show that this mechanism is accurate, reliable, and proper for transport in wireless sensor networks.

A Multilayer Perceptron-Based Electric Load Forecasting Scheme via Effective Recovering Missing Data (효과적인 결측치 보완을 통한 다층 퍼셉트론 기반의 전력수요 예측 기법)

  • Moon, Jihoon;Park, Sungwoo;Hwang, Eenjun
    • KIPS Transactions on Software and Data Engineering
    • /
    • v.8 no.2
    • /
    • pp.67-78
    • /
    • 2019
  • Accurate electric load forecasting is very important in the efficient operation of the smart grid. Recently, due to the development of IT technology, many works for constructing accurate forecasting models have been developed based on big data processing using artificial intelligence techniques. These forecasting models usually utilize external factors such as temperature, humidity and historical electric load as independent variables. However, due to diverse internal and external factors, historical electrical load contains many missing data, which makes it very difficult to construct an accurate forecasting model. To solve this problem, in this paper, we propose a random forest-based missing data recovery scheme and construct an electric load forecasting model based on multilayer perceptron using the estimated values of missing data and external factors. We demonstrate the performance of our proposed scheme via various experiments.

Compressive sensing-based two-dimensional scattering-center extraction for incomplete RCS data

  • Bae, Ji-Hoon;Kim, Kyung-Tae
    • ETRI Journal
    • /
    • v.42 no.6
    • /
    • pp.815-826
    • /
    • 2020
  • We propose a two-dimensional (2D) scattering-center-extraction (SCE) method using sparse recovery based on the compressive-sensing theory, even with data missing from the received radar cross-section (RCS) dataset. First, using the proposed method, we generate a 2D grid via adaptive discretization that has a considerably smaller size than a fully sampled fine grid. Subsequently, the coarse estimation of 2D scattering centers is performed using both the method of iteratively reweighted least square and a general peak-finding algorithm. Finally, the fine estimation of 2D scattering centers is performed using the orthogonal matching pursuit (OMP) procedure from an adaptively sampled Fourier dictionary. The measured RCS data, as well as simulation data using the point-scatterer model, are used to evaluate the 2D SCE accuracy of the proposed method. The results indicate that the proposed method can achieve higher SCE accuracy for an incomplete RCS dataset with missing data than that achieved by the conventional OMP, basis pursuit, smoothed L0, and existing discrete spectral estimation techniques.

The Influence of Service Recovery Justice on Intention to Recommend for Retailer

  • SHIN, Yongsun;KIM, Moonseop
    • Journal of Distribution Science
    • /
    • v.18 no.2
    • /
    • pp.91-98
    • /
    • 2020
  • Purpose: This research aimed to suggest retailing companies some ways to enhance customer satisfaction with service recovery and recommendation intention towards these companies. For this purpose, current study examined the relationships among service recovery justice, service failure severity, customer trust, recovery satisfaction and intention to recommend and the moderating role of ego-resilience. Research design, data and methodology: Current study developed a structural equation model in which perceived service recovery justice is a predictor, service failure severity, customer trust, recovery satisfaction are mediators, intention to recommend is a dependent variable and the ego-resilience is a moderator between the perceived service recovery justice and the customer trust and the recovery satisfaction. Data were collected from customers who experienced service failures from retailers. A total of 400 questionnaires were collected and 365 samples were used for analysis after deleting data having missing value. SPSS 25.0 and AMOS 24.0 were used to test the validity, reliability, and structural equation modeling. Results: Empirical results showed that the perceived service recovery justice had a negative influence on the perceived service failure severity and a positive influence on the customer trust and the recovery satisfaction. These results indicate that when customers perceive the service recovery justice more highly, they perceive the service failure less severe but they perceive the retailer more trustworthy and are satisfied with service recovery. In addition, the customer trust and the recovery satisfaction had a positive influence on the intention to recommend. These results indicate that when customers perceive the retailer more trustworthy and are satisfied with service recovery, they are more intend to recommend the retailer. Moreover, the influence of the perceived service recovery justice on the customer trust and the recovery satisfaction was moderated by the ego-resilience. Conclusions: This study contributed to the service recovery literature by proving the relationship among service recovery justice, service failure severity, customer trust, recovery satisfaction and intention to recommend. Moreover, current research introduced the ego-resilience into service recovery research area and revealed the moderation role of the ego-resilience. Managerially, this research suggested retailing companies some ways to effectively recover from service failure.