• Analytical Science and Technology
• /
• v.25 no.4
• /
• pp.214-222
• /
• 2012

ASTM (American Society for Testing and Materials) D6751-10 suggests analytical methods as well as specifications for biodiesel quality. However, it is expensive and time-consuming to follow the ASTM testing methods to analyze biodiesel and various impurities. This paper develops a quantitative analysis system for biodiesel and impurities based on Infrared spectroscopy and a multivariate statistical method, PLS (partial least squares). In addition, four different pre-processing techniques were compared for spectrum correction and noise reduction. Savitzky-Golay pre-processing showed the best performance.

• Journal of the Korean Geographical Society
• /
• v.51 no.4
• /
• pp.509-523
• /
• 2016

The soil prediction can provide quantitative soil information for sustainable mountainous ecosystem management. Visible near infrared spectroscopy, one of soil prediction methods, has been applied to predict several soil properties with effective costs, rapid and nondesctructive analysis, and satisfactory accuracy. Spectral preprocessing is a essential procedure to correct noisy spectra for visible near infrared spectroscopy. However, there are no attempts to evaluate various spectral preprocessing methods. We tested 5 different pretreatments, namely continuum removal, Savitzky-Golay filter, discrete wavelet transform, 1st derivative, and 2nd derivative to predict soil carbon(C) and nitrogen(N). Partial least squares regression was used for the prediction method. The total of 153 soil samples was split into 122 samples for calibration and 31 samples for validation. In the all range, absorption was increased with increasing C contents. Specifically, the visible region (650nm and 700nm) showed high values of the correlation coefficient with soil C and N contents. For spectral preprocessing methods, continuum removal had the highest prediction accuracy(Root Mean Square Error) for C(9.53mg/g) and N(0.79mg/g). Therefore, continuum removal was selected as the best preprocessing method. Additionally, there were no distinct differences between Savitzky-Golay filter and discrete wavelet transform for visual assessment and the methods showed similar validation results. According to the results, we also recommended Savitzky-Golay filter that is a simple pre-treatment with continuum removal.

• Journal of information and communication convergence engineering
• /
• v.20 no.2
• /
• pp.125-130
• /
• 2022

In modern society, as many data are used for research or commercial purposes, the value of data is gradually increasing. In related fields, research is being actively conducted to collect valuable data, but it is difficult to collect proper data because the value of collection is determined according to the performance of existing sensors. To solve this problem, a method to effectively reduce noise has been proposed, but there is a point in which performance is degraded due to damage caused by noise. In this paper, a device capable of collecting time series data was designed to correct such data noise, and a correction technique was performed by giving an error value based on the representatively collected ultrafine dust data, and then comparing before and after Compare performance. For the correction method, Kalman, LPF, Savitzky-Golay, and Moving Average filter were used. Savitzky-Golay filter and Moving Average Filter showed excellent correction rate as an experiment. Through this, the performance of the sensor can be supplemented and it is expected that data can be effectively collected.

• Korean Journal of Soil Science and Fertilizer
• /
• v.37 no.4
• /
• pp.259-265
• /
• 2004

A quicker method was developed for foliar analysis in diagnosis of nitrogen in apple trees based on multivariate calibration procedure using partial least squares regression (PLSR) and principal component regression (PCR) to establish the relationship between reflectance spectra in the near infrared region and nitrogen content of fresh- and dry-leaf. Several spectral pre-processing methods such as smoothing, mean normalization, multiplicative scatter correction (MSC) and derivatives were used to improve the robustness and performance of the calibration models. Norris first derivative with a seven point segment and a gap of six points on MSC gave the best result of partial least squares-1 PLS-1) model for dry-leaf samples with root mean square error of prediction (RMSEP) equal to $0.699g\;kg^{-1}$, and that the Savitzky-Golay first derivate with a seven point convolution and a quadratic polynomial on MSC gave the best results of PLS-1 model for fresh-samples with RMSEP of $1.202g\;kg^{-1}$. The best PCR model was obtained with Savitzky-Golay first derivative using a seven point convolution and a quadratic polynomial on mean normalization for dry leaf samples with RMSEP of $0.553g\;kg^{-1}$, and obtained with the Savitzky-Golay first derivate using a seven point convolution and a quadratic polynomial for fresh samples with RMSEP of $1.047g\;kg^{-1}$. The results indicate that nitrogen can be determined by the near infrared reflectance (NIR) technology for fresh- and dry-leaf of apple.

• Journal of the Semiconductor & Display Technology
• /
• v.22 no.3
• /
• pp.84-89
• /
• 2023

In water resource management, data prediction is performed using artificial intelligence, and companies, governments, and institutions continue to attempt to efficiently manage resources through this. LSTM is a model specialized for processing time series data, which can identify data patterns that change over time and has been attempted to predict groundwater level data. However, groundwater level data can cause sen-sor errors, missing values, or outliers, and these problems can degrade the performance of the LSTM model, and there is a need to improve data quality by processing them in the pretreatment stage. Therefore, in pre-dicting groundwater data, we will compare the LSTM model with the MSE and the model after normaliza-tion through distribution, and discuss the important process of analysis and data preprocessing according to the comparison results and changes in the results.

• Clean Technology
• /
• v.17 no.3
• /
• pp.215-224
• /
• 2011

It is an important issue to develop quality assessing system for biofuel for the purpose of accelerating the mass production of biofuel. It is particularly challenging to conduct testing method in the mass production of bioethanol while meeting quality specifications such as ASTM (American Society for Testing & Materials) D4806-10. In order to address this challenge, this paper proposes on-line spectroscopic quality assesment system based on Near Infrared spectrum and Partial Least Squares method in Chemometrics. As a result of testing a number of preprocessing methods and Partial Least Squares, it was found out that Savitzky-Golay method showed the best performance in terms of spectrum correction, noise reduction, and model maintenance. The proposed system allows us to assess multiple quality components continuously using spectroscopic facilities with the cheap cost. Since the value of R2 is more than 0.99, it is possible to replace the laboratory analysis.

• Proceedings of the Korean Society for Agricultural Machinery Conference
• /
• 2000.11c
• /
• pp.560-568
• /
• 2000

The aims of this research were to investigate the preprocessing effect of spectrum data on prediction performance and to develop a robust model to predict SSC in intact apple. Spectrum data of 320 Fuji apples were measured with the on-line transmittance measurement system at the wavelength range of 550∼1100nm. Preprocess methods adopted for the tests were Savitzky Golay, MSC, SNV, first derivative and OSC. Several combinations of those methods were applied to the raw spectrum data set to investigate the relative effect of each method on the performance of the calibration model. PLS method was used to regress the preprocessed data set and the SSCs of samples, and the cross-validation was to select the optimal number of PLS factors. Smoothing and scattering corection were essential in increasing the prediction performance of PLS regression model and the OSC contributed to reduction of the number of PLS factors. The first derivative resulted in unfavorable effect on the prediction performance. MSC and SNV showed similar effect. A robust calibration model could be developed by the preprocessing combination of Savitzky Golay smoothing, MSC and OSC, which resulted in SEP= 0.507, bias=0.032 and R$^2$=0.8823.

• Korean Journal of Fisheries and Aquatic Sciences
• /
• v.30 no.1
• /
• pp.16-30
• /
• 1997

Application of digital filter to the wave analysis is studied using the observed data by wave gauge. Sea wave data obtained from wave gauge always include long wave frequency components. In order to estimate the sea wave parameters, we must re-analyzed wave data by using a digital filter and the concept of mean sea level correction method. By the wave by wave analysis and spectral methods, sea wave parameters on the basis of wave data obtained by the conventional method and digital filter are compared. The best-fitted design filter determined by the necessary conditions of frequency responses, can be obtained by calculating various transfer functions. Thus, to get the best the digital filter design, both Butterworth filter and Savitzky-Golay filter of digital filter are used in the frequency and time domain, respectively. Three cases of observation wave data are calculated by applying digital filter. The components of different frequency bands in the surf zone are coexisted in three cases. The wave data for wind wave components is computed using the digital filter the surf zone and off-surf zone, and based on the filtered data, wave parameters are calculated by the spectral analysis and wave by wave analysis methods, respectively. As a results, when sea wave data observed by wave gauge are analyzed, the Savitzky-Golay method is recommended which can well appear cut-off frequency by experimental choosing filter length in the time domain. The better mean sea level correction method is the Butterworth filter in the frequency domain.

• Applied Chemistry for Engineering
• /
• v.23 no.1
• /
• pp.35-41
• /
• 2012

This paper proposes a new methodology for the real-time on-line quality monitoring of biofuel processes through the integration of infrared spectroscopy and chemometrics. A method of Partial Least Squares (PLS) in Chemometrics is employed for quantitative analysis of key components in bioethanol products. After a number of preprocessing methods and variable importance in projection (VIP) are used, Savitzky-Golay method showed the best performance in terms of spectrum correction, noise reduction, and model maintenance. The proposed method allows us to economically forecast the concentration of multiple impurities encountered with the production of bioethanol. The proposed system is also accurate enough ($R^2$ > 0.99) to replace the laboratory analysis.

• Proceedings of the Korean Society of Near Infrared Spectroscopy Conference
• /
• 2001.06a
• /
• pp.1041-1041
• /
• 2001

Removal of variability in spectra data before the application of chemometric modeling will generally result in simpler (and presumably more robust) models. Particularly for sparsely sampled data, such as typically encountered in diode array instruments, the use of Savitzky-Golay (S-G) derivatives offers an effective method to remove effects of shifting baselines and sloping or curving apparent baselines often observed with scattering samples. The application of these convolution functions is equivalent to fitting a selected polynomial to a number of points in the spectrum, usually 5 to 25 points. The value of the polynomial evaluated at its mid-point, or its derivative, is taken as the (smoothed) spectrum or its derivative at the mid-point of the wavelength window. The process is continued for successive windows along the spectrum. The original paper, published in 1964 [1] presented these convolution functions as integers to be used as multipliers for the spectral values at equal intervals in the window, with a normalization integer to divide the sum of the products, to determine the result for each point. Steinier et al. [2] published corrections to errors in the original presentation [1], and a vector formulation for obtaining the coefficients. The actual selection of the degree of polynomial and number of points in the window determines whether closely situated bands and shoulders are resolved in the derivatives. Furthermore, the actual noise reduction in the derivatives may be estimated from the square root of the sums of the coefficients, divided by the NORM value. A simple technique to evaluate the actual convolution factors employed in the calculation by the software will be presented. It has been found that some software packages do not properly account for the sampling interval of the spectral data (Equation Ⅶ in [1]). While this is not a problem in the construction and implementation of chemometric models, it may be noticed in comparing models at differing spectral resolutions. Also, the effects on parameters of PLS models of choosing various polynomials and numbers of points in the window will be presented.