APPLICATION OF MACHINE LEARNING FOR RECOGNIZING SURFACE WELDING DEFECTS IN VIDEO SEQUENCES

: The paper offers a solution to the problem of detecting and recognizing surface defects in welded joints that appear during tungsten inert gas welding of metal edges. This problem belongs to the machine vision. Welding of stainless-steel edges is carried out auto - matically on the pipe production line. Therefore, frames of video sequences are investigated. Images of some welding defects are shown in the paper. An algorithm proposed by the au - thors is used to detect welding defects in the video sequence frames, the efficiency of which has been confirmed experimentally. The problem solution of welding defects recognition is based on the use of traditional machine learning methods: support vector machine and artifi - cial neural network. To build classification models, a labeled dataset containing automatically extracted texture features from the areas of welding defects detected in the video sequences was created. An analysis was performed to identify the strength of the correlation of texture features between each other and the dependent variable in the dataset for dimensionality reduction of the feature vector. The models were trained and tested on datasets with dif - ferent numbers of features. The quality of the classification models was evaluated based on the accuracy metric values. The best results were achieved by the classifier built using the support vector machine with a chi-square kernel on a training sample with two features. The build models allow automatic recognition of such welding defects as lack of fusion and metal oxidation. The computational experiments with real video sequences obtained with a digital camera confirmed the possibility of using the proposed solution for recognizing surface weld - ing defects in the process of manufacturing stainless steel pipes.


Introduction
Nowadays, machine learning methods are popular for solving problems in various spheres.Machine learning is many different methods of discovering patterns in data.Algorithms based on machine learning methods are also used for pattern recognition.Pattern recognition is a computer vision task that consists in detecting and classifying any visual objects in digital images.The purpose of applying machine learning for pattern recognition is to create models that allow to obtain information about the belonging of a visual object to one or another class.Pattern recognition involves the tasks of detecting and classifying surface defects of products [1] occurring in industrial production, in particular, defects in fabric [2], marble slabs [3], steel plates [4], rolled steel sheets [5], and various steel products [6].One of the research directions is the recognition of welding defects in metal products: [7], welding defects on an assembly line of fuel injectors [8], for pipelines and pressure vessels [9], on the surface of the engine transmission [10].
The paper deals with the production of welded flexible stainless steel pipes.Such pipes are used in heating and water supply systems.Flexible pipes are made of certain grades of stainless steel.Standard pipe diameter can be 15, 20, 25, or 32 mm, and the wall thickness of the pipe is about 0.3 mm.This is stated in the manufacturer's technical documentation.The manufacturing process of such pipes is quite complex.All stages of production are carried out on an automated line.After forming the steel strip, the edges of the metal are welded in automatic mode.Tungsten inert gas (TIG) welding is used for this purpose.The quality of TIG welding can be influenced by various factors: metal quality, quality of welding materials, voltage surges in the electrical network, etc.Under the influence of various factors, surface defects of welding may appear, such as burn-through, lack of fusion, metal oxidation, and other defects described in normative documents.Fig. 1 shows images of (a) a flexible corrugated stainless steel pipe with a lack of fusion welding defect, (b) a non-corrugated pipe with a lack of fusion, (c) a non-corrugated pipe with metal oxidation, (d) a non-corrugated pipe with burn-throughs, (e) a non-corrugated pipe with a defect-free welded joint.Welding defects lead to leaks, low performance of manufactured pipes, and manufacturing defects.Welding quality is controlled by personnel who constantly visually inspect the welded joints after welding on the production line.Subjectivity and human factors can influence the result of visual inspection.If a defect appears, the personnel should promptly identify and eliminate the cause.The timely response of personnel will reduce the number of defective products in production.As a result, production costs are reduced.Therefore, it is advisable to automate the quality inspection of metal edge welding on the production line.When a defect is automatically detected, personnel are alerted and problems are corrected.This inspection should be carried out immediately after welding the pipe before corrugating the pipe.
It can be seen in Fig. 1, that welding defects contrast with a defect-free welded joint.When the metal is not fused, the surface shows a lack of connection between the edges.Metal oxidation can be either slight or strong with the appearance of steel temper colors.Therefore, welding defects can be detected and recognized quite accurately in images automatically.However, it is necessary to take into account various factors that can affect the obtained images: changes in illumination, camera characteristics, etc.
Various image processing methods are used to recognize surface welding defects in images to perform preprocessing, segmentation, localization, and classification of defects [11].Preprocessing is performed to improve image quality: contrast enhancement and noise removal.An important stage of recognition is to determine the presence of a welding defect in the image, and if it is present, to separate the defect area from the main background of the image and localize it.Then, to clarify the defect type, the classification is solved using machine learning methods.
To solve the problem of recognizing welding defects, specifically to build classifiers, researchers use digital image sets obtained from a camera [7,9], digital radiographic images [11,12], and frames of video sequences.Researchers propose various solutions based on both traditional machine learning and deep learning methods to recognize welding defects in images.The review paper by Wenhui Hou et al. [11] summarizes the results of scientific works on weld defect detection in radiographic images.The paper states that textural and geometric features are widely used to train models for classifying welding defects.A key factor is the selection of an algorithm for building the classifier.The review [11] mentions k-NN (k-Nearest Neighbor), ANN (Artificial Neural Network), SVM (Support Vector Machine), and CNN (Convolutional Neural Network).Hongquan Jiang et al. used a dataset consisting of texture features of welding defects to build a classifier and evaluate its quality [13].The researchers observed that texture features are partially redundant concerning each other.Therefore, the authors [13] applied the PCA (Principal Component Analysis) method for dimensionality reduction.They used SVM to build a model based on the obtained principal components.This solution allowed them to achieve a classification accuracy of 90.4%.Rajesh V. Patil and Y. P. Reddy [14] presented methods for the detection and classification of weld defects using texture features based on SVM and ANN algorithms, and the maximum accuracy values based on test data were 98.75 and 97.5% respectively.Dong Shaohua and co-authors [15] extracted 14 geometric and textural features of welding defects, created a dataset and built an SVM multi-classifier.The accuracy of the model was more than 90%.In the presence of a sufficiently large amount of labeled data, and computational and time resources, researchers use DNNs (Deep Neural Networks) [8,16].Such networks do not require feature extraction and show a high percentage of recognition accuracy.Shiraz Ajmi et al. [17] solve the problem of classifying weld defects by increasing the dataset using augmentation and deep learning.In this work [17], the defect detection problem is not solved.
The authors of the works investigate welding defects occurring when using different types of welding and materials.Research is needed in the area under consideration -tungsten inert gas welding.The authors of the paper recorded video sequences using a digital camera during the manufacturing of corrugated stainless steel flexible pipes.The obtained video sequences contain frames with such TIG welding defects as lack of fusion and metal oxidation.The set of welding defect images is limited because welding defects do not frequently appear in the production process.For this reason, the authors limited themselves to traditional machine learning methods to solve the problem of recognizing welding defects.
The purpose of this paper is to solve the problem of recognizing surface defects of TIG welding in video sequences using traditional machine learning methods.It requires continuous processing of frames, defect fixation in case of its appearance in the frame, and defect classification.DOI: 10.37943/16RYGE2979 © Mariya Yemelyanova, Saule Smailova A distinctive feature of this work is the use of an algorithm proposed by the authors to automatically detect various TIG welding defects in the frames of video sequences.In addition, a labeled dataset created by the authors is used to train and test the classifiers.

Materials and Methods
One of the important steps in recognizing surface welding defects in a video sequence is to determine the presence or absence of a defect in the current frame.If there is a defect, segmentation of the defect area is performed.To solve this problem, the algorithm proposed by the authors of this paper in [18] is used.The algorithm is based on the method of modeling and background subtraction.The idea of the algorithm is to subtract the current frame of the video sequence from the image model, in case of a defect in the frame, the difference image in the form of a binary mask will contain a significant number of white pixels, which indicates a defect.In addition, the author proposes an algorithm for the automatic detection of weld defects in video sequences based on a comparison of brightness histograms [19].The similarity of brightness histograms of the current frame and the model image is determined by the correlation method.Based on the research, the proposed algorithms showed good enough results [18,19].Therefore, these algorithms can be used for the detection and localization of TIG welding defects in video sequences.
There are the following steps for building the classifiers shown in Fig. 2: feature extraction from the area of detected defects, feature selection, data labeling, splitting the dataset into training and test samples, selecting and applying machine learning algorithm, building models, and evaluating the quality of classification models.The following describes the building of classifiers according to these steps.

Figure 2. Stages of classifier building
There were 545 fragments of stainless steel pipe images with welding defects as a result of processing the obtained video sequences using the algorithm given in [18].In addition, 32 areas with defects were obtained manually from pipe images with welding defects.There was a lack of fusion in 265 fragments, and metal oxidation was visible in 312 fragments.
Twelve texture features were extracted to create a dataset from all defect areas automatically highlighted in the frames using the algorithm given in [18]: angular second moment, contrast, correlation, sum of squares, inverse difference moment, sum average, sum variance, sum entropy, entropy, difference variance, difference entropy, information measures of correlation.Textural features are described in [14].The values of texture features are used to create a feature vector that describes the defect.The created dataset consists of 577 vectors containing real values of texture features.The obtained numerical values of texture features have different ranges of values and have no units of measurement.
Texture features show different aspects of texture.The second angular momentum reflects the homogeneity and coarseness of the texture.The contrast reflects the sharpness of the image and the depth of "furrows".The correlation reflects the consistency of the image texture.Inverse difference moment characterizes the "smoothness" and homogeneity of the image.Entropy is associated with randomness and non-uniformity.Other statistical estimates correspond to the concepts of mathematical statistics.
Since a strong correlation can be observed between some texture features, feature selection is required to eliminate multicollinearity.Correlation is a measure of similarity between two features.If two features are linearly dependent, their correlation coefficient is ±1.If there is no relationship between features, the correlation coefficient is 0.
The CFS (Correlation-based Feature Selection) method estimates features based on the hypothesis that good datasets contain features that are not correlated with each other but are correlated with the target variable.A simple way to identify highly correlated features is to construct a matrix of pairwise correlation coefficients (1).
The correlation coefficient thresholds are set heuristically for feature selection.The selection of a machine learning algorithm is an important step in solving the classification problem.We propose SVM and ANN algorithms for building classification models.
The SVM algorithm finds the optimal hyperplane in an N-dimensional space to separate objects into classes while maximizing the distance between the hyperplane and the nearest objects of different classes.The nearest objects to the hyperplane are the support vectors.There are kernels in SVM: linear kernel, Radial Basis Function (RBF) kernel, chi-square kernel, and others.Each kernel has its parameters that require tuning to improve the quality of the model.
ANN consists of artificial neurons connecting each other in various ways and is organized into layers.ANN is a machine learning model that is used in supervised learning.In this research, a labeled structured dataset is used for the training and testing models.The neural network requires tuning of training parameters.
Various metrics, such as accuracy, precision, and recall are used to evaluate the quality of classification models [20].Accuracy is the proportion of objects for which the class is correctly defined.This metric is used when there is no class imbalance.In this research, we use the created dataset with a uniform number of feature vectors for two classes of defects -lack of fusion and metal oxidation.

Results and Discussion
To solve the problem of recognizing surface defects of TIG welding in frames of video sequences used: the integrated development environment Visual Studio 2019 (programming language C#), library EmguCV, and platform Accord.NET.
The dataset consisting of texture features of welding defects is generated as a CSV file.The data is split into training and testing samples randomly in the ratio of 70% to 30%.
Feature reduction in the dataset can affect the speed of feature extraction for recognition when processing video sequence frames as well as the accuracy of the trained model depending on the training algorithm.In case of a strong correlation between several features, one of them can be deleted from the dataset.The matrix of pairwise correlation coefficients is calculated according to (1) to estimate the correlation between traits.The result is shown in Fig. 3, where 1 -angular second moment, 2 -contrast, 3 -correlation, 4 -sum of squares, 5 -inverse difference moment, 6 -sum average, 7 -sum variance, 8 -sum entropy, 9 -entropy, 10 -difference variance, 11 -difference entropy, 12 -information measures of correlation.

Figure 3. Matrix of paired correlation coefficients of dataset features
There is a strong correlation between some textural features (correlation coefficient more than 0.7).Correlation coefficients between the dependent variable (class) and 12 textural features were also calculated.The following values according to the numbering were obtained: -0.310, 0.811, 0.117, 0.818, -0.377, 0.821, 0.824, 0.717, 0.809, -0.738, 0.848, 0.592.The target variable is practically not influenced by such features as angular second moment, correlation, inverse difference moment, and information measures of correlation.The other features (contrast, sum of squares, sum average, sum variance, sum entropy, entropy, difference variance, difference entropy) influence the dependent variable, but there is a multicollinearity between them.
To investigate the influence of features and their number on the quality of the classification model, training was carried out with the different number of features using SVM and ANN algorithms with defined parameters.When training the classifier using SVM, the following kernels were used: linear kernel, RBF kernel, and chi-square kernel with the parameters C=100, gamma=0.005.Parameter C limits the importance of each object.The gamma parameter specifies the degree of proximity of object locations.
A neural network (perceptron) was selected to build the classifier.The backpropagation of the error algorithm was used for training.The neural network was trained with the following parameters: number of inputs (number of features in the dataset -12); number of neurons in the hidden layer -12; number of neurons in the output layer -2 (number of classes); activation function of neurons -sigmoid; number of iterations -500, momentum -0.5, RMSE -0.001.Fig. 4 shows the architecture of the neural network.Table 1 shows the results of the quality evaluation of the built models.It can be seen from the table that the dimensionality reduction increased classification accuracy when using the SVM algorithm to train the model.The classifier with a chi-squared kernel has the highest accuracy value.ANN-based classifiers performed well when trained on a training set of 12 features.The algorithm [18] and models were tested using real video sequences containing more than 40000 frames with weld joints.When processing the frames, a region of interest of 1010×135 pixels was selected.Fig. 5 shows some results of the joint work of the algorithm for the detection of welding defects proposed by the authors of this paper in [18] and the classifier trained based on SVM, which showed the best results in the testing process.When a defect appears on a video sequence frame, the defect is automatically detected and localized [18].After that, textural features are automatically extracted from the highlighted defect area, and the trained classifier determines the defect class.The defect areas were marked with a red rectangular frame, and such defects as oxidation metal and lack of fusion were classified in frames.

Conclusion
The paper proposes a solution to the problem of recognizing surface defects of TIG welding in video sequences using traditional machine learning methods.To solve the problem, a dataset consisting of texture features extracted from the areas of detected defects is created.The values of texture features were analyzed and multicollinearity between some features was revealed.Considering the analysis, classification models were trained and tested on datasets containing different numbers of features using SVM and ANN algorithms.It can be concluded that the solution is applicable recognizing welding defects in video sequences based on the research carried out.The proposed solution may be one of the solutions for the visual quality inspection system of welded joints.
Since there were not enough images with other welding defects, for example, burn-through, the built models can identify only two classes of defects: lack of fusion and metal oxidation.In the future, authors will plan to increase the created dataset and train and test classifiers on the updated dataset to recognize other types of defects.

Figure 1 .
Figure 1.Images of welded joints: (a) a flexible corrugated stainless steel pipe with a lack of fusion; (b) a non-corrugated pipe with a lack of fusion; (c) a non-corrugated pipe with metal oxidation; (d) a non-corrugated pipe with burn-throughs; (e) a non-corrugated pipe with a defect-free welded joint

Figure 5 .
Figure 5.Some results of recognizing welding defects in video sequences using the SVM-based model: (a) metal oxidation; (b) lack of fusion

Table 1 .
Accuracy metric values for models based on SVM, ANN