Brain MR Multimodal Medical Image Registration Based on Image Segmentation and Symmetric Self-similarity

Abstract

With the development of medical imaging technology, image registration has been widely used in the field of disease diagnosis. The registration between different modal images of brain magnetic resonance (MR) is particularly important for the diagnosis of brain diseases. However, previous registration methods don't take advantage of the prior knowledge of bilateral brain symmetry. Moreover, the difference in gray scale information of different modal images increases the difficulty of registration. In this paper, a multimodal medical image registration method based on image segmentation and symmetric self-similarity is proposed. This method uses modal independent self-similar information and modal consistency information to register images. More particularly, we propose two novel symmetric self-similarity constraint operators to constrain the segmented medical images and convert each modal medical image into a unified modal for multimodal image registration. The experimental results show that the proposed method can effectively reduce the error rate of brain MR multimodal medical image registration with rotation and translation transformations (average 0.43mm and 0.60mm) respectively, whose accuracy is better compared to state-of-the-art image registration methods.

1. Introduction

With the continuous updating of medical imaging equipment, a large number of advanced imaging technologies have emerged. Among them, magnetic resonance (MR) imaging technology has the advantages of protecting patients from radiation, high image resolution and obtaining stereoscopic images of brain and spinal cord, which has wide applications in medical diagnosis. In order to better identify the various organs and tissues of the human body, MR medical images can be generally divided into T1, T2 and PD weighted images (different weighted images display different pixel values on the same part), but medical image analysis needs to compare the different modal images of the same patient together to obtain the patient's comprehensive information. Firstly, it is necessary to solve the problem of strict alignment of different modal images, that is, image registration. However, factors such as intensity non-uniformity and noise of various modal images pose great challenges for multimodal medical image registration, which are caused by imaging instruments and differences in pixel value of different modal images for the same imaging site.

Multimodal medical image registration algorithms can be mainly divided into three categories: pixel-level registration [1-7], feature-level registration [8-14] and deep learning based registration [15-29]. One type of image registration method is the pixel-level registration which mainly investigates the pixel relationship between two images. It uses the whole pixel information of the image to perform coordinate transformation and determine the similarity measure, which transforms the registration problem into a unified optimization problem. The state-of-the-art methods of this category include the spindle centroid method [1-2], the cross-correlation method [3-4], and the maximum mutual information method [5-7]. Those methods have the disadvantages of low efficiency and sensitivity to noise, since it is registered by using the entire pixel of the image.

The second type of image registration method is the feature-level registration, and this type method firstly extracts the features reflecting the important information of the image based on the similar features between different modal images, and then finds the registration parameters to maximize the similarity. The feature selection strategy in this category are target edge [8-9], texture [10-12], and curvature [13-14]. This type of methods has a much higher registration efficiency than pixel-level registration methods, since it only uses pixels that describe the features. However, the registration accuracy of this type of methods depend largely on whether the extracted features are invariance between different modal images and whether the feature are easily extracted among all modal images.

The third type of image registration method is based on deep learning, and this type of registration approach is mainly divided into two categories:

• Supervised or weakly supervised based registration [15-18]. The production of gold standards for supervised learning is a big problem. Therefore, weak supervision is currently a research hotspot.

• Unsupervised based registration [19-22]. The unsupervised piece of self-supervised learning based on similarity measures is currently a research hotspot. But for multimodal registration problems, the choice of similarity measures is a big challenge. The deep learning based registration method did not develop until recent years, especially for multimodal images, and it still faces many challenges until now. In contrast to other fields in medical image process and analysis such as image segmentation [23-25] and denoising [26-28], deep learning based methods have yet not settled on the best way to apply this technique into medical image registration. The main challenges are as follows:

• Poor applicability. A method or set of parameters only applies to a particular modality or even to a specific dataset, and over-fitting problems are easy to occur during training of deep learning based methods. Some techniques had been used to overcome over-fitting problems. Tian et al. proposed a deep learning based approach called enhanced convolutional neural denoising network (ECNDNet) [28] which applied residual learning and batch normalization to make training easier.

• The data tagged is extremely limited. Deep learning is extremely dependent on data. However, a large number of annotated data is difficult to achieve. Some solutions have been proposed in recent years, for example, Pujitha et al utilized the generative adversarial network-based solution to overcome the sparsity issue of annotated data in medical domain [29]. For multimodal images, we should not only take how to process untagged data but also how to construct labels into consideration. For example, whether we should label data with mode property and standard transformation matrix, if so, how to co-training to ensure efficiency and training convergence.

• Evaluation of the registration is a difficult problem without gold standards. Many unavoidable factors, such as training set and hyperparameters, affect the result of deep learning based registration. Therefore, it is difficult to find a gold standard to evaluate such methods. For multimodal images, how to design registration cost function is a difficult problem, when we should take the accuracy and the training convergence into consideration.

In recent years, with the rapid development of medical image registration technology, the performance of multimodal MR medical image feature-level registration has been greatly improved. In order to improve the accuracy and efficiency of registration, the registration scheme is not only limited to the metrics between pixel-level of the floating image and reference image or the consistency information of feature- level, but also takes the local self-similarity information of the floating image and the reference image themselves into consideration. Rivaz et al. applied image local self-similar information to image registration [30]. They added local structural information of the image to the mutual information (MI) similarity measure, called self-similarity α MI (SeSaMI), which overcomes the drawback of MI similarity measure that only takes image grayscale information into consideration. Subsequently, they proposed contextual conditioned MI (CoCoMI) [31], which incorporates the contextual information of the image into the conditional mutual information metric and achieves good results in multimodal medical image registration. In addition, the modality independent neighborhood descriptor (MIND) [32] has also been widely used in multimodal medical image registration, which can extract immutability features in the local domain of the image itself, such as corners, edges, textures and so on. Subsequently, based on MIND, a novel neighborhood self-similar descriptor called self-similarity context (SSC) [33] was proposed, which is more robust to noise. Recently, Jiang et al. proposed the modality-independent local binary pattern (miLBP) local texture feature descriptor, which has also achieved good results in the field of registration [34]. The descriptor is robust to the intensity non-uniformity of the image. The registration accuracy in multimodal image registration is generally better than SSC. Subsequently, the robust self-similarity descriptor (RSSD) was proposed by Kurutach et al. [35] to overcome the shortcoming that the miLBP descriptor does not have rotational invariance, which further improves the accuracy of multimodal image registration.

Based on the advantages and disadvantages of these above methods, this paper proposes a registration method based on image segmentation and symmetric self-similarity of brain images. Our method utilizes an algorithm based on fuzzy C-means (FCM) clustering [36-37] to segment the brain MR images, and uses the approximate symmetric self-similarity information of MR brain slice images to eliminate the negative effects of asymmetric points under the SSD multimodal medical image registration framework. More particularly, two novel approximate symmetric self-similarity operators are proposed to constrain the floating and reference images. Experiments show that the registration error of the proposed method in brain MR multimodal medical image registration is lower than state-of-the-art image registration methods.

The contributions and novelty of the proposed method are as follows:

• Artificial modal image for registration based on image segmentation and image approximate symmetry is proposed for the first time.

• To guarantee consistency of each modal image, two kinds of constraint operators are proposed to handle images with approximate symmetry.

• Through the theory and experiment, the validity and applicable conditions of the proposed constraints are analyzed.

We organize this paper as follows, section II proposes the method, section III introduces the simulation experiment and analysis, section IV gives results and discussion and gives the concluding remarks is given in section VI.

2. The proposed Method

This section will briefly introduce a fuzzy C-means clustering algorithm called bias corrected fuzzy C-means (BCFCM) [38] for bias-field estimation and image segmentation. Moreover, an image symmetric axis extraction algorithm by self-registration method named mirror symmetry via registration (MSR) [39] will be sketched out. In addition, we will introduce the proposed registration scheme and explain how to take advantage of our proposed self-similarity symmetric operators to constrain segmented images for converting multimodal medical images into a unified modal in detail. Finally, we will analyze computational complexity of our proposed method.

2.1 Bias Corrected Fuzzy C-Means

Segmentation of medical images has always been a problem due to low image resolution, susceptible to external environmental interference, imaging tissue diversity and so on. Therefore, for the characteristics of specific image segmentation problems, different segmentation algorithms need to be designed. For example, mammogram image has a shortcoming of low contrast, Gupta et al. proposed an approach named linearly quantile separated histogram equalisation-grey relational analysis (LQSHE-GRA) [40], which improves overall contrast of given MI and segmentation accuracy. Ahmed et al. proposed a segmentation algorithm called BCFCM [38] in allusion to MR images, which can solve the intensity non-uniformity of MR image. Therefore, we use BCFCM algorithm to process our MR brain image.

Bias corrected fuzzy C-means is an algorithm for medical image segmentation and bias-field correction based on FCM which is a partition-based clustering algorithm. Its main idea is to make maximum similarity between objects divided into the same cluster, and the similarity between different clusters is minimum. The objective function is given as follows:

\(J_{m}=\sum_{i=1}^{N} \sum_{j=1}^{C} u_{i j}^{m}\left\|x_{i}-c_{j}\right\|^{2}, 1 \leq m<\infty\)       (1)

Where x_i represents the pixel value of the image to be divided, N indicates the total number of pixels. c_j represents the cluster centroid and C indicates the total number of categories of pixels to be divided. m is an arbitrary real number greater than zero. u_ij represents the membership degree, that is, the probability that the i -th pixel belongs to the j -th class, which needs to satisfy \(\sum_{j=1}^{C} u_{i j}=1\) (In image segmentation, the matrix U composed of u_ij is called the partition matrix).

Medical images usually have problems such as bias-field and noise, which have a negative impact on the segmentation of medical images. Therefore, on the basis of FCM, Yamany et al. proposed BCFCM [38] to add bias-field correction and neighborhood pixel information to image segmentation optimization formula. The object function is given as follows:

\(J_{m}=\sum_{i=1}^{N} \sum_{j=1}^{C} u_{i j}^{m}\left\|y_{i}-\beta_{i}-c_{j}\right\|^{2}+\frac{\alpha}{R^{2}-1} \sum_{i=1}^{N} \sum_{j=1}^{C} u_{i j}^{m}\left(\sum_{y_{r} \in \mathrm{N}_{R}}\left\|y_{r}-\beta_{r}-c_{j}\right\|^{2}\right), 1 \leq m<\infty\)       (2)

Where y_i represents the observed image pixel, that is, the pixel is affected by the bias-field. β_i represents bias-field, x_i = y_i − β_i indicates the image pixel after deviating from the field. α is the scale factor, which can control the influence of pixel neighborhood on image segmentation. Generally, when the image is noisy, the value of α increases appropriately. N_R represents the set of neighborhood pixels with the i -th pixel as the center of the neighborhood, and R is the side length of the neighborhood square window. y_r and β_r represent the neighborhood pixel and the bias-field of this pixel value respectively.

2.2 Mirror Symmetry via Registration

Mirror symmetry via registration is a self-aligned image symmetry axis detection method. It has a good effect on the detection of symmetric axis for approximate symmetric image, and it is robust to environmental factors such as noise and uneven illumination. The main steps of the method are given as follows:

• Set the vertical line passing the image center point as the initial axis of symmetry.

• Obtain a mirror image by mirror-transformed about the initial axis of symmetry.

• The original image is set as the reference image, and the mirror image is set as the floating image for rigid registration, then the rigid transformation matrix parameters are obtained.

• The symmetry axis of the original image can be obtained by the rigid transformation matrix parameters which obtained in the third step. The detailed process is described in [39].

In order to overcome the problem that the complete mirror symmetry condition is difficult to satisfy and the false detection or misdetection of the axis of symmetry due to uneven illumination and environmental noise, Cicconet et al. [39] did the following work in the third step to improve the performance of detecting the axis of symmetry.

• A sampling strategy based on random sample consensus (RanSaC) [41-42] was used to sample floating image (mirror image) blocks and obtain multiple sub-images, which were respectively registered with the target image (original image). The main purpose of the block is to detect the symmetry target sub-image (the registration image and the image to be registered are not two images satisfying the mirror symmetry, but the symmetry of an object in the image)

• Use normalized cross correlation (NCC) [43-44] as the registration similarity measure. This method uses the normalized correlation metric to quantitatively describe the correlation between the pixels of two images, which performed better than other similarity measures under the MSR framework.

2.3 Proposed Multimodal Image Registration Scheme

In this section, we propose a multimodal brain MR medical image registration scheme based on BCFCM and MSR. Since the pixel values of the corresponding positions of the multimodal MR brain images are quite different, the sum of squared differences (SSD) [45-46] similarity metric cannot be accurately registered directly, so we transform the original image and the reference image into a unified modality, so that the pixel values of the corresponding locations are as consistent as possible, and finally the registration is done using SSD metrics.

The flow chart of the framework is shown in Fig. 1. The explanation of steps is shown as follows:

• The BCFCM algorithm is used to segment the original floating image and the original reference image.

• The MSR method is used to detect the symmetry axis of the original reference image and the original floating image, and extract the symmetry axis equation respectively.

• Segment the images after processing of step 1 further by binarization processing.

• Approximate symmetry constraint on the image processed in step 3 according to the symmetry axis equation detected in step 2, the resulting images are recorded as a reference image and a floating image to register, respectively.

• Initialize rigid transformation matrix registration parameters.

• Get optimal transformation matrix registration parameters by registering images with SSD similarity metrics.

Fig. 1. Multimodal brain MR medical image registration framework

2.4 Image Segmentation and Binarization Proccessing

The process uses the BCFCM segmentation algorithm described above. The segmentation task of the image is to detect the target region of the original image and accurately distinguish the target object from the background. Therefore, the optimal membership degree u*_ij is obtained by using the formulation (2), and combined into an optimal partition matrix U* (the dimension of this matrix is one dimension more than the original image. The size of the last dimension is C , so this matrix is a multi-category grayscale partition matrix). Since the segmentation task at least distinguishes the target object and the background, and the background of the MR brain medical image is usually black, the total number of pixel categories to be divided when C needs to be satisfied with C ≥ 2 , and one of the cluster initial values should be set to 0, which is recorded as c_j=0 = 0. Finally, the segmentation sub-matrix with the background gray value U'^*_j=0 is selected from U^* as the segmentation matrix of the original image to segment the image. The formulation can be expressed as follows:

\(\mathbf{U}^{*}=\left[\begin{array}{c} \mathbf{U}_{j=0}^{\prime *} \\ \mathbf{U}_{j=1}^{\prime *} \\ \ldots \\ \mathbf{U}_{j=C}^{\prime *} \end{array}\right], C \geq 2, c_{j=0}^{*} \leq c_{j=1}^{*} \leq \ldots \leq c_{j=C}^{*}\)       (3)

Let the size of the pixel matrix U'^* be ×xy, and then we have:

\(\mathbf{U}_{j}^{\prime *}=\left[\begin{array}{cccc} u_{i=1, j}^{*} & u_{i=x+1, j}^{*} & \ldots & u_{i=x(y-1)+1, j}^{*} \\ u_{i=2, j}^{*} & u_{i=x+2, j}^{*} & \ldots & u_{i=x(y-1)+2, j}^{*} \\ \ldots & \ldots & \ldots & \ldots \\ u_{i=x, j}^{*} & u_{i=2 x, j}^{*} & \ldots & u_{i=x y, j}^{*} \end{array}\right], x y=N\)       (4)

Where N indicates the total number of pixels, u*_ij is the optimal membership degree, which indicates the optimal probability that the i -th pixel point belongs to the j –th class.

The U′*_j=0 is actually a background probability matrix, that is, the normalized pixel value of the segmented image U′*_j=0 is determined by the pixel distance of the original image from the background category’s cluster centeriod c*_j=0. The smaller the difference, the greater the probability that the original pixel value belongs to the background, and the closer the normalized pixel value of the segmented image at this position is to 1. In order to facilitate the execution of the subsequent steps, the segmented image is subjected to a binarization threshold segmentation process. The image segmentation matrix after binarization threshold segmentation is denoted as P , and the formula is given as follows:

\(\mathbf{P}=\mathrm{p}\left(\mathbf{U}_{0}^{\prime *}\right)\)

\(\mathrm{p}(g)=\left\{\begin{array}{l} 1, g<\delta \\ 0, g \geq \delta \end{array}\right.\)       (5)

Where p(g) is the threshold segmentation function, and δ represents the threshold, under ideal condition, we set δ =1 (According to the above description, the BCFCM partition matrix is actually a probability matrix. This task is divided according to the background. Ideally, the probability that the background belongs to the background class is 1).

2.5 Symmetry Detection of Image and Symmetry Constraint of Binary Segmentation Images

Since the multimodal MR brain slice image satisfies the approximate symmetry property, the MSR can be used to detect the symmetry axis of the original image and obtain the symmetry axis equation, which can be denoted as l . Let the matrix P after symmetry constraint be denoted as S , and then the formula can be given as follows:

S = s(P)

\(\mathrm{s}_{\text {sparse }}(\mathrm{h})=\left\{\begin{array}{l} 0, h=0 \\ 0, h=1 \text { and } \sum_{a=1}^{A^{2}} h_{a}^{\prime}=0 \\ 1, h=1 \text { and } \sum_{a=1}^{A^{2}} h_{a}^{\prime} \neq 0 \end{array}\right.\)

\(\mathrm{s}_{\text {dense }}(h)=\left\{\begin{array}{l} 1, h=1 \\ 1, h=0 \text { and } \sum_{a=1}^{A^{2}} h_{a}^{\prime} \neq 0 \\ 0, h=0 \text { and } \sum_{a=1}^{A^{2}} h_{a}^{\prime}=0 \end{array}\right.\)       (6)

Where s_sparse (h) and s_dense (h) are the symmetric constraint functions of sparsity and density, respectively.

In actual, s(h)= s_sparse (h) or s(h) = s_dense (h). h′ is the symmetry pixel point about the axis of symmetry l , and h′_a represents the neighborhood pixel value centered on h′ ( h′ included), A² is the square window size of the symmetric point neighborhood, indicating the number of pixels in the square window of the symmetric point neighborhood. By means of the constraint, the pixel points that do not satisfy the approximate symmetry of the target image can be set to the same value as the symmetry point, which can eliminate asymmetrical points and improve the consistency of the registration of each mode image under the SSD similarity measure.

The underlying reasons to propose the two symmetry constraints are as follows:

• Premises and conditions: the multimodal MR brain slice image satisfies the approximate symmetry property, the MSR can be used to detect the symmetry axis of the original image and obtain the symmetry axis equation

• Challenges and motivation: The noise caused by the shooting instrument and the environment (missing part of the image, uneven illumination, etc.) adds inconsistency to different modal images. Moreover, different modal images are difference in pixel’s values. Therefore, it is a challenging task to register mutimodal images.

• The proposed constraints effects: Use image approximate symmetry to effectively ensure the consistency of each modal image, thereby reducing the difficulty of registration.

The flow chart of the proposed method is shown in Fig. 2 and Fig. 3.

Fig. 2. The flow chart of MR image processing with the sparse symmetric constraint

Fig. 3. The flow chart of MR image processing with the dense symmetric constraint

Fig. 2 is the diagram of using s_sparse (h) constraint condition, Fig. 2 (a) is the original image selected from a real patient MR PD modal brain image of the RIRE dataset [47], Fig. 2 (b) shows the axis of symmetry colored by green, Fig. 2 (c) is the image divided by the background class after BCFCM image segmentation, Fig. 2 (d) is the binarized image processed according to the threshold δ , which is set to 0.8, Fig. 2 (e) is the image constrained by sparse symmetry condition and parameter A is set to 1. Fig. 3 is the diagram of using s_dense (h) constraint condition, Fig. 3 (a) is the original image selected from a real patient MR T2 modal brain image of the RIRE dataset, the process of Fig. 3 (b)-(d) is the same as Fig. 2 (b)-(d), Fig. 3 (e) is the image constrained by dense symmetry condition and parameter A is set to 1.

2.6 Computational complexity of proposed method

We will refer [40] to analyze the computational complexity of our proposed algorithm in this subsection. The complexity of our proposed method at each stage is shown in Table 1 (we are considering the image size as n₁ x n₂ in this paper).

Table 1. Computational complexity of the proposed method at different stages

Here, m₁ ×m₂ is the matrix size composed of pixels selected from Step 2, so we have m₁m₂  ≤ n₁n₂. And S is the area of neighborhood domain whose size is ×AA. Hence, the total computational required is O(S·m₁m₂)+O( n₁n₂). This computational complexity is very efficient compared to miLBP and RSSD whose computational is at least O((S +1) n₁n₂).

3. Experimental Results and Analysis

In order to better evaluate the proposed method, we will briefly introduce the datasets and evaluation criteria used in this section. And then, the evaluation experiments are introduced emphatically.

3.1 Datasets

The RIRE dataset [47] is based on the retrospective medical image registration assessment (RIRE) project led by Professor Fitzpatriek of Vanderbilt University, the world's most well-known medical image registration assessment. This dataset provides a large number of patient case data, including three-dimensional brain images (T1, T2 and PD) with image sizes as 256×256×25 voxels.

The BrainWeb dataset [48] is selected from the BrainWeb Anti-Shake Brain MR Image Database at the Brain Imaging Center of the University of McGill University. This database contains simulated brain MR image data based on two anatomical models: normal and multiple sclerosis (MS), and provides three-dimensional brain MR images under three synthetic modalities i.e. T1, T2, PD. These providing images are all registered brain images and contain different scan thicknesses, noise, and bias-fields. The image size is 181×217×181 voxel, and the voxel size of each dimension is 1mm.

3.2 Evaluation Standard

At present, when comparing registration algorithms, researchers use target registration error (TRE) [35,49] as the evaluation standard in most situations, which is defined as the distance between the registered image and the physical target position. It can be expressed by the following formula:

\(\operatorname{TRE}(D)=\sqrt{\left(D_{x_{i=1}}-D_{x_{i=1}}^{\prime}\right)^{2}+\left(D_{x_{i=2}}-D_{x_{i=2}}^{\prime}\right)^{2}+\ldots+\left(D_{x_{i=N}}-D_{x_{i=N}}^{\prime}\right)^{2}}\)       (7)

Where D represents the pixel point of the registered image, D_xi represents the coordinate position of pixel points in the image after registration, D′_xi is the target physical coordinate position corresponding to D_xi , and N indicates the size of the image dimension. For ease of research and analysis, the above equation can be transformed into the following form for registration involving only translation and rotation transformation:

\(\operatorname{TRE}(D)=\left\|\mathbf{T}_{0}-\mathbf{T}_{D}\right\|\)       (8)

Where T₀ is the gold standard transformation matrix for converting the image to be registered into the target reference image, and T_D is the transformation matrix of the image to be registered into the image that has been registered by the algorithm. In the actual simulation experiment, the two images that have been registered are respectively used as a floating image and a reference image, and then the reference image is subjected to rigid transformation (translation and rotation transformation), which is the gold standard transformation T₀ . The evaluation experiments of this paper all use the TRE index as the evaluation standard.

3.3 Evaluation Experiment

In order to analyze and explain that the proposed method is effective to what kind of noise, we select a completely symmetrical image as the experimental object, which is shown in the first picture of Fig. 4. The rest parts of Fig. 4 are BCFCM segmented image, binary image by binary threshold segmentation, sparse constraint image and dense constraint image, respectively. The first picture of Fig. 5 has been added background noise and the first picture of Fig. 6 simulates the image with part missing. The rest parts of processing are the same as Fig. 4. The free parameter δ is set to 0.8 and A is set to 1.

Fig. 4. Noiseless image processing effect

Fig. 5. Image with noise in the background processing effect

Fig. 6. Image with part missing processing effect

From the comparison of the forth image and the fifth image in Fig. 5, we found that if the source image is polluted by slightly background noise, sparse constraint operator has a good performance of denoising and dense constraint operator plays a side effect. Such phenomenon is opposite when using dense constraint operator to process image with part missing from the comparison of the forth image and the fifth image in Fig. 6.

In order to further test the registration on how to choose the symmetry constraint in actual complex scene and the influence of different symmetry constraints on the registration performance, we compare the effects of sparse constraints and dense constraints by TRE evaluation standard. We selected the RIRE dataset MR brain slice T1, T2, and PD modal images to perform experiments as floating images and reference images, respectively, and the PD and T2 modal images are shown in Fig. 2 and Fig. 3. The source data image size is 256×256 pixels. The experiment is divided into two parts: the first part only considers the influence of multimodal brain MR image registration caused by the rotation of the image around the center of the image, where the rotation angle range is [−20° , 20°] ; The second part only considers the effect brought by the image displacement, where the horizontal and vertical displacement ranges are [−20, 20] pixels. The experimental comparison results are shown in Table 2. The content before “-” indicates the floating image and the content after “-” indicates the reference image. For example, “sparse-dense” in the table indicates that the floating image is sparsely constrained, and the reference image is densely constrained. As shown in Fig. 3 (b), it can be found that the segmented image of the T2 mode is too much affected by the noise and the bias-field, so the image after using the sparse constraint will be inferior, but the T1 and PD modal images do not have this problem, so the “sparse-dense” condition contrast experiments only consider T1 and PD sparse constraints, T2 dense constraints.

Table 2. Symmetric constraint comparsion experiment

It can be found from Table 2 that when both the floating image and the reference image use sparse constraints, only T1-PD can be registered, but the performance is the best of all. The reason is that the segmentation image after sparseness constraint of the T2 modal image is seriously missing, which results in the registration requirement is not met, but this phenomenon does not exist in T1-PD registration. In addition, the use of dense constraints may cause image noise to expand and bring about image segmentation by mistake, which has a certain negative impact on the registration. When the T2 mode uses dense constraint and the T1 or PD mode uses sparse constraint, the registration performance is worse than the floating image and the reference image are both densely constrained. Therefore, after comprehensive consideration, if the segmented images of reference image and the floating image used in registration are little influenced by bias field and noise, sparse symmetric constraint should be adopted. Otherwise, the dense symmetric constraint should be used to process the image.

In conclusion, the sparse constraint is suitable for images that are less affected by noise, especially the image does not have a large area missing. When the image noise is large, especially there is a large area missing, the dense constraint is used. That is, dense constraint is the second option. It can be understood that the sparse constraint solves the influence of slight noise (background noise) and the inconsistency of each modal image itself on image registration; yet the dense constraint solves the image registration problem with severe noise more, especially for missing part of image. The purpose of the sparse-dense experiment is to use the sparse and dense constraints respectively for the two images with different noise conditions to improve the registration accuracy, but unfortunately, experiments show that the registration of the reference image and the floating image using the same constraint operator has better performance.

In order to evaluate the performance of the proposed method for the registration of brain MR multimodal images, we selected the T1, T2 and PD modal images of the normal brain in the BrainWeb dataset for rigid registration experiments. The parameters of the dataset are: slice thickness is 1mm, noise intensity is 3% (calculated relative to the brightest tissue), and intensity non-uniformity is 40%. In the rotation experiment, the rotation angle range is [−20° , 20°] , in the translation experiment, the translation range is [−20,20] mm. The comparison methods used in this experiment are MI, SSC, miLBP, and RSSD. The experimental results of the comparative experimental methods are derived from the literature [20] (the experimental datasets and experimental parameters used are the same as those described in [20]). In this experiment, our proposed method uses sparse symmetric constraints for each modal image, where the threshold value δ of the segmented image is 0.8; the size of the neighborhood in the sparse symmetric constraint function is 3×3, that is, A = 3. The experimental results are shown in Table 3.

From the comparison experiments in Table 3, it can be found that the registration performance of our proposed method is significantly better than other methods. Whether in the rotation or translation experiments, our proposed method (average 0.43mm and 0.60mm) has a big improvement compared to RSSD (average 0.72mm and 1.22mm). Especially for the translation experiments, the proposed method error is below 1mm, while the errors of the other methods all beyond 1mm.

Table 3. Multimodal rigid registration experiment

In addition, in Table 2 and Table 3, it can be found that on the RIRE dataset, the registration of the T1 and PD modal images obtained by our proposed method is better, and the T2 modal images are relatively poor. On the BrainWeb dataset, the T2 and PD modal images have better registration results for the images to be registered, and the T1 modal images are relatively poor. It can be seen that our proposed method is greatly affected by the noise caused by each modal shooting environment, but not the modes themselves.

The registration performance of the proposed methods T1-T2 and T1-PD is similar, and PD-T2 can achieve almost error-free registration. It can be seen that the proposed method has excellent performance for rigid body registration of multimodal MR brain images. The visual assessment of proposed method is shown in Figs. 7-9, respectively.

Figs. 7-9 are the visual assessment of registration performance. Here, the registered images are consisted of the reference image and registered floating image. We can find that no matter which mode the reference images and the floating images are, the registered images are of a good performance of alignment in human eyes.

Fig. 7. Visual assessment of multimodal rigid registration with T1-T2

Fig. 8. Visual assessment of multimodal rigid registration with T1-PD

Fig. 9. Visual assessment of multimodal rigid registration with PD-T2

The proposed method first uses binarization threshold segmentation for the segmented image, which overcomes the difference of gray information of different modes to some extent. Subsequently, our proposed the symmetric constraint operator is used to further transform the image to an artifical mode for registration. Since noise generally does not have symmetric approximation characteristics, this step eliminates noise interference while preserving the inherent information of the image. So our proposed method is significantly better than other methods.

3.4 Some suggestions for obtaining free parameters on our proposed method

δ and A are two free parameters for our proposed method. The purpose of δ is to distinguish background from foreground. Sometimes, owing to noise and shooting instrument affection, the probability of medical image’s background belongs to background class processed by BCFCM cannot be 1 but close to 1. After many experiments, we find that the range of δ between 0.8 and 0.9 can distinguish background from foreground preciously. Of course, some complex images need adjustment. If foreground is mistaken for background, δ should be increased, and if background is mistaken for foreground, δ should be decreased.

The free parameter A adjusts the size of neighborhood domain using for constraint operators. If choosing dense operators, A =1 is recommended. However, if there are seriously image missing on both sides of the symmetry, A should be increased. If choosing sparse operators, visual assessment of variation can be seen in Fig. 10. We can find that if the background noise is dense on both sides of the symmetry, A =1 is recommended. Otherwise A = 3 is recommended due to error of interpolation. However, if the axis detection by MSR has a big error due to incomplete symmetry of the image, A should be increased.

Fig. 10. Visual assessment of variation according to the change of the parameter value

4. Results and Discussion

In this section, based on the previous introduction and experiments, we mainly discuss how each contribution is achieved. Three parts are given as follows:

• We use the BCFCM algorithm as the image segmentation method. The BCFCM algorithm considers the influence of image illumination unevenness on image segmentation, and adds the corresponding constraint operator. It is an algorithm that is very suitable for MR image segmentation. On this basis, in order to ensure the consistency of the modal images, we performed threshold binarization. We also use the MSR symmetry axis detection algorithm to propose the symmetry axis of the image. This method is based on the registration detection algorithm. It has the following advantages: only the symmetrical object in the image can detect the symmetry axis, and the whole image is not needed to satisfy the symmetry; Symmetrical objects do not need to be completely symmetrical but only satisfy approximate symmetry. Finally, by using one of our proposed two operators, we can transform the images into a unified artificial mode for registration.

• By reading relevant literature and experiments, we found that different MR modal images are quite different in pixel level, and cannot be completely solved even after segmented by BCFCM, since the noise seriously affected during the actual shooting process. Therefore, we propose two approximate constraint operators to solve the problems of noise of each modal image and retain the uniform information of each mode as far as possible.

• Through the theory and experiment, we can get the conclusions as follows: the sparse and dense constraint are two proposed constraint operators for multimodal images with approximate symmetry property. Here, the sparse constraint is suitable for the image without large area missing and the dense constraint is the second option for images with large area missing.

5. Conclusion

Under the SSD similarity metrics registration framework, we propose a brain MR multimodal medical image registration scheme based on image segmentation and symmetric self-similarity, which converts different modal images into a unified artificial modality. In our work, we use BCFCM to segment multimodal medical images and extract target regions in medical images. This method overcomes the influence of intensity non-uniformity in images to some extent; on this basis, we perform binarized threshold segmentation on the segmented image processed by BCFCM, which is beneficial to eliminate the low-frequency noise in the target image. Finally, considering that the multimodal brain MR medical image has the approximate symmetry of the left and right brain, we propose two kinds of the self-similarity constraint operator, which are beneficial to further eliminate the local noise in the images and ensure the consistency between the multimodal images. Compared with the current registration schemes for processing modal images with self-similar descriptors (such as SSC, RSSD, and miLBP), our proposed method has obvious advantages in registration accuracy.