A Survey on Content Aware Image Resizing Methods

Abstract

With the advancement in the field of image processing, images are being processed using various image processing algorithms. Nowadays, many efficient content-aware image resizing techniques are being used to safeguard the prominent regions and to generate better results that are visually appealing and pleasing while resizing. Advancements in the new display device with varying screen size demands the development of efficient image resizing algorithm. This paper presents a survey on various image retargeting methods, comparison of image retargeting results based on performance, and also exposes the main challenges in image retargeting such as content preservation of important regions, distortion minimization, and improving the efficiency of image retargeting methods. After reviewing literature from researchers it is suggested that the use of the single operator in image retargeting such as scaling, cropping, seam carving, and warping is not sufficient for obtaining satisfactory results, hence it is essential to combine multiple image retargeting operators. This survey is useful for the researchers interested in content-aware image retargeting.

1. Introduction

Nowadays multimedia contents such as images and videos available on the internet are growing day by day. Various display devices having different resolution and aspect ratio are used to access multimedia contents. To fit the images on varying screen size efficient image resizing algorithms are required that can preserve important image contents. To achieve image retargeting two traditional image retargeting operators are used: uniform scaling and cropping. Due to some drawbacks in these operators some advanced and efficient methods are required to be developed. Resizing of the image with the help of a scaling operator produces image distortion if the value of scaling factors is too large or too small. On the other hand, the cropping operator produces information loss during image resizing. Avidan et al. [1] suggested a simple image retargeting operator i.e. seam carving that can be used for a variety of image manipulations such as aspect ratio change, image retargeting, content amplification and object removal. A data structure for multi-size images had been defined to support continuous resizing ability in real-time. The content-aware image resizing methods comprises of 2 main steps: (1) computation of saliency map which measures important regions, (2) Images resizing operation which use image resizing operator. This paper introduces various methods of saliency measures and operators of image resizing. Image retargeting algorithms preserve important regions of the image hence estimation of the location of salient regions present in the image is an important part. Pixels, regions which belong to salient object contains low and high-level features which attract human visual attention and must be preserved in image retargeting operation.

Dynamic programming and graph cut techniques are widely used in various image retargeting operators. To solve the defined energy minimization problem seam carving technique comprises the concept of dynamic programming. In the seam carving technique, dynamic programming is used to find the optimal seams, which are the path of pixels with low energy in both horizontal and vertical directions. Using graph cut technique energy minimization problem can be solved by computing, the maximum flow in a graph. In real-time processing of the images, the overall performance of image retargeting operation depends upon the selection of best image retargeting operators, optimization techniques, and saliency map generation techniques. The computational complexity of retargeting operators is also dependent upon the number of parameters. Images having low-level features require low computation time and are suitable for real-time processing, but high computational efforts are required to process the saliency map which contains complex features. Hence it is concluded that image retargeting operation requires high computational efforts as image and its saliency map contain low-level and complex features.

The strategy behind selecting different papers for this review was simply based on three factors such as keywords, timeline, and relevancy. Certain important keywords for segregating the important papers were selected and searched online from the huge amount of available databases. Some of these keywords include visual saliency, saliency detection, visual attention models, content-aware image retargeting, region of interest (ROI), seam carving, warping, scaling, cropping, and multiop, etc. A total of 97 available literature from the year 2003 to 2019 have been selected and summarized in this survey as different content-aware image retargeting techniques and visual saliency models were optimized and enhanced in this duration. Saliency detection and its estimation is the fundamental step of image retargeting algorithm.

This survey paper starts by reviewing the problems and various improvements in the field of visual saliency detection and its estimation and hence presents a review on year-wise improvements in different methodologies of image retargeting techniques. This survey does not intend to cover every single published paper in the area of content-aware image retargeting, however, provides a comprehensive view of the approaches by sorting them into groups of similar techniques. The table given below shows the year-wise selection of the literature to complete this survey paper. This survey will be useful to researchers and practitioners interested in image retargeting.

S.NO	Category	Year	Count of Papers
1	Visual Saliency Detection	1998-2019	30
2	Cropping	2003-2019	16
3	Scaling	2008-2012	3
4	Seam Carving	2007-2019	23
5	Warp	2008-2017	9
6	Multioperator	2009-2019	16

In this paper, section 1 provides an introduction to image retargeting and its usefulness in computer vision. Sections 2 explains visual attention analysis and saliency map measures, top-down and bottom-up saliency detection models, and presents a survey on saliency map detection and estimation algorithms. Section 3 presents a brief explanation of various image retargeting operators. Section 4 presents discussion and conclusions.

2. Visual Attention Analysis and Saliency Measure

The fundamental step of image resizing algorithms is the estimation of the saliency map of the input image. Pixels, regions which belong to salient object contains low and high-level features which attract human visual attention and must be preserved in image retargeting operation. Features that come under low-level categories are gradient energy, image frequencies, contrast, or color features. Features that belong to the second category i.e. high-level features are faces, people, or objects.

A saliency map of the image can be produced by considering low and high-level features present in the image. Intensity variation can be computed from the pixel to its neighborhood in the horizontal and vertical directions. Due to less complexity and low computation time this function is used widely. To identify pixels that move differently optical flow or motion vectors are analyzed to create a temporal saliency map. Importance maps that combine motion, image depth saliency is used specifically for stereoscopic video. For saliency estimation, the bottom-up approach utilizes intensities of pixels, the direction of edges, and the color of pixels which are low-level features present in the image. Top-down methods utilize semantic information available in the image, like the place of salient objects (e.g., text, bodies, faces), symmetries. For better results of image retargeting, both approaches of saliency estimation can be combined. Three categories of the bottom-up model are (1) attention point prediction, (2) salient region detection, and (3) salient object detection. According to literature,contrast-based classification is accepted widely in the field of saliency detection as it leads to visual attention based on low-level features that are frequently used to compute contrasts. The three categories of the bottom-up model are given below.

2.1 Attention Point Prediction

As the movement of the eye provides important information, hence attention models can be validated based on human observation through the movement of the eye. This important information can be extracted through visual search, reading, searching, and scene perception. Visual information that is collected from a visual attention point can reduce the processing complexity if the selection process of visual information is efficient and advanced. The attention point can be predicted accurately by referring to the structure of the human eye. An area of the human eye that is called visual nerve cells perceives the visual information.

The different eye terminologies such as the macular area, retina, central fovea, etc. play a vital role in the visual information perception by any human. This perception greatly depends on the dynamic selectivity of the surrounding sensing environment. Sometimes these dynamic samples work as a procedure of the visual attention point transfer. A huge amount of such image information can be controlled largely by human vision. The above points explain the biological basis of attention point prediction. The major basis of attention prediction models in the early stages was mostly concentrated on eye movement prediction and human visual attention. Itti et al. [3] in their study explained the basis of subsequent saliency detection models by simulating the sequences of visual attention points.

2.2 Salient Region Detection

Several models have now been developed to recognize salient regions. For identifying such regions contrast plays a significant role. All bottom-up models can be categorized based on prior foreground and background.

2.2.1 Detection of the Important Region Based on Local Contrast

Contrast can be represented by the variation in the brightness and color of the object with other objects within the same field of view. In local contrast detection, subsets of pixels in the object are compared with adjacent subsets of pixels present in other objects. The saliency value is the distance between features. As edges are highly noticeable to the human eye. The available saliency detection algorithm can produce high saliency value for edges.

2.2.2 Detection of the Important Region Based on Global Contrast

Models based on global contrast compare all regions within the entire image. The following are some advantages of important region detection which is based on the global contrast. (1) To produce high saliency in global contrast detection the flaws coming from local contrast detection can be compensated in the contour position. (2) To highlight the salient regions saliency values are distributed in similar regions. If the background of the image is too complex and the salient region present in the image is large, then it can be possible that rather than highlighting the salient area global contrast-detection highlights background.

2.2.3 A Prior Foreground and Prior Background

Various existing aprior foreground models directly calculate salient objects based on scarcity, singularity, and uniqueness of visual information. Some aprior foreground models focus on acquiring high edge values for an image and ignore the low values at the inner of an object. Wei et al. [2] proposed first aprior background model. The objective of this model is to extract background according to aprior information and evaluate the difference of all pixels at backgrounds.

2.2.4 Learning Algorithm

Intensities of pixels, the direction of edges, and the color of pixels which are the low-level features can be utilized by a bottom-up approach. Top-down methods utilize semantic information available in the image, like the place of salient objects (e.g., text, bodies, and faces), symmetries, and structures. For better results of image retargeting, both approaches of saliency estimation can be combined. In recent years, due to the ability to extract the high semantic information Convolutional Neural Network (CNN) fascinate much attention and it has shown satisfactory results.

2.3 Salient Object Detection

Researchers showed that contrasts fascinates human visual attention, and to calculate the contrast of image low-level features are normally used. Various bottom-up models suggested by researches depend on several prior knowledge.

Itti et al. [3] suggested an approach that was motivated by the human visual system to compute bottom-up saliency by focusing on low-level features. A pyramid technology was employed to estimate three saliency maps for each low-level feature such as texture, direction, and intensity. Estimated feature maps are further combined to generate a single saliency map. After some iteration salient regions converged to a few key points. To compute saliency in the image Stentiford [4] suggested a method that is based on variations between neighboring pixels present in the image. The method suggested by him results from the larger and smoother salient region, and it is more suitable than Itti's method.

To analyze contrast Ma et al. [5] suggested a heuristic-based method that is effective than the method proposed by Itti. Ma and Guo [6] suggested a method in which firstly the image is segmented based on color and texture using the fuzzy k-means clustering method, and then saliency is computed based on relative locations and area of the regions, entropy concerning the source image. Setlur et al. [7] suggested a method to segment the image using mean-shift image segmentation. Histogram intersection is used to measure the color similarity between regions. Harel et al. [8] suggested a bottom-up graph-based visual saliency model. In the first step of proposed model activation maps on the feature, channels are formed. The activation map is further normalized to highlight conspicuity and confesses mixture with other maps. Santella et al. [9] suggested a method for cropping photographs in which eye tracking is used to find out locations of high-saliency areas present in the image. Gal et al. [10] and Golub [11] proposed a method in which point of interest are measured manually. In papers [12, 7, 13] it is mentioned that to avoid unsatisfactory results from automatic algorithms importance map of the image could be specified manually. Liu et al. [14] proposed a method to segment image into regions. For each region of the image, saliencies are assigned by considering the size of the region, relationships among neighboring regions, and their position in the image.

Wang et al. [15] suggested a warping method to retarget image. In this method for each local region, optimal scaling factors are calculated iteratively and a warped image gets updated which matches optimal scaling factors as closely as possible. This technique diverts and distributes the distortion in all spatial directions. Achanta et al. [16] suggested efficient saliency maps that can easily avoid artifacts generated by a conventional seam carving algorithm. Hasan et al. [17] proposed a method that follows a similar approach for the images without having faces. Goferman et al. [18] proposed a context-aware saliency. The usefulness of the suggested method can be observed in 2 applications where the context of the prominent objects is important as the objects themselves. Authors suggested giving importance to the regions which are closure to the salient objects. The method suggested by them is supported by local low-level considerations, global considerations, and visual organization rules.

These are 4 basic principles of human visual attention. For saliency detection in the image, a multitude of methods which are explained in [19, 20, 21, 22] can be used. In image and video compression the outcome of the saliency detection algorithm can be used [23]. In comparison to stationary objects, moving objects having different movement directions can fascinate the viewer's attention [24].

Image, depth saliency, and motion can be combined to generate saliency maps for stereoscopic video [25]. To get more details on various aspects of human visual perception and visual attention readers of this paper can refer [26, 18, 27, 28, 29]. Annum et al. [30] proposed a scheme, in which firstly, the contrast of the image is enhanced using weighted approximated histogram equalization afterward edges, semantics are maintained and undesirable details like texture are suppressed using edge-preserving guided filter. To perform scale aware operations and to obtain optimize cue, iterative rolling guidance filter and Cellular automata are used respectively. Ji et al. [31] suggested a graph model-based bottom-up salient object detection framework. Under a manifold ranking framework, multiple saliency maps are fused using low-level and objectness features. To refine the saliency map, saliency optimization is applied and in the final stage, saliency maps are integrated into different features using multilayer cellular automata.

3. Literature Review on Content-aware Image Retargeting Operators

3.1 Content-aware Cropping

To resize the image cropping is a naïve technique. In this technique a rectangular cropping window is placed at the center of the image, the cropping window keeps the central part of the image intact and discards the rest of the part of the image. Using this technique salient objects can be cut and most of the information can be lost. Preserving salient objects and finding a perfect location of the cropping window over the image is a very challenging task. Suh et al. [32] suggested an approach in which the importance map is computed and a rectangular cropping window is searched using the greedy approach, which covers a maximum percentage of salient points from the image. Ma et al. [6] proposed a method in which initially image is segmented based on color and texture using the fuzzy k-means clustering method and saliency is computed based on relative location and area of the regions, entropy concerning the source image. Zhang et al. [33] proposed a method in which energy function is defined to optimize the problem of auto-cropping. The defined energy function comprises 3 terms: composition, conservative, and penalty. The objective function is maximized for an optimal solution by employing the particle swarm optimization (PSO). Setlur et al. [7] proposed a method to segment input image using mean-shift image segmentation. Histogram intersection is used to measure the color similarity between regions. For region simplification regions are merged which are adjacent and region saliencies are assigned by combining bottom-up and top-down features. Santella et al. [9] suggested a method in which gaze points are documented and used in image cropping. The image is subdivided into various regions based on their color match, and based on graze tracker results a saliency map is generated. Fixation data is used for recognizing salient content and also for obtaining the best results of cropping. To obtain optimal results of cropping it is required to minimize cost function to include the region of interest, an important area of the image can be maximized, cuts through background objects can be avoided. Ciocca et al. [34] proposed a self-adaptive image cropping technique in which using a CART classifier images are classified into semantic types such as landscape, close-up, and "other". The proposed technique makes use of both visual and semantic information. Stentiford [35] proposed a method which is based on visual attention measure to reveal the informativeness of the image while performing cropping. To obtain the saliency map of the image similarities among a neighborhood of the image are analyzed and the image is copped based on the obtained saliency map. For semi-automatic image cropping Golub [11] proposed a system in which form input image users choose a point of interest for semi-automatic image cropping. The proposed system advises few cropping candidates and taking photography rules-of-thumb into consideration points of interest are placed. Based on regions of interest Amrutha et al. [36] found the optimal crop gained from the combined saliency models of Itti's and Stentiford's. By using a quality classifier Nishiyama et al. [37] proposed an automatic cropping technique. The quality classifier evaluates whether the image region which is selected to be cropping is agreeable to the user. An optimal cropped region can be found after applying the quality classifier to the candidates. Quality classifier comprises 4 steps: (1) Identifying multiple subject areas in the image, (2) Extracting features, (3) Estimating posterior probabilities, and (4) finally, combining posterior probabilities. Kopf et al. [38] extended the automatic cropping approach suggested by Suh et al. [32]. As compared to cropping, browsing is a more refined technique. In the browsing technique, a rectangular cropping window is placed on different regions on the screen. By taking pictures of each region of the screen a sequence of the image is created which shows a different aspect of the image. Cavalcanti et al. [39] proposed that using four feature extractors images can be analyzed to compute related content regions. To obtain the results from these feature extractor, the genetic algorithm (GA) optimization problem was set up. She et al. [40] proposed an approach in which images can be cropped through learning the structure. After the classification of photos, a graph-based visual saliency map was extracted. Based on the visual saliency map a dictionary was built for each category of image. After giving a solution to the sparse coding problem a cropped area is obtained. Tang et al. [41] considered regional and global features to suggest a photo quality assessment. Chen et al. [42] proposed an efficient system for cropping image which combines visual composition, boundary simplicity, and content preservation models. To preserve salient regions of the image Jaiswal et al. [43] suggested an automatic image cropping approach based on saliency map detection. The entire process of image cropping is repeated for the optimum result of cropping. Chen et al. [44] suggested an automated cropping technique based on a new model describing the relationship between attention preserving and region cropping. Kao et al. [45] proposed a cropping technique that is based on the aesthetic map. Using aesthetic map image regions can be distinguished based on the aesthetic quality category. Edge spatial distribution can be represented by the gradient energy map. To assess the quality of composition and related model is learned with the aesthetic map and gradient energy map. Moreover, to compute the aesthetic information an aesthetic preservation model is developed. To perform image cropping Guo et al. [46] suggested a cascaded cropping regression technique. In the proposed method 2 step learning approach is used to state the problem related to lacking labeled cropping data. To solve the problem a classifier is trained using CNN. Afterward, it is designed to fetch features from a database of cropping. Using a deep learning strategy by Rahman et al. [47] several images are trained to get precise importance maps by graph-based segmentation and adjustment of ray levels. To represent prominent objects, Gaussian filter and image scaling are used. The objective of the framework is to preserve prominent image contents and identify the region of interest.

3.2 Content-aware Scaling

Image resizing using conventional scaling operators results for information loss. In real-time applications global visual effects of the image can be preserved using scaling operators when interpolation scaling methods are employed. However, these methods can produce aliasing effects. Scaling operator results image deformation if the difference of aspect ratio of input and output image is very high. Shi et al. [48] proposed a technique which is based on morphological edge interpolation. Opening and closing image morphological operations are used to eliminate noise and to make the image smooth. For plain and edge regions present in the image two interpolation algorithms were used. Jiang et al. [49] proposed an efficient edge-adaptive scaling algorithm, in which the input image is segmented in four types of image blocks having a directional edge detector. Interpolation is applied along the direction of edges. Liang et al. [50] proposed a patch-wise scaling method that focuses on the salient area of the image and also tries to preserve the global visual effect. Local bidirectional similarity measures and smoothness measures are used to assess the quality of the retargeted image. Based on the saliency map source image is segmented in important and unimportant patches.

3.3 Content-aware Seam Carving

Seam carving is a popular image resizing approach, which was proposed by Avidan and Shamir [1]. In this technique, optimal seams are computed by comparing the energy of neighboring pixels (left, top, right). Neighboring pixels with the least energy is selected and considered in the seam. The seam is the 8-connected path of pixels in the horizontal and vertical direction, which contains one pixel per row. Based on the importance map, which represents the variation of intensity in both, the direction, this algorithm eliminates the path of pixels from the homogenous region of the image where the variation of intensity is very less or negligible. After the elimination of a single seam from the image, pixels are shifted from right to left direction or top to bottom depending on the algorithm to compensate for the eliminated seam. After seam elimination viewer can observe the changes only in those regions of the image where important contents do not exist. In a seam carving technique, optimal seams can be defined using dynamic programming. The seam carving technique can also be used in image enlargement by duplicating optimal seam in each iteration. Seam elimination and duplication processes may lead to image distortion and can produce artifacts in the salient area of the image. Rubinstein et al. [51] suggested a forward energy criterion, which deals with the problem associated with a seam-carving algorithm after the elimination of optimal seams. After performing seam elimination nonadjacent pixels become neighbors and produce artifacts in the retargeted image. Hence, more energy can be introduced to the importance map. The forward energy criterion stated that the optimal seams, which are chosen for elimination, might be responsible for reintroducing a minimum amount of energy. Achanta et al. [16] suggested an improved image resizing technique based on the seam-carving algorithm and used a novel saliency detection scheme. In the proposed method, saliency maps are used to identify important regions of the image by measuring global saliency of pixels, which consider intensity, and color features. The proposed saliency map is capable to avoid artifacts and robust to the noise present in the image. In the seam carving, algorithm gradient maps are required to be computed in each iteration but the proposed saliency map is required to be computed only once. Goferman et al. [18] suggested context-aware saliency. The usefulness of the suggested method can be observed in two applications where the context of the prominent objects is important as the objects themselves. He suggested giving importance to the regions, which are closure to the salient objects. The method suggested by him is supported by local low-level considerations, global considerations, and visual organization rules. Noh et al. [52] proposed energy function, which was based on the difference of forwarding gradient to preserve regular structures such as straight line, circle, ellipse, para-curve present in the images. After seam removal, the pixels, which become adjacent are tested for curvature irregularity using energy function and include the variation of gradient orientation and magnitude of the pixels. The main objective was to suppress the effect of seam removal and optimize the performance of dynamic programming. Mishiba et al. [53] proposed a block-based seam carving method, which was the extended version of the seam-carving algorithm. In the proposed algorithm seam is the path of pixel blocks and each element in the seam path is pixel block. In the image, the shrinking process blocks on the seam path are down-sampled. The proposed algorithm is faster than the traditional seam-carving algorithm and results in fewer distortions in image resizing. Zhou et al. [54] proposed an efficient energy function after considering constraints on object geometry to optimize the seam carving technique. Tan et al. [55] improved energy by a perceptually related energy function and this enhancement safeguards original structures in a better way for seam carving. To generate an adaptive importance map Liu et al. [56] accepted a multi-scale contrast-based saliency map. To overcome integral shortcomings of the seam carving algorithm the reserving ratio map is introduced and a scheme of mapping and resampling was used. A resized image could safeguard important objects and maintain the layout of the scene in a better way by using a continuous seam carving method. Wu et al. [57] improved the image-resizing process based on Cao's algorithm after determining the eliminated seams by accumulated energy and the neighboring probability. After the improvement, eliminated seams were dispersed. Han et al. [58] proposed improved seam carving based on wavelet. The proposed method is capable of estimates the local energy map by weighing multi-scale subbands suitably. In the image, retargeting process the semantic information of images could be a safeguard. In the wavelet transform domain, Mishiba et al. [59] proposed the seam-carving algorithm to suppress the breaking of spatial continuity. Conger et al. [60] proposed a generalized seam-carving algorithm and developed a multi-scale analysis model. By employing filter banks, seam traveling from the salient image features can be avoided. The proposed method could safeguard the salient image structure, as this method is less sensitive to a fine texture. To formulate image resizing as a binary graph-leveling problem Mansfield et al. [61] applied a visibility map. The use of Parallel programming is profitable to improve the speed of image retargeting (Thilagam et al. [62]). As energy, function ignores the important image structure the conventional seam carving and its improved algorithms adopt pixel importance. Hence, the structure of objects in the source image is often distorted. To preserve the structure, Mishiba et al. [63] proposed seam merging and a new merging energy criterion. Due to the lack of use of pixel importance, the method could not safeguard salient contents. Hence, to enhance seam merging Mishiba et al. [64] used importance and structure energies. When estimating the structure energy of the original image he tried to safeguard the main structures by taking a cartoon version of the source image. A new energy term has introduced to overwhelm the distortion produced by extreme shrinking or enlargement at the time of iterative merging or inserting (Mishiba et al. [65]). Wang et al. [66] propose a video seam-carving algorithm. Using the proposed algorithm structure of important objects can be preserved. Curves of source and retargeted video frames are compared and additional matching costs are computed. In the energy map of the seam carving algorithm matching, the cost is then added. Chen et al. [67] proposed a method through which an energy map can be obtained by combining the traditional L−1 norm of the gradient with depth-aware saliency (3D saliency). In the analysis of the image, these two operators proposed energy function contains both local and global information, which is verified to be effective in the seam-carving algorithm. Choi et al. [68] proposed a modified efficient seam-carving algorithm that can preserve the salient part of the image and also maintain the important structure of the image which is easily visible to the human eye. In the proposed algorithm author put a constraint that seams are sparsely assigned to each other. Lin et al. [69] proposed a saliency-based seam-carving algorithm. According to the greyscale of saliency detection images are classified. The foreground and detailed area of a subject and edge can be protected by adding various protection methods afterward cumulative energy map is calculated. Single-pixel wide seams are chosen for elimination by considering the principle of deleting the minimum energy pixel. Li et al. [70] proposed method, which utilizes suboptimal and discontinuous seams for seam carving. Spatio-temporal coherence can be preserve by allowing optimal seams to travel in homogeneous regions of the image and to obtain the discontinuous seams. To reduce the computational complexity genetic algorithm is used. In this manner, each frame can be resized to the targeted size by successively eliminating the seams. Patel et al. [71] proposed a method in which multiple pixel wide seam is inserted and removed in a single iteration to accelerate the process of seam carving. The energy of pixels that are to be eliminated and inserted energy after the elimination of a multiple pixel seam is also minimized to avoid the presence of false edges. The width of a multiple pixel seam is a serious factor that is made adaptive in the image resizing process to prevent the energy of an image. Patel et al. [72] proposed an adaptive multi-pixel wide seam carving in which the width of the optimal seam is expanded to make it multi-pixel wide seam. The energy of the pixels to be eliminated and according to the width of the seam energy of pixels across the multiple-pixel wide seam has improved. To avoid the growth in energies, the width of the seam has made adaptive.

3.4 Warping

Warping-based image retargeting methods are also referred to as continuous methods which are applied to the image to obtain the retargeted image. The objective of the warping based method is to minimize the local distortion from salient areas of the image and the regions which are not prominent are permitted to be distorted more. Several methods have been proposed which adopt various constraints and optimization methods to generate smoother and pleasing results as compared to traditional image retargeting methods. Liu et al. [12] proposed a method in which a piecewise linear warping scheme has been employed which has high distortion in uninteresting regions of the image and produces less distortion in the region of interest. According to the proposed method, there is only one region of interest per image, and by using the contrast-based method the saliency map can be generated. Gal et al. [10] proposed a technique to map textures into various surfaces, which avoids deformation of salient features present in the image. Salient areas are specified manually and deformation of these areas is constrained to be a similarity transformation, within a Laplacian image editing optimization framework. Ren et al. [73] proposed an image retargeting technique for mobile devices with a small screen size. By taking important regions, the layout of the image, and edge integrity as constraints, the process of content-aware image resizing is framed as a constrained sampling task. Every pixel in the image can be expressed in terms of the vector encoding the constraints. By combining all the pixels having a similar value of vector for making blocks the source image is transformed into a graph representation. Thereafter, to determine the sampling ratio of each framed block balanced minimum cost flow algorithm is applied. The results of the proposed method can be produced by an interpolated sampling scheme and direct scaling. Kim et al. [74] proposed an adaptive image and video resizing technique using Fourier analysis. In the proposed algorithm source image is divided into various strips using the gradient information. Each strip is scaled according to its saliency measure. Using Fourier transform, distortions produced by the scaling procedure are formulated. To define the sizes of scaled strips to reduce the sum of distortions Lagrangian multiplier technique is used. Wang et al. [15] suggested a "scale-and-stretch" warping method. For each local region present in the image proposed method works by computing optimal local scaling factors. Afterward, the warped image updated with optimal scaling factors as closely as possible. The proposed technique diverts the distortion in all the spatial directions in such a way that the impact of distortion on salient features should be as minimum as possible. Pavi´c and Kobbelt [75] presents how to use two‐colored pixels as a generic tool for image processing. Zhang et al. [76] suggested a method that efforts to safeguard important local regions and to preserve the structure of edges available in the image. Handles are used to define both local regions and edge in the image, and each handle weight is assigned which are based on an importance map. In each handle, shape distortion is measured using quadratic distortion energy. For each handle weighted sum of the quadratic distortion, energies are decreased to obtain the retargeting results. Guo et al. [13] proposed an image retargeting method using saliency-based mesh parametrization. A mesh image representation is constructed which can preserve image structure while image retargeting. Based on constructed mesh representation, the image retargeting problem is formulated as a constrained image mesh parametrization problem. Image mesh is associated with image saliency to preserve salient objects and to minimize distortion which is visible to the human eye. Lin et al. [77] proposed a patch-based retargeting scheme with an extended significance measurement to preserve the structure of salient objects, line structures, and minimizes distortions. Constraints which are based on similarity transformation is used to force visually salient content to undergo as-rigid-as-possible deformation, while an optimization process is performed to smoothly propagate distortions. Kaufmann et al. [78] introduce the framework of content-aware image warping tasks using the finite element method. Li et al. [79] proposed a warping-based stereo image retargeting approach to preserve the structure of important objects and the depth of 3D scenes. Warping functions are used to characterize the depth distortion and the impact of a warping function on depth distortion is analyzed. Binocular visual characteristics of stereo images are exploited to derive region-based depth-preserving constraints that control the warping functions directly. Afterward, a novel warping-based stereo image retargeting framework and a quad-based implementation of the proposed framework is presented. Gallego et al. [80] proposed an SR system that is implemented on cell phone devices. For capturing images a camera is used and a classical multiple-image super-resolution (SR) method is applied which uses a set of images having low resolution. Images that are taken using the camera of cell phones are subjected to the proposed filtering scheme wherein images having high noticeable blur are rejected to avoid outliers from disturbing the produced images with high resolution. Islam et al. [81] proposed a warping-based approach that can resize and remap the depth of stereoscopic video simultaneously to create a better 3D viewing experience. In the proposed method significant map is computed for each frame of stereoscopic video, then using non-homogeneous scaling optimization volume warping is performed to resize the stereoscopic video. In the warping process to remap the depth and to preserve the important contents a depth remapping and other constraints are applied.

3.5 Multi-operator

Single image retargeting operators are not capable to provide the better result of image resizing. Some researchers have suggested the use of a multi-operator to attain better results without image deformation. Han et al. [82] proposed an efficient energy function to improve the seam carving technique. The traditional seam carving technique employs a gradient-based energy function which results in distortion of prominent objects present in the image objects based on their layout and shapes. Object shape deformation can be preserved in the proposed method using a wavelet-based energy function. To suppress excessive deformation of the objects present in the images, a switching scheme is also proposed which is based on the energy distribution. Kumar et al. [83] proposed an approach in which local gradient information with a thresholding technique is used to control the process of optimal seam selection process. The process of seam selection halts when further seam selection and removal results in undesirable distortion of image contents while resizing. Antialiasing is used to make image content smooth which is distorted by the seam elimination process. Rubinstein et al. [84] proposed an efficient content-aware image retargeting technique based on multi-operators to avoid the deficiencies of a single operator. In the proposed technique multiple operators are combined to define a resizing space using a dynamic programming algorithm. Bi-directional warping was used to evaluate various resizing results. Dong et al. [85] proposed image resizing by combining image retargeting operators such as seam carving and scaling. Dong et al. [86] proposed a retargeting scheme in which to formulate operator cost function an image energy function and the dominant color descriptor is used. To meet users' preferences, a coefficient is utilized to revise the operator costs. They also proposed a framework based on a multi-operator to integrate users' real visual preferences tightly. Dong et al. [19] suggested a resizing method based on summarization. In this method, object carving is performed based on a multi-operator framework. The proposed method uses unidirectional seam carving and does not exploit homogeneous information along the retargeting direction. Shi et al. [87] proposed a bi-directional seam carving technique that considers both directions simultaneously. The deigned significance map and saliency measure was computed by a multi-resolution saliency model. According to the objective function, they decided the number and the sequence of the vertical and horizontal seam carving. Bi-directional image resizing combines unidirectional seam carving in both horizontal or vertical direction, and uniform scaling. The proposed method avoids the distortion of important objects. Su et al. [88] suggested a multi-operator retargeting by combining three operators i.e. content-aware cropping, seam carving (SC), and scaling. Cropping operator is utilized to eliminate the unimportant region of the image. Afterward, a new seam carving technique is used. The efficiency of the proposed image retargeting scheme is the utilization of a seam carving algorithm. After performing a seam carving operation finally, the image is scaled or normalized directly. Tsai et al. [89] proposed an image retargeting method that is based on multiple operators to improve the efficiency of the image retargeting operation. In the proposed method firstly cropping operator is utilized after that an aspect ratio adjusting method is adopted if the result of retargeting is not the desired size of the image. Finally, the Uniform scaling operator is used to adjust the size of the retargeted image. Zhu et al. [90] proposed a saliency and structure-preserving Multi-operator image retargeting method. The proposed method categorizes the images which utilized SIFT density and helps mitigate negative influence from the center-bias property of most existing saliency detection models. To improve the performance of image retargeting and to obtain an optimal sequence of image retargeting operators the proposed method adopts the principles that are Earth Mover's Distance and Gray-Level Cooccurrence Matrix. Zhu et al. [91] proposed a method that combines image retargeting operators. The concept of a genetic algorithm is employed with a proposed method to enhance performance. The applicability of genetic algorithms in the proposed method is to obtain the optimal operator ratio. The proposed method reduces the computational complexity and safeguards the important objects present in the image. Zhang et al. [92] proposed a scheme in which an optimized importance map is utilized which comprises the effect of the gradient map, skin map, canny edge map, and context-aware saliency map. The novelty of the proposed scheme is the exploitation of seam carving and scaling operators in a systematic way to get a decent balance between content loss & image enlargement. They also proposed an optional step to accelerate the seam carving process. Zhang et al. [93] proposed a new multi-operator image resizing to combine seam caving and warping image retargeting operators. The proposed method safeguard the benefits of warping operators in keeping highly noticeable areas of the image without dumping contents of the image which are unnoticeable. At the same time seam, a finding scheme is utilized to address the difficulty of focus detection. Zhang et al. [94] combined seam carving with scaling operators. To ensure the quality of important objects which are highly noticeable, the source image is stretch in both horizontal and vertical direction and then indirect seam carving and scaling methods are applied to perform similarity transformation. To shrink the image the direct SC method with the GVF method is applied. Abhayadev et al. [95] proposed a Multi-Operator based content-aware image retargeting technique. The proposed multi-operator combines seam carving and scaling. Using proposed multi-operator distortion in the image can be minimized during image retargeting. Chang et al. [96] proposed an image retargeting technique and a unique irregular interpolation method. Proposed content and mask-aware image resizing technique which can resize the source image into several shapes to show the prominent part of the main ROI. The proposed method supports the elimination of pixels which are not highly noticeable and frames a possible number of neighboring objects in the provided mask.

4. Discussion and Conclusion

Nowadays content-aware image retargeting is one of the latest research areas. This survey paper presents the work of researchers related to various existing image retargeting operators. The main challenge of content-aware image retargeting operators is the preservation of content loss, object distortion, and generating image retargeting results that are visually attractive to the human eye. In conclusion following research area can be improved to generate a better image retargeting algorithm. 1. All content-aware image resizing techniques are based on salience measure algorithms. More efficient techniques can be to improve this process. 2. Efficient techniques can be implemented to preserve complex structures and relationships among objects. 3. Resizing techniques can be applied on some special occasions, such as irregularly-shaped images. To evaluate the quality of retargeted image more objective criteria can be considered as current image retargeting criteria are generally subjective. This paper reviews the current academic attainments in the content-aware image retargeting field and also discuss the benefits and shortcomings of these techniques in various applications. Conventional image retargeting operators such as scaling and cropping focus only on geometric constraints and neglect the important information of ROIs present in the image. To retarget the original image scaling operator can be applied uniformly and it may be possible that important objects cannot be recognized. When the aspect ratio of the original image change then the scaling operator causes image distortion. Cropping operator preserves image content but the important information of the image which lies outside to cropping window boundary may be lost completely. Content-aware image retargeting techniques preserve the important content and also maintaining the geometrical structures and semantic information present in the image. Although image retargeting techniques are efficient still some shortcomings exist. It has been observed that the definition of importance is subjective. If the image contains complex objects which are higher closer to other objects many content-aware image resizing techniques generate inadequate results. Although various efficient image retargeting techniques have now been proposed by researchers, still existing techniques cannot generate optimal results and needs improvement. The main objective of each algorithm is to optimize a proper energy function to generate pleasing image retargeting results. They eliminate or shrink image contents that are less noticeable to the human eye and safeguard important information without any deformation. The seam carving technique eliminates unimportant pixels to form the image regions such as sky, water, and sand. Eliminated optimal seams from the image can produce image artifacts if the image contains line structures, complex objects which are very close to each other. The warping technique reduces the image aspect ratio by eliminating information which is not important after distortion. Warping methods efficiently preserves the geometric structure smoothly and may generate unpleasing results if the image contains salient features. Multi-operator methods take advantage of existing image retargeting operators by combining them. Instead of developing new techniques for image resizing using a single operator, it is observed that better image retargeting results can be obtained from multi-operator based image retargeting. However, they are time-consuming on account of the conversion measures among different operators. Researchers in their study are focusing their attention on content preservation, global visual impacts, and computational complexities. Still, there are no image retargeting techniques is available which work well in all the conditions and can generate satisfactory results completely. Resizing quality completely depends upon the image contents. It is observed that the cropping operator is suitable when the image contains a single important object in a small region of the image. Uniform scaling is suitable when the image is full of important information and structures which are very complex such as scenery image. Seam carving is the most appropriate technique for image retargeting when most of the regions of the image contain low energy pixels. Warping methods are suitable when it is required to maintain the structural information in a better way. Multi-operator based methods are used to preserve salient objects and the global structure of the objects in very less time. Researchers have been suggested framework for multi-operator based image retargeting in which seam carving, scaling, and cropping is combined to achieve better image retargeting results. Combining several operators in an ordered sequence is a multi-operator sequence that defines a directed path in resizing space beginning at the origin and following the path’s operator sequence. In the multi-operator framework, it is suggested that a certain type of operator can appear multiple times in different places in the sequence. In the sequence of multi-operators, some single operator can be used to increase the size of the image and others to reduce the size of the image in different directions (width and height). In some hybrid image retargeting framework optimized seam carving is combined with scaling and also allows user interaction to specify a preference between the two operators. For better user interaction, a switching factor can be included to optimize user interaction and to control the switching point. The switching point can be considered and critical point in image retargeting where a single operator starts the deformation of the image contents. Better image retargeting results can be generated after defining the optimal switching points among different operators. In recent years, salient object detection algorithms play a vital role in image and video retargeting techniques to produce pleasing results. In our opinion, salient object detection techniques can be further improved by the inclusion of CNN that can suggest a new direction for saliency detection algorithms. The development of CNN based salient object detection techniques can easily distinguish the salient and non-salient regions and also overcome the limitations of traditional techniques. Results produced by CNN based saliency detection techniques can be used further to generate better retargeting results. In the field of image retargeting, several techniques have been proposed by the researchers. All the proposed techniques are not well-suited on different types of images. The quality of image retargeting results can depend upon the performance of the algorithm. Each image retargeting technique has some inherent shortcomings, as retargeting cannot be done without loss of information. In our opinion, the image, video retargeting techniques, and quality assessment algorithms could be automated for producing and assessing better retargeting results. The results of such algorithms could produce a large discrepancy between human perception and automatic measurements.

To resolve the limitations associated with single image retargeting operators many authors suggested using a combination of multiple operators. In our opinion, a multi-operator based image retargeting techniques can be used that combine single operator in conjunction with the simple operators such as cropping and scaling. Image and video retargeting techniques cannot generate aesthetic results if scenes contain few homogeneous areas and very complex scenes. In such a case, the main objective can be the preservation of faces or the structure of the person, as they fascinate the attraction of the viewer. The quality of results produced by various existing retargeting techniques is very high especially in the case of multi-operator based techniques, still, some improvements are required to produce better results that can reduce noticeable artifacts. Although various automatic retargeting techniques have been suggested by researchers, still these techniques do not reach the quality level of manual adaptation yet.