Efficient Screen Splitting Methods - A Case Study in Block-wise Motion Detection

Abstract

Screen splitting is one of the fundamental tasks in different methods including video and image compression, screen classification, screen content coding and the like. These methods in turn support various applications in data communications, remote screen sharing, remote desktop delivery to assist teaching-learning, telemedicine, Desktop as a Service etc. In the literature we find systems requiring splitting assumes a fixed size split that do not change dynamically, also there is no analysis why that split is chosen in terms of performance. By doing mathematical analysis this paper first finds the efficient splitting schemes that can be easily automated to make a system adaptive. Thereafter, taking the screen motion detection as a case study, it demonstrates the effects of various splitting methods on motion detection performance. The simulation results clearly shows how classification performances varies with different splitting which will facilitate to choose the best splitting for a specific application scenario as well as making the system adaptive by providing dynamic splitting.

1. Introduction

Splitting is used in a variety of applications; image or video compression, high efficiency video coding (HEVC), screen or image block classification, remote desktop delivery and so on. Although the generic methods presented in this paper are applicable to any field, in our context, splitting refers to the breaking down of image or desktop screen into equal sized non-overlapping blocks. Research works on those fields usually assume a fixed size small block (e.g. 4×4, 8×8, 16×16 ) and analyze their methods accordingly. Likewise, 8×8 DCT is very popular in jpeg image compression [1] which needs to split a still image into a number of 8x8 blocks. Various literatures deal with image and video compression [2][3][4], classification and screen content coding [5][6][7][8][9][10][11][12] which uses splitting into 8×8, 16×16, 32×32 sized blocks. Similarly, virtual desktop delivery protocol [13] and even a camera application [14] also require splitting. In addition, some researches on provisioning Quality of Experience (QoE) in remote desktop delivery using VNC-RFB protocol as well assume splitting in a fixed block size [15][16][17]. The advantage of making the blocks equal size is obvious; it is possible to apply same processing, scaling and coefficient constants on every block. However, in some cases it is impossible to split into non-overlapping blocks of a fixed size because it depends on the resolution and in those cases splitting creates a different sized reminder blocks in each row which should be dealt differently than the equal sized blocks adding extra burden in processing [2]. Sometimes block shapes are also important and to gain extra advantage in provisioning better Quality of Experience it is desirable to choose square or even rectangular block shapes. If there is an efficient algorithm, it will enable us to split and adjust the block sizes dynamically. But no such algorithm which can efficiently split the image or screen as per user defined dimension with the facility of automation and thus no way to verify whether the chosen fixed splitting is the best suited for the problem scenario or not.

To solve the above problem, this paper deals with providing algorithms for efficient splitting of images and desktop screen and then analyzing these algorithms for a specific scenario of screen motion detection. We provide three algorithms for splitting; first algorithm will split into largest square shaped (n×n pixels) non-overlapping blocks. Second algorithm will also split into square shaped blocks but in this case maximum number of desired blocks will be given. So, if the maximum number of desired square-shaped blocks increases it will produce smaller size blocks while the third algorithm will split into blocks of arbitrary shape and size based on the desired number of blocks in the width and height direction.

The remainder of this paper is organized as follows. Section 2 describes related works. Section 3 describes the desktop splitting with mathematical analysis while section 4 presents different approaches of splitting in algorithmic forms. As a specific use case, detailed analysis employing splitting in motion detection by means of experiments, simulation and results is presented in section 5, followed by the conclusion in Section 6.

 

2. Related Work

2.1 HRDP

There are several advantages of intercepting the hardware layer for desktop delivery to thin clients and Hybrid Remote Display Protocol (HRDP) [15] utilizes this fact. The HRDP server application intercepts the desktop pixel data from frame buffer of graphic card. Then, the display desktop is divided into several rectangular areas. Depending on the number of changed pixels in a block, the motion detector will calculate the motion rate by comparing two adjacent frames to find high motion areas in which the display updates are delivered by the MJPEG module. The remaining areas are low motion where the display updates are handled by the VNC module.

2.2 QHRDP

An adaptive desktop delivery scheme for DaaS was proposed in [17] which is an improvement on the HRDP by selecting encoding adaptively. The paper recommended a Quality of Experience (QoE) model that can quantify the QoE scores for different encodings (MJPEG and VNC) and find out the most suitable scheme for a block. Part of the motion detection algorithm is about finding the maximum sub-matrices which accumulates smaller motion blocks into a larger one. Finally, With the increasing number of users, if the system cannot meet the desired service requirement, the system negotiate with the users to decrease their requirement for keeping satisfactory QoE.

Both HRDP and QHRDP used a fix splitting that needs to be chosen beforehand thus they cannot take the advantage of adaptive splitting. Moreover, finding maximum sub-matrices in QHRDP is very time consuming and dynamic programming approach of this task have a complexity of O(n3) where n represents the number of blocks. In this context, we propose some algorithms for efficient splitting which can be easily automated and thus making the system adaptive.

 

3. Screen Splitting

As discussed earlier, we need to split the whole screen area into equal-sized and non-overlapping blocks of integer dimensions where screen refers to the image, video or desktop. If the dimension is ScreenWidth × ScreenHeight, the splitting process divides it into RecWidth × RecHeight blocks termed as the outer rectangle. As a result, we get blocks of dimension BlockWidth×BlockHeight where BlockWidth = ScreenWidth/RecWidth and BlockHeight=ScreenHeight/RecHeight.

From the outer rectangle and the block dimension we obtain the original screen dimension by the relationships, ScreenWidth= BlockWidth*RecWidth and ScreenHeight= BlockHeight*RecHeight. Fig. 1 illustrates screen splitting through an example. The mathematical analysis presented below will clarify more.

Fig. 1.Relationships between splitting parameters

3.1 Mathematical Analysis

Let us consider the screen frame of dimension SW×SH where SW and SH are the screen width and height respectively, the pixel data matrix FM of that frame can be represented as

If we want to split the screen into non-overlapping blocks of dimension BW×BH, then the block width should divide the screen width and block height divides the screen height without reminder, i.e.

After split, number of blocks in height and width directions are

And, total number of generated blocks,

The rectangle matrix RM is defined as below

Each entry Bxx in rectangle matrix is a block that can be expressed as matrix BM of dimension BH×BW pixels. The Kth block on Nth row is expressed as Bnk and the corresponding matrix BMnk is represented by means of original pixels as below

From the above generic matrix we can easily find the first block matrix BM11

let, GCD be the greatest common divisor of screen dimensions SW and SH. In mathematics, the greatest common divisor (gcd) of two or more integers is the largest positive integer that divides the numbers without a remainder. Thus, the split with largest possible blocks have the dimension GCD×GCD. By definition,

Taking BW=BH=GCD, and using eq. 3 and 4 we get

from e.q. 10 and 11 we get block aspect ratio in the rectangle matrix

So, the block aspect ratio in e.q. 12 is equal to the original aspect ratio, also the blocks are square shaped.

Now, let's split the screen frame into square blocks in smaller sizes. To do this we find the divisors of the GCD. For integers m and n where m divides n, m is a divisor of n and is written as m|n. The vector GCDdivs contains all divisors from 1 to GCD.

by definition, GCD|SH and GCD|SW, for any divisor di in GCDdivs we get

Again, we get block aspect ratio in the rectangle matrix

also preserves the original aspect ratio. MaxDB is the maximum number of desired blocks and the split should satisfy

On the contrary, to generate arbitrary shaped blocks we take the number of desired blocks both in width and height direction as DWidth and DHeight respectively where

The divisor vectors for SH and SW are referred to as HeightDivs and WidthDivs and defined as

There are total ScreenWidth*ScreenHeight possible combinations of desired blocks while the actual number of possible splitting combinations is length(WidthDivs)*length(HeightDivs). For any pair of (DHeight, DWidth) we need to find the rectangle dimension (RecH, RecW) from the vectors (HeightDivs, WidthDivs) keeping the shortest difference among the corresponding elements as defined below

But we don't know whether TB1 in e.q 24 satisfies e.q 16 or not. Thus we find more rectangle dimension pairs using neighboring divisors as follows.

then the total block TBi which is closest but not exceeds MaxDB is found and corresponding rectangle dimension is determined.

 

4. Splitting Algorithms

In this section, considering different approaches of screen splitting we will organize into formal algorithms. The 'flower.jpg' image file of dimension 240×160 pixels (available at http://layek.itrrc.com/files/flower.jpg) will be used as a running example.

There are many choices of split depending on number of desired blocks, the size of the blocks and the shapes such as square, horizontal or vertical rectangular. When choosing a rectangular shape the question again arises what should be the aspect ratio of the outer rectangle.

In the simplest case, let us consider we have only two information available i.e. width and height. Our example image has the aspect ratio 3:2 and can be split the image into 3 * 2 = 6 blocks. Here, the outer rectangle matrix will also have the same ratio and the created blocks are square sized of 80×80 as in Fig. 2.

Fig. 2.Split generating largest blocks based on aspect ratio

Algorithm 1 splits the screen preserving block aspect ratio in the rectangle dimensions which creates largest blocks that is less than the whole screen. Equations 1-8 in the mathematical analysis presented on section 3 define the general parameters. Then we find the greatest common divisor (gcd) of screen dimensions which is the largest value that can divide both of them without remainder (equations 9- 12). Thus, to get largest non-overlapping square blocks, we take this gcd value as the block dimensions and number of blocks in both screen dimensions (RecHeight and RecWidth) are calculated accordingly. The variable GCD in the algorithm refers to this value of greatest common divisor of screen width and height. As an exception, if the GCD of screen width and screen height is 1 then there will be no split. GCD of our example image is gcd(240,160)=80.

Besides, considering square shaped blocks in smaller sizes, for example 40×40 block also preserves the block aspect ratio where RecWidth=6, RecHeight=4 and total blocks TB=24, 20×20 blocks (RecWidth=12, RecHeight=8, TB=96) and so on. As a result, in this second approach another parameter MaxNumberOfBlocks is used to set the upper limit for the number of blocks after split. When the desired number of blocks is less than the number of largest blocks it will return the whole screen itself which is an exception. The divisor vector GCDdivs of example image is given as, GCDdivs=divisors(80)=[1,2,4,5,8,10,16,20,40,80]

Algorithm 2 is based on equations 13-16 illustrates the detail approach whereas Fig 3 shows the splitting for two different values of maximum number of blocks for example image. Here, both splittings have the block aspect ratio 6:4=15:10=3:2 which preserves the original image aspect ratio 240:160=3:2. Each of Every divisors in GCDdivs also divides both screen height and width giving square blocks. As the maximum number of desired square-shaped blocks increases it will produce smaller size blocks. We only need to check the number of blocks does not exceed the maximum number of blocks (step 6). The last step is to deal with the exception case when the gcd is 1.

Fig. 3.Splitting based on maximum number of desired blocks preserving the block aspect ratio.

In the first two approaches, only square shaped blocks is obtained that preserve block aspect ratio in the outer rectangle. However, square blocks are rarely found and in case of example image square split can be generated only into 6 (3×2), 24 (6×4), 96 (12×8), 150 (15×10), 384 (24×16), 600 (30×20), 1536 (48×32), 2400 (60×40), 9600 (120×80), 38400 (240×160) blocks. But users may not always interested or require square blocks. Based on the analysis through e.q. 17 to 29, Algorithm 3 formally states the step by step arbitrary splitting process while the detailed flow diagram is given in Fig. 4 for easy illustration.

Fig. 4.Flow diagram for algorithm 3, splitting into arbitrary sized blocks

In Fig. 5, example image is split into 1×16, 16×1, 5×6 and 12×32 blocks. Here, our goal is to split the image into non-overlapping equal sized blocks that is very much close to specified dimension of outer rectangle. Let's see in a bit details how the algorithm works when splitting the image into 18 blocks in width direction and 14 blocks in the height direction totaling 18*14=252 blocks indicating the maximum number of blocks. The divisor vectors are:

Fig. 5.Splitting in arbitrary sized blocks

In this case, at first the algorithm will find divisors closest to 18 and 14 from the vectors WidthDivs and HeightDivs which is 16 in both directions making the total number of blocks 256 that exceeds the total desired number 252. After that, the algorithm checks the nearest block dimension pairs which are 15×10, 15×16, 15×20, 16×10, 16×20, 20×10, 20×16 and 20×20 from the divisor vectors. From these pairs the algorithm will selects 15×16 (Fig. 5-d) generating 240 blocks that is the closest to but not exceeds 252.

 

5. Application in Motion detection

The current section will present a motion detection approach and perform some experiments to see the effects of different splitting on the performance of desktop screen motion detection.

5.1 Motion detection

The motion detection approach in this section is based on the procedure discussed in [17]. Here, only the essential parts are used which is sufficient to explain the algorithms. Steps are described in Fig. 6 and detail is as follows:

Fig. 6.Motion detection process

a. Breaking up the display screen into meshed rectangular areas: This process divides up the desktop screen into small blocks with same sizes using the algorithms defined in the previous sections. In this way we get a rectangle matrix of dimension RecHeight×RecWidth, where RecHeight is the number of small vertical blocks and RecWidth is the number of small horizontal blocks.

b. Obtaining the update count in each block and Ignoring small noise: In this phase, the number of pixels are counted that are different from the previous frame for each small block and are stored as corresponding element in the matrix CV[RecHeight][RecWidth] which can be defined as follows.

c. Labelling the blocks as motion and non-motion or different motion classes : The next stage compute the percent of motion pixels out of the total number of pixels in a block then label as motion or non-motion based on a threshold value. Conversely, it is also possible to divide the blocks into several motion classes to encode them with different coders. Fig. 7 further illustates the process by means of an example.

Fig. 7.Example of screen splitting and classification

5.2 Performance metrics

Determining high motion and low motion is essentially a classification problem, thus classification evaluation methods can be applied. Detected blocks are compared with the original clip and the confusion matrix is formed as described in Table 1.

Table 1.Confusion matrix for classification

From the confusion matrix we can evaluate classification performance with several following metrics.

Accuracy: accuracy means proportion of true results (true positives and true negatives) with the total number of cases i.e.

Precision: the proportion of the true positives against all the positive results (both true positives and false positives) is termed as precision i.e.

Recall: recall is the proportion of positives that are correctly identified i.e.

Specificity: specificity is the proportion of negatives that are correctly identified i.e.

Balance Accuracy: balanced accuracy is defined as the arithmetic mean of sensitivity and specificity which is used to avoid inflated performance estimates on imbalanced datasets i.e.

F1-Score: f1-score also avoids inflated performance estimates on imbalanced datasets and defined as,

Classification error: classification error is the proportion of false results among the total number of cases i.e.

Balance error: to avoid inflated performance estimates on imbalanced datasets balanced error is used and defined as,

The error properties i.e. classification and balance errors are just the reverse metric of accuracy and balance accuracy respectively thus these metrics are excluded from the figures.

5.3 Experiment setup

To show the effect of splitting algorithms in screen motion detection, several high motion and low motion elements are arranged on desktop and the screen is captured with TinyTake video capture program then the screen movie is saved as a video file. Screenshot of the video file is given in Fig. 8 that can be downloaded from http://layek.itrrc.com/files/MixedDesktop.mp4. The general properties of the video file is given in Table 2.

Fig. 8.Screenshot of the mixed screen movie file.

Table 2.General properties of the captured mixed screen video file

5.4 Block-wise motion rate calculation

The experiments were done on MATLAB R2014b. First, we read the video file and store the data in a variable. Splitting algorithms then determine the rectangle size and the block size. Maximum changes of pixel value that can be happened to a color pixel is 765 (255+255+255). In our motion detection approach, average changes for each pixel in the first 30 frames were calculated and filtered by only considering a pixel as motion if the change of the pixel value is at least 115(about 15% of maximum possible change). Fig. 9 and Fig. 10 shows the plot of the changes without and with filtering respectively where white color represents motion pixels. After this process the data matrix becomes one zero matrix.

Fig. 9.Plot of average pixel changes for 30 frames

Fig. 10Plot of pixel changes after filtering

In the next step, block-wise percent of motion pixels of the total number of pixels in a block is calculated as shown in Fig. 11. Now, it is the time to classify the blocks based on the percent of motion. Dividing the screen into multiple class of motion rates with multiple threshold values is also possible and then different encoding schemes can be used for those classes. However, as said earlier to keep the analyses and calculation simple only high motion and low motion classes are used in the following experiments.

Fig. 11.Block-wise percent of motion

5.5 Determining the threshold

At this moment, the vital task is to select threshold value that will determine the high and low motion blocks because the performance of detection depends on the threshold. In Fig. 12 detected motion regions for the thresholds 1, 3, 10 and 20 are shown which clearly demonstrates the differences. Still, we cannot determine the best threshold for current scenario therefore performances should be compared with the varying thresholds. The binary matrices before and after the detection are compared and the confusion matrix is formed where Table 3 is used to find True Positives (TP),True Negatives (TN), False Positives (FP) and False Negatives (FN) for every pixel. Finally, some arbitrary splitting are taken and average performances for every thresholds varying from 1 to 50 are plotted which is shown in Fig. 13 whereas the best threshold value is approximated as 3.

Fig. 12.Motion detection for thresholds 1, 3, 10 and 20

Table 3.Computing confusion matrix for a single pixel

Fig. 13.Classification performances with varying threshold values

Choosing a threshold value is application dependent and on the basis of the metrics that are given importance. In our scenario both recall and precision have been given higher preference. Recall measures the percent of motion from original is detected and precision measures the percent of detected motion which is detected correctly. The figure shows the intersection point of these metrics is about 3 and is selected as the threshold value.

5.6 Performance analysis

Present section analyzes the classification performances from different viewpoints. Initially, taking different maximum number of blocks Algorithm 2 generates aspect ratio preserving splits and performance simulations are performed accordingly, Fig. 14 shows the results. In the performance plots some steady parts are found because the same split happens for a range of maximum desired blocks. The GCD of 1920 and 1080 are 120, GCDdivsors=divisors(120)=[1,2,3,4,5,6,8,10,12,15, 20,24,30,40,60,120], total number of elements is 16 hence there are only 16 distinct splits in this case as compared to the number of choices for maximum desired blocks which is 1 to 1920*1080.

Fig. 14.Performances with varying Maximum number of blocks

Fig. 15 shows the performance simulations for arbitrary splitting generated by Algorithm 3. We observe that the same number of blocks but different aspect ratio i.e. varying shaped blocks happens many times and the performance also varies with the aspect ratio. For instance, in our screen video the three splits (1,4),(2,2) and (4,1) each generates 4 blocks with block dimensions (1920×270, (960×540) and (480×1080) respectively whereas the accuracies are 54.9%, 79.69%, 74.91% and recalls are 82.58%, 81.10% 49.40% respectively. So, the performance improvement with the increasing number of blocks is not regular in all the points therefore the effect of block aspect ratio is evident which pushes us to analyze further in different ways.

Fig. 15.Performances with varying number of blocks for arbitrary splitting

As a consequence, all 1024 possible splitting are simulated which is the product of width and height divisors length. Different combination of rectangular dimensions can also have same ratio so the results are filtered keeping the ratio fixed and the performances with the increasing number of blocks are plotted. Fig. 16 and Fig. 17 show the plots for ratio 16/9 and for several other ratios respectively. These two figures clearly reveal the strong relationship between number of blocks and classification performances. Specificity is showing best and most stable performance which means detection has smaller number of false positives. On the other hand, precision tends to increase steadily with the increasing number of blocks.

Fig. 16.Performances with varying number of blocks for the ratio 16/9

Fig. 17.Performances with varying number of blocks for different ratios

It is time to see the effect of ratio on the performances. All combinations of ratios lies between 1 and 0 where RecWidthRecHeight and reverse (RecHeight/RecWidth) where RecWidth

Fig. 18.Performances with varying ratio deviation for the fixed number of blocks 3840

Fig. 19.Performances with varying ratio deviation for different fixed number of blocks

At this point, it is clear that classification performance increases with the increasing number of blocks and decreases with increasing ratio deviation where number of blocks have stronger influence. But increasing number of blocks incurs additional cost and thus degrade performance. Elapsed time increases almost linearly with the number of blocks (Fig. 20). Although MATLAB scripts consisting loops usually take longer time than actual implementation with other languages, the proportional increase of time is significant. Moreover, we notice considerable increase in memory usage as well as CPU utilization when running the motion detection algorithm with large number of blocks. Additionally, we plot the elapsed time with ratio deviation for fixed number of blocks in four cases which does not reveal any key effect of ratio deviation on elapsed time (Fig. 21).

Fig. 20.Computation time with the increasing number of blocks

Fig. 21.Computation time with varying ratio deviation for different fixed number of blocks

 

6. Conclusions

This paper first analyses the screen splitting approaches and organized them into formal algorithms. Then it took desktop motion detection as a case study and through different experimental evaluations demonstrated how the classification performances are affected by splitting strategies. The screen scenario made by arranging several motion/animations and non-motion/document areas then experiments were performed and measured the performances for different splitting. From the results, it is clear that performance increases with increasing number of blocks and decreases as the ratio deviation increases though the rates are not same. However, increasing number of blocks increases the resource usage massively while changing only the ratio keeping the number of blocks same does not show any significant effect on system resource consumption. The basic splitting algorithms proposed in this paper can be applied in a variety of applications and easily automated. By run-time verification it can find suitable splits in specific scenarios, for instance the remote desktop delivery solution can incorporate dynamic splitting to support better Quality of Experience.