Temporal Search Algorithm for Multiple-Pedestrian Tracking

Abstract

In this paper, we provide a trajectory-generation algorithm that can identify pedestrians in real time. Typically, the contours for the extraction of pedestrians from the foreground of images are not clear due to factors including brightness and shade; furthermore, pedestrians move in different directions and interact with each other. These issues mean that the identification of pedestrians and the generation of trajectories are somewhat difficult. We propose a new method for trajectory generation regarding multiple pedestrians. The first stage of the method distinguishes between those pedestrian-blob situations that need to be merged and those that require splitting, followed by the use of trained decision trees to separate the pedestrians. The second stage generates the trajectories of each pedestrian by using the point-correspondence method; however, we introduce a new point-correspondence algorithm for which the A* search method has been modified. By using fuzzy membership functions, a heuristic evaluation of the correspondence between the blobs was also conducted. The proposed method was implemented and tested with the PETS 2009 dataset to show an effective multiple-pedestrian-tracking capability in a pedestrian-interaction environment.

1. Introduction

For a long period of time, tracking has been an important theme in terms of computer vision, and it remains a challenging task. Computer-vision tracking involves the detection of the moving regions of images, the separation of foreground pedestrians from the background, and movement prediction. The tracking of a pedestrian is straightforward, as the pedestrian can easily be found in each frame, and the elapsed time can be used to determine the trajectory of the detected pedestrian; however, the generation of the trajectories of multiple tracked pedestrians is more complicated. During the tracking of multiple pedestrians, it is possible to mistakenly identify one of the other pedestrians as the target pedestrian, or to miss the detection of a pedestrian due to inter-pedestrian interactions such as occlusions.

Multi-object tracking is generally divided into the following three categories [1]: (1) Point tracking represents an object as a point and associates each point of a frame with a point in another frame. (2) Kernel tracking uses template- and density-based appearance models to track an object based on its shape and appearance. (3) Silhouette tracking extracts the contours of objects and tracks objects by matching silhouette shapes.

The point-tracking method involves the expression of the tracking target into a point, whereby the attribute of the target is extracted and attached to each point. This method requires a motion-correspondence problem, and in terms of classification, comprises statistical methods and heuristic methods. Multi Hypothesis Tracking (MHT) and Joint Probabilistic Data Association Filters (JPDAF) are examples of the statistical methods, and while they solve the interaction problem by finding jointly optimized trajectories, these methods can suffer from a combinatorial hypothesis space. Although global optimization can track a complete sequence, it is limited to a variety of assumptions in terms of an experiment. Globally, it is difficult to achieve success with optimum tracking because the combinatorial assignment problem is NP-complete. We propose a method that finds locally optimal trajectories using a heuristic function, which can be computed using consecutive 3 frames, i.e. (t)th frame, (t−1)th frame and (t−2)th frame. Even though this method does not guarantee to find globally optimal trajectories, it can be used for tracking pedestrians in real time by reducing search time. Further, while the local-tracking method that is used in the Kalman filter has a high accuracy regarding precision and localization, it needs to be modified to detect objects, despite the interactions between the objects, and so that the objects can be corresponded across frames for multi-object tracking [2].

The point-tracking method [1][3] requires an external mechanism to detect all of the pedestrians in every image frame. The tracking problem then requires an association of the detected blobs in the current frame with the tracked pedestrians of previous frames [4]. Our proposed earlier work presents a method for the generation of the trajectories of multiple pedestrians through the use of the motion-correspondence method [5], whereby each pedestrian is represented as a point. The experiment result of this earlier work shows an excellent trajectory-generation capability regarding detected positions in situations of added Gaussian noise; however, in a real-life tracking system, a pedestrian can sometimes appear as blob pieces due to detection-phase errors. Alternatively, several pedestrians can be detected as a merged blob due to their interactions with other pedestrians such as walking close together or crossing paths. It is therefore necessary to segment merged blobs that contain more than two pedestrians, and to also group separated pedestrian blobs. According to a recent method [8], the detected neighboring blobs can be represented as a graph, and the shortest-path algorithm can be applied to the group blobs. The processed blobs are associated with the tracked pedestrians of previous frames; however, their work only allows for cases of merged blobs, whereby either a many-to-one or a one-to-many correspondence must occur between the blobs. A multi-objective path finding problem was the research that has been resolved by combines artificial immune system (AIS), chaos operator, and particle swarm optimization (PSO) [6]. The non-deterministic polynomial-time hard combinatorial optimization problem was solved by a research of the fitness-scaling adaptive genetic algorithm with local search [7].

In this paper, we propose a multi-pedestrian tracking system that assigns a blob to each pedestrian. For the first stage of our method, we detect the pedestrian blobs in each frame of the images. The detected blobs are then applied to decision trees that identify whether the blobs are to be merged or split, whereby the blobs are assigned to either a merged-blob decision tree or a split-blob decision tree. The merging and splitting processes regarding the identified blobs occurs next, and each blob is then assigned to a pedestrian. During the second stage, for each blob in the current frame, the A* algorithm is used to search for the corresponding blob in the previous frame. Fuzzy clustering is used to develop a heuristic function for the evaluation of the possibility of blob association, whereby membership is computed from the velocities of the blobs that are under a smooth-motion constraint.

The rest of this paper is organized as follows: section 2 describes the proposed system; section 3 explains the results of the experiment regarding the algorithm, including a performance analysis; and lastly, section 4 comprises the conclusion of the paper, as well as a discussion of future works.

 

2. Proposed Method

The block diagram of the proposed method is shown in Fig. 1. We separated the foreground from the background by using a Gaussian mixture model whereby the separated foreground consists of blobs. Each pedestrian blob is represented by features such as area, perimeter, aspect ratio, center of mass, average color, and the distance to the closest blob in the previous frame; based on training-set learning, these features are used for the construction of the decision trees. The resultant learned decision trees are used to classify the blobs as either “split” or “merged,” and the splitting and merging processes assign blobs to each pedestrian.

Fig. 1.Block diagram of the proposed method

To associate a blob in the current frame with a tracked pedestrian from a previous frame, we devised a modified A* heuristic search algorithm. For each blob, the search for the most-probable pedestrian from the previous frame is based on the Euclidean distance and the difference between the velocity angles of the blob and the pedestrian. The required heuristic function that computes the possibility that a tracked pedestrian corresponds with a blob was designed using a fuzzy-C-means (FCM) clustering algorithm. It is the association of the blobs with the pedestrians over the entire frame sequence that produces the trajectories, each of which is a trajectory for a single pedestrian.

2.1 Merging and splitting of blobs using Decision Trees

In images, the number of detected blobs is different from the number of pedestrians. During this stage, the multiple detected blobs of a pedestrian should be merged into one blob, and a single blob that consists of two pedestrians should be divided into two blobs; the decision trees are used for the performances of these merging and splitting processes. A decision-tree-based classification method was selected, since the process is relatively easy to understand and the effective features of the tree-structure-derived logical expression are easy to modify.

The first decision tree Merge tree TM decides whether or not it will merge two adjacent blobs, whereas the second tree Split tree TS decides whether it will split a blob or not; these decision trees are generated from an ID3 learning algorithm. A set of blob pairs BM={,…,} is collected as a training set of TM, and each pair is labeled as either a positive sample or a negative sample. For each pair , the geometric features fM={fM1,fMn} are extracted and used as blob-pair descriptors, and it is these features that are used for the construction of TM. A set of blobs Bs={b1,…,bm} is collected as a training set of TS. Each blob is labeled as either a positive sample or a negative sample. For each blob, the geometric features fs={fs1,fsm} are extracted to describe the blob and they serve as the features for TS.

The ID3 decision tree is an information-theoretical algorithm invented by Ross Quinlan [10]. Each iteration of the ID3 involves the attainment of the largest information gain (or smallest entropy value), and a probability of p is determined from the occurrence frequency. The entropy of a given BM is computed as follows:

where C is a set of classes in BM that is either of a positive (merging) class or a negative (not-merging) class, and p(Ci) is the proportion of the number of pairs labeled as class Ci in BM to |BM|. The information gain(IG) from the partitioning of BM according to the value of the feature fM is computed using eq. (2), as follows:

In eq (2), is the entropy of the subset that is constructed through the collection of pairs, each of which has a value of vj for the feature fMi, from BM. VMi denotes a set of values for fMi; therefore, The probability p(vj) is computed using eq. (3), as follows:

At each iteration of the construction of a decision tree, the feature that provides the maximum information gain is selected as a decision node. To deal with the numeric features, the C4.5 [11] ID3 algorithm is used.

The pair of blobs selected by TM are merged, and the blob selected by TS is split into two decision trees, whereby the merging and splitting processes are respectively used. A blob is split vertically in consideration of the shape of the person. After the blob-processing stage, each blob Bi is represented as , as these are the coordinates of the center of mass.

2.2 Blob association

2.2.1 Heuristic-function evaluation

After the blobs have been processed, an A* search algorithm is used to associate the blobs in (t−1)th frame with the blobs in (t)th frame To develop a heuristic-evaluation function for the association of the pair for the A* algorithm, we must assume that each blob in (t)th frame is a cluster center. Using FCM clustering, we measure the confidence degree of the blob in the (t−1)th frame that belongs to a cluster wherein the center of is in (t)th frame. FCM clustering is a method whereby classifications are made according to the membership degree that is commensurate with the distance of the clusters and the input data [12]. The same process is repeated for the calculation of the cluster center until the change is within the acceptable range of the threshold value. The first step of FCM clustering is an establishment of the number of clusters and an initialization according to the exponential weight. The second step is the calculation of the center point of the fuzzy clustering. The third step is a calculation of the new membership function for the center of the cluster. Lastly, if the change in the cluster is less than the threshold value, the repeated process ends; however, if the change in the cluster is greater than or equal to the threshold value, the process is repeated from the second step onward.

We defined the joint feature where dj=(dj∙x,dj∙y)∈R2 represents the distance vector of a point and θj represents the orientation of this vector. A set of joint features can be represented as the following

where is the j th blob in the (t)th frame and is an i th blob in the (t−1)th frame.

A set of the joint features of a blob center can be represented as the following

where is an index of a blob in the (t−2)th frame, which is connected to the i th blob in the (t−1)th frame, . Thus, denotes velocity vector of (i)th blob of (t−1)th frame in polar coordinates.

As shown in Fig. 2, is the confidence degree for blob to associate with blob . This confidence degree is estimated using motion constraints that are based on the velocity vector and orientation angle of each blob. The confidence-degree functions for blobs between the (t−1)th frame and the (t)th frame satisfies the following constraints:

Fig. 2.(t−1)th frame and (t)th frame for blob matching of each blob

If we assume that each blob in the (t)th frame is a cluster center, then we can estimate the confidence degree by minimizing the following function Z:

where q is a parameter that controls the fuzziness. It is a smoothness constraint of the velocity. It minimizes total differences between velocity of each blob at (t−1)th frame and velocity of the corresponding blob at (t)th frame.

The confidence degrees can be derived from the following Lagrange function:

where λi are the Lagrange multipliers. By differentiating L with respect to and applying normalization constraints.

Then the confidence degree can be obtained as the following function:

For each blob in the (t)th frame, the computed confidence degree is used as the value of the heuristic function in the A* search algorithm. This heuristic function is represented as the following (m+1)×(m+1) cost matrix CM. CMij is an element of the i th row and j th column of the CM, and it represents the heuristic values for the and blob correspondence that is . If the element is less than the threshold ε, it is set to zero. The last row of the CM defines et-1 so that the sum of the confidence degrees for each blob in the (t−1)th frame is 1.

2.2.2 A* search

The search tree is constructed as the nodes of depth i that are the candidate nodes for the i th blob in the (t−1)th frame. Firstly, the root node S becomes the start node that is a null node, and S is placed into a priority queue. S is selected from the priority queue and the successor nodes are subsequently generated; these are the blobs in the (t)th frame with a non-zero confidence degree of association with in eq. (11), and they are inserted in the priority queue and sorted according to the confidence degree. From the priority queue, a node with the maximum confidence degree is selected and expanded. In level 2 of the search tree, nodes with a non-zero confidence degree regarding an association with will appear. After is selected, the successors of and from eq. (11) are generated and inserted into the priority queue; from this queue, a node with a maximum confidence degree is selected. The successors and are generated, since they have non-zero confidence degrees for an association with Note that a node that appears in the path from the root node to the current selected node is not generated as successor. This process is iterated until the goal node is selected from the queue. The goal node is the node in depth m of the search tree if, through this iterative process, it finds the best matching pairs between the blobs in the (t−1)th frame and the blobs in the (t)th frame. After the search is finished, the algorithmtraces back from the goal node to the starting node by following the path to obtain association pairs. From the optimal path found in the search graph of Fig. 3, the following association pairs are derived:

Fig. 3.Blob-association search tree

In the search graph of Fig. 3, the dotted line represents the associations of the blobs along the found path at each level of the search tree.

 

3. Experiment Results

We used the Caviar "Crowd of four people meet, walk and split" sequence , PETS 2009 dataset S2 from the L1 sequence view 001 and the L2 sequence view 001 as shown in Fig. 5. The Caviar sequence is used only for tracking in clean environment. WEKA (Waikato Environment for Knowledge Analysis) version 3.7, written in JAVA, is an experimental tool used for machine learning [27] that we used for ID3 machine learning and the visualization of the decision trees. Blob detection and a trajectory-generation algorithm were implemented in OpenCV 2.4.2 with Visual Studio 2010.

The sequence diagram of the proposed method of this paper is shown in Fig. 4, whereby the data flow of the system is shown from left to right. In Fig. 4, the image sequence is provided as an input to the system from the left side. The detected pedestrian blobs are the data that are generated next, followed by the decision trees that are the next output data for the ID3 learning algorithm. After the merging and splitting of the blobs, each blob is represented as a center of the mass point. Lastly, the trajectories of the pedestrians are generated as the output data of the blob-association process.

Fig. 4.Sequence diagram of the proposed method

Fig. 5.data set used in the experiment

3.1 Dataset and Condition Description

The video footage used in the experiments features multiple walking pedestrians whose motions are generally meeting and breakup situations. In our study, one pedestrian is detected as one blob so that a trajectory can be generated using the point-correspondence method.

Fig. 6 shows splitting or merging situations for the detected blobs in the image. Two pedestrians are detected as one blob, so the blob number 0 of Fig. 6 (a) needs to be split; furthermore, one pedestrian is detected as two blobs, so the blob number 0 and the blob number 1 of Fig. 6 (b) need to be merged.

Fig. 6.Merge or split situation

3.2 Trajectory Generation

3.2.1 ID3 Decision Tree

The decision trees TS and TM are generated in terms of the blob attributes that are extracted from the training-input image sequence. The ID3 decision tree that is obtained from the calculation of the entropy of the divided sets is shown in Fig. 7. Fig. 7 a) shows the split decision tree where “MoveDistance” is an attribute denoting the distance from the closest blob in the (t−1)th frame. The “HeightDiff” attribute represents the difference between the blob height and the height of the closest blob in the (t)th frame. “AreaDiff” represents the difference between the areas of two blobs. In Fig. 7 b), “MaxArea” denotes the larger area of two adjacent blobs, and “AreaDiff” denotes the area difference between two adjacent blobs.

Fig. 7.Blob-merge and blob-split decision trees

In TS, the threshold of the moving distance is 37 and the threshold of the area difference is 358. If it is decided that blobs are to be split from TS, the blob is divided into two blobs using the splitting procedure. If it is decided that two blobs are to be merged from TM, the two blobs are merged into a single blob. After the splitting and merging processes, the center of the mass of each blob that is calculated is used for the confidence-degree computation.

3.2.2 Multiple-Pedestrian Trajectory in Clean Environment

In the first scenario, we tested the proposed method in a clean environment, whereby we assumed that interferences between pedestrians such as occlusion do not occur; however, in cases of occlusion, we manually assigned a blob for each pedestrian. Table 1 shows the cost matrix that was obtained from the calculation of the confidence degree of the association of two blobs that were selected from the (t)th frame and the (t−1)th frame for eight sequential frames of the L1 sequence. For the computation of the confidence degree, ε = 0.05, and if the confidence degree is less than 0.05, it is set to 0. The number in (i)th row and (j)th column of each table represents confidence degree of each frame. For example, the second table in top row is a cost matrix for corresponding blobs in the 2nd frame to blobs in the 1nd frame. 0.22 in 2nd row and 3rd column of this table denotes confidence degree for corresponding 2nd blob of 2nd frame to the 3rd blob of 1st frame, i.e.

Table 1.Cost Matrix (CM)

Fig. 8 shows the generated trajectories for the S2 views of the L1 and L2 sequences from the PETS 2009 dataset. The pedestrian group consists of nine people in the (a) L1 sequence and 33 people in the (b) L2 sequence. The pedestrians of (a) frequently change their moving direction in comparison with those of (b). The L1 and L2 sequences generated trajectories of 30 frames and 26 frames, respectively. Fig. 8 (c) shows the trajectories that were traced using the proposed method for the L1 sequence, and Fig. 8 (d) shows the generated trajectories for the L2 sequence. For both cases, the generated trajectories are exactly matched with the ground truth.

Fig. 8.Multiple trajectories of pedestrians

Table 2 shows the tracking performance for the trajectory generation of Fig. 8 (d) and CAVIAR data set. The generated trajectories of the L2 sequence show the ratio of the 100 % correctly detected tracks (CDTR), the ratio of the 100% correctly corresponding pairs (CCPR), and the 0 % average ID change (AIDC) in the first row, where there is no added noise. We added noise for the blob locations, and for each added noise, we measured the CCPR, CDPR, and AIDC [5].

Table 2.Tracking performance in clean environment

3.2.3 Tracking in Real Environment

In the second scenario, we tested the proposed method in a real environment where interferences between pedestrians were considered. In this scenario, the generated decision trees are used for splitting a blob and merging blobs. For the quantitative performance evaluation for the experiment in this scenario, we use the MOTA (Multiple Object Tracking Accuracy) that is provided by the Classification of Events, Activities, and Relationships (CLEAR) consortium. The MOTA is a performance index for the measurement of tracking accuracy according to the sum of the following three errors: ratio of missed tracking target, false detection rate, and ratio of mismatched blobs. A comparison between the tracking results obtained from the experiment and the GT trajectories yielded the following findings: The miss rate is 3.42 % and the false positive rate is 27.82 %. As shown in Fig. 9, our proposed method shows a MOTA of 68.75 %. We also compared the performance indexes of other participants to the PETS 2009. Regarding the 2D MOT benchmark features, the tracking results were compared for the PETS 2009 S2 L2 images. The index used in Fig. 9 is taken from the state-of-the-art methods of the MOT Challenge Benchmark [24]. Our method produced favorable results in comparison with those of the MOT Challenge Benchmark; for the comparison, our method-exclusion process is based on the multi-view images.

Fig. 9.Tracking-performance comparison with other methods

 

4. Conclusion

During the formulation of a method for the generation of trajectories for multiple pedestrians, numerous difficulties arise. Our point-correspondence algorithm for multiple-object tracking that was previously proposed has been modified and expanded in this work. To track multiple pedestrians in a real environment, we consider the interactions between pedestrians as the most important issue. We therefore developed a decision method for the splitting and merging of blobs, whereby decision trees learned from the ID3 algorithm are used to resolve the occlusion problem of pedestrians. For the identified situations, the blob-splitting or blob-merging processes are applied for the redefinition of a blob for each pedestrian and to calculate the association costs. We used the A* algorithm to successfully generate the pedestrian trajectories, whereby a smooth constraint is used to devise the heuristic function for the association of a blob in the (t)th frame with a blob in the (t−1)th frame. In our experiment, we used the MOTA index for a performance evaluation regarding multiple-pedestrian tracking. In a comparison between the proposed method and the state-of-the-art methods of the 2009 MOT Challenge Benchmark, a favorable performance was observed. We did, however, restrict the compared methods to the tracking systems from single-view image sequences, but we are going to improve the system so that it can handle more-frequent pedestrian interactions.