Real-Time Cattle Action Recognition for Estrus Detection

Abstract

In this paper, we present a real-time cattle action recognition algorithm to detect the estrus phase of cattle from a live video stream. In order to classify cattle movement, specifically, to detect the mounting action, the most observable sign of the estrus phase, a simple yet effective feature description exploiting motion history images (MHI) is designed. By learning the proposed features using the support vector machine framework, various representative cattle actions, such as mounting, walking, tail wagging, and foot stamping, can be recognized robustly in complex scenes. Thanks to low complexity of the proposed action recognition algorithm, multiple cattle in three enclosures can be monitored simultaneously using a single fisheye camera. Through extensive experiments with real video streams, we confirmed that the proposed algorithm outperforms a conventional human action recognition algorithm by 18% in terms of recognition accuracy even with much smaller dimensional feature description.

1. Introduction

On livestock farms, it is important to monitor the health and anomalies of the livestock inorder to reduce costs and welfare impacts [1]. One of the measures to assess animal health ismovement. Unhealthy animals suffering from limb disorders or disease are much less active than healthy animals. Animal movement also indicates estrus. The most observable sign of the estrus phase is the mounting action, where a cow in estrus follows another female cow for afew seconds, and then mounts the cow. Estrus detection in cattle helps determine the optimaltime for artificial insemination, which leads to increasing not only conception rates for the herd but also milk production. In this paper, we propose a machine vision-based cattle action recognition algorithm to detect mounting events during estrus and present a real-time cattlemonitoring system exploiting the algorithm.

There have been several attempts to recognize the movement of animals [2-7]. Radiofrequency transmitters have been used to determine presence and duration at feeding and watering locations, with this data being correlated to health status [2-4]. Data loggers withaccelerometers have been used to analyze postural behavior patterns of cattle [6].

In order to detect lameness in cows, wireless 3D accelerometers were used to measuretemporal gait characteristics on all four limbs of the cows [8], and accelerometer data from cow collar sensors were collected by clustering them in accordance with activity level. Then, mounting events were identified using a change-detection technique on the high-activity index derived from clustered time series data.

However, attachment of sensors may cause stress and, in some cases, is impractical to usedue to their cost and vulnerability. Automatic computer vision systems can be an effectivealternative to avoid such problems in monitoring animals [9].

Poursaberi et al. [10] evaluated the status of lameness in an individual cow by extracting thearc of the back from side-view videos during standing and walking. Kashiha et al. [11]quantified pig locomotion associated with lameness with top-view videos. Specifically, foreground objects representing pigs were extracted using image processing, and then each pig was located by fitting each object into an ellipse model. In order to extract pigs moreaccurately in an unevenly illuminated pigpen, Guo et al. [12] applied adaptive partitioning and multilevel thresholding. By extracting object contour information and utilizing it, the method can observe several objects at the same time, but the contour information becomes severely degraded in complex backgrounds.

Recently, automatic estrus detection requiring more complicated motion recognition has been investigated [9], [13], [14].

Nasirahmadi et al. [9] presented a mounting-detection algorithm for pigs, where moving objects are initially fitted into ellipses, and then, the major and minor axis length ratio of the fitted ellipse is utilized to determine mounting events. However, ellipse fitting is not suitable for cow localization because cattle usually move their heads to a much larger extent than pigs, as shown in Fig. 1. In addition, not only adult cows but also young calves live in the barn. As aresult, the major and minor axis lengths can be estimated inaccurately, and thus mounting detection can fail.

 

Fig. 1. Feasibility of ellipse fitting for cows and pigs. (a) A cow image, and (b) ellipse fitting result for (a). (c) A pig image, and (d) ellipse fitting result for (c).

Chung et al. [13] presented a cattle mounting detection system in which the height of the moving object is examined with side-view videos, since the mounting action leads to a suddenincrease in object height. However, there are usually many livestock in a single barn, which leads to complex scenes in which animals are frequently occluded by each other and move 2150 Eui-Ju Heo et al.: Real-Time Cattle Action Recognition for Estrus Detection around continuously. Despite easy mounting detection when using side-view videos, objectsize varies largely depending on distance to the camera and the cow closest to the camera canocclude the mounting event. In addition, side-view videos can be affected by direct s unlight and are easily degraded.

Tsai and Huang [14] detected estrus behavior of Holstein cattle having black-and-whitemarkings based on the most observable sign of estrus, where a cow follows another cow for a few seconds, and then mounts the cow. Using motion detection and morphological operations, moving objects were extracted from top-view videos. Then, the mounting event is detected based on the changes of moving object lengths, i.e. the extracted moving object is much longer, compared to the length of a normal cow during the following behavior, and then, the moving object shrinks suddenly during the mounting event. However, foreground object extraction iseasily affected by changes, such as sudden light changes, dynamic backgrounds, ground urinestains. That is why the method requires many parameters to be tuned. Furthermore, if the object of interest in the herd includes taurine cattle having no special patterns on the bodies, simple image processing tends to fail to extract foreground object.

In order to deal with these problems, motion recognition for a practical estrus detectionsystem needs to be more robust to dynamic complex scenes while keeping computational complexity low.

As computing power increases exponentially, sophisticated computer vision-based motion recognition, mainly for human movement, has been researched and developed [15-23].

Among them, a robust representation of movements, called the motion history image (MHI), has been widely used to describe motion information within a video. The MHI can represent not only the location of object motion generated in the scene but also the process of how the object moves, and with very low complexity [15]. Furthermore, it is suitable for expressing human posture, gait, and gestures, because it is not sensitive to silhouette noises, such as holes, shadows, and missing parts [17].

 

Fig. 2. The configuration of the proposed cattle monitoring system. Cattle in the three enclosures aremonitored using a fisheye camera. When interesting actions including mounting action are recognized, a short video clip is stored in the server, and a message is sent to the person in charge for checking the events.

In this paper, we present a vision-based cattle monitoring system for estrus detection as shown in Fig. 2. Using a top-down fisheye camera, cattle in the three enclosures are monitored in real-time. If any cattle action of interest is recognized, the system stores a short video clip on the server and notifies the person in charge of the event by sending a message. Then, the person can access the video clip to check the event.

In order to recognize cattle actions, we propose novel MHI-based features that caneffectively describe cattle movements, especially mounting action. Since mounting actionoccurs during the rare estrus period, it is difficult to acquire a number of training dataset. Through learning the proposed features with the support vector machine (SVM) framework that is well suited for small or medium sized dataset, various cattle actions, such as mounting, walking, tail wagging, and foot stamping, can be recognized in complex scenes thanks to MHI-inherited robust representation and silhouette noise insensitivity.

Furthermore, since the proposed cattle action recognition algorithm is very efficient, a visual cattle monitoring system using it can recognize the actions of cattle in three enclosuressimultaneously, which reduces the system cost.

The main contributions of this paper can be summarized as follows:

• We propose an effective feature description for cattle action recognition, which is robust to complex scenes. Compared with a conventional action recognition method using the same MHI, the proposed feature description improves therecognition accuracy significantly.
• Since the proposed feature description has a sufficiently small dimension, multiplecattle can be monitored simultaneously in real time, which leads to the system costreduction.

To the best of our knowledge, the proposed algorithm is the first vision-based actionclassification for livestock.

In the following section, before presenting the proposed features, MHI (the intermediateinformation from which the proposed features are extracted) is first investigated. In Section 3, we present the proposed cattle action recognition algorithm in detail. Through extensive experiments, the proposed algorithm is compared to conventional motion recognitionalgorithms in Section 4. Finally, concluding remarks are given in Section 5.

2. Motion History Image

Thanks to both robust representation and the simplicity of MHI, several features based on MHI have been proposed to effectively represent characteristics of motion [18-20], [22]. Bradski and Davis [19] applied Hu moments [24] to the gradient of the MHI silhouettes torecognize an object pose.

Weinland et al. [18] proposed a motion history volume that consists of multiple MHIsobtained from multiple cameras for free-viewpoint motion representation.

Kim et al. [20] presented depth motion appearance to describe the global 3-D shape of body movement using a modified MHI. An entire sequence of depth maps was encoded to a 4096-dimensional histogram of gradients (HoG) descriptor for motion recognition.

Since cattle actions are well represented in MHI, we also exploit MHI to produce the proposed feature. Therefore, we review MHI and examine its distinguishable patterns withrespect to cattle actions.

Let \(I(x, y, n) \text { and } D(x, y, n)\) be an image sequence with temporal index , and successive frame difference \(D(x, y, n)=|I(x, y, n)-I(x, y, n-1)|\), respectively.

Then, MHI denoted by \(H^{\Gamma}(x, y, n)\) is obtained as follows:

\(H^{\mathrm{\Gamma}}(x, y, n)=\left\{\begin{array}{cc} {\Gamma,} & {\text { if } D(x, y, n)>\delta} \\ {\max \left(0, H^{\mathrm{\Gamma}}(x, y, n-1)-1\right),} & {\text { otherwise }} \end{array}\right.\)       (1)

where Γ indicates the maximal temporal extent of the movement and is a threshold value fordetecting motion occurrence.

If the motion occurrence is sufficiently certain, i.e. \(D(x, y, n)>\delta\), then\(H^{\Gamma}(x, y, n)\) is setto its maximal value, \(\Gamma . H^{\Gamma}\) decreases gradually to zero over time unless another motionoccurrence is detected at the position. We omit the positional indices for simple notation unless ambiguity may be caused.

Fig. 3 visualizes the process of MHI generation. Given an input image sequence \(I(n)\)motion occurrence is detected as shown in Fig. 3 (b). The latest motion-detected regions have the maximal value, H^Γ(n) while other regions are diminished. Since MHI leaves trails about any movements, not only the location of the object’s current motion but also the process of how the object moves are represented in MHI.

Fig. 4 demonstrates MHIs for typical cattle actions. Letting a connected region having positive values within H^Γ(n) be a motion pattern M_i(n), where indicates the pattern index, each MHI has very unique and distinguishable patterns. In the MHI for mounting action, the motion pattern is quite a bit larger than in other actions. The motion pattern is enlarged instantaneously, and then decreases gradually for 3 ~ 7 seconds. Walking action produces adirectional motion pattern. Tail wagging and foot stamping occurring over a very short termyield small and disperse motion patterns. In the following section, we constitute a small dimensional feature that effectively represents such unique patterns in MHI.

3. Proposed Cattle Action Recognition Method

A block diagram of the proposed cattle action recognition method is shown in Fig. 5. Initially, the image frame obtained using a fisheye camera is geometrically undistorted. After MHI is obtained as explained in the previous section, moving objects are searched for based on the MHI.

If a moving object is detected, a candidate motion pattern series (CMPS) for the object is gathered, and a feature vector representing the object movement is obtained using the CMPS. Finally, the action for the CMPS is determined using an SVM classifier. A detailed description of each processing step is given in the following subsections.

 

Fig. 5. Block diagram of the proposed cattle action recognition method.

3.1 Preprocessing

In the proposed system, a single fisheye camera is utilized to monitor multiple enclosures. This system configuration reduces costs effectively, but the fisheye lens causes geometric distortion leading to inaccurate motion information. In the preprocessing step, the fisheye lensdistortion is corrected, and the region of the enclosures to be monitored is extracted from each video frame, F(n). The extracted and undistorted region image is denoted by I(n).

 

Fig. 3. The process of MHI generation. (a) An input image sequence . (b) Binary maps indicating \(\begin{equation} D>\delta . \text { (c) MHIs } H^{\Gamma} \end{equation}\)

 

Fig. 6. Preprocessing result: (a) a fisheye image, and (b) an distorted image.

Fig. 6 shows an example of the preprocessing step. Since cattle in both the left and rightenclosures look shrunken due to the fisheye distortion, their motion is also represented assmaller than the actual movement. Through the preprocessing step, the sizes of adult cattle become similar, regardless of location, as shown in Fig. 6 (b).

Since the installed camera is fixed, the geometric transformation for correcting the lensdistortion does not change over time. Therefore, the coordinate mapping for the transformation can be reused to alleviate huge amounts of re-computation, once it iscalculated.

3.2 Candidate Motion Pattern Detection

In a dynamic complex scene, a number of motion patterns can appear in the MHI. In order to improve both computational efficiency and the robustness of the recognition method, only alarge motion pattern is regarded as cattle movement. Specifically, if the size of M_i(n) becomes larger than a predefined threshold at the n₀-th frame, we hypothesize that a certainaction of interest is about to begin.

 

Fig. 4. MHIs for the cattle actions of interest: (a) mounting, (b) walking, (c) tail wagging, and (d) foot

As a candidate for an action of interest, a series of motion patterns \(M_{i} \cdot\left (n_{0}+n\right)\)’scorresponding to \(M_{i}\left(n_{0}\right)\), called CMPS, are gathered for the following frames \((N<\Gamma)\). Then, the proposed features are extracted from the CMPS.

Object tracking has been one of the most important and still active research areas in the field of computer vision [26-29]. A recent research [28] can detect and track multiple objects successfully even in complex background. However, thanks to the slow cattle movement, asimple tracking algorithm is used in our method.

Since the corresponding motion patterns in successive frames typically overlap due tointrinsic slow motion of the cattle, the Jaccard similarity coefficient \(\mathcal{J}\) can be a simple buteffective measure for temporal association. Given two regions,\(R_{i} \text { and } R_{j}, \mathcal{J}\)is obtained as;

\(\mathcal{J}\left(R_{i}, R_{j}\right)=\frac{\left|R_{i} \cap R_{j}\right|}{\left|R_{i} \cup R_{j}\right|}\)       (2)

where |⋅| indicates the region area. That is, the more the two regions overlap, the larger \(\mathcal{J}\) becomes.

For a motion pattern \(M_{i}(n)\), the corresponding motion pattern \(M_{i^{*}}(n+1)\) within \(H^{\Gamma}(n+1)\) is determined as follows:

\(i^{*}=\underset{j}{\operatorname{argmax}} \mathcal{J}\left(M_{i}(n), M_{j}(n+1)\right)\)       (3)

The CMPS obtained from an MHI sequence generated with Γ is denoted by \(P_{n}^{\Gamma}=\)\(M_{i^{*}}\left(n_{0}+n\right),(0 \leq n \leq N)\) The CMPS tracked in an MHI sequence generated with Γ is denoted by \(P^{\Gamma}(n)=M_{i *}\left(n_{0}+n\right),(0 \leq n<N)\).

3.3 Proposed Features

Although MHI provides a rich representation of object movement, tremendous amounts of data and training time are required for learning MHI itself, due to its high dimensionality. In order to alleviate this problem, we extract low-dimensional features consisting of unique and distinguishable information from MHI, and an SVM classifier is trained using the features.

As mentioned above, cattle behavior can be characterized by the movement area that is represented in the size of the motion pattern in the MHI. The tail wagging and foot stamping behaviors show small-sized motion patterns, whereas walking and mounting behaviors havelarge motion patterns because the entire body moves. In contrast to the walking behaviorshowing continuous locomotion in a certain direction, the large amount of movement in the mounting behavior occurs instantaneously, and then rapidly disappears.

In order to discriminate such motion patterns, information about both the amount and direction of movement is utilized as features.

Three motion area-related values \(\mathcal{A}\left(P_{n}^{\Gamma_{1}}\right), \mathcal{A}\left(P_{n}^{\Gamma_{2}}\right), \text { and } \nabla\left(\mathcal{A}\left(P_{n}^{\Gamma_{1}}\right)\right)\) are exploited as the features for the amount of movement, where \(\mathcal{A}(\cdot)\) represents the region size and \(\nabla\left(\mathcal{A}\left(P_{n}^{\mathrm{\Gamma}_{1}}\right)\right)\) is obtained as

\(\nabla\left(\mathcal{A}\left(P_{n}^{\Gamma_{1}}\right)\right)=\mathcal{A}\left(P_{n}^{\Gamma_{1}}\right)-\mathcal{A}\left(P_{n-1}^{\Gamma_{1}}\right)\)      (4)

When \(\Gamma_{1}>\Gamma_{2}, \quad \mathcal{A}\left(P_{n}^{\Gamma_{1}}\right)\) represents the amount of all the movements detected within the past Γ₁ frames, whereas \(\mathcal{A}\left(P_{n}^{\Gamma_{2}}\right)\), indicates the amount of more recent movement, and thus \(\left(P_{n}^{\Gamma_{1}}\right) \geq {A}\left (P_{n}^{\Gamma_{2}}\right)\). The ratio between\(\mathcal{A}\left(P_{n}^{\Gamma_{1}}\right) \text { and } \mathcal{A}\left(P_{n}^{\Gamma_{2}}\right)\) can describe a uniquecharacteristic of the action. If \(\mathcal{A}\left(P_{n}^{\Gamma_{2}}\right)\) is close to \(\mathcal{A}\left(P_{n}^{\Gamma_{1}}\right)\), it implies that abrupt largemovement has occurred recently. Although both the mounting and walking actions producelarge motion patterns, walking action tends to keep \(\mathcal{A}\left(P_{n}^{\Gamma_{2}}\right)\) small, compared to \({A}\left (P_{n}^{\Gamma_{1}}\right)\), whereas with mounting action \(\mathcal{A}\left(P_{n}^{\Gamma_{2}}\right)\) fluctuates greatly.

To determine whether a specific direction exists in an action, positional information (i.e. the centroid of the motion pattern) is exploited. With mounting action, the center of the motion pattern does not move much, in spite of a large amount of movement. On the other hand, forwalking action, the centroid of the motion pattern continuously moves in one direction.

First, the MHI-weighted centroid position of the candidate motion pattern obtained with \(\Gamma_{1}\) is calculated as

\(\begin{aligned} &\bar{x}_{n}=\frac{1}{z} \sum_{\left(x_{k}, y_{k}\right) \in P_{n}^{\Gamma_{1}}} x_{k} \cdot H^{\Gamma_{1}}\left(x_{k}, y_{k}, n\right)\\ &\bar{y}_{n}=\frac{1}{z} \sum_{\left(x_{k}, y_{k}\right) \in P_{n}^{\Gamma_{1}}} y_{k} \cdot H^{\Gamma_{1}}\left(x_{k}, y_{k}, n\right) \end{aligned}\)       (5)

where \(z=\sum_{P_{p}^{\Gamma_{1}}} H^{\Gamma_{1}}\left(x_{k}, y_{k}, n\right)\) is the normalization factor.\(\bar{x}_{n} \text { and } \bar{y}_{n},(0 \leq n \leq N)\) contain the history of center position of the candidate motion pattern.

Then, using the principal component analysis with a set of points consisting of\(\bar{x}_{n} \text { and } \bar{y}_{n}\) two eigenvalues, \(\lambda_{0} \text { and } \lambda_{1}\), are obtained. Each eigenvalue is proportional to the variance of the given data along the direction of each principal component. As a result, if \(\lambda_{0}\) is quite greater than \(\lambda_{1}\), it can be determined that the centroid of the motion pattern moves in a specific direction. Therefore, the two eigenvalues are also used as features to represent existence of directionality in the candidate motion pattern.

Conclusively, once a candidate motion pattern is detected from \(H^{\Gamma_{1}}\left(n_{0}\right), \text { i.e. }\left|M_{i}\left(n_{0}\right)\right|>\xi\) , \(M_{i}\left(n_{0}\right)\)is tracked for the following frames to obtain \(P_{n}^{\Gamma_{1}} \text { and } P_{n}^{\Gamma_{2}},(0 \leq n \leq N)\). The threemotion area-related feature values, \(\mathcal{A}\left(P_{n}^{\Gamma_{1}}\right), \mathcal{A}\left(P_{n}^{\Gamma_{2}}\right), \text { and } \nabla\left(\mathcal{A}\left(P_{n}^{\Gamma_{1}}\right)\right)\) are determined in each frame, while the two motion directionality-related feature values, \(\lambda_{0} \text { and } \lambda_{1}\), are obtained once for the candidate motion pattern series. Consequently, for each candidate motion patternseries, a \((3 N+2)\)-dimensional vector is produced as the feature vector, and is then fed into the SVM classifier to identify the cow action pattern.

4. Experimental Results

In the experiments, a total of 104 video clips, 26 each for the four cattle behaviors, were used. The video sequences were taken at 20 frames per second at a farm in Cheonan, South Korea, from August 2016 to April 2017. Each video clip includes one of the four behaviors.

The proposed method was executed every fifth frame in the video clip (that is, at four framesper second) for effective dimension reduction of the feature vector without degrading performance. The image frame of the three adjacent enclosures, each of which was 8m long and 4m width, were undistorted to I_n with 400 × 600 resolution in the preprocessing step.

δ and ξ were set to 20 and 2000, respectively. The detected movement remains within MHI for at least three seconds by setting Γ₁ to 12; and Γ₂ (for keeping more recent movement) wasset to 1. Once a candidate motion pattern was detected, it was tracked for 7 seconds according to the mounting behavior duration. That is, the candidate motion pattern series consists of = 28 temporally corresponding motion patterns, and thus, 86-dimensional feature vectors were used in the experiments.

In order to evaluate the performance of the proposed cattle action recognition method, the MHI-histogram of gradients (MHIHoG)-based method [25] was implemented and compared. In this implementation, a 108-dimensional MHIHoG feature vector was extracted from each candidate motion pattern within the same undistorted MHI as in the proposed method.

For both the proposed method and MHIHoG identically, 28 video clips, seven each for the four actions, were employed to train the SVM classifier, and the other 76 clips were used forrecognition test.

Tables 1 and 2 summarize the feature matrices of the MHIHoG-based method and the proposed method, respectively, obtained using 5-fold cross validation. The general idea of confusion matrix is to count the number of times instances of class actual are classified as class predicted. The MHIHoG-based method had low recognition rates for mounting and foot stamping actions. Mounting and walking actions were confused with each other, and 22% of the two actions were misclassified with MHIHoG features.

The proposed method classifies cattle action patterns with a much higher recognition percentage than the MHIHoG-based method. In particular, for the mounting action (the mainaction to be recognized for estrus detection), the true positive rate became 0.89, whereas both false positive and false negative were relatively small. In particular, mutual misclassification between the mounting and walking actions was just 8% of the two actions.

Table 1. Confusion matrix of the MHIHOG method.

 

Table 2. Confusion matrix of the proposed method.

 

Table 3. Comparison of the proposed and the MHHOG method.

 

The proposed method successfully classified cattle actions in terms of the amount of movement. The mounting/walking actions involving large amounts of locomotion, and the tail wagging/foot stamping actions with small amounts of motion, were classified with 94% accuracy. In contrast, the MHIHoG method misclassified 15% of the data, in terms of the amount of movement, and in particular, the misclassification percentage increased to 33% for the foot stamping action. Table 3 shows the overall recognition rates for the two methods interms of accuracy, precision, and recall. For each action, the three metrics are calculated as abinary classification problem. Then, the overall evaluation for a certain metric is determined by averaging the corresponding metrics for the four actions. The proposed methodoutperformed the MHIHoG-based method in all the three metrics.

The processing time of the proposed method was 25ms for each frame. Specifically, the preprocessing that geometrically undistorts the fisheye image frame requires about 21.5ms. The remaining processes including MHI calculation, feature extraction, and classification takeonly 3~4ms, ensuring real-time monitoring.

5. Conclusion

In this paper, we presented MHI-based features representing cattle behaviors. The SVM classifier learned with the proposed features recognized cattle mounting action with anaccuracy of 89%. Compared with the conventional motion recognition method for human motion, the proposed feature effectively distinguishes the action of the cattle even with a lowerdimension. It was confirmed that the proposed action recognition method outperformed the conventional method. Thanks to its low complexity, the proposed method can be applied tomonitor multiple enclosures simultaneously. The proposed method is also inexpensive and can be applied in real complex environment.