Abnormal Event Detection and Localization Based on Crowd Analysis in Video Surveillance

: With the rapid development of economy and urban construction, abnormal events detection has arouse spread attention in need of public security. When an abnormal event occurs, the pedestrians in crowd escape or run instinctively which will lead to sharp change in the collectiveness feature and kinetic energy of crowd. This paper proposes a method based on the Collectiveness Energy Index (CEI) which combines the two features mentioned above to detect the abnormal events because it is not unreliable to utilize either of the two features singly. Besides, this paper also presents a means to locate abnormal behaviours in the anomalous scenes. Firstly, we obtain spatial coordinate of particles existed on individuals in each frame using generalized Kanade-Lucas-Tomasi


Introduction
With the rapid development of economy and urban construction, video surveillance technology has arouse spread attention in need of public security.As one of the main tasks of video surveillance, the detection and analysis of crowd abnormal events have become a hot research topic in recent years.The timely detection of abnormal events is important to protect the safety of people and public property.
Abnormal events are defined as those that deviate from regular pattern [1].Generally speaking, abnormal events are caused by objects with unexpected appearance or motion patterns.Thus, abnormal event detection is equivalent to detecting the irregular motion or appearance of pedestrians or other non-human objects [2].Recently, considering the outstanding performance of deep learning on various computer vision tasks [3,4], a variety of deep neural networks such as deep convolutional autoencoder (CAE) [5,6] and generative adversarial network (GAN) [7,8] have been utilized for video anomaly detection.However, most of deep learning-based methods process the entire video sequences in an end-to-end manner, without modelling the local interaction relationship between objects in the scene, which causes the loss of spatial context in the video sequence.
To this end, we present a novel video anomaly detection strategy based on the detailed analysis of object interaction.All of objects under a surveillance scene can be regarded as a crowd and the collective motion is the most significant feature of the crowd systems.We utilize the collectiveness [9] for modelling the spatially coherent structure of collective motion, which illustrates one important structural property of collective motion: behaviour consistency remains high among individuals in local neighbourhood, while low among those that are far apart, despite in the same collective manifold.Therefore, the collectiveness measurement is effective for monitoring the transition of a crowd system from ordered to disordered states, which further is capable of detecting crowd anomaly such as escaping.As for the crowd anomaly without apparent order transition such as running suddenly, we integrate the collectiveness with kinetic energy (reflecting the motion state of crowd) for accurate detection.Besides, this paper investigate the individual collectiveness and motion velocity for local anomalous object identification.
The primary contributions of this paper are listed as below: 1) An algorithm that fuses the collectiveness measurement and motion state is proposed to detect various types of anomalies.2) The proposed method is capable of detecting crowd abnormal events and locating individual anomalous action simultaneously; 3) Experimental results on two public benchmark datasets validate the effectiveness and robustness of the proposed method.The remainder of this paper is organized as follows: In section 2, we describe the detection method of abnormal events and present the localization scheme of abnormal behaviours in detail.In section 3, the experiment and evaluation results of two different datasets are provided to demonstrate the advantages of the proposed method.The final section gives a conclusion of the study.

Methodology
The system of this paper is shown in Fig. 1 and Fig. 2. We firstly use Kanade-Lucas-Tomasi key point tracker (gKLT) to extract individuals trajectories that are expressed by different spatial coordinates of particles existed on individuals at different time.After all required features are obtained, the abnormal events detection and anomaly localization are conducted.

Individuals Tracking
All particles' spatial coordinates in each frame got by gKLT algorithm are expressed as follows:   12 , ,...
where t H is the particles set at time t,   , ti ti XY is the coordinate of particle i at time t, I is the number of particles, tix V and tiy V are horizontal and vertical velocity of particle i respectively.

The Collectiveness Feature
A crowd is more than a gathering of individuals.Under certain circumstances, individuals in a crowd are organized into a unity with different levels of collective motions [9].Collectiveness measures the behaviour similarity of an individual with others in its neighbour.It is calculated by the cosin of an angle between two adjacent particles' direction of velocity.Assuming i and j are two adjacent particles, the behaviour similarity   , ti ti XY between i and j is defined as: Existing empirical studies of collective motion show that animals maintain local interaction among neighbours with a fixed number of neighbours on topological distance, rather than with all neighbours within a fixed spatial distance [10].Therefore, the     We define a set l P that contains all paths of length l between i and j, so the behaviour similarity of l-path is represented as: After utilizing the generating function to make the sum of   , l s i j converge, the individual collectiveness of particle i is denoted as: where z is a real-valued regularization factor, and l z can be interpreted as the weight for l-path similarity.The individual collectiveness would not increase with l when z < l.

Crowd Feature Extraction
The Kinetic Energy: The kinetic energy of crowd represents the intensity of the crowd's movement.We utilize the particles obtained by the gKLT algorithm to denote the pedestrians.Therefore, the kinetic energy of crowd equals to the summation of all particles' kinetic energy.According to the definition of kinetic energy in physics, the kinetic energy of crowd is expressed as: where i m is the velocity weight of particle i, and i m usually adopts the same constant.
ti V is the velocity of particle i at time t.
Crowd Collectiveness: The crowd collectiveness is defined as the mean of all particles' individual collectiveness at time t: where

 
CollectivenessSet i is the individual collectiveness of particle i.

Collectiveness Energy Index
In some scenes, pedestrians walk randomly all the time and begin to escape suddenly such as the UMN dataset [11], and the crowd collectiveness maintain a low value from start to end.Thus, using the crowd collectiveness singly is unreliable to detect the abnormal events.While in some other circumstances, pedestrians in crowd move quickly and towards the same direction at the initial time.Then pedestrians begin to escape.It is apparent that the kinetic energy is high all the time so that utilizing the kinetic energy singly to detect the abnormal events is unsuitable.
In conclusion, we should utilize feature multiplication method to combine crowd collectiveness with kinetic energy of crowd, which is expressed by Collectiveness Energy Index (CEI).The value of the crowd collectiveness ranges from 0 to 1, which is far lower than that of kinetic energy.So it is necessary to normalize the kinetic energy before combination.The normalized kinetic energy is represented as: where N is the number of frames,

  n k
Et is the normalized kinetic energy The Collectiveness Energy Index is defined as: where CEI(t) denotes the Collectiveness Energy Index at time t.

Abnormal Behaviours Localization
Abnormal behaviours includes crowd abnormal behaviours and individual abnormal behaviours.
In some cases, all of pedestrians would escape or run from dangerous region because of instinctive response when an abnormal event happens, which are considered as crowd abnormal behaviours.Therefore the localization of crowd abnormal behaviour equates to locate each escaping or running pedestrian.The main task of individual abnormal behaviour localization is to locate pedestrians or objects whose velocity and direction are obviously different.
We propose a novel method to locate the abnormal behaviours.Each frame of the video sequences is split into many blocks of size M * M, with block whose coordinate is (i, j) at frame t is represented by   , t B i j .The x-coordinate of blocks range from 1 to w/M, and the y-ordinates of blocks range from 1 to h/M, where w and h is the width and height of video frame.

Block Feature Extraction
Velocity: We estimate the velocity feature of block   , t B i j by computing the optical flow between two consecutive frames using Lucas-Kanade which is represented as: ,, where n x v and n y v are the optical flows of pixel n in the vertical and horizontal directions respectively, N is the total number of pixels within block   For some big objects such as carts, the number of particles existed on them obtained by gKLT tracker is more than that of small objects and these particles have the same direction of motion.So the value of   , t I i j belonging to block   , t B i j that covers the big objects is larger than that of normal block

Block Classification
In order to locate the abnormal behaviours, we just need to determine that which block   , t B i j is labelled as anomalous.Each block is examined by two classifiers.If the block is judged as anomaly by the first classifier, the second classifier is not carried out.Given certain velocity thresholds V, individual collectiveness threshold I.The block   , t B i j is considered as anomalous if either of the following formulas is workable: The first equation is a motion check, where objects with irregular speed are detected as anomalous such as the riding bike in the crowd.The second equation is an appearance check, where objects with irregular appearance are detected as anomalous such as the driving car in the crowd.

Experimental Results
Quantitative evaluation of our proposed method is conducted utilizing the accuracy index, which is calculated as below: where true positive (TP) is abnormal sample that is correctly detected, true negative (TN) is normal sample that is correctly identified, N is the total number of frames for the test video.

Crowd Abnormal Events Detection and Localization
The UMN dataset are shot in 3 different scenes, including lawn, indoor and plaza.In each scene, pedestrians walking randomly at the initial time is considered to be normal events and pedestrians fleeing suddenly is regarded as abnormal events.There are 11 abnormal events in total corresponding to 11 videos that are partitioned from the whole video set according to the 3 different scenarios.We select two videos from two typical scenes (lawn and plaza) in UMN dataset for experiments.Fig. 3 shows the CEI curves of the two videos.
From Fig. 3(a), the threshold is 0.3930, the CEI curve exceeds the threshold at 496th frame when the pedestrians start to escape.So the abnormal event occurs at the 496th frame.After 574th frame, the curve begins to below the threshold with pedestrians moving away from the field of view.We compare the proposed method with Collectiveness Model [9] (denoted as CM), Social Force Model [11] (denoted as SFM) and Energy Model [12] (denoted as EM) in identifying the crowd escape events.Table 1 shows the accuracy of the four methods for crowd escape event detection.Our method achieves an accuracy of 93.26%, which is higher than that of CM (81.60%),EM (89.66%) and SFM (87.62%).

Individual Abnormal Behaviours Localization
We carried out experiments on UCSD Anomaly Detection dataset [13] by the proposed localization algorithm.The dataset is acquired with a stationary camera and includes two subsets (Ped1 and Ped2) with diverse crowd density.The image size in Ped1 is 238×159 pixels, while on Ped2 it is 360×240.Anomalies commonly occurring in the dataset include: bikers, skaters, cars and other objects, while the videos that contain only pedestrians are regarded as normal.Some qualitative localization results of UCSD dataset are shown in Fig. 4, which confirms that the proposed method is capable to locate individual abnormal behaviours efficiently.

Conclusion
In this paper, we propose a brand new scheme for video anomaly detection based on the crowd interaction analysis.The proposed method employs the collectiveness measurement for modelling the spatially coherent structures of crowd, which is combined with the kinetic energy for multiple types of crowd anomalies detection.In addition, the motion velocity and individual collectiveness are jointly applied for local anomalies identification.Experiments are conducted on the UMN and UCSD datasets, and the testing results demonstrate that our method is effective for anomaly detection and localization simultaneously.Besides, the detection performance is superior to several compared methods in terms of accuracy.However, there are two factors that deteriorate the proposed method: 1) The gKLT tracker often tracks some particles that do not exist on objects, which may lead to producing wrong detection results; 2) The objects' size of the same kind vary greatly owing to the scene's perspective, and the equal-sized block structure in our method is not appropriate to deal with this issue.

Figure 1 :
Figure 1: The framework of abnormal events detection.

Figure 2 :
Figure 2: The framework of abnormal behaviours localization.

j
are in a neighbourhood, it cannot accurately estimate the behaviour similarity when i and j are at a distance.We should calculate the behaviour similarity based on topological structure: The path behaviour similarity based on l  is expressed as: Individual collectiveness indicates the consistency in movement direction of an individual with others.The individual collectiveness of block   is the individual collectiveness of particle n with coordinate (i, j), M is the number of particles whose value of individual collectiveness are not equal to 0 within block   , t B i j .
Fig. 3(b) shows 185 the same tendency like Fig. 3(a).The threshold is 0.2838 and the abnormal event is detected at the 554th frame.

Figure 3 :
Figure 3: The CEI curve of two videos in UNM dataset.

Figure 4 :
Figure 4: The localization results of UCSD dataset.

Table 1 :
Accuracy (%) comparison of proposed method with CM, SFM and EM on UMN dataset.