1. Introduction

Fire is one of the major hazards of the modern society as it causes grave and significant losses of lives, properties and socio-economic infrastructures around the world every year. In the past decades, different technologies have been developed to detect and control fire at very early stage. These technologies can be broadly classified into conventional methods and video-based techniques. Conventional methods employ ion or particle sensors, heat sensors, optical sensors (infrared, visible, ultraviolet), relative humidity sampler or air transparency sampler; whereas video-based fire detection (VFD) systems use video camera and computational techniques in image processing, machine vision and pattern-recognition to intelligently detect fire in a manner like the way humans sense fire.

Conventional methods have proven to be inefficient and unreliable in many applications and this could be attributed to many reasons, such as proximity of sensors to the source of the fire – to reduce transport delay. In addition, they are oftentimes difficult to use in places with excessive ceiling heights or large areas such as warehouses, tunnels, and outdoors. They are also not suitable in harsh environments and in areas with strong airflow- since the air flow may easily dilute the concentration of the smoke.

Video-based Fire Detection (VFD) techniques detect fire by recognizing either smoke or flame anywhere within the field of view of the camera at a distance by using numerical analysis to model the monitored area [13]. Vision-based detection techniques can be used to sense the presence of flames within the camera’s field of view, reflected fire light when flames are covered, presence of ambient or pluming smoke clouds, and intrusion into monitored property. VFD techniques are becoming viable alternatives to the conventional fire detection methods and have shown to be useful in solving several problems associated with conventional fire sensors [1]. VFD techniques have numerous advantages such as fast response, indoor and outdoor detection at a distance, non-contact, absence of spatial limits, ability to provide fire progress information, and forensic evidence for fire investigations [13].

Currently, available VFD algorithms mainly use models that are trained with observable characteristics of flame or smoke. In early studies, flame detection was the main subject of investigation. Recently, more attention is being focused on smoke detection. This is because smoke is usually produced before flames and can readily be observed from a long distance; therefore, it is an important sign for early fire detection [2].

Many smoke detection algorithms using video images captured in visible-spectrum have been proposed [3-13]. These algorithms extract structural and statistical features from visual signatures such as motion, colour, edge, obscuration, disorderliness, growth rate, contour, geometry, texture and energy of smoke regions. The extracted features are then used as inputs for rule-based, Bayesian, or rule-first-Bayesian-next analysis to detect the presence of smoke. A survey of different methods used for smoke detection is discussed in our earlier study [13]. [9] proposed a wavelet-based real-time smoke detection algorithm, in which both temporal and spatial wavelet transformations were employed. The temporal wavelet transformation is used to analyse the flicker of smoke like objects, while the spatial wavelet transformation is implemented to calculate the decrease in high-frequency content corresponding to edges caused by the blurring effect of smoke. [10] proposed a method targeted at reducing the false alarms of the smoke detection systems in their previous works. The smoke is represented as a texture using the parameters obtained from background estimation, wavelet transform, and colour information. The model is trained using SVM, and promising simulation results were obtained. [12] proposed a smoke-detection approach that utilizes block-based spatial and temporal analyses. A candidate-region extraction step is firstly performed using a combination of temporal difference and GMM background subtraction techniques. Then, the method extracts energy-based and normalized-RGB colour-based features within the spatial, temporal, and spatial-temporal wavelets domains [13]. The three features are combined and fed to a Gaussian kernel-based SVM for classification. To reduce the false alarm rate and maintain a high detection rate with a short reaction time, a temporal-based alarm decision unit (ADU) is introduced. An average detection rate of 83.5 %, false-alarm rate of 0.1% with average reaction time of 1.34 seconds was reported.

Smoke detection has been recognized as part of dynamic texture (DT) segmentation. Nonetheless, DT segmentation is very challenging due to their unknown spatiotemporal extension and stochastic nature of the motion fields. Leveraging on the remarkable results obtained by researchers in dynamic textures segmentation, [2] proposed feature extraction methods that exploit dynamic characteristics of smoke for video-based smoke detection [13]. The algorithm is made up of various block-based processing stages which include candidate smoke blocks detection using motion and colour in RGB colour space; and candidate smoke blocks verification using accumulative motion orientation, Histograms of Equivalent Patterns (HEP)-based spatial texture descriptors, and Space-time Feature Analysis which consists of inter-frame difference and dynamic texture Descriptors on Three Orthogonal Planes [13]. They carried out extensive comparative studies on major spatial and dynamic texture descriptors. They introduced Edge Orientation Histogram (EOH) in three orthogonal planes. The performances of the proposed features are evaluated using SVM classifiers. Their experimental results show that improved detection accuracy and false alarm resistance are achieved compared with state-of-the-art technologies.

Due to the irregular shapes of smoke, varied lighting conditions, occlusions, shadows, scene complexity, video-based smoke detection remains a challenging task. This study proposes an effective smoke-detection method using spatial-temporal wavelet energy analyses and Weber Local Descriptor in three orthogonal planes (WLD-TOP) as dynamic texture descriptor. In this paper, we introduce a novel method for smoke detection that exploits variations in wavelet energy of a scene covered with smoke and dynamic textural properties of smoke. [16] demonstrated that WLD outperforms in texture recognition than state-of-the-art best descriptors like LBP, Gabor, and SIFT. The basic WLD descriptor is a histogram where differential excitation values are integrated according to their gradient orientations. In this study, we generalize the spatial mode of WLD to a spatiotemporal mode as it was done in previous studies whereby LBP was generalized as a spatiotemporal descriptor, i.e., LBP in three orthogonal planes (LBP-TOP), which is very promising in DTs recognition. Likewise, we refer to the spatial-temporal mode of WLD as WLD-TOP.

The rest of the paper is organized as follows. The methodology used for the Smoke Detection system is discussed in Section 2. Experimental results are presented in Section 3. Section 4 concludes the paper.

2. Methodology

2.1. VSD System Architecture

The proposed Video-based Smoke Detection (VSD) system uses a combination of attributes (motion, colour, energy and texture) whose mutual occurrence leaves smoke as their only combined possible cause; and detect smoke using SVM classifier by block processing. The block diagram of the proposed VSD system is shown in Fig. 1. The architecture comprises of data acquisition, pre-processing, feature extraction, classification and post-processing stages.

2.2. Pre-Processing

The pre-processing stage comprised many sub-units which were interconnected to obtain regions that were suspected to contain smoke pixels. The sub-units in this stage were: image sub-blocking, RGB-Greyscale image conversion, colour analysis in HSI colour space, GMM background modelling and subtraction. For the purpose of model training and testing, video clips were collected from the test data previously used in [10], along with additional video clips downloaded from the following websites: http://signal.ee.bilkent.edu.tr/VisiFire/Demo/SampleClips; http://cvpr.kmu.ac.kr and http://imagelab.ing.unimore.it/visor. The frame rate of the video data varied from 15 to 30 Hz and the size of the video clips varied between 320 by 240 and 640 by 480 pixels. Brief description of the video clips used for system training and testing is shown in Table 1. Using video clips employed in previous studies allows quantitative comparison of proposed method with state-of-the-art methods in smoke detection. Since smoke is a non-rigid object, object-based image segmentation is inefficient for smoke detection in video. Therefore, a block-based technique, which provided a more effective smoke detection was used. Input images were subdivided into non-overlapping ‘N by N' square blocks, and the features were extracted from these local regions for smoke detection. Generally, there is a trade-off between complexity and performance of block-based image segmentation. It was observed that as the number of sub-blocks increased, the classification accuracy improved. However, the computational cost and the complexity of the detection also increased. The block size of 16 by 16 pixels was found to be a good compromise between accuracy and complexity. Video clips with frame height or width that were not multiple of 16 were padded by repeating border elements of the frame to make the frame height and width to be a multiple of 16. Sub-blocking was followed by a moving-block detection algorithm implemented using Gaussian Mixture Models (GMM). The video frames were then transformed into HSI colour space, where further analyses were performed to differentiate smoke candidate blocks from other non-smoke moving blocks. Smoke is semi-transparent when it initially starts to expand, which leads to a decrease in the chrominance values of pixels. YUV, YCbCr, HSV and HSI colour spaces were investigated on smoke of different colours to find robust colour model that will adequately characterize smoke of different colours with low computational complexity. Since smoke may have any colour (which can be grey, light grey, white, dark grey or black) depending on compositions of the fuel material, the chrominance based methods were found to be inadequate for smoke detection. HSI colour space was used for the proposed VSD as it provided colour-invariant characteristic feature that reliably differentiated smoke from other moving objects. Every pixel in each block was transformed from RGB colour space to HSI colour space using Eq (1) to (3). Every frame in a video sequence was multiplied by the foreground mask obtained from GMM. Saturation (S) and Intensity (I) were obtained from the resulting frame. A pixel is considered to be a smoke candidate if its saturation and intensity were less than empirically determined thresholds. A binary mask, (Colour_Motion_mask), was then obtained which indicated whether a given pixel was a smoke candidate or not. Since the Hue component of HSI was not required in this analysis, only Saturation and Intensity were computed to reduce computational complexity of the smoke detection system. The Colour_Motion_mask, φ(i, j) is defined by Eq (4).

Hue, H is given as

[1]

where

[1]

Saturation, S is given as

[2]

while intensity, I is given as

[3]

[4]

The flowchart for the HSI colour analysis of moving block is shown in Fig. 2.

Fig. 1
The VSD Architecture.

Fig. 2
Flowchart of the HSI Colour Analysis of a Moving Block

Pixels with the maximum number of changes across frames were selected from a large number of video sequences (smoke and non-smoke video clips). Saturation and Intensity values were computed for the selected pixels in each frame of the sequences. The results obtained for the selected pixels in a typical smoke video clip and non-smoke video clip are shown in Fig. 3(a to d). From the result, it was observed that variations in intensity and saturation of smoke pixels are significantly different from those of other moving objects. While variations in smoke pixels are gradual and irregular, that of rigid moving objects tends to be spontaneous and regular. The gradual and irregular nature of variations in intensities of smoke pixels could be attributed to irregular nature of smoke motion. Also, in the absence of strong wind, the intensity of smoke pixels varied slowly when compared with that of a pixel of a non-smoke solid moving object.

As shown in Fig. 3(a) and Fig. 3(c), saturation values of the selected smoke pixel varied between 0.061 and 0.318; while that of the selected non-smoke moving pixel varied between 0.005 and 0.19. The results indicated that a smoke pixel could not have zero value -since saturation of zero value could only be obtained from objects with pure black colour. Though the minimum and maximum saturation values obtained for the selected non-smoke moving pixel were 0.005 and 0.19 respectively, it should be noted that the saturation value of a non-smoke pixel could vary between 0.000 (for pure black object) and 1.000 (for pure white object). To reduce the possibility of categorizing a smoke pixel as non-smoke a threshold value of 0.7 was selected for maximum saturation value that could be obtained from smoke pixel. As shown in Fig. 3(b) and Fig. 3(d), it was observed that the intensity values of the selected smoke pixel varied between 0.319 and 0.616 while that of selected non-smoke object varied between 0.290 and 0.990. The intensity value of an ordinary non-smoke object can range from 0.000 (for black object) to 1.000 (for shining white object). Though the maximum intensity value of the selected smoke pixel was 0.616, a threshold value of 0.900 was used to differentiate smoke pixel from other moving objects. This was important to reduce the possibility of rejecting a smoke candidate pixel at the pre-processing stage. Though the selected threshold allowed other non-smoke blocks to pass the pre-processing stage, the non-smoke blocks were easily rejected during SVM classification.

Thus, in generating colour mask, a moving pixel was considered to be smoke candidate if its saturation was less than 0.7 and its intensity less than or equal to 0.9. An "AND" logical combination of the colour mask and motion mask was performed to obtain smoke candidate blocks that were fed to the feature extraction stage.

2.3. Feature Extraction

The input data (smoke candidate blocks in the video frame) was transformed into a reduced representation set of features at this stage. The outputs (marked blocks) of the pre-processing stage were used as the inputs for the feature extraction stage. Two processes used in this stage were 2-D spatial-temporal wavelet analysis and dynamic texture analysis.

2.3.1. 2-D spatial-temporal wavelet analysis.

Previous studies have shown that wavelet sub-images contain the spatial texture and edge information of the original image in form of local extrema. Discrete Wavelet Transform (DWT) has become an efficient tool in evaluating the energy variation of an intensity image for smoke detection [8]. DWT has also proved useful in obtaining decomposed images through various sub-bands that allow extraction of smoke’s features at different resolutions and frequencies. The two-dimensional DWT is made up of approximation and detail parts. The original image is decomposed into four sub-images, which are Wф, WφH, WφV and WφD. The scale j + 1 approximation coefficients are again divided into four sub-images of smaller size. In other words, the output of DWT is a vector of the form [An, (Hj, Vj, Dj)j = 1,…, n] where An is a low-resolution approximation (low-frequency data of row and column, LL) of the original image, Hj is wavelet sub-image containing the image details in horizontal direction (high-frequency data of row and low-frequency data of column, HL), Vj is wavelet sub-images containing the image details in vertical direction (low-frequency data of row and high-frequency data of column, LH) , Dj is wavelet sub-image containing the image details in diagonal direction (high-frequency data of row and column, HH) at the j-level decomposition. The output of n-level DWT decomposition on the original image will produce 3n + 1 sub-images. A 1-level 2D decomposition is shown in Fig. 4.

Fig. 3
Saturation and intensity variation of selected pixels of video frames with smoke.

(a) Saturation variation

Fig. 3
Saturation and intensity variation of selected pixels of video frames with smoke.

(b) Intensity variation

Fig. 3
Saturation and intensity variation of selected pixels of non-smoke video frames

(c) Saturation variation

Fig. 3
Saturation and intensity variation of selected pixels of non-smoke video frames

(d) Intensity variation

Fig. 4
2-D Discrete wavelet transform

High-frequency energy of each block was computed using single stage 2-D discrete wavelet decomposition of the current image block using Eq (5).

[5]

where bk is the kth block of the active frame and It is the input image at time t; H, V and D are vertical, horizontal and diagonal sub-band details respectively.

One-level Daubechies (Haar) wavelet was used for the wavelet analysis. To extract spatial-temporal energy variations from video frames, wavelet energy was computed for W consecutive video frames and five derived features were calculated from the sub-band energies. The five statistical parameters were variance, standard deviation, skewness, kurtosis and sum of inter-frame energy differences.

Blocks with maximum temporal variation across frames were selected from a smoke video clip and non-smoke video clip. Each block’s energy was computed over the video sequences. Fig. 5 shows the results of a comparison of the energy analysis for smoke and an ordinary moving object. Smoke produced a smoother variation in the energy value. In contrast, non-smoke solid moving objects produced large instantaneous variations in the energy value. To obtain these variations irrespective of the absolute value of the block energy, several statistical parameters were computed. The computed parameters were: variance, standard deviation, skewness, kurtosis and sum of inter-frame energy differences.

Fig. 5(a) and 5(b) show that changes in variance of wavelet energy produced when a selected block was obscured by smoke were more irregular than when a selected block was obscured by non-smoke moving objects. The variance of wavelet energy in the case of non-smoke moving block was predictably regular and varied from near zero to maximum value along the consecutive video frames. Fig. 5(c) and 5(d) show the plot of sum of energy change for the selected block in smoke and non-smoke block respectively over time. It could be observed that the sum of energy change for non-smoke moving objects exhibits more predictable, regular and oscillatory change than that of smoke. From the results, all the statistical parameters obtained from wavelet energy analysis were relevant in discriminating smoke from other moving objects.

Fig. 5(a)
Variance of energy change for selected block in video with smoke

Fig. 5(b)
Variance of energy change for selected block in non-smoke video

Fig. 5(c)
Sum of energy change for selected block in video with smoke

Fig. 5(d)
Sum of energy change for selected block in non-smoke video

The results obtained showed that the spatial-temporal features obtained from wavelet energy of a video clip could easily be used to distinguish smoke from non-smoke moving object.

2.3.2. Dynamic Texture analysis using Weber Local Descriptor (WLD)

Weber Local Descriptor (WLD) descriptor is based on Weber’s Law which states that the change of a stimulus (such as sound or light) that is just noticeable is a constant ratio of the original stimulus. Inspired by this law, [16] proposed WLD descriptor for texture representation. WLD descriptor represents an image as a histogram of differential excitations and gradient orientations and possesses several desirable properties such as robustness to noise and illumination changes, elegant detection of edges and powerful image representation [16]. WLD is computationally simple and effective for texture classification, and it is complementary to LBP. Three steps which are required in computing basic WLD descriptor are: finding differential excitations, gradient orientations and building the histogram.

To get differential excitation ℰ(xc) of a pixel xc, firstly the intensity differences of xc with its neighbours xi, i = 1, 2, …, p are calculated as

[6]

Then the ratio of total intensity difference of xc with its neighbours xi to the intensity of xc is calculated as follows:

[7]

Arctangent function can be used as a filter on fratio to enhance the robustness of WLD against noise which results in:

[8]

where Ixx is the intensity difference between two pixels on the left and right of the current pixel xc, and Iyy is the intensity difference of two pixels directly below and above the current pixel,

The gradient orientations are quantized into T dominant orientations as:

where

[10]

In our case T = 12 and the dominant orientations are

[11]

References

[1] E.A. Çetin, K. Dimitropoulos, B. Gouverneur, H.Y. Habiboglu, B.U. Töreyin, and S. Verstockt, et al., “Video fire detection – review,” Digital Signal Processing, vol. 23, no. 6, pp. 1827–1843, 2013.

[2] J. Chen and Y. You, “Early Fire Detection Using HEP and Space-time Analysis,” Computer Vision and Pattern Recognition (cs.CV); Multimedia (cs.MM) arXiv:1310.1855, pp. 228 –312, 2013.

[3] J. Chen, Y. You, and Q. Peng, “Dynamic analysis for video-based smoke detection,” International Journal of Computer Science Issues, vol. 10, no. 2, pp. 298–304, 2013.

[4] L. Wen-hui, F. Bo, C. Xiao-lin, W. Ying, and L. Pei-xun, “A Block-based Video Smoke Detection Algorithm,” Journal of Jilin University (Science Edition), vol. 50, no. 5, pp. 979-986, 2012.

[5] J. Gubbi, S. Marusic, and M. Palaniswami, “Smoke detection in video using wavelets and support vector machines,” Fire Safety Journal, vol. 44, no.8, pp. 1110 –1115, (2009).

[6] R.A. Gonzalez-Gonzalez, V. Alarcon-Aquino, O. Starostenko, R. Rosas-Romero, J.M. Ramirez-Cortes, and J. Rodriguez-Asomoza, “Wavelet-based smoke detection in outdoor video sequences,” in Proceedings of the 53rd IEEE Midwest Symposium on Circuits and Systems (MWSCAS), pp. 383 –387, 2010.

[7] F. Yuan, “A fast accumulative motion orientation model based on integral image for video smoke detection,” Pattern Recognition Letters, vol. 29, no. 7, pp. 925 - 932, 2008.

[8] B. U. Töreyin, Y. Dedeoglu, and A.E. Çetin, “Wavelet-based real-time smoke detection in video,” in European Signal Processing Conference (EUSIPCO), pp. 246- 302, 2005.

[9] B. U. Töreyin, Y. Dedeoglu, and A.E. Çetin, “Flame Detection in Video Using Hidden Markov Models,” in International Conference on Image Processing (ICIP 2005), pp. 1230–1233, 2005.

[10] B. U. Töreyin, Fire Detection Algorithms Using Multimodal Signal and Image Analysis. PhD thesis, Dept. Elect. Eng., Bilkent University, Ankara, Turkey. Available at: http://www.arehna.di.uoa.gr/thesis/uploaded_data/Fire_Detection_Algorithms_Using_Multimodal_Signal_and_Image_Analysis_2009_thesis_1232106137.pdf, 2009.

[11] P. Piccinini, S. Calderara, and R. Cucchiara, “Reliable smoke detection system in the domains of image energy and colour,” in Proceedings of International Conference on Image Processing, pp. 1376 –1379, 2008.

[12] L. Chen-Yu, L. Chin-Teng, H. Chao-Ting, and S. Miin-Tsair, “Smoke Detection using Spatial and Temporal Analyses,” International Journal of Innovative Computing, Information and Control, vol. 8, no. 6, pp. 200-300, 2012.

[13] J.A. Ojo and J.A. Oladosu, “Video-based Smoke Detection Algorithms: A Chronological Survey,” International Institute for Science, Technology and Education (IISTE), ISSN 2222-1719 (Paper), ISSN 2222-2863 (Online) vol. 5, no. 7, 2014.

[14] B.U. Töreyin, Y. Dedeoglu, A.E. Çetin, S. Fazekas, D. Chetverikov, and T. Amiaz, et al., “Dynamic texture detection, segmentation and analysis,” Proceedings of ACM International Conference on Image and Video Retrieval (CIVR), pp. 131 –134, 2007,

[15] B.U. Töreyin and A. E. Çetin, “Wildfire detection using LMS based active learning,” in Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 12-100, 2009.

[16] U. Ihsan, H. Muhammad, M. Ghulam, A. Hatim, B. George, and M. M. Anwar, “Gender Recognition from Face Images with Local WLD Descriptor,” IWSSIP 2012, Vienna, Austria, vol. 5, no.6, pp. 11-13, 2012.

[17] D. Sloven, P. Renaud, and M. Michel, “Characterization and Recognition of Dynamic Textures based on 2D+T Curvelet Transform,” Signal, Image and Video Processing, pp. 146- 272, 2013.

Additional information

Cómo citar: J. A. Ojo and J. A. Oladosu. “Effective Smoke Detection Using Spatial-Temporal Energy and Weber Local Descriptors in Three Orthogonal Planes (WLD-TOP”, Journal of Computer Science & Technology, vol. 18, no. 1, pp. 38-50, 2018