# Parallel implementation of color-based particle filter for object tracking in embedded systems

- Mai Thanh Nhat Truong
^{1}and - Sanghoon Kim
^{1}Email author

**7**:2

https://doi.org/10.1186/s13673-016-0082-1

© The Author(s) 2017

**Received: **17 August 2016

**Accepted: **8 December 2016

**Published: **9 January 2017

## Abstract

Recently, embedded systems have become popular because of the rising demand for portable, low-power devices. A common task for these devices is object tracking, which is an essential part of various applications. Until now, object tracking in video sequences remains a challenging problem because of the visual properties of objects and their surrounding environments. Among the common approaches, particle filter has been proven effective in dealing with difficulties in object tracking. In this research, we develop a particle filter based object tracking method using color distributions of video frames as features, and deploy it in an embedded system. Because particle filter is a high-complexity algorithm, we utilize computing power of embedded systems by implementing a parallel version of the algorithm. The experimental results show that parallelization can enhance the performance of particle filter when deployed in embedded systems.

## Keywords

## Background

Object tracking has important roles in many vision-based applications such as traffic monitoring [1], surveillance systems [2], and recently, augmented reality [3]. Generally, the goal of object tracking algorithms is to locate the moving objects of interest in the video data retrieved from image acquisition devices and to produce a record of the trajectory of the objects. In practice, object tracking can be more difficult because of object shapes, light conditions, occlusions, sudden change in object motions, camera motions, etc. Owing to various difficulties that cannot be solved simultaneously, object tracking methods are usually designed to track objects with specific properties in certain environments [4]. For instance, one object tracker produces good results in various environments under different lighting conditions, but produces low accuracy results when the target shape or silhouette is changed because of camera angles. One tracking method can predict target movement accurately, but it may fail when tracking bouncing objects as a result of sudden changes in movement direction. Therefore, object tracking is still a high-complexity and time-consuming task. The complexity is increased when the tracking task is performed in environments with complex surroundings, or the requirement is to track objects with various appearances.

Approaches for object tracking can be categorized into three groups: point tracking, kernel tracking, and silhouette tracking [5]. In point tracking, the target objects are represented by points that contain information of the object properties. Objects are tracked using the relation of the points, and their locations and movements are calculated based on the previous state of these points. However, an external mechanism is required to locate the objects in every frame. Kernel tracking relies on the object shape and appearance, which are called kernels. For instance, the kernel can be a rectangular region or an elliptical shape with an associated histogram. Objects are tracked by calculating the motion of the kernel in consecutive frames. This motion is usually defined as a parametric transformation such as translation, rotation, and affine. In silhouette tracking, the target objects are tracked by their estimated region in each frame. Silhouette tracking approaches use the features extracted from the object region, which are also called models. After having obtained the models, object silhouettes are tracked by using shape matching or contour evolution. Essentially, both these methods can be considered as object segmentation applied in the temporal domain using the object state from the previous frames.

Object tracking is also an important application for embedded systems. Recently, embedded systems and SoC (systems-on-chip) platforms have become popular because of the rising demand for portable, low-power devices. One of the most popular manufacturers of this type of platform is ARM Holdings, the developer of the ARM (Acorn RISC Machine) platform, a family of RISC (reduced instruction set computing) architectures for computer processors. Currently, processors based on the ARM platform are widely used in smartphones, tablets, smart wearable devices, and are being introduced into many other electronic devices. Therefore, developing efficient algorithms for such systems has attracted considerable attention. Many vision-based applications for embedded systems, including object tracking, have been already developed [6–8]. However, embedded systems have limited computing power and memory. This limitation prevents embedded systems from performing complex tasks, such as object tracking, within a reasonable execution time. For example, robust tracking algorithms based on high-complexity computing methods, like principal component analysis or image features such as SIFT (scale-invariant feature transform) and SURF (speeded-up robust features), are able to run in real-time only on modern powerful computers [9]. There is a trade-off between speed and reliability. Fast algorithms are less universal under certain conditions, and vice versa. For this reason, the development of algorithms which are both fast and reliable remains an ongoing problem.

In this research, particle filter is selected as the tracking framework among the common approaches for object tracking. Particle filter, a point tracking method, is used to perform estimation in state-space models, where the information of the state is acquired over time. It has been proven to produce high performance when applied to nonlinear and non-Gaussian estimation problems [10], hence particle filter is a robust solution for object tracking. This method approximates a posterior probability density of the state. In this case, it is the positions of the object in image sequence. The approximation is conducted by using point mass representations of probability densities, which are called particles. Usually contours, color features, or appearance models are used as particles when applying particle filter to object tracking [10–14]. A significant disadvantage of particle filter is computational complexity. The complexity increases as the area of the tracked region and the number of particles increase. When the dimensionality of the state space increases, the number of particles required for the sampling increases exponentially. A powerful computer is required for the purpose of producing results in a reasonable time. Fortunately, as a result of the rapid development of the semiconductor industry, embedded systems have become more powerful. This research presents a method for utilizing modern hardware architecture of embedded systems in an object tracking task. Parallel programming has been applied successfully in computer vision, showing its effectiveness in reducing the execution time of high-complexity tasks [15]. This is another reason for particle filter to be chosen, because parallel programming can be applied to increase computing speed while retaining tracking accuracy. In this paper, we introduce a parallel implementation of particle filter algorithm, for the purpose of enhancing the performance of the particle filter based object tracking method when deployed in embedded systems.

## Methods

### Particle filter

*t*indicate the time step. Then the advancement of the state \(x_t\) of a discrete dynamic system is defined as

*t*, an observation \(z_t\) of the state \(x_t\) is acquired as

*k*, the set of measurements \(Z_k = \{ z_t, t = 1, \ldots , k\}\) is available and the initial

*p.d.f*(probability distribution function) \(P(x_0)\) is given.

*p.d.f*\(P(x_{k-1} | Z_{k-1})\) at time \(k-1\), the prediction \(P(x_k | Z_{k-1})\) is calculated as

*p.d.f*\(P(x_k | Z_{k-1})\) at time

*k*is updated using Bayes’ rule

*p.d.f*\(P(x_0)\) is estimated from this particle set. Let \(x^i_k\) denote the particle

*i*, and \(w^i_k\) denote its corresponding noise, both at time

*k*. Each particle has its own transition function and advances independently. The prediction stage produces an approximation of the a priori

*p.d.f*

*N*is the number of particles. After prediction, the weight of each particle is updated using the obtained observations. Let \(\rho ^i_k\) be the weight of particle

*i*at time

*k*that can be calculated as

*N*, after redistribution. However, resampling leads to another problem called sampling impoverishment. In this phenomenon, high-weight particles have a greater chance of being drawn multiple times during redistribution, whereas low-weight particles have a small chance of being drawn at all. This decreases the diversity of the particles after a few resampling steps. In the worst case scenario, all particles might be “merged” into a single particle. Sampling impoverishment decreases the performance of the tracking process drastically. Therefore, an appropriate particle redistribution method is required. A review of resampling methods for particle filter is provided in [17].

### Color model

In tracking framework, the particle filter-based tracker requires a similarity measurement for the purpose of calculating the likelihood between particles and the target object. Before presenting our measurement, we will provide a brief introduction regarding visual features that are usually used in object tracking. As in [5], common visual features are color, edge, optical flow, and texture. For color features, the color of an object is affected mainly by two physical elements, the spectral power distribution of the illuminant and the surface properties of the object. In digital images and videos, colors are organized into specific sets called color spaces. This is an abstract mathematical model which simply describes the range of colors as tuples of numbers. There exist several color spaces for various purposes, and RGB (Red-Green-Blue) is the most commonly used color space [18]. However, the RGB space is not a perceptually uniform color space, in other words, small changes in RGB values may lead to large color differences perceived by the human eye. In contrast, L*u*v* and L*a*b* are perceptually uniform color spaces, while HSV (Hue-Saturation-Value) is an approximately uniform color space. However, these color spaces are sensitive to noise, which is a significant disadvantage because it is impossible to obtain noise-free data from image acquisition devices. Currently, there is no color space that is suitable for all vision-based applications. The choice of color space depends heavily on the goal of the application.

Edges are defined as strong changes in intensity values between object boundaries. Techniques used to identify these changes are called edge detectors. Edges are less sensitive to illumination changes than color features. This is an important property in object tracking because the algorithms are required to operate in various environments. The most popular edge detection technique is the Canny edge detector because of its simplicity and accuracy. Optical flow is the motion pattern of objects, surfaces, and edges in a visual scene caused by the relative motion between an observer (human eye or camera) and the scene. It is computed using the brightness constraint, which assumes brightness constancy of corresponding pixels in consecutive frames. Other than object tracking, optical flow is also used in motion estimation or video compression. For texture, it is a measure of the intensity variation of a surface which quantifies properties such as smoothness and regularity. Image texture gives us information about the spatial arrangement of color or intensities in an image or local regions of an image. This feature requires preprocessing procedures to extract texture information from images before being applied in desired algorithms. Texture features, like edge features, are also less sensitive to illumination changes.

Particle filter is a high-complexity algorithm; hence, a simple visual feature descriptor is required in order to prevent the increase of execution time of the whole tracking algorithm. Color distribution similarity is chosen because of its simplicity in calculation. We will combine particle filter and color information by integrating color distribution into particles. Each particle is defined as a rectangular region in the video frame and equal in size to the target object. Color distributions of particles are represented by three-dimensional RGB histograms. This color space is selected because of its robustness against noise and occlusion.

*M*bins to a given color pixel at location \(y_l\). In detail, we produce a histogram of a local region by discretizing the color pixels in the region into bins, and then count the number of pixels in each bin. For the discretization step, first we normalize the color values of pixels by dividing the value of each component by

*L*. Then, we quantize the normalized values into

*m*bins.

*L*and

*m*are the number of levels and bins for each component, respectively. These parameters must be the same for all three components. Let

*r*,

*g*, and

*b*be the values of the Red, Green, Blue components, respectively, of the pixel at location \(y_l\). Then, the equation \(h(\cdot )\) for assigning the histogram bin is defined as

*i*,

*j*, and

*k*are indexes of the chosen bin. For example, with \(L = 256\) (8-bit RGB) and \(m = 8\), the pixel at location \(y_0\) with RGB values (45, 172, 103) will be assigned to the bin with index (1, 5, 3). The total number of bins is \(M = 8 \times 8 \times 8 = 512\). Then, the number of pixels in each bin is used to construct the final histogram. Figure 1 shows an illustration for the output three-dimensional histogram from our calculation. The number of pixels in each bin is represented by size of the spheres.

*I*is the index of histogram bins, the term \((\sqrt{\bar{H_1} \bar{H_2} M^2})^{-1}\) is used for normalization. The proposed tracker employs the Hellinger distance to update the a priori distribution calculated by the particle filter.

### Tracking framework

The particle filter in this research, which is developed to track movement of objects, is based on the Condensation algorithm [10]. The state vector \(X_t = \{x_0, \ldots , x_t\}\) describes the state of the tracked object, and \(Z_t\) is the vector which stores all the observations of object movements \(\{z_0, \ldots , z_t\}\) up to time *t*. The Condensation algorithm estimates the object state by calculating the conditional probability density \(P(X_t | Z_t)\) using a nonlinear filter. \(P(X_t | Z_t)\) represents the possible states of the tracked object based on previous states and measurements. Usually, the posterior density \(P(X_t | Z_t)\) and the observation density \(P(Z_t | X_t)\) are non-Gaussian, which increases the complexity of the tracking process.

*N*is the number of particles. This particle set is used for approximating a probability distribution, which represents the object movement. For each particle,

*s*is the location of the image region with its associated histogram, and \(\pi \) is the corresponding weight which obeys

*N*particles are drawn using resampling techniques. The tracked object is localized by the estimation of mean state, which is calculated at each time step by

## Parallel implementation

Parallel computing is a type of computation in which many calculations or the execution of processes are carried out simultaneously. Large and complex calculations can often be divided into smaller ones, which can then be solved simultaneously. Parallelization can utilize the multi-core architecture of modern embedded systems, in which all cores of the processor take part in the calculation process. Parallel implementation can decrease the processing time of these steps, which leads to higher performance in object tracking due to higher processed frames per second. However, parallelization has its own limitation. Generally in parallel computing, a complex task is first split into discrete parts that can be solved concurrently. Following this, these parts are distributed to several threads, and are then solved separately in each thread. An overall coordination mechanism is employed to control the status of the threads. After completing the computation, these threads will communicate with each other using this mechanism in order to produce final results. In some cases, the time required for communication is higher than that for solving the split tasks. The overall execution time of parallel implementation in these cases may be higher than that for sequential implementation. This phenomenon is called parallel slowdown.

In the particle filter algorithm, the most complex and time-consuming steps are resampling of the particles and calculating the likelihood between the particles and the target object. Another complex step is calculating the weighted mean of the particles, however implementing this step will lead to parallel slowdown due to data dependency. Therefore, we will use parallel programming to reduce the execution time of the resampling and likelihood calculation. The detailed algorithms for these steps are listed below.

Since there are no data dependencies between particles in these algorithms, parallel threads do not need to communicate with each other during processing. For example, in the resampling step, newArr(1), newArr(2), newArr(3), and newArr(4) can be calculated simultaneously using four threads, and hence reduce the execution time of the whole process while avoiding parallel slow down.

## Experimental results

In this section, we present the performance of the object tracking method. Both versions of the particle filter algorithm, sequential and parallel, are implemented in C++ on a Linux operating system. OpenCV library is used for processing the video frames. For parallelization, we use the OpenMP library for the implementation of parallel particle filter. The program is deployed in embedded board Odroid U3, which has a quad-core processor running at 1.7 GHz and 2 gigabytes of memory. The videos used for testing are acquired from [21]. All videos has a resolution of \(320 \times 240\). In the next sections, we will show the efficiency of the particle filter object tracker and the comparison of performance between sequential and parallel implementation.

### Tracking accuracy

### Computational performance

## Conclusion

Recently, embedded systems have become popular with many applications. Developing efficient algorithms for such systems has attracted considerable attention. This research aims to increase the performance of the particle filter-based object tracking method when the algorithm is deployed in embedded systems by using parallel programming. A significant limitation of embedded systems is their computing power, hence it is difficult to use embedded systems for high-complexity computation. After several years of development, embedded systems have become more powerful. Currently, modern systems have high memory and multi-core architecture. However, the application must also utilize this advantage. In this research, we have implemented a parallel particle filter for object tracking in order to utilize the multi-core architecture. The proposed object tracker has shown its efficiency in different tests. The program is also deployed in Odroid, an embedded board with a four-core processor. With 300 particles, the object tracker with sequential implementation can process 11 frames per second, whereas parallel implementation can process 16 frames per second while retaining the tracking accuracy. The experimental results show that multi-core embedded systems can produce higher performance if the hardware is used at its maximum potential. However, the color model is not effective if the illumination of the environment changes. The proposed object tracking method also loses track of the object if there are other objects with similar color pattern. For future research, we will combine multiple features to increase the tracking accuracy. On the other hand, the tracking method must be optimized for the purpose of deploying in embedded systems.

## Declarations

### Authors' contributions

MTNT designed and developed the algorithm and took suggestions as and when necessary from Professor SK. Both authors read and approved the final manuscript.

### Acknowledgements

This research was supported by Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (2015R1D1A1A01057518).

### Competing interests

The authors declare that they have no competing interests.

**Open Access**This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

## Authors’ Affiliations

## References

- Kenan M, Fei H, Xiangmo Z (2016) Multiple vehicle detection and tracking in highway traffic surveillance video based on sift feature matching. J Inf Process Syst 12:183–195Google Scholar
- Juhyun L, Hanbyul C, Kicheon H (2015) A fainting condition detection system using thermal imaging cameras based object tracking algorithm. J Converg 6:1–15Google Scholar
- Bostanci E, Kanwal N, Clark AF (2015) Augmented reality applications for cultural heritage using kinect. Hum Cent Comput Inf Sci 5:1–18View ArticleGoogle Scholar
- Smeulders AWM, Chu DM, Cucchiara R, Calderara S, Dehghan A, Shah M (2014) Visual tracking: an experimental survey. IEEE Trans Pattern Anal Mach Intell 36:1442–1468View ArticleGoogle Scholar
- Yilmaz A, Javed O, Shah M (2006) Object tracking: a survey. ACM Comput Surv 38:1–45View ArticleGoogle Scholar
- Fiack L, Cuperlier N, Miramond B (2015) Embedded and real-time architecture for bio-inspired vision-based robot navigation. J Real Time Image Process 10:699–722View ArticleGoogle Scholar
- Hammoudi K, Benhabiles H, Kasraoui M, Ajam N, Dornaika F, Radhakrishnan K, Bandi K, Cai Q, Liu S (2015) Developing vision-based and cooperative vehicular embedded systems for enhancing road monitoring services. Proced Comp Sci 52:389–395View ArticleGoogle Scholar
- Lu F, Lee S, Kumar Satzoda R, Trivedi M (2016) Embedded computing framework for vision-based real-time surround threat analysis and driver assistance. In: The IEEE conference on computer vision and pattern recognition (CVPR) workshops, vol 1, pp 83–91Google Scholar
- Varfolomieiev A, Lysenko O (2016) An improved algorithm of median flow for visual object tracking and its implementation on arm platform. J Real Time Image Process 11:527–534View ArticleGoogle Scholar
- Isard M, Blake A (1998) Condensation-conditional density propagation for visual tracking. Int J Comput Vis 29:5–28View ArticleGoogle Scholar
- Isard M, Blake A (1998) In: Burkhardt H, Neumann B (eds) Icondensation: unifying low-level and high-level tracking in a stochastic framework. Springer, Berlin, pp 893–908Google Scholar
- MacCormick J, Blake A (2000) A probabilistic exclusion principle for tracking multiple objects. Int J Comput Vis 39:57–71View ArticleMATHGoogle Scholar
- Wu Y, Huang TS (2004) Robust visual tracking by integrating multiple cues based on co-inference learning. Int J Comput Vis 58:55–71View ArticleGoogle Scholar
- Khan Z, Balch T, Dellaert F (2004) A rao-blackwellized particle filter for eigentracking. In: Proceedings of the 2004 IEEE computer society conference on computer vision and pattern recognition, vol 2. CVPR, pp 980–986Google Scholar
- Moulkheir Naoui SM, Belalem G (2016) Feasibility study of a distributed and parallel environment for implementing the standard version of AAM model. J Inf Process Syst 12:149–168Google Scholar
- Stepanov OA, Vasil'ev VA (2016) Cramér-rao lower bound in nonlinear filtering problems under noises and measurement errors dependent on estimated parameters. Autom Remote Control 77:81–105MathSciNetView ArticleMATHGoogle Scholar
- Li T, Bolic M, Djuric PM (2015) Resampling methods for particle filtering: classification, implementation, and strategies. IEEE Signal Process Mag 32:70–86View ArticleGoogle Scholar
- Barthélemy Q, Larue A, Mars JI (2015) Color sparse representations for image processing: review, models, and prospects. IEEE Trans Image Process 24:3978–3989MathSciNetView ArticleGoogle Scholar
- Vojir T, Noskova J, Matas J (2014) Robust scale-adaptive mean-shift for tracking. Pattern Recognit Lett 49:250–258View ArticleGoogle Scholar
- Nummiaro K, Koller-Meier E, Van Gool L (2002) In: Van Gool L (ed) Object tracking with an adaptive color-based particle filter. Springer, Berlin, pp 353–360Google Scholar
- Tracking Dataset. http://cmp.felk.cvut.cz/~vojirtom/dataset/