MHT algorithm selects the optimal association hypothesis for the same object at two consecutive frames. And the calculation of hypothesis probability is critical. From the tracking start time to the *k*th time step, all the measurements are recorded as *Z*^{k} = {*Z*(1), *Z*(2),…, *Z*(*k*)} and all the hypothesis sets obtained by MHT algorithm at *k*th time step are recorded as \( \varOmega^{k} = \left\{ {\left. {\varOmega_{i}^{k} ,i = 1,2, \ldots ,I_{k} } \right\}} \right. \). The hypothesis probability \( P_{i}^{k} \) is calculated at the *k*th time step by the hypothesis \( \varOmega_{i}^{k} \) as follows:

$$ p_{i}^{k} \, = \,p\left( {\{ \varOmega_{i}^{k} \text{|}Z^{k} \} } \right) $$

(8)

Assumed that \( \varOmega_{i}^{k} \) is obtained by the correlation hypothesis *φ*_{k} between the hypothesis \( \varOmega_{g}^{k - 1} \) at previous frame and the measurements *Z*(*k*) at the current frame. Bayes theorem is utilized to compute the hypothesis probability:

$$ p\left( {\varOmega_{g}^{k - 1} ,\varphi \text{|}Z\left( k \right)} \right)\, = \frac{1}{c}p\left( {z\left( k \right)\text{|}\varOmega_{g}^{k - 1} ,\varphi_{k} } \right)\, \times p\left( {\varphi_{k} \text{|}\varOmega_{g}^{k - 1} } \right) \times \,p\left( {\varOmega_{g}^{k - 1} } \right) $$

(9)

where *c* is the normalized constant. In terms of the association hypothesis *φ*_{k}, the number of the measurements associated with the new object at the current frame is marked as *N*_{NT} (*h*), the number of the measurements associated with the false object is set as *N*_{FT} (*h*), the number of the measurements associated with the previous object is labeled as *N*_{DT} (*h*), and the number of all the object is *M*_{K}. Assumed that the number of the existing detected object obeys binomial distribution, the number of new objects is subject to Poisson distribution, and the number of false objects also obeys Poisson distribution, we can obtain:

$$ P(N_{DT} ,N_{FT} ,N_{NT} \left| {\varOmega_{g}^{k - 1} } \right.) = \left( \begin{aligned} N_{TGT} \hfill \\ N_{DT} \hfill \\ \end{aligned} \right)P_{D}^{{N_{DT} }} (1 - P_{D} )^{{(N_{TGT} - N_{DT} )}} \times F_{{N_{FT} }} (\beta_{FT} V)F_{{N_{NT} }} (\beta_{NT} V) $$

(10)

where *P*_{D} is the detection probability, β_{FT} is the probability density of the false alarm, β_{NT} is the probability density of the new objects, *F*_{n} (*λ*) is the Poisson distribution with the rate parameter \( \lambda \). Then we get:

$$ P(\varphi_{k} \left| {\varOmega_{g}^{k - 1} } \right.) = \frac{{N_{FT} !N_{NT} !}}{{M_{k} !}}P_{D}^{{N_{DT} }} (1 - P_{D} )^{{(N_{TGT} - N_{DT} )}} \times F_{{N_{FT} }} (\beta_{FT} V)F_{{N_{NT} }} (\beta_{NT} V) $$

(11)

The hypothesis probability is provided by:

$$ P_{i}^{k} = \frac{1}{c}P_{D}^{{N_{DT} }} (1 - P_{D} )^{{(N_{TGT} - N_{DT} )}} \times \beta_{FT}^{{N_{FT} }} \beta_{NT}^{{N_{NT} }} \times \left[\prod \begin{aligned} N_{DT} \hfill \\ m = 1 \hfill \\ \end{aligned} N(Z_{m} - H\tilde{X},S)\right]P_{g}^{k - 1} $$

(12)

After the probability of each possible association hypothesis is obtained, all the association probabilities are represented by one correlation matrix. The hypothesis *H* with the maximum association probability is selected as the optimal hypothesis.