The
probability density function for the
d-dimensional
normal distribution with
mean vector
μ and
covariance matrix Σ is given by the formula:
pμ,Σ(x) =
(e
-(x-μ)TΣ-1(x-μ)/2
)
/ ((2π)d|Σ|)1/2,
where
x and
μ are
d-dimensional
column vectors and Σ is a
d×
d matrix.
If the covariance matrix is singular, the naive application of this formula will run into problems, as in that case Σ will not be invertible and have a determinant of 0.
The singularity of Σ implies that the normal distribution is flat along one or more directions. The distribution is not really d-dimensional - it's less than d-dimensional. Specifically, the distribution lives in an affine subspace of the d-dimensional space. The rank of Σ gives the dimensionality of that affine subspace.
So the trick to handling singular Σs is this: first check if x is on that affine subspace. If it's not, then the probability density is automatically 0.
If it is, then we reduce the dimensionality of the problem and evaluate the density on the affine subspace (Hint: use the SVD of the covariance matrix).