A system is said to be at an unstable equilibrium when a slight pertubation will knock it out of equilibrium. For example, a ball balanced on the top of a bowl is at an unstable equilibrium because any slight force that acts on the ball will cause it to roll off. Compare with stable equilibrium, an example of which is a ball at the bottom of a bowl. Any force on this ball and it will roll around a little bit, but then come back to rest at its initial position.
In Classical Mechanics, an unstable equilibrium is at a point where
dU
-- = 0
dx
and
d2U
--- < 0
dx2
where U is the potential energy.