Your scrotum, the sack that holds your two spheres of love just below Mr. Lucky, is an incubator for your sperm. Sperm need to live at a constant temperature in order to survive; your scrotum will retract in order to bring the balls closer to the body for warmth when they're cold, and vice versa when they're hot.
On the other hand, sometimes fear induces a physiological response that causes your scrotum to retract. This is probably to make sure that the reproductive organs aren't harmed while experiencing dangerous situations. Examples of such situations include dragon slaying and systematic downvoting.