Computers have a finite amount of space for the
stack. When a program uses up more
stack space than has been allocated for it, a
stack overflow occurs. This is a common problem, because the fixed amount of space can be easily overrun if a program allows
recursion to get out of hand or uses too many large stack-based
variables.
In the bad old days, operating systems like MS-DOS had no real protection against this. The stack would typically grow downward towards the top of the heap, and when they eventually met, the heap would become corrupt. The program would usually hang the computer, or at least cause strange behavior.
Nowadays, things are better: modern operating systems can use hardware memory paging to control a program's behavior when it exceeds its alloted stack. The OS puts an unmapped page at the bottom of the stack. When the stack grows downwards into this boundary page, the operating system traps the page fault exception. It can either grows the stack by paging in some real memory, or it can shut down the program safely. Java, with its virtual machine, has even more control. Not only does it shut down an offending program, but it provides a nice stack trace, usually pointing to the offending code.
Stack overflows are usually accidental and are difficult to exploit. They shouldn't be confused with buffer overflow attacks or stack smashing attacks. These types of attacks deliberatly overflow a stack based variable. This corrupts other data on the stack, usually the return address from a function call, but the stack itself does not grow beyond it bounds.