Malloc bombs are programs that keep asking for memory. If resource limits are not properly set up, such program will probably make some systems thrash to death.
For example, the following program is a effective malloc bomb on most systems (because malloc has to save some bookkeeping information about allocated blocks, the system will have to give out some real memory).
main(){while(1){malloc(1);}}
Note that because sometimes the system does not actually allocate memory until the program touches it, certain malloc bombs are not effective on certain systems. For example, malloc(400000000) won't have much effect on many systems, but calloc(1,400000000) does in my linux box, because the C library cleared the space, which counts as "touching" the space.
It is even more effective if you "touch" the space continuously, with memset or similar things.
On modern systems, intelligent virtual memory manager will make such malloc bombs run mostly on swap, and when swap runs out, overcommiting control system will make malloc fail, so such programs have little chance to actually drive the system down. Nonetheless, they still can greatly reduce the responsiveness of the system by swapping out a lot of useful code and data.