Note: This writeup requires some basic knowledge in assembly and the inner workings of a computer to make sense.
The most common buffer overflow exploit involves the abuse of the stack. In most operating systems, each program will be allocated a stack, which is used to pass arguments, store local variables, and hold the return address.
Some background information must be covered first, before we can start talking about the attack outright.
The return address is a memory location that will have control transferred to it when the ret opcode is executed. This is used to implement functions; when a program uses call foo to launch a function foo, the computer will push the address of the next instruction into the stack, and perform other diagnostic operations before control is transferred to foo. When foo returns via ret, the return address is popped off the stack and used to replace the instruction pointer, thereby relinquishing control to the instruction immediately after call foo, as it should.
When a function is called, it will typically do a number of things to set up the computer state so the function can... erm... function. One of these things would be to allocate some of the space on the stack for local variables. If, for example, I have a local character buffer of size 100 bytes, 100 bytes will be allocated off the top of the stack for this variable.
Thus, when the actual function code begins, the stack will look somewhat like this:
+-----------+----------------+-----+-----------------+
| arguments | return address | etc | local variables |
+-----------+----------------+-----+-----------------+
Except, it doesn't
really look like this, at least in the case of the
x86 architecture, because on
intel-based systems, the stack
grows downwards; the
bottom of the stack has the
highest memory address while the
top has the
lowest. Think of it as something like this (excuse the
lousy ascii art):
Low memory address -------------> High memory address
+-----------------+-------+-----+------+------+-----+
|Buffer..... |blah |0x20 |0x46F |arg2 |arg1 |
+-----------------+-------+-----+------+------+-----+
| variables | fp | ret | arguments |
Top of stack <------------- Bottom of stack
Note that while the stack seems to be "upside down", variables still grow from left to right.
Okay. With that set, we can now go into the actual exploitation technique.
The way a stack buffer overflow works is that it uses the overflow to overwrite the return address. When a static-sized buffer has data larger than its size copied into it, the extra data is written into the memory space following it. This implies that if a sufficiently big chunk of data is copied into the buffer, it will cause the return address to be replaced.
Since the return address is meant to point to the code which foo should secede control to once it finishes, replacing the return address would allow an attacker to cause the computer to execute code from just about anywhere in the program's allocated memory space.
What most exploit programs do is replace the return address with a pointer to the beginning of the buffer. The data stored at the beginning of this buffer is actually some machine code which can perform various operations using the priviledge offered by the program being exploited. This is typically something that spawns a shell, aka shellcode.
While this seems to be quite simple, in reality it is much trickier. An exploit writer must take into consideration the size of buffer space available, as well as if any characters will be filtered out by the function before data is written into the buffer. Also, because the location of the buffer in memory is dynamic and can change with each separate run of the program, overwriting the return address with the memory location of the buffer requires various workarounds and to some extent guessing.
A more detailed discussion on buffer overflow exploits, as well how to overcome the issues mentioned in the preceding paragraph, can be found in "Smashing the Stack for Fun and Profit", written by Aelph One.