A set of rules that define the probibility of an event occuring in a roleplaying game. Anything from how hard it is to hit a monster with a weapon to the chance that an angry sea captain will accept a business proposal can be represented by numbers. Usually, some sort of difficulty is set, and then a player character must roll dice to beat that difficulty. If the die roll is high (or low) enough, then the action succeeds. If not, the game master gets to have his fun.
Of course, player interaction is only one of many things that may be incorporated into a system. Things such as random treasure generation, number/type of monsters appearing in an encounter, etc. are also often part of a system.