In the statistical/experimental sense, a sample is defined by the AHD as "A set of elements drawn from and analyzed to estimate the characteristics of a population." Sampling is the act of taking samples in order to test or examine something. It is used primarily in research, to select a representative sample of a population (of people, nodes, research papers, frogs, whatever), so that we may understand the whole better by analyzing the sample.
For more information, you should really read The Statistics Project metanode (or statistical sampling if you want to just know about that sort of thing).
In the musical sense, sampling occurs when an artist reuses a sound from another source in their music. Sampling is especially common in rap (where dialogues are often built up between rappers by sampling each other's songs), techno and electronica (where the songs are often entirely built from samples), though the technique is also used in many recent works by pop artists. Several music creation programs, most notably Acid Pro by Sonic Foundry, revolve entirely around the use of samples, and come with many, many samples to get you started.
There are in fact two types of sampling that are to be found. The first revolves around using a "chunk" of sound, like a few phrases of speech, or a few stanzas of singing, or even a melody, like the theme from "The Price Is Right." Songs that include such samples often use them for comedic effect, or sometimes just to play with the sound. The second type is the sampling of instruments - for instance, drum machines use recorded samples of real drums in order to create their sound. Fruity Loops uses sampled instruments for a lot of its percussive effects. There are also synthesizers into which you can feed a sample, and then play like an instrument, modulating the playback of the sample.
About the history and culture of sampling, there is a lot to say, and I don't feel that I'm at all qualified to discuss it. Perhaps mcc or PMDBoi could add something to the mix (no pun intended), but until then, let me leave you with pingouin's words on the topic...
"A whole era of rap was built on samples from James Brown LPs. Public Enemy made a weird gumbo from widely varied sources. Depeche Mode and Art of Noise sometimes built entire structures out of everyday sounds. Its use in pop is now as commonplace and cliched as a Chuck Berry riff."
In the signal processing sense, sampling refers to the act of taking a sample from a signal, usually at a certain rate, known as the sampling rate. You can supersample and oversample, and if you sample at a "bad" frequency, you can get artifacts. DSP, or Digital Signal Processing deals a lot with samples.