So! Whether you are here because you want to give shape to your abstract aural fantasies, or because you want to break out of the guitar-bass-and-drums straightjacket by opening up some new sonic dimensions, you are really here because you feel that you should at least know a DX-7 from a Minimoog before you buy either. Good thinking! Buying a guitar (a very good thing to buy indeed!) can be difficult enough, but at least you think you can spot the difference between a Les Paul and a Strat.
Some terminology..
First, it is important to realize that there is no clear definition as to what a synthesizer really is. In fact, you'd best consider it a whole class of electronic musical instruments that may have nothing more in common than their ability to synthesize an audio signal. To control the floodgates, we will leave drum synthesizers and samplers out of this discussion.
Hardware considerations
Some synths have keyboards and some don't. If you are looking for a synth to play live you will have to take this into consideration. Further, most synths are equipped with cheap plastic keys that do not respond like piano keys. There are synths, however, that have very nice weighted keys. They play like real piano keys but they are very expensive. Further, the feel of piano keys isn't always desirable. When you play sounds with a slow attack, like string sounds, it will feel unnatural to have a simulated hammer effect in your keyboard. Generally, if you are planning to play your synth from the keyboard primarily, as opposed to sequencing it from a computer or hardware sequencer, you should spend some time behind the keyboard before buying the synth.
If you like to experiment, and I believe that is the point of buying a synth in the first place, you should look for a synth with lots of MIDI controllers. They allow you to change certain parameters of the sound in real time so you can perform your music with expression similar to that of more traditional instruments, if that is what you want of course. Controllers also allow you to do anything from subtle morphing to radical filtersweeps and whatnot. To put it as it is: controllers are essential. No matter what style of music you are into. Note that really old synths have no MIDI, but you can still control their sound using ordinary knobs.
The main disadvantage here is that you cannot record the movement of those knobs in a sequencer because they do not send out any information on the MIDI port.
Synth architecture
Although there can be a world of difference between synths in terms of sonic qualities and the means by which they synthesize sound, there is some common ground when it comes to internal organization. It is necessary to understand a bit about general synth architecture before you should decide on a synth.
Roughly, a synth has two different types of parts. First, there are the generative parts of the synthesizer (the oscillators) that produce a pitched complex waveform. Second, there are parts that allow you to manipulate that waveform. An examle of these are filters, ring modulators, etc. We will discuss both types of components below.
Generative components
This is where the most important differences between synthesizers are to be found. There are many protocols by which a synthesizer can generate sound. Each protocol has it's own characteristic idiom of timbres. Each protocol can produce its own universe of diverse sounds.
First, you should check out what type of synthesizing protocols a synth supports. Each protocol has its own merits. For instance, frequency modulation is cool for bright, highly synthetic string and piano-like sounds, but 'real' instrument simulation requires a physical modelling protocol or advanced wave table synthesis. Similarly, if you like acid bleeps you should consider subtractive synthesis. Many modern synths have more than one protocol. There is only one thing to go by here: your own ears!
Apart from the aforementioned protocols, there are many, many more exotic ways of synthesizing a sound, like granular synthesis, additive synthesis, variable phase modulation, comb filter oscillation, sync modulation, cross modulation, ring modulation, etc.
Signal manipulation components
The most common parts in this class are resonant frequency selective filters. Most synths have them, and they allow you to modify the sound of the synth by creating anything from subtle changes in brightness to dramatic tearing acid bleeps. Other signal manipulation components are ring modulators, wave clipping devices, resonators, and built in effects like chorus, reverb and delays. When it comes to signal manipulation: the more options the better.
Fixed versus modular architecture
If you are afraid of technology you should buy a synth with nice presets. However, synths can (and should) be reprogrammed. This is where you can do some serious sonic discovery. Most synths have a fixed signal path. That means that the manufacturer decides how the internal signal routing is arranged. In a modular system, you can decide what is controlled by what. More importantly: you can abuse that power! Modularity is desirable, but not for the fainthearted.
Polyphony and multitimbrality
Two very important concepts that are often overlooked by the unsuspecting buyer. Polyphony is the number of notes a synth can play at one time. Many old synths, like the TB-303 or the MiniMoog, are monophonic. They can play only one note at a time, so chords are out of the question. Modern examples of monophonic synths are the Waldorf Pulse and the Korg Prophecy. In general, anything above 16 voices of polyphony is ok, but if this is actually enough depends on the multitimbrality as well.
Multitimbrality is the ability of the synth to play multiple parts with different sounds on different MIDI channels. For instance, a piano part on channel one and a trumpet on channel 5. Again, old synths often do not support this at all and neither do the modern monophonic ones. If you have only one synth to create a whole piece of music, you have got to get a multitimbral synth.
On the other hand, if you already have a nice multitimbral synth and you need something really impressive for leads, then a really cool monophonic synth can be a very good option indeed. Further, if you have a 16 parts multitimbral synth you do not have to worry if you have only 24 voices of polyphony. In practice you will almost never have so many notes sounding at the same time.
A word of caution here: a multitimbral synth shares its polyphony over all voices. If you have a polyphony of two then you can play only one note on the trumpet and one note on the piano at the same time in the example above, or two notes on each alone.
MIDI and CV
When a synth is equipped with a MIDI port, it can communicate with other synths and computer hardware. This way, you can use a MIDI sequencer on a computer to 'play' the instrument for you, or you could use the keyboard one one synth to play notes on another. All serious synth manufacturers equip their products with MIDI. If you buy a a really old synth from around the early 1980's and before, it may not have MIDI. Then, you can only play it remotely using a control voltage (CV). There are special boxes that can convert MIDI to CV, but you should generally stay away from all this if you want a synth for making music with your computer.
Well that's about it. Good luck, and have fun!