As a small addendum to TheLutenist's excellent writeup, I think some mention should be made of the contributions of the French gambists.
The film Tous les matins du monde presents this side of the the Gamba. It mainly focuses on the early virtuosos, Sainte-Colombe and Marin Marais. Along with changes in playing style, the Gamba changed somewhat in France through the 17th and 18th centuries (and somewhat in the rest of Europe as well, though most of this originated in France). The use of the viol consort fell into disuse through the 17th century, and the bass viola da gamba became the most frequently used, though the pardessus de viole, similar in size to the treble, was also relatively common. This last is very infrequently played today, since most music for it is written for any upper instrument - flute, recorder, violin, pardessus de viole, or even hurdy-gurdy and bagpipes (of the musette variety, usually)
Some basic changes to the bass viol included a generally heavier and often slightly larger construction. The most obvious part of this was the addition of a seventh string in the bass, which was normally tuned to the A below the bass clef - a very low note indeed! The playing style was expanded to include more flashy techniques - lots of notes, of course, and also chords, pizzicato, and special effects like playing with the wood of the bow.
From here the instrument spread, like most things french, throughout Europe, and Bach, Telemann and other composers wrote for it. Another french composer of note is Forqueray, whose pieces were also arranged for harpsichord by his son.
This music, in combination with the consort pieces mentioned above make up the bulk of the modern gambist's repertoire, though they are also very frequently used as continuo instruments, playing the bass lines in orchestras and smaller ensembles. In this capacity, they mimic the more frequent use of the cello, but provide a pleasant contrast to its more assertive tone.