A topological vector space is a real (or complex, but that ultimately makes no difference) vector space X, which is given a topology in which addition is a continuous map X × X → X and scalar multiplication is a continuous map R × X → X. Corresponding notions exist over other fields of scalars, but being an analyst I'm most familiar with the classical real case.

General topological vector spaces may be very badly behaved by the comfortable standards of Rn. They need not be metrizable, or even first countable or Hausdorff. Their topology need not derive from a norm, or a family of seminorms. Thus to prove interesting theorems you usually have to restrict the category of topological vector spaces under study. Progressively more specialized categories include:

locally convex spaces
This is the broadest category of TVS commonly encountered outside the abstract theory of TVSs;
Fréchet spaces
This is a fairly large group of TVS which are still complete metric spaces, and is the largest category on which a reasonable calculus including the inverse function theorem and similar results exists; also, since they are complete, in Fréchet spaces one has the classical consequences of the Baire category theorem (the open mapping, closed graph, and uniform boundedness theorems);
Banach spaces
This is a comfy category where every space has a norm, calculus is easy to do, and a rich operator theory exists -- although the geometry of Banach spaces can be complicated and includes some startling infinite-dimensional phenomena; Banach manifolds are interesting animals which crop up when you try to do differential geometry on manifolds of mappings between manifolds;
Hilbert spaces
These spaces are similar enough to finite-dimensional vector spaces that their operator theory follows fairly classical lines and was worked out in the early part of this century; quantum mechanics takes place mostly in operator algebras over Hilbert spaces;
Rn
and of course our familiar Euclidean space is a topological vector space, although one of a very special kind.
For more information consult Helmut Schaefer, Topological vector spaces, or if you're not familiar at all with functional analysis, better start with something like John B. Conway's Functional analysis: a first course (if you already know measure theory) or Introductory real analysis by Kolmogorov and Fomin (if you don't).