Just to clarify: a graviton is actually a spin 2 tensor boson, not a vector. For deep reasons, a particle's spin relates directly to which irreducible representation of the Lorentz group it fits into (vectors, tensors, spinors, etc.). Here's a quick glossary:
Spin 0: Scalar
Spin 1/2: Spinor
Spin 1: Vector
Spin 2: Tensor
Particles higher than spin 2 cause problems in quantum field theory. However, they happen naturally in perturbative string theory. Some of us don't like string theory. Some do.
But yes, the graviton is associated with the gravitational force, which naturally is tensorial (consider the metric in general relativity). One can think of the graviton as a wave-like perturbation away from the flat Minkowski background. But then, don't we really want a background-free formulation of physics? I know I do.