All movement is relative, so if you say that the time-traveler would be left behind, you somehow have to pick a point of reference relative to which the Earth is moving and the victim is not.
Since we're inventing the technology, it wouldn't be unreasonable to say that, for instance, the object that is traveling through time retains the same physical location with respect to the object that has exerts the largest amount of gravitational pull on it.