In the beginning, the original 13 colonies were land grants. Given the relatively cursory survey work at the time, easy-to-find geographical landmarks, especially rivers, were often used as dividers. This practice continued despite some silly results, such as cities essentially divided in half by state borders.
Eventually, however, things switched to using more logical geometric lines. This avoided problems with river drift as well as preventing the division of resources (particularly watery ones, again) in odd ways. In fact, one of James K. Polk's Presidential slogans, concerning the Oregon territory, went "54°40' or Fight!", referring to 54°40' north latitude as the boundary with the British territory that would later become Canada.
Interestingly, it has been proposed to redraw the lines of the 50 states differently, often reducing the number of states in the name of saving money, since (proponents of the idea claim) many of the costs of a state government are constant. In addition, they would keep large cities from being divided by state lines. Of course, conspiracy theorists took some of these proposed maps and claimed that was what the One World Goverment would do with their control structures...