A famous quote (from Feynman, I think) is that physics is to maths as sex is to masturbation.
By this is meant that the processes are similar, but physics produces tangible results whereas maths is just for fun (for the analogy I think we have to ignore the tangible results of masturbation). It sounds good, but it's not a brilliant analogy.
The real difference is that physics is a science, and maths is not. Physics therefore works on the principle of formulating a hypothesis, testing it, and discarding it if disproven. Maths works on the principle of playing around, sometimes with an aim (perhaps to prove something you suspect), sometimes without.
The confusing part is that the two are so closely linked. A scientific hypothesis can usually be best framed mathematically, and once that's done you can apply mathematical principles to it, which will usually work with reality as well. For example, if you hypothesise about something circular you can use geometry (in maths) to turn that into a statement about π and angles. This isn't always the case, because sometimes mathematics makes assumptions different to what holds in the real world. If I draw a triangle around the sun, the angles will add up to more than 180° because gravity curves space. Mathematics works only within the assumptions you put into it.
The beauty is that the real world often does correspond to something elegant mathematically. However, it is important to remember that the processes are quite different.