If I get your goat, I have annoyed or irritated you.
The root of this idiom may come from an old practice (which may or may not have been English or Welsh) of keeping a goat with one's cows. Apparently, the goat was a calming influence and thus lead to the production of more milk.
If one were feeling nefarious, one might get one's neighbour's goat (presumably by illegal means), which would remove the calming influence and make one's neighbour's cows less productive and one's neighbour more irritated or annoyed.
A similar practice in the early 20th century occured in stables of race horses. Again, the goat is supposed to be calming and its removal would presumably have a negative effect on the performance of the horse and the mood of the owner.
"To get somone's goat" was in the 1904 edition of the Random House dictionary with another explanation for its being. According to Random House, "goat" is prison slang for "anger". Thus, to get one's goat is to provoke one's anger.
The meaning that seems most plausible to you depends, I suppose, whether you have a stronger affinity with prisons or with stables. Both are equally likely to be accurate.
Or maybe it's just that goats get annoyed and irritated, and so do people.
kthejoker says A true story: Seabiscuit's trainer bought a goat to put in his stall to settle him down. The horse was rather indignant, and picked up the goat
by the tail and tossed it out of the stall.
The Debutante says You might be interested to know that lonely or fractious racehorses are still offered goats for company. The goat we kept was called Rachel...
Sources
The Wordsworth Dictionary of Idioms (1993)
The Bedtime Browser at http://www.briggs13.fsnet.co.uk/book/g.htm