The best put down of this phrase I ever saw was on a sketch show (I forget which one).

Person 1: I'm not racist but I do believe people of different ethnic groups have different abilities.
Person 2: I'm not sexist but I do think women should stay at home while the man goes out to work.
Person 3: I'm not a chiropodist but I do remove corns from people's feet for a living...