During a discussion prior to an exam for psychopharmacology while I was at U-M, my professor (who does research in that field) related some information on Aspartame... a colleague of his had done research on the substance and his conclusions were that Aspartame causes perceptible memory impairment in the short term; long-term effects were not a part of the study. Hearsay? Oh yes. But nonetheless, here's a professor in the field of neurology (Tim Scherer) who doesn't drink anything with Aspartame in it EVER - and that's good enough for me (especially since I've never really liked the way it tastes in the first place).
The fact that Monsanto owned Nutrasweet for a while is also not exactly reassuring - Monsanto is currently undergoing (as of 07/06) litigation for PCB dumping, and is not altogether well known for its sterling ethical record.