LSD can permanently mess up your visual memory
According to James B. Appel, Professor of Behavioral Pharmacology at USC, LSD causes permanent changes to visual perception in lab rats, and I assume in humans. However, my assumption might be confounded by effective dosage and/or compensatory strategies. Unfortunately, there isn't much scientific research that looks at LSD's pharmacology in humans.
Due to personal experience with LSD and based on my training in cognitive psychology, I hypothesize that LSD disrupts iconic memory, i.e. the visual mode of sensory memory, because, in the widely accepted information-processing system models of cognition, sensory memory is the next stage of processing after sensory input. Iconic memory retains information for between 0.5 and 1.0 second, whereas echoic memory, the auditory mode sensory memory, lasts between 3.0 and 4.0 seconds, just long enough for working memory to grab hold of salient information. In particular, working memory recognizes patterns and feeds distilled information to long term memory where comprehension processes take over (i.e. the lexicon in the case of reading). Regardless of whether attention is the result of bottom-up or top-down phenomena (or both), sensory memory is a constantly renewed copy of low level perception.
Photographic memory occurs when iconic memory is precisely tuned and orderly to the extent that incoming visual stimuli do not rapidly decay. As to what this mechanism must be like, it does not employ afterimages or normal mental imagery. The human visual system is "hard wired" at infancy, and there is no hope for the full recovery of visual perception, as stated earlier. Also, considering that occular dominance columns must develop before a critical age, it is clear that the foundation the visual system does not deal very well with insult. Concerning the gradual extinction of visual tracers, this is probably due to attentional compensation provided through working memory, and is not the result of an increase in available visual information.
After using LSD, the representation of a visual stimulus in iconic memory is no longer governed by precise neuronal firing patterns, but is rather subject to whatever alterations has LSD wrought. LSD is not neurotoxic, yet it does alter synaptic configurations. In the early visual pathway, these changes are not readily reversed. Thus, as goes the early visual pathway, so goes the quality of information in iconic memory. Just as a photograph's detail is limited by the lens, so to is the detail of a memory limited by perceptual fluency. In conclusion, you can kiss your perfect photographic memory good bye forever the first time you take LSD!