The hand gesture that went along with the phrase was something like removing a monocle. The left thumb and forefinger are held together, making a loop. This is put up to the left eye, with the tip of the thumb and forefinger against the nose. If it is done correctly, it is possible to see through the loop of the thumb and forefinger. Then move the hand down in a dismissive motion, until it is at the level of the waist and the arm is almost straight. It should be slightly to the left of the body.

It's really hard to describe such a simple motion.

glassonion says: "The hand gesture forms a "6" which is interesting". It's interesting, of course, since the main character of The Prisoner is number 6. Well, he's not a number, he's a free man...