The Black box argument is probably the best rebuttal to the "Turing Test" which basically states that since consciousness is subjective, the only way we can know if a computer is intelligent is to ask it questions. If its answers can't be distinguished from those of a human, the computer has to be considered capable of thought, and by extention sentient.
The philosopher John Searle's counter-argument ran like this: Suppose I'm locked in a black box with two slots in it marked "Input" and "Output." Pieces of paper with black squiggles on them are periodically shoved through the Input slot. My job is to look up the squiggles in a rule book I've been given and shove pieces of paper marked with other black squiggles through the Output slot as the rule book directs.
Unbeknownst to me, the black squiggles are Chinese characters. Outside the black box, scientists have been inputting questions in Chinese, and I've been sending back Chinese responses. My answers have convinced the scientists that the black box understands Chinese. But I don't understand Chinese at all! So how can a computer, which operates in the same way, be said to understand Chinese--by extension, to think?
Proponents of the Turing Test replied that although the person in the box didn't understand Chinese, but the system as a whole (you + the rule book + the box) does. Nonsense, replied Searle. Suppose I memorize the rule book and dispense with the black box. Now I constitute the whole of the system. People hand me symbols; I respond with other symbols based on the rules. I appear to understand Chinese, but I don't. I merely display a facility in Chinese syntax. Chinese semantics, the essence of thought, eludes me. Just so with computers.
the conclusion of the black box argument States that although artificial intelligence may be possible, it can't arise from computers as they currently understood. There's strong evidence that computers and the brain are fundamentally different. Nobody really knows how consciousness arises, but it seems evident there's more to it than just computer programs.