This expression has Shakespearean origins. In the tragic play, Julius Caesar this expression occurs during a conversation between Casca and Cassius, two conspirators againt Julius Caesar. The pun comes from the fact that a Roman senator, Cicero is speaking Greek and his fellows don't understand him. The context is as follows:
Cassius: Did Cicero say anything?
Casca: Ay, he spoke Greek.
Cassius: To what effect?
Casca: Nay, an I tell you that, I'll ne'er look you i' the face again: but those that understood him smiled at one another, and shook their heads; but, for mine own part, it was Greek to me. I could tell you more news too: Marullus and Flavias, for pulling scarfs off Caesar's images, are put to silence. Fare you well. There was more foolery, if I could remember it.