"By the by" is a phrase that is, as far as I know, identical in meaning to "by the way." Both of these phrases can be joined to a sentence to indicate that it is tangential to the conversation at hand. That is, the speaker is addressing an issue partially or entirely unrelated to the present subject. The use of this phrase demarks the sentence as being separate from the main thread of conversation, and therefore spares the listener the challenge of finding the relevence. All too often, though, despite the use of this phrase, the focus of conversation may shift, leaving the original conversation inelegantly discarded.
Although used primarily in informal, spoken language, it can also often be found in electronic chat and even formal written documents. In the latter case, to avoid an overly conversational tone, it is best to parenthesize the whole sentence.
The literal meaning of the "by the way" is actually obvious, but most people probably don't stop to consider it. In that phrase, "way" is used in the sense of "road." "By the way" indicates that an idea is an aside, is near but not on the "path" of the conversation. There are implied metaphors between "conversation," and the "way," and between the newly offered, somewhat relevent idea, and a token found by the side of the road.
However, a parallel interpretation of "by the by" doesn't seem to hold up. I find the use of this phrase disconcerting as a result. I'm not sure what the second "by" is supposed to mean, and it doesn't appear to even be grammatical. Perhaps "by" is an archaic synonym for "way."