Apparently the term originates from the French "tour d'ivoire" coined by the French literary critic Charles Augustin Sainte-Beuve in 1837 to characterise the poet Alfred Victor de Vigny. Apparently Vigny led a very isolated life, which explains part of the isolationistic connections to the phrase "ivory tower". The term entered the English language towards the end of the 19th century.
Interesting comment: There seems to have been some initial misunderstanging for "tower" as the word as it is used in French at the time had certain phallic and sexual connotations, whereas the white of the ivory connotes purity. It has been speculated that this contradiction had a lot to do with the popularity of the term.
Herman Hesse's book "The Glass Bead Game" [1943] provides a brilliant commentary on the Ivory Tower concept.
References:
Encyclopedia of Word and Phrase Origins [1987]
Brewer's Dictionary of Phrase and Fable [1981]