One theorised origin of the phrase, to "get right down to the nitty gritty" comes from African-American slave slang. The "nitty gritty" is the bottom of a woman's vagina, so to get right down to it is, in fact, blatantly sexual.
Another theory, which may not be mutually exclusive with the above, says that the nitty-gritty referred to the contents of the hold of the slave-trading ships, and a trip to the hold was to go 'down to the nitty-gritty'.
Clearly, the modern meaning of the phrase has lost almost all of those connotations, and is not associated with contemporary Standard African-American English. However, British police officers have been forbidden from using the word because of possible racist implications. This came to light when, on May 14, 2002, the UK's Home Office minister John Denham used the phrase in a speech to a Police Federation conference, and was told that he shouldn't have.