According to the
A.Word.A.Day mailing list, a "
brown study" is a state of absorption in
thought, from the idea of brown being
gloomy. The
Oxford English Dictionary online adds "especially an
idle or purposeless
reverie" and records the phrase as far back as
1532.
The word "study" has an old meaning of anxious thought or abstraction, but its use in this phrase makes me think of the artistic uses of the word, "an object presenting effects of colour (and the like) attractive to an artist" and "a literary work executed as an exercise or as an experiment in some particular style or mode of treatment." (OED) My interpretation is that someone in a brown study could be the subject of a painting done in shades of brown.