The previous writeups here discuss the difference between ambient music and complex works necessitating full attention. In that respect, I'm going to be going off on a little tangent here.
There's a lot of fun to be had in listening to music. I don't disagree. It's art, and I've had great emotions pricked in me often by what's stimulating my stirrups.
That said, there's a lot of enjoyment and fun in music beyond sitting in front of the stereo and letting it bowl you over. It's part of our shared culture, and in addition to the hours racked up listening, there's those times where you play it as a sonic backdrop while you work or to lure others into social interactions, lend it out, spread it around, pick up on references, try to figure out interconnections ("hey, skip back a few seconds...I'd swear I just heard Emerson play something like the Star Wars theme...no, really..."), categorise people to put them in boxes, and best of all, argue about it.
I think that's the greatest thing about music. Like literature or film, it's not just about how it affects you, but others. You can use it as a handy Rorscach test to look into others. And you can spend days informally discussing it with friends, or anyone who'll listen. It's handy like that. A lot of people invest their egos in the music they listen to, making it all the more fun to discuss it and get heated up. There's no better feeling than justifying the music you listen to. On the flip side, it's wonderful to be able to hold your music out as a fuck off signal.
I think music is for sharing.