I think there are two reasons that many people cannot enjoy
Science Fiction There is some badly written SF featuring
misogyny, wooden characters, stilted plot and an excess of
technobabble. Not all SF is good, but that's true of any genre.
SF grew up in the thirties and forties, and came of age in the era of
pulp magazines. These books were designed to appeal to
adolescent boys and as such featured implied sex, lurid covers, and fantastically virile
heroes. Adventure stories and space battles were common, some fantastically unrealistic.
Because this audience was not at all selective in their purchases, some very poor stories were bought and published. Many noted science fiction writers actually learned to write while selling stories. Robert Heinlein Ray Bradbury and Robert E. Howard were writers who came of age in the pulp era. E.E. "Doc" Smith's Lensman series was cited by George Lucas as the inspiration behind Star Wars. The Lensman books are a good example of the space opera a genre term for stories involving space battles and adventure heroes. They have a certain attraction, but Smith's fascination with bone structure and how a man need's beefsteak make the work easy to riducule.
The quality of writing and stories began to sharply improve during the forties and fifties, primarily due to efforts of editors John W. Campbell and Hugo Gernsback, both of whom gave their names to major awards for quality work in speculative fiction. Nor was the writing uniformly bad. Writers like Hal Clement brought a rigorous scientific accuracy to the genre. Poul Anderson, Alice B. Sheldon {writing as James Tiptree jr.} and Alfred Bester all were fine writers from SF's golden era.
This impression is often reinforced by media science fiction. Some media SF has been excellent. Harlan Ellison wrote "City on the Edge of Forever" for the original Star Trek series, and the episode won an Emmy. Babylon 5 showed real brilliance and took home several more Emmys, as have individual episodes of other series, particularly the early seasons of Star Trek: The Next Generation. The Stanley Kubrick film 2001: A Space Odyssey captured many imaginations, and in later years films like Blade Runner, and GaTtaca explored themes of real importance. Forbidden Planet and The Day the Earth Stood Still continue to stand out as fine, relevant films many years after their release. The new Battlestar Galactica might be the finest piece of drama ever put on television, and won a Peabody award for its exploration of what it means to be human.
However, much media SF can readily be categorized as drek. Leaving aside the occasionally hilarious misadventures of Ed Wood, much media SF has simply offered monster and cowboy movies with aliens filling in for ghosts and Indians. The excruciating Event Horizon highlights another cliche of bad science fiction, the universe without rules. As SF is sometimes seen as 'kid stuff", it is often seen as acceptable to ignore such niceties as a coherent plots and believable characterization.
The second reason is what I see as a failure of imagination. In reading speculative fiction there is a contract between reader and writer. In the first few pages of a story the author is expected to provide the reader a basic understanding of how the universe operates in this story. The author may refine these 'rules, but may not violate them. The reader in term "orients" themselves to the universe of the story. They literally adapt themselves to this new set of rules.
Children do this naturally. After all, this entire world is new to them. Why not another? But some adults, long trained to adapt to the nuances of our world, can no longer make that imaginative leap. They dismiss the world as 'unrealistic'. In fact, society often supports limiting your imagination. Fantasy stories are a staple of early childhood, yet often discouraged after puberty. We are trained to adapt to reality, and so many lose the capacity to put that aside.
Mainstream critics and literary marketing further this stereotype. Many seem to assume that any story without combat, spaceships and lots of bug eyed aliens cannot be SF. I recently heard one critic describe Connie Willis as 'sort of a science fiction writer'. Margaret Atwood's The Handmaid's Tale is clearly science fiction, but more often pitched as a feminist tale in order to reach a mainstream audience. The producers of Tom Cruise's film Vanilla Sky never once mentioned it is really an SF movie. An entire SF subgenre, the technological disaster story, whose leading proponent is Michael Chrichton, is very popular among the technologically leery. Many would deny outright that the stories are at their core science fiction.
Science fiction is not about bad writing and wooden plots. Those were largely banished from written speculative fiction when the last pulp magazine failed. It is about ideas and people in fascinating circumstances. Because of the intellectual distance provided by the alternative worlds, it can teach us things about our world in ways other literatures cannot. But many people see only the bad work out there, or cannot make the imaginative leaps required.