Uberfetus:
The problem you are running into is that is not a very meaningful thing to say "so-and-so is a conservative." This is for two reasons. The first one is that it is far too general. Pat Buchanan is a conservative. Charles Murray is a conservative. Bill O'Reilly is a conservative. You are a conservative. What do these folks have in common? Not much. If you wanted to be more specific, you could say "I am a minarchist," or "I am a anti-federalist," or whatnot.
The second problem is the verb "to be" is insufficient for discourse. (see E-Prime) Instead of trying to say what you are, why not say what you believe? Instead of "I am a libertarian," why not "I believe that the government should play a smaller role in people's lives than it currently does." The latter is a statement of fact, the former is just a label.
Labels are nice for simplifying, but sometimes they can be taken too far. Certainly most people's minds can grasp that political beliefs are more complex than "liberal" vs. "conservative." Why then, should our politics turn into some sort of strange football game between the Democrats and the Republicans?
But I digress. The question now at hand should be: why do you feel compelled to label yourself as a conservative? You say that conservative means only someone who wishes to shrink the size of the government, but that's simply not true. In common parlance, conservative usually refers to someone who dislikes the federal government, but has additional connotations of one who is in favor of the social status quo, and the strengthening of "traditional values". Indeed the federalist position might simply be seen as an extension of a love of tradition, and a desire to emulate the philosophies of certain founding fathers.
Since the word itself refers to being resistant to change, or risk-averse, I see it as more likely that you are in fact not a conservative by the common definition, than that people are wrong when they sense a disparity between their perceptions of you and your self-proclaimed political affiliation. If it were up to me, I'd call you a libertarian, but like I said, I'm no big fan of labels.
I can understand why you might prefer the "conservative" label. After all, many people have a lot of positive connotations with that word. Perhaps you were even brought up to believe that you were a conservative, and the conservatism = goodness, so now you label yourself that way regardless of what you believe. Unfortunately, you must accept the negative connotations of the word along with the positive ones. So if it truly bothers you, the things people associate with conservatism, perhaps you should find a better way to express yourself than a one-word team-designation.
And now, an obligatory on-topic paragraph. Are conservatives mean-sprited? Well, some are and some aren't I suppose. I could argue that their philosophy makes them more likely to be, but so what? We could volley "conservatives are mean," and "liberals are dumb" back and forth all day, but haven't we had enough flag-waving yet?