I think it needs to be mentioned here that we're talking about
humans. Remember, we're not the only
species on the planet? Birds, for example, have their Y chromosome in the
female of the species. Some animals have their male as a
haploid organism, and the female is
diploid. In most
angiosperms, gender differentiation doesn't even occur until the
flower is formed, and even then, it's only a matter of turning on or off a few
genes. In
yeast, gender is determined by
one gene located on chromosome three; there are also spare copies of both gender type genes that are silenced, but can be activated by a recombination event. (So yes, you can change the gender of your yeast.)
Naturally, though, since most genetic studies have gone on in organisms in which the norm is XX-females and XY-males (mainly humans and fruit flies), we tend to think of the Y chromosome as the male chromosome.