A few gotchas:
While compressed Source code is about the same size as object code (debian and suse manage to fit their source and binaries on the same number of CDs), but it often expands by a factor of 5 times or more when it's unpacked and compiled (MAME goes from 7Mb in a solid archive to 220Mb unpacked and compiled on my machine)

Compilation takes ages, especially on a 486. It could take weeks to install a CDs worth of source code, and some packages (X windows, Mozilla, The Kernel, to name a few) are extremely large and difficult to compile (GCC takes hours to compile, even on high-end systems). This would effectively rule out installing on systems with less than 64Mb of RAM. (And one of linux's big plus points is that it can be installed on a 486 and still be usable.

Having source code sitting on the hard disk while the system is installing itself would mean that it takes up at least twice the hard-disk space it normally would, and once it's finished, the hard-disk would end up half-full. Reading it off the CD would be better, but it would mean that the CD was whirring all night.

Looking at things like MAME, where the majority of code is spent in a loop, and you'd expect the compiler options to have a significant impact, the difference in speed between different processor versions is negligable (less than 10 percent). You'd see roughly the same performance just using binaries optimised for i586 instead of i386 (which distributions like mandrake provide).

The whole point of distributions is that they come compiled 'so you don't have to'. The kind of person that would use something like this is quite capable of getting the source and installing themselves.

A better, but more time-consuming solution to 'linux bloat' would be for the distributer to put some effort into hand-picking packages, so that it doesn't by default install 15 different shells, 20 different tetris games, 5 different window managers, etcetera... Fewer packages rather than smaller packages. Who could be arsed to choose between 25 different text editors (especially with redhat's installer, or dselect)?

It would be nice to see a linux distribution that contained only a few of each type of package, and a wide variety of types, that all fitted well together, all had the same widgets/command-line syntaxes, and could install sensible combinations of these packages with a minimum of fuss. If Microsoft can do this with windows and office, why can't a company like redhat?

ssd - Under most open-source licenses, it is sufficient to provide an offer to supply source-code en lieu of the actual source, which is how companies like red hat can release binary-only distributions.
2003 - And plus ca change. Now there is Gentoo Linux, which does this very thing. Installations can take up to a week on non-bleeding-edge hardware. But it's optimized, which makes it all the more 1337. On the flip side come heavyweights like debian woody, availible on 5 CDs of binaries, or 1 DVD (turn it over for the sourcecode). Same 5 inch disk redhat 5.2 took up, but now there's 5 times more stuff.