The problem, really (apart from some early symptoms of MPAS), is definitions. Just what are you calling infinity, and what do you mean when you say number?
The thing is, "number" is such a basic concept to us, we forget it's actually an abstract creation of our minds. As such, we should probably try to pin down what we mean, when we discuss numbers. In a way, this is what mathematics is all about: inventing an abstract construct which seems to catpture some elementary pattern, defining it, and then studying how that construct behaves. Often, when we say "number", what we really mean is natural number, or rational number, or real number; sometimes even complex number. These are well-defined algebraic structures. Infinity is not a number in any of these senses; any element of these structures is evidently not "infinity". (Read the definitions and convice yourself, if you doubt this.)
So that settles it, right? Well, not quite. Mathematicians and physicists sometimes "add infinity" to their numbers (reals, or complex numbers, or whatever), depending on what they're going to do with them. There's nothing inherently wrong with this, so long as your definitions work... You don't "prove" that 1/0 is infinity; you can define it that way, and see if you get something workable. As it happens you don't! But then, mathematicians never were bothered with dividing by zero -- it's not meant to be defined. But now, start with the natural numbers, and add a new number X, and define that X+n=X (for any n). You might as well get used to calling it "infinity". Now see where that gets you...
Infinity is hardly something you run into in everyday life. The patterns abstracted by mathematicians to form their fields of study are, by and large, finite. But they might naturally lead to the study of infinity (e.g. the sets of natural and real numbers are clearly infinite); conversely, considering "infinity" as a "number" or "point" is often a mathematician's ploy to return to the realms of "finite" intuition, as exemplified by the Riemann sphere mapping.
More generally, the infinite is tackled by mathematicians in several ways. Try the concepts of series, limit, infinitesimal, and transfinites for starters. It's certainly there, in one guise or another...
Then, must we conclude that infinity is a number? It is, if you say so.