There is no question of "is" versus "isn't". It's all a question of definitions. If infinite sets are part of your mathematical world, and your preferred notion of number is the cardinality of sets, then indeed infinity is a number. If you're giving names to points on a line, or extending the natural numbers to include solutions to algebraic equations, then infinity is not a number.
For more on why mathematicians get to choose on this issue, see under infinity is not a number.