No, Doom does not use 3d matrix maths for rotations and the like. This is because it is not really 3d. It uses a technique called raycasting. Raycasting works by traversing a tree of all the walls in the level for each column of pixels, and drawing the floor leading up to that wall, and then the wall itself, and finally the ceiling. None of this involves 3d matrix mathematics.