Two elements of a group, a and b, commute if ab = ba. It is common in the study of linear algebra to consider whether two matrices commute. The group product in this case is just matrix multiplication.
In quantum mechanics, if two matrices that represent physical observables commute, then the observables can have simultaneous precise values. As an example, the operators representing energy and linear momentum of a free particle commute, so it is possible to know a free particle's energy and linear momentum simultaneously (neglecting relativistic effects, E = p2/2m). On the other hand, the operators representing linear momentum and position of a particle never commute, implying that a particle's position and velocity cannot be known simultaneously (see Heisenberg Uncertainty Principle).
The commutator of a and b, [a, b], is defined as ab - ba. If a and b commute, [a, b] = 0.