The branch of
algebra that concerns itself with
groups (see definition given there).
That is, we have an operation with two arguments that is associative, has an identity element, and an inverse for every member.
Group theory is only interested in how the operations combine. It doesn't care what the set and the operations stand for. The main exception are permutation groups. They are of particular interest because every group
is isomorphic to a permutation group.
It turns out to be extremely difficult to map out the world of groups, even in the finite case. Group theory distinguishes all kinds of relations between groups (one group being a subgroup of another, one group being a normal divisor of another, etc.), properties related to group-to-group mappings (group homomorphisms), and so forth. Specific classes of groups have their own theories, with specific concepts and theorems.
Group theory is extremely abstract and often mentioned as a paradigmatic example of pure mathematics. The discovery of a practical application for Lie groups, one of those specific classes developed in theory, is often cited in defense of the need for society to support pure science.