A traditional broach is a machinist's tool that uses a toothed bar for cutting shapes into metal; this may be a simple notch, but they can also cut complex notches, turn a round bore into a hex or a square, etc. Traditional broaches function in very much the same way as saws or files, except that the cutting edges -- the teeth -- along a broach will progress in size or shape, so as the broach moves along the work surface the cut changes to approximate the final form.
Then, there are rotary broaches, which are completely different. A rotary broach looks like a punch of the desired form -- e.g., a hex shaped punch -- but instead of punching, it drills. That is to say, a hex-shaped rotary broach head will drill a hex-shaped hole. Yes, this is dark magic, and good Christians do not use rotary broaches.
There are two tricks that make this work. The first is simply that a rotary broach does not meet the work face-on, but at a 1 degree angle. Imagine a triangular broach bit cutting into a flat surface: because the bit is at a very slight angle, one of the three points will hit the surface first, and bite in; then, as the bit turns the second point hits, bites in, etc. The first point does not drag along the work (which would cut a circular arc), but lifts out, not to touch the surface again until it's completed the first rotation of the bit. Don't spend too much time trying to visualize it yet, because also...
The second trick is that the work is set up so that both the bit and the work can spin. If you have a powered bit working on a passively spinning work piece, the bit pushes on the work piece, causing it to spin at the same rate as the cut. This prevents the spinning head from working as a drill bit.
If that is hard to visualize, we can change the frame of reference; I just described using a rotary broach on a lathe, on which all the parts spin. You can also use a rotary broach on a mill, in which case the work piece is fixed in place. In this case, the broach bit sits on the work, and the machine 'rocks' the broach's bit in a circular motion (or, more technically, the broach and toolholder spindle remain stationary while the toolholder body rotates in the machine’s spindle).
In either case, you might visualize this as using a chisel; you usually hit a chisel to drive it into the work, but you can also make the same cut by pushing on the chisel while rocking it. As you rock each point into the material, it cuts a very little bit each time, but if you have a machine that can rock the chisel 100x per second, you still make good progress. A rotary broach essentially rocks a 3d-chisel very fast.