throw in the towel

To give up. To surrender. Comes from the sport of boxing. To end a match, a manager can throw a towel into the ring indicating surrender much like a white flag.

In many areas a manager does not have the ability to throw in the towel. This is a rule that varies by location in boxing. Sometimes only the referee can stop a bout. In modern Boxing matches the refs will stop the fight way before any manager would be throwing towels around.