OR: Guaranteed way to win $1 by betting on a sequence of coin tosses

A somewhat strange setup, based on betting "double or nothing" on a fair coin with a martingale. While not a paradox in the strict sense of the word, the name was given in the 19th century, and has since stuck.

This is the precise setup: you may wager any sum of money against the bank. The bank then flips a fair coin. If it comes up heads, you lose your stake; otherwise, you win the amount of your stake (i.e. if you bet $3 and the coin came up tails, you'll have $3 more than when you started). Under these conditions, a win of $1 (or any other sum) is guaranteed by the following:

  1. Stake $1. If you win, you've won $1 -- leave the game.
  2. So now you're $1 down. Stake $2! If you win, your net gain is $2-$1=$1, so you've won $1 -- leave the game.
  3. So now you're $3 down. Double your stake again, to $4! If you win, your net gain is $4-$3=$1, so you've won $1 -- leave the game.
    ...

If you reach step k>1, you've lost $2k-1-1, and you wager $2k-1. So the process may continue. Note, moreover, that the only way of the process not terminating is if all the coin tosses are heads -- something which occurs with probability 0.

So you win (almost surely -- this means roughly "always") $1 every time you do this!

Clearly, something here should be wrong. But a great deal of probability theory had to be worked out to straighten things out!