In respect of paddling, swimmer refers to to a former paddler. That is someone who's fallen out of their boat.