I'm really good at swimming - ask my choad partners.
The paddle-swimming idea really does work, especially in "oh shit" situations when you need to get to shore fast.
Boats are a pain in the ass and easier to recover than paddles. You really shouldn't be on the water if your buddies can't get your boat.
I say try to manage both if you can, but getting to shore is a chore (rhymes, doesn't it?). If things get complicated, ditch the boat, and paddle-swim to shore, then run downstream so your buddies don't have to wrangle your boat too long. You can catch your breath later.
Oh yeah - if you exercise any unintended water exits, you should definitely chug a beer from your booty or kharma will catch up with you. . .