I also recommend taking a whitewater rescue course. My husband is not the most comfortable swimmer, and we are taking a class in 2 weeks. I am hopeful that it will give him some confidence, that if something ever does happen, he'll know what to do. Not just know it in his mind, but in his body, having performed many of the "rescues" before. I believe we also practice some defensive swimming, which would probably also give you some confidence that you've practiced.
It's totally natural to feel afraid of what you aren't comfortable with. Just like in boating, one way to get past it is to get some training so you have help stepping through what you need to learn.
good luck, it's fun going downriver!
I am a river, babe - I've got plenty of time, I don't know where I'm going, I'm just following the lines..... - "We are water" by Shaye