I wear them as well. I only lost one contact in about 6 years. The biggest issue other folks with contacts complain about is when the water get's real splashy the contacts tend to float around in your eye. Usually in the middle of a rapid, blink hard a few times and it helps. Don't open your eyes underwater because the river grit tends to get between the lens and your eye, you don't want to see the rock your gonna hit anyways.
. Always bring some extras in your vest or drybag just in case. A friend got some oakley wire frames with lenses that darken in the sun. It helped him a bunch. I don't like sungalsses becasue then tend to fog, but was recommended by my eye doctor.