If you are wearing Motocross gear you need to start from the beginning again! False security are not safe.
To each their own. I found I never needed full-face or elbow pads, but when it came to actually swimming, it could have helped keep my my legs and stuff a bit more functional if I had more leg protection. I never seem to hit my elbows or face on anything, regardless of where I'm upside down though. Swims through blast-rock hell are so rare for me that I'm not really as methodical and skilled at it as I could be; so I do kind of need the protective gear to compensate (not just because I don't swim, but admittedly, because I don't get nearly enough miles on Clear Creek or anywhere). If I do need to jump in their and swim, or happen to be the guy swimming when he shouldn't be (which I never plan to be again, ever), I don't want to have to worry about protecting my knees shins, when they're busy protecting the rest of me. I don't want to worry about my knees and shins when I'm kicking either.
Chances are, I won't have a swim like that again once I get the knee/shin guards, but I'd like to keep myself as competent as possible, when the time comes. Kind of like how all you really only need is a dry-top in cold water, provided you don't swim or have to get in the water and rescue a pinned boater.