Ski vests don't have the bottom buckle which is very important in whitewater hydraulics. When you are pulled down, you tend to raise your hands up and most life jackets would slip off of you over your head. Whiteater PFDs all come with a bottom buckle, which you clip under your rib cage and that buckle will most likely safe your life some day.
River safety is not something you want to save money on. Invest on a most flotation PFD and you will be very happy the next time you are swimming a class 3 rapid. I'd also recommend looking for a PFD with multiple pockets, adjusting straps and a knife buckle.
Most whitewater PFDs will come in 16 lbs of flotation, this is enough for most cases, but the more flotation you have the safer you feel in the water. I got MTI Big Bouy MTI Big Buoy Whitewater PFD
for both me and my wife and totally love the jacket, 26.5 lbs of flotation is nice! I've swam plenty of class 4 rapids to know how it feels to be pulled down under.
A good PFD should be the first investment you do when you get into rafting/kayaking.