okay, I'm still clocking in under my first year of paddling too, so one newby to another, I guess....
We're short on paddle shops where I live (the southeast), but we do have an REI-- most of the crap REI sells is just that, crap for the whitewater enthusiast, great for flatwater paddlers. You can get your neoprene there and whatnot, but the vests and paddles they sell are generally the touring variety (I don't know what it is with REI and this whitewater discrimination crap!

). That said, any of the IR splashtops can go either recreational or whitewater, but i'm guessing you probably already had that part figured out.

I get most of my gear from shops out in CO, or Rock/ Creek Outfitters out of Chattanooga. Also, I believe Colorado Kayak Supply has an eBay store, so you can at least get some cut rates by checking there. Search for the store name, though, not the equipment. It may just be me, but I'm a little wary about buying gear-- especially when it comes to things like skirts and PFDs-- from perfect strangers on eBay.
You might want to check into paddle or conservation clubs in your area for swiftwater rescue classes once you've got your basic paddling skills downpat... for some folks, this is 4 months in, some others, 6 months to a year. Alot of times, either river enthusiast or conservation groups will have paddle clinics (or hell, even shops where you guys are), and many will offer a couple of swiftwater classes to intermediate and advanced paddlers several times a year. My advice would be to wait until you've taken one of these classes, or spent plenty of time with a seasoned pro (at least you have that option out there! LOL) who can show you how to throw a bag, etc., before you start investing in safety equipment. I always carry a first aid bag in my drybag, but I've also had basic responder courses and wilderness rescue/ first aid. It's always nice to have an extra swab for bee stings or patch for blisters on hand for the minor stuff, though, and I recommend everyone carry at least a small first aid kit. And the whistle is a must.
On PFDs, what you need to look for is fit and color. I totally concur with CoreyD about the color of vests above. If something happens, it's alot easier to pick out a bright yellow or red vest in the water or on the shore than it is to spot a navy blue or dark green. Also, when you're draining the boat and getting your gear together at the end of the day, you're gonna be alot less likely to leave that PFD on the rocks at the takeout if it's bright and yellow instead of black, just because it's alot easier to see. And try on several vests before you plop down the $$-- your local paddle shop can be a great resource for this, and don't be shy about asking the folks there to help you out. Stohlquists don't fit me for crap, but the Lolitas by lotus are cut perfectly for my body, chest, range of motion, etc. It's all about figuring out what works for you.
Finally, on the booties... you're gonna wanna go ahead and grab booties, not just chacos or Keens. Keens are great to have around at the end of the day, I love 'em for kicking around the put in and takeout, or back at the campsite. But the fact is, they're bulky and sticky and won't fit in the bulkhead of most playboats. And I've personally tried paddling in my Keens before, got my toes all squished up in the front of the boat, then flipped and had to swim, and thought my legs were staying in when I kicked out of my boat. I don't think the straps caught on anything, I think my toes really were just wedged, and when I had to kick out, my ankles were in an awkward position to begin with. Take it from me, though, panic & stuck feet do NOT make for happy additions to your worry list when you're trying to pick up a new sport!
And on the note of swimming-- do make sure you have a good helmet, just like everyone else has said. Corey mentioned the SWEET helmets, and they're awesome-- pricey, but awesome. You have to play with the pads a little, but once you get the right fit, it's as if the thing was custom designed just for you... and when you roll or smack your head, it absorbs the blow to the point you can't even feel an impact. Aside from your boat though, make your helmet your other "big" investment. Both you and the folks who like ya will be thankful for it later!