My knees are fine but I've dealt with hesitation from shoulder injuries. I took it slow, got my confidence back by paddling with a good crew, and taking it slow. It took a LOT of patience for my limitations. (A good crew is one that has each other's back, doesn't get pissed and frustrated when anyone swims, considers the safety of everyone, but also one that doesn't let you punk out on things you KNOW you are capable but are unreasonably afraid.) I think the easier you are on yourself now the better off you will be in the long run. It took about a year for me to trust my shoulder and myself.
My suggestion is to keep in mind that paddling is a personal journey, and it's supposed to be about HAVING FUN. I agree with what was said about not worrying about your ego. Maybe you were class 5 at one point but accepting current limitations is part of the game. Also protecting your knee with padding probably couldn't hurt to try and see if it does anything for your head. If you're bummin about and looking for a good read check out A Rock Warrior's Way by Arno Illgner. It's geared towards climbing but the mental training it outlines is applicable to all walks of life. Dealing with fear is something I think we all go through, and yes, I agree that it is irrational. A healthy amount of fear is necessary to not recklessly go balls to the wall, but don't make decisions based on fear, make them based on facts, and accept where you are in your paddling journey. Confidence is not cockiness. And don't forget to have fun and don't worry about what everyone else thinks of you. High school is over (and it shouldn't have mattered then either