I think the sum of the group will have a great impact on wether you will have success in taking him or not. If you have a group that otherwise is known to be self sufficient, and you have a good relationship with your brother, I'd invite him.
If you need every person in their group to pull their own weight, and your relationship is strained, I probably wouldn't. It could jeopardize the relationships you have built with the rest of your group, if they end up viewing him as a liability.
You have to remember, bad things can happen on any trip, and he needs to be able to respond and help. We had two broaches with our gear barge and witnessed two medivac rescues (both from guided trips) and the private trip that launched after us had a multitude of flips with the smallest raft in their group.
I've heard of some married couples getting so mad at one another that one of them almost walked out at Phantom ranch. I'm not a brother, but I could see a sketchy relationship with a family member get tested also.
Those are things to think about. Honestly, it's a once in a lifetime trip for most of us, so you would be offering him a gift that he wouldn't be able to fully appreciate until your feet hit the ground at the end. I went when I was 16, and I am about to turn 35. I am just starting to think applying for myself. (My Dad applies for the middle fork every year, and I get nervous and excited when the drawing comes up. No luck yet!)
It is a once in a lifetime trip, and stories will be made either way.