Plenty of folks do it, but as you know it's totally a personal judgment call & not without risks. We did a SJ trip in June last year with 8 kids, ages almost 2 to 6, with 12 adults. IMO the most dangerous part of the trip is on the ramps and at camps - they will probably fall asleep on the boat. There are many folks on this board that have taken kids on trips at younger ages, and other people that think it's a bad idea. So, as they say- "opinions vary".
The first question you have to ask is if you're doing it for him/her, or yourself. It will be harder than a normal trip because of sleeping, naps, and always having to have eyes directly on them for 24 hrs x 5 days. It might lead to wonderful memories for them if they have a ball (our 3 year old still talks about his trip), but you will likely hear "I wanna go home" when the're tired, hungry, etc. And your fun factor will be less than a normal trip.
So you might consider just the top, or just the bottom, rather than the whole 5 days.
There are other good threads on this topic on this board, with others' experiences, concerns and encouragement.