I have kayaked that section (the section below Ouray and above KOA) I think you could DO a raft down that stretch but if would have to be one of those specialized mini-rafts from the east coast with the foot straps in it...
My concern with a raft would be lack of ability to avoid MOST of the hazards. AND if anyone was in the water there is tons of debris from the mining days (the danger to a raft is greater than the danger to a creekboat for example). If the river didn't hurt the swimmer the shredded metal in the river might.
I don't recommend this section for a raft but I am just a kayaker who guided for a while.