I live in Loveland and drive Trail Ridge frequently on weekends to get to Grand County. Its not faster to Fraser but I think its probably faster to Granby and parts beyond...besides, slowly cruising through the high country instead of doing stop and go at the twin tunnels seems much nicer to me.
If you know the road, its usually very easy to scare the bejesus out of the tourons cruising at 10 MPH above treeline. Just pull up on their bumper and keep pulling out like you want to pass. (This works even better if there is a 500 ft drop off next to the road). They will usually freak and pull over or practically stop so you can zoom by.
Now I don't advise it, but there is also some very good visibility up there so you just might consider passing the motor homes on the double yellow. This even more fun. The only probllem is you can't turn to look at their faces because you need to drive. This is a good job for your navigator.