I go by two rules of thumb:
1. If two boats are going downstream, the downstream boat has the right of way.
2. If one boat is going downstream and another is in an eddy or surfing a wave, the boat moving downstream has the right of way.
So if I'm going faster than someone in front of me, I yield to them until it's safe to pass. If I'm surfing a wave and see someone coming downstream, I jump out of the wave and let them pass. If I'm in an eddy and there is downstream traffic, I wait until there is a break big enough before leaving the eddy.