There is no right of way in the sense that we usually observe it (upstream boater has right of way, larger craft have right of way over smaller, etc.)
The best thing to do with fishermen is to not spook the fish - if you do, they will be gone for a good long time. I usually let my boat just drift and stop paddling till I'm well past them. I've heard it both ways, but supposedly it's actually better to pass close to them to avoid messing up their chance of a catch. Someone who fishes should comment on that...
I rarely speak to fishermen. Most seem to value solitude in the wild - something most of us also enjoy, and I try to respect that with a simple nod. I don't get what they do, they don't get what I do, but we both love the river, so I just try to keep it chill.
Join up, suckas.
"People demand freedom of speech to make up for the freedom of thought which they avoid."
- Soren Kierkegaard