There's a middle road that you might explore. Get to know why he has been an obstacle from his own point of view. Then judge whether he's open to discussion as part of the work. Discussions over the issues might open up practical solutions to some of the conflict that has occurred, could create a shift in his attititudes by virtue of the relationship you develop with him, etc. Then you can assess on-going your comfort from an integrity perspective, your comfort in the day to day working environment, etc. Just don't get locked into the positions he and boater's he's clashed with have jumped into. If you feel too strongly that he's impossible, unreasonable, or whatever, then quit.