Fighters aren't afraid of one another. They genuinely believe they can beat anyone.
The person who stopped the fight from happening was Bob Arum. Once one issue was resolved, immediately another one would rise, and every time the common factor was Bob Arum.
The drug testing was a ridiculous hang up to begin with, it wasn't a big deal until Pac's team made it one, but then later on decided that it wasn't a big deal.
Last year's debacle was just a flatout disgrace from Arum. Floyd calls out Pacquiao demanding more than a 50/50 split. Pacquiao openly agreed to it. When it came time to make it official, Pac refused anything less than 50/50. Arum, who already said Pac was going to fight May 5, the same day Mayweather called for, then suddenly pushed Pac's fight day back a month (to when Floyd was in jail) when he saw Floyd was very serious about making the fight happen.
He also wanted to go and build some massive stadium in a short amount of time, stating that's why they couldn't fight on May 5, which was even more ridiculous seeing how they could make just as much money at the MGM.
I could be wrong, but I believe common oponents between Maywather and Pacquiao have all said that Mayweather is actually much easier to negotiate with and more reasonable than Arum and Pac are.