Oh I don't think the brothers were scared of someone as rudimentary as Valuev - though he does pose some problems with his height, jab and weight. Frankly it would have made for a stinker of a fight. They just weren't interested because they wouldn't have gotten all the things they like in negotiations. Sauerland and King would have been too difficult to out-manoeuvre on everything.
Valuev was champion before then. Chagaev had only been champion in recess since 2008 when he had to pull out of a fight with Valuev. Valuev then won the vacant title and had to fight Chagaev but since he failed the medical tests he lost the right to challenge Valuev. Then he fought Wlad.