Why would Ken Norton look down when both boxer's have to stare each other down before a fight, I don't think he was scared. But why, it shows a kind of weakness if you can't look the other man in the eye.
If he habitually did it, he probably didn't care for this prefight ritual. It can be stressful to have staring contests even if you're the winner Hoya would also avoid the staredown, and he has a great resume.