Keeping Calm

This is exactly my point. Defense isn't merely about defending against soft, weak punches. It also means defending against hard "take your head off" shots. That's the distinction here. And i'm not talking about simple blocking and slipping, but also movement/footwork to make him miss and think twice about wasting energy and being countered.
If you're fighting scared and firing back against a bigger, more experienced guy - guess who's gonna get the short end of the stick?

So back to my original point...calmness comes with confidence in your defense.

 

I've had my share of trouble with taller guys.

But to give you an answer to your problem about staying calm.

Answer yourself this.

What are your own strengths?
Follow them and keep on training hard.

Having a bad day with a tall fatass is nothing, you have to realize that you'll lose and you'll win sometimes, depending on how you come back that's what separates you from the rest.