It is impossible to adequately secure a machine against an attacker who has physical access to it; there are too many ways the attacker can compromise it.
The bastion host provides much of the actual functionality of your Internet connection, and if it is lost, damaged, or stolen, your site may effectively be disconnected. You will certainly lose access to at least some services.
 Practical UNIX Security by Simson Garfinkel and Gene Spafford (O'Reilly & Associates, second edition, 1996) contains an excellent and extensive discussion of physical security.
Never underestimate the power of human stupidity. Even if you don't believe that it's worth anyone's time and trouble to get physical access to the machine in order to break into it, secure it to prevent well-meaning people within your organization from inadvertently making it insecure or nonfunctional.
Your bastion host should be in a locked room, with adequate air conditioning and ventilation. If you provide uninterruptible power for your Internet connection, be sure to provide it for the bastion host as well.