Stealth DoS Attacks on Secure Channels

We initiate study of the use of ‘secure tunnel’ protocols, specifically IPsec, and its availability and performance guarantees to higher-layer protocols, in particular TCP, against Denial/Degradation of Service (DoS) attacks. IPsec is designed to provide privacy and authentication against MITM attackers, and employs an anti-replay mechanism to ensure performance. For our analysis, we define a new family of adversaries, the stealth denial and degradation of service (DoS) adversaries. These adversaries are weaker than the classical MITM adversary, and may be of interest in other works. We analyse their ability to launch (DoS) attacks on secure channels, and show realistic amplification attacks, disrupting TCP communication over secure VPNs using IPsec. In particular, we show that anti-replay mechanism is critical for performance by launching a DoS attack on communication over IPsec without anti-replay window. We present attacks exploiting insufficient IPsec anti-replay window size, and show how to calculate correct window size. Finally we present attacks on IPsec with correctly adjusted anti-replay window size thus showing that even large anti-replay window does not ensure performance to TCP flows. We then suggest a fix to TCP in IPsec gateway designed to prevent the above attacks, and to provide secure channel immune to degradation and other DoS attacks. Our solution involves changes (only) to the sending gateway machines running IPsec. In addition to their practical importance, our results also raise the challenge of formally defining secure channels immune to DoS and degradation attacks, and providing provably-secure implementations.