We approach the task of network congestion control in datacenters using Reinforcement Learning (RL). Successful congestion control algorithms can dramatically improve latency and overall network throughput. Until today, no such learning-based algorithms have shown practical potential in this domain. Evidently, the most popular recent deployments rely on rule-based heuristics that are tested on a predetermined set of benchmarks. Consequently, these heuristics do not generalize well to newly-seen scenarios. Contrarily, we devise an RL-based algorithm with the aim of generalizing to different configurations of real-world datacenter networks. We overcome challenges such as partial-observability, non-stationarity, and multi-objectiveness. We further propose a policy gradient algorithm that leverages the analytical structure of the reward function to approximate its derivative and improve stability. We show that these challenges prevent standard RL algorithms from operating within this domain. Our experiments, conducted on a realistic simulator that emulates communication networks' behavior, show that our method exhibits improved performance concurrently on the multiple considered metrics compared to the popular algorithms deployed today in real datacenters. Our algorithm is being productized to replace heuristics in some of the largest datacenters in the world.
|Title of host publication||IAAI-22, EAAI-22, AAAI-22 Special Programs and Special Track, Student Papers and Demonstrations|
|Publisher||Association for the Advancement of Artificial Intelligence|
|Number of pages||7|
|ISBN (Electronic)||1577358767, 9781577358763|
|State||Published - 30 Jun 2022|
|Event||36th AAAI Conference on Artificial Intelligence, AAAI 2022 - Virtual, Online|
Duration: 22 Feb 2022 → 1 Mar 2022
|Name||Proceedings of the 36th AAAI Conference on Artificial Intelligence, AAAI 2022|
|Conference||36th AAAI Conference on Artificial Intelligence, AAAI 2022|
|Period||22/02/22 → 1/03/22|
Bibliographical noteFunding Information:
Work done during an internship at Nvidia Research.
Copyright © 2022, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved.