Recovery of Distributed Iterative Solvers for Linear Systems Using Non-Volatile RAM

Yehonatan Fridman; Yaniv Snir; Harel Levin; Danny Hendler; Hagit Attiya; Gal Oren

doi:10.1109/FTXS56515.2022.00007

Recovery of Distributed Iterative Solvers for Linear Systems Using Non-Volatile RAM

Yehonatan Fridman, Yaniv Snir, Harel Levin, Danny Hendler, Hagit Attiya, Gal Oren

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

HPC systems are a critical resource for scientific research and advanced industries. The increased demand for computational power and memory ushers in the exascale era, in which supercomputers are designed to provide enormous computing power to meet these needs. These complex supercomputers consist of numerous compute nodes and are consequently expected to experience frequent faults and crashes. Mathematical solvers, in particular, iterative linear solvers are key building block in numerous large-scale scientific applications. Consequently, supporting the recovery of distributed solvers is necessary for scaling scientific applications to exascale platforms. Previous recovery methods for iterative solvers are based on Checkpoint-Restart (CR), which incurs high fault tolerance overhead, or intrinsic fault tolerance, which require extra computation time to converge after failures. Exact state reconstruction (ESR) was proposed as an alternative mechanism to alleviate the impact of frequent failures on long-term computations. ESR has been shown to provide exact reconstruction of the computation state while avoiding the need for costly checkpointing. However, ESR currently relies on volatile memory for fault tolerance, and must therefore maintain redundancies in the RAM of multiple nodes. This not only incurs high memory overhead but also prevents ESR from being fully resilient, that is, resilient against a full system crash. Recent supercomputer designs feature emerging non-volatile RAM (NVRAM) technology, for example, the exascale Aurora that is planned to consist of Intel Optane™ DCPMM. This paper investigates how NVRAM can be utilized to devise an enhanced ESR-based recovery mechanism that is more efficient and provides full resilience. Our mechanism, called in-NVRAM ESR, provides full resiliency while significantly reducing both the memory footprint and the time overhead in comparison with the original ESR design (in-RAM ESR). In-NVRAM ESR is based on a novel MPI One-Sided Communication (OSC) over RDMA implementation, which was optimized and applied for using NVRAM to store recovery data for iterative linear solvers.

Original language	English
Title of host publication	Proceedings of FTXS 2022
Subtitle of host publication	Workshop on Fault Tolerance for HPC at eXtreme Scale, Held in conjunction with SC 2022: The International Conference for High Performance Computing, Networking, Storage and Analysis
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	11-23
Number of pages	13
ISBN (Electronic)	9781665488471
DOIs	https://doi.org/10.1109/FTXS56515.2022.00007
State	Published - 2022
Externally published	Yes
Event	12th IEEE/ACM Workshop on Fault Tolerance for HPC at eXtreme Scale, FTXS 2022 - Dallas, United States Duration: 13 Nov 2022 → 18 Nov 2022

Publication series

Name	Proceedings of FTXS 2022: Workshop on Fault Tolerance for HPC at eXtreme Scale, Held in conjunction with SC 2022: The International Conference for High Performance Computing, Networking, Storage and Analysis

Conference

Conference	12th IEEE/ACM Workshop on Fault Tolerance for HPC at eXtreme Scale, FTXS 2022
Country/Territory	United States
City	Dallas
Period	13/11/22 → 18/11/22

Bibliographical note

Funding Information:
This work was supported by Pazy grant 226/20, ISF grant 1425/22, the Lynn and William Frankel Center for Computer Science, and Intel Corporation (oneAPI Center of Excellence program). Computational support was provided by the NegevHPC project [79]. The authors would like to thank Gabi Dadush, Israel Hen and Emil Malka for their hardware support

Funding Information:
This work was supported by Pazy grant 226/20, ISF grant 1425/22, the Lynn and William Frankel Center for Computer Science, and Intel Corporation (oneAPI Center of Excellence program). Computational support was provided by the NegevHPC project [79]. The authors would like to thank Gabi Dadush,IsraelHenandEmilMalkafortheirhardwaresupport.

Publisher Copyright:
© 2022 IEEE.

Keywords

ESR
Exascale
HPC
Intel Optane DCPMM
Iterative Solvers
MPI OSC
NVRAM
PCG
RDMA
Recovery

Access to Document

10.1109/FTXS56515.2022.00007

Cite this

Fridman, Y., Snir, Y., Levin, H., Hendler, D., Attiya, H., & Oren, G. (2022). Recovery of Distributed Iterative Solvers for Linear Systems Using Non-Volatile RAM. In Proceedings of FTXS 2022: Workshop on Fault Tolerance for HPC at eXtreme Scale, Held in conjunction with SC 2022: The International Conference for High Performance Computing, Networking, Storage and Analysis (pp. 11-23). (Proceedings of FTXS 2022: Workshop on Fault Tolerance for HPC at eXtreme Scale, Held in conjunction with SC 2022: The International Conference for High Performance Computing, Networking, Storage and Analysis). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/FTXS56515.2022.00007

Fridman, Yehonatan ; Snir, Yaniv ; Levin, Harel et al. / Recovery of Distributed Iterative Solvers for Linear Systems Using Non-Volatile RAM. Proceedings of FTXS 2022: Workshop on Fault Tolerance for HPC at eXtreme Scale, Held in conjunction with SC 2022: The International Conference for High Performance Computing, Networking, Storage and Analysis. Institute of Electrical and Electronics Engineers Inc., 2022. pp. 11-23 (Proceedings of FTXS 2022: Workshop on Fault Tolerance for HPC at eXtreme Scale, Held in conjunction with SC 2022: The International Conference for High Performance Computing, Networking, Storage and Analysis).

@inproceedings{83552c98d0d54d8495683273cbe46b87,

title = "Recovery of Distributed Iterative Solvers for Linear Systems Using Non-Volatile RAM",

abstract = "HPC systems are a critical resource for scientific research and advanced industries. The increased demand for computational power and memory ushers in the exascale era, in which supercomputers are designed to provide enormous computing power to meet these needs. These complex supercomputers consist of numerous compute nodes and are consequently expected to experience frequent faults and crashes. Mathematical solvers, in particular, iterative linear solvers are key building block in numerous large-scale scientific applications. Consequently, supporting the recovery of distributed solvers is necessary for scaling scientific applications to exascale platforms. Previous recovery methods for iterative solvers are based on Checkpoint-Restart (CR), which incurs high fault tolerance overhead, or intrinsic fault tolerance, which require extra computation time to converge after failures. Exact state reconstruction (ESR) was proposed as an alternative mechanism to alleviate the impact of frequent failures on long-term computations. ESR has been shown to provide exact reconstruction of the computation state while avoiding the need for costly checkpointing. However, ESR currently relies on volatile memory for fault tolerance, and must therefore maintain redundancies in the RAM of multiple nodes. This not only incurs high memory overhead but also prevents ESR from being fully resilient, that is, resilient against a full system crash. Recent supercomputer designs feature emerging non-volatile RAM (NVRAM) technology, for example, the exascale Aurora that is planned to consist of Intel Optane{\texttrademark} DCPMM. This paper investigates how NVRAM can be utilized to devise an enhanced ESR-based recovery mechanism that is more efficient and provides full resilience. Our mechanism, called in-NVRAM ESR, provides full resiliency while significantly reducing both the memory footprint and the time overhead in comparison with the original ESR design (in-RAM ESR). In-NVRAM ESR is based on a novel MPI One-Sided Communication (OSC) over RDMA implementation, which was optimized and applied for using NVRAM to store recovery data for iterative linear solvers.",

keywords = "ESR, Exascale, HPC, Intel Optane DCPMM, Iterative Solvers, MPI OSC, NVRAM, PCG, RDMA, Recovery",

author = "Yehonatan Fridman and Yaniv Snir and Harel Levin and Danny Hendler and Hagit Attiya and Gal Oren",

note = "Publisher Copyright: {\textcopyright} 2022 IEEE.; 12th IEEE/ACM Workshop on Fault Tolerance for HPC at eXtreme Scale, FTXS 2022 ; Conference date: 13-11-2022 Through 18-11-2022",

year = "2022",

doi = "10.1109/FTXS56515.2022.00007",

language = "אנגלית",

series = "Proceedings of FTXS 2022: Workshop on Fault Tolerance for HPC at eXtreme Scale, Held in conjunction with SC 2022: The International Conference for High Performance Computing, Networking, Storage and Analysis",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "11--23",

booktitle = "Proceedings of FTXS 2022",

address = "ארצות הברית",

}

Fridman, Y, Snir, Y, Levin, H, Hendler, D, Attiya, H & Oren, G 2022, Recovery of Distributed Iterative Solvers for Linear Systems Using Non-Volatile RAM. in Proceedings of FTXS 2022: Workshop on Fault Tolerance for HPC at eXtreme Scale, Held in conjunction with SC 2022: The International Conference for High Performance Computing, Networking, Storage and Analysis. Proceedings of FTXS 2022: Workshop on Fault Tolerance for HPC at eXtreme Scale, Held in conjunction with SC 2022: The International Conference for High Performance Computing, Networking, Storage and Analysis, Institute of Electrical and Electronics Engineers Inc., pp. 11-23, 12th IEEE/ACM Workshop on Fault Tolerance for HPC at eXtreme Scale, FTXS 2022, Dallas, United States, 13/11/22. https://doi.org/10.1109/FTXS56515.2022.00007

Recovery of Distributed Iterative Solvers for Linear Systems Using Non-Volatile RAM. / Fridman, Yehonatan; Snir, Yaniv; Levin, Harel et al.
Proceedings of FTXS 2022: Workshop on Fault Tolerance for HPC at eXtreme Scale, Held in conjunction with SC 2022: The International Conference for High Performance Computing, Networking, Storage and Analysis. Institute of Electrical and Electronics Engineers Inc., 2022. p. 11-23 (Proceedings of FTXS 2022: Workshop on Fault Tolerance for HPC at eXtreme Scale, Held in conjunction with SC 2022: The International Conference for High Performance Computing, Networking, Storage and Analysis).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Recovery of Distributed Iterative Solvers for Linear Systems Using Non-Volatile RAM

AU - Fridman, Yehonatan

AU - Snir, Yaniv

AU - Levin, Harel

AU - Hendler, Danny

AU - Attiya, Hagit

AU - Oren, Gal

PY - 2022

Y1 - 2022

N2 - HPC systems are a critical resource for scientific research and advanced industries. The increased demand for computational power and memory ushers in the exascale era, in which supercomputers are designed to provide enormous computing power to meet these needs. These complex supercomputers consist of numerous compute nodes and are consequently expected to experience frequent faults and crashes. Mathematical solvers, in particular, iterative linear solvers are key building block in numerous large-scale scientific applications. Consequently, supporting the recovery of distributed solvers is necessary for scaling scientific applications to exascale platforms. Previous recovery methods for iterative solvers are based on Checkpoint-Restart (CR), which incurs high fault tolerance overhead, or intrinsic fault tolerance, which require extra computation time to converge after failures. Exact state reconstruction (ESR) was proposed as an alternative mechanism to alleviate the impact of frequent failures on long-term computations. ESR has been shown to provide exact reconstruction of the computation state while avoiding the need for costly checkpointing. However, ESR currently relies on volatile memory for fault tolerance, and must therefore maintain redundancies in the RAM of multiple nodes. This not only incurs high memory overhead but also prevents ESR from being fully resilient, that is, resilient against a full system crash. Recent supercomputer designs feature emerging non-volatile RAM (NVRAM) technology, for example, the exascale Aurora that is planned to consist of Intel Optane™ DCPMM. This paper investigates how NVRAM can be utilized to devise an enhanced ESR-based recovery mechanism that is more efficient and provides full resilience. Our mechanism, called in-NVRAM ESR, provides full resiliency while significantly reducing both the memory footprint and the time overhead in comparison with the original ESR design (in-RAM ESR). In-NVRAM ESR is based on a novel MPI One-Sided Communication (OSC) over RDMA implementation, which was optimized and applied for using NVRAM to store recovery data for iterative linear solvers.

AB - HPC systems are a critical resource for scientific research and advanced industries. The increased demand for computational power and memory ushers in the exascale era, in which supercomputers are designed to provide enormous computing power to meet these needs. These complex supercomputers consist of numerous compute nodes and are consequently expected to experience frequent faults and crashes. Mathematical solvers, in particular, iterative linear solvers are key building block in numerous large-scale scientific applications. Consequently, supporting the recovery of distributed solvers is necessary for scaling scientific applications to exascale platforms. Previous recovery methods for iterative solvers are based on Checkpoint-Restart (CR), which incurs high fault tolerance overhead, or intrinsic fault tolerance, which require extra computation time to converge after failures. Exact state reconstruction (ESR) was proposed as an alternative mechanism to alleviate the impact of frequent failures on long-term computations. ESR has been shown to provide exact reconstruction of the computation state while avoiding the need for costly checkpointing. However, ESR currently relies on volatile memory for fault tolerance, and must therefore maintain redundancies in the RAM of multiple nodes. This not only incurs high memory overhead but also prevents ESR from being fully resilient, that is, resilient against a full system crash. Recent supercomputer designs feature emerging non-volatile RAM (NVRAM) technology, for example, the exascale Aurora that is planned to consist of Intel Optane™ DCPMM. This paper investigates how NVRAM can be utilized to devise an enhanced ESR-based recovery mechanism that is more efficient and provides full resilience. Our mechanism, called in-NVRAM ESR, provides full resiliency while significantly reducing both the memory footprint and the time overhead in comparison with the original ESR design (in-RAM ESR). In-NVRAM ESR is based on a novel MPI One-Sided Communication (OSC) over RDMA implementation, which was optimized and applied for using NVRAM to store recovery data for iterative linear solvers.

KW - ESR

KW - Exascale

KW - HPC

KW - Intel Optane DCPMM

KW - Iterative Solvers

KW - MPI OSC

KW - NVRAM

KW - PCG

KW - RDMA

KW - Recovery

UR - http://www.scopus.com/inward/record.url?scp=85147947653&partnerID=8YFLogxK

U2 - 10.1109/FTXS56515.2022.00007

DO - 10.1109/FTXS56515.2022.00007

M3 - ???researchoutput.researchoutputtypes.contributiontobookanthology.conference???

AN - SCOPUS:85147947653

T3 - Proceedings of FTXS 2022: Workshop on Fault Tolerance for HPC at eXtreme Scale, Held in conjunction with SC 2022: The International Conference for High Performance Computing, Networking, Storage and Analysis

SP - 11

EP - 23

BT - Proceedings of FTXS 2022

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 12th IEEE/ACM Workshop on Fault Tolerance for HPC at eXtreme Scale, FTXS 2022

Y2 - 13 November 2022 through 18 November 2022

ER -

Fridman Y, Snir Y, Levin H, Hendler D, Attiya H, Oren G. Recovery of Distributed Iterative Solvers for Linear Systems Using Non-Volatile RAM. In Proceedings of FTXS 2022: Workshop on Fault Tolerance for HPC at eXtreme Scale, Held in conjunction with SC 2022: The International Conference for High Performance Computing, Networking, Storage and Analysis. Institute of Electrical and Electronics Engineers Inc. 2022. p. 11-23. (Proceedings of FTXS 2022: Workshop on Fault Tolerance for HPC at eXtreme Scale, Held in conjunction with SC 2022: The International Conference for High Performance Computing, Networking, Storage and Analysis). doi: 10.1109/FTXS56515.2022.00007

Recovery of Distributed Iterative Solvers for Linear Systems Using Non-Volatile RAM

Abstract

Publication series

Conference

Bibliographical note

Keywords

Access to Document

Other files and links

Fingerprint

Cite this