Similarity based deduplication with small data chunks

L. Aronovich, R. Asher, D. Harnik, M. Hirsch, S. T. Klein, Y. Toaff

Research output: Contribution to journalArticlepeer-review

6 Scopus citations

Abstract

Large backup and restore systems may have a petabyte or more data in their repository. Such systems are often compressed by means of deduplication techniques, that partition the input text into chunks and store recurring chunks only once. One of the approaches is to use hashing methods to store fingerprints for each data chunk, detecting identical chunks with very low probability for collisions. As alternative, it has been suggested to use similarity instead of identity based searches, which allows the definition of much larger chunks. This implies that the data structure needed to store the fingerprints is much smaller, so that such a system may be more scalable than systems built on the first approach. This paper deals with an extension of the second approach to systems in which it is still preferred to use small chunks. We describe the design choices made during the development of what we call an approximate hash function, serving as the basic tool of the new suggested deduplication system and report on extensive tests performed on a variety of large input files.

Original languageEnglish
Pages (from-to)10-22
Number of pages13
JournalDiscrete Applied Mathematics
Volume212
DOIs
StatePublished - 30 Oct 2016

Bibliographical note

Publisher Copyright:
© 2015 Elsevier B.V.

Keywords

  • Approximate hashing
  • Deduplication
  • Similarity
  • Small data chunks

Fingerprint

Dive into the research topics of 'Similarity based deduplication with small data chunks'. Together they form a unique fingerprint.

Cite this