Similarity based deduplication with small data chunks

Lior Aronovich, Ron Asher, Danny Harnik, Michael Hirsch, Shmuel T. Klein, Yair Toaft

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

6 Scopus citations


Large backup and restore systems may have a petabyte or more data in their repository. Such systems are often compressed by means of deduplication techniques, that partition the input text into chunks and store recurring chunks only once. One of the approaches is to use hashing methods to store fingerprints for each data chunk, detecting identical chunks with very low probability for collisions. As alternative, it has been suggested to use similarity instead of identity based searches, which allows the definition of much larger chunks. This implies that the data structure needed to store the fingerprints is much smaller, so that such a system may be more scalable than systems built on the first approach. This paper deals with an extension of the second approach to systems in which it is still preferred to use small chunks. We describe the design choices made during the development of what we call an approximate hash function, serving as the basic tool of the new suggested deduplication system and report on extensive tests performed on an variety of large input files.

Original languageEnglish
Title of host publicationProceedings of the Prague Stringology Conference, PSC 2012
Number of pages15
StatePublished - 2012
EventPrague Stringology Conference, PSC 2012 - Prague, Czech Republic
Duration: 27 Aug 201228 Aug 2012

Publication series

NameProceedings of the Prague Stringology Conference, PSC 2012


ConferencePrague Stringology Conference, PSC 2012
Country/TerritoryCzech Republic

Bibliographical note

Place of conference:Czech Republic


  • Approximate hash scheme
  • Compression
  • Deduplication


Dive into the research topics of 'Similarity based deduplication with small data chunks'. Together they form a unique fingerprint.

Cite this