Entropy-Based Approach to Efficient Cleaning of Big Data in Hierarchical Databases

Eugene Levner; Boris Kriheli; Arriel Benis; Alexander Ptuskin; Amir Elalouf; Sharon Hovav; Shai Ashkenazi

doi:10.1007/978-3-030-59612-5_1

Entropy-Based Approach to Efficient Cleaning of Big Data in Hierarchical Databases

Eugene Levner
, Boris Kriheli
, Arriel Benis
, Alexander Ptuskin
, Amir Elalouf
, Sharon Hovav
, Shai Ashkenazi

Department of Management

Holon Institute of Technology
Ashkelon Academic College
Bauman Moscow State Technical University
Clalit Health Services
Ariel University

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

1 Scopus citations

Abstract

When databases are at risk of containing erroneous, redundant, or obsolete data, a cleaning procedure is used to detect, correct or remove such undesirable records. We propose a methodology for improving data cleaning efficiency in a large hierarchical database. The methodology relies on Shannon’s information entropy for measuring the amount of information stored in databases. This approach, which builds on previously-gathered statistical data regarding the prevalence of errors in the database, enables the decision maker to determine which components of the database are likely to have undergone more information loss, and thus to prioritize those components for cleaning. In particular, in cases where the cleaning process is iterative (from the root node down), the entropic approach produces a scientifically motivated stopping rule that determines the optimal (i.e. minimally required) number of tiers in the hierarchical database that need to be examined. This stopping rule defines a more streamlined representation of the database, in which less informative tiers are eliminated.

Original language	English
Title of host publication	Big Data – BigData 2020 - 9th International Conference, Held as Part of the Services Conference Federation, SCF 2020, Proceedings
Editors	Surya Nepal, Wenqi Cao, Aziz Nasridinov, MD Zakirul Alam Bhuiyan, Xuan Guo, Liang-Jie Zhang
Publisher	Springer Science and Business Media Deutschland GmbH
Pages	3-12
Number of pages	10
ISBN (Print)	9783030596118
DOIs	https://doi.org/10.1007/978-3-030-59612-5_1
State	Published - 2020
Event	9th International Conference on Big Data, BigData 2020, held as part of the Services Conference Federation, SCF 2020 - Honolulu, United States Duration: 18 Sep 2020 → 20 Sep 2020

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	12402 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	9th International Conference on Big Data, BigData 2020, held as part of the Services Conference Federation, SCF 2020
Country/Territory	United States
City	Honolulu
Period	18/09/20 → 20/09/20

Bibliographical note

Publisher Copyright:
© 2020, Springer Nature Switzerland AG.

Funding

The research of Eugene Levner, Arriel Benis and Shai Ashkenazi is supported by the grant of the Ariel University and Holon Institute of Technology (Israel) “Appli-cations of Artificial Intelligence for enhancing efficiency of vaccination programs” [no. RA19000000649]. The research of Alexander Ptuskin is supported by the grant of the Russian Foundation for Basic Research “Development of economic-mathematical methods for evaluation of alternative options and identification of the best available technologies” [no. 18-410-400001]. Acknowledgement. The research of Eugene Levner, Arriel Benis and Shai Ashkenazi is supported by the grant of the Ariel University and Holon Institute of Technology (Israel) “Applications of Artificial Intelligence for enhancing efficiency of vaccination programs” [no. RA19000000649]. The research of Alexander Ptuskin is supported by the grant of the Russian Foundation for Basic Research “Development of economic-mathematical methods for evaluation of alternative options and identification of the best available technologies” [no. 18-410-400001].

Funders	Funder number
Holon Institute of Technology (Israel) “Appli-cations of Artificial Intelligence
Holon Institute of Technology (Israel) “Applications of Artificial Intelligence	RA19000000649
Russian Foundation for Basic Research	18-410-400001
Ariel University

Keywords

Data cleaning
Entropy evaluation
Entropy-based analytics

Access to Document

10.1007/978-3-030-59612-5_1

Cite this

Levner, E., Kriheli, B., Benis, A., Ptuskin, A., Elalouf, A., Hovav, S., & Ashkenazi, S. (2020). Entropy-Based Approach to Efficient Cleaning of Big Data in Hierarchical Databases. In S. Nepal, W. Cao, A. Nasridinov, MD. Z. A. Bhuiyan, X. Guo, & L.-J. Zhang (Eds.), Big Data – BigData 2020 - 9th International Conference, Held as Part of the Services Conference Federation, SCF 2020, Proceedings (pp. 3-12). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 12402 LNCS). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-030-59612-5_1

Levner, Eugene ; Kriheli, Boris ; Benis, Arriel et al. / Entropy-Based Approach to Efficient Cleaning of Big Data in Hierarchical Databases. Big Data – BigData 2020 - 9th International Conference, Held as Part of the Services Conference Federation, SCF 2020, Proceedings. editor / Surya Nepal ; Wenqi Cao ; Aziz Nasridinov ; MD Zakirul Alam Bhuiyan ; Xuan Guo ; Liang-Jie Zhang. Springer Science and Business Media Deutschland GmbH, 2020. pp. 3-12 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{d623bae7aa404e3fb93ea0518b9ab172,

title = "Entropy-Based Approach to Efficient Cleaning of Big Data in Hierarchical Databases",

abstract = "When databases are at risk of containing erroneous, redundant, or obsolete data, a cleaning procedure is used to detect, correct or remove such undesirable records. We propose a methodology for improving data cleaning efficiency in a large hierarchical database. The methodology relies on Shannon{\textquoteright}s information entropy for measuring the amount of information stored in databases. This approach, which builds on previously-gathered statistical data regarding the prevalence of errors in the database, enables the decision maker to determine which components of the database are likely to have undergone more information loss, and thus to prioritize those components for cleaning. In particular, in cases where the cleaning process is iterative (from the root node down), the entropic approach produces a scientifically motivated stopping rule that determines the optimal (i.e. minimally required) number of tiers in the hierarchical database that need to be examined. This stopping rule defines a more streamlined representation of the database, in which less informative tiers are eliminated.",

keywords = "Data cleaning, Entropy evaluation, Entropy-based analytics",

author = "Eugene Levner and Boris Kriheli and Arriel Benis and Alexander Ptuskin and Amir Elalouf and Sharon Hovav and Shai Ashkenazi",

note = "Publisher Copyright: {\textcopyright} 2020, Springer Nature Switzerland AG.; 9th International Conference on Big Data, BigData 2020, held as part of the Services Conference Federation, SCF 2020 ; Conference date: 18-09-2020 Through 20-09-2020",

year = "2020",

doi = "10.1007/978-3-030-59612-5\_1",

language = "אנגלית",

isbn = "9783030596118",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "3--12",

editor = "Surya Nepal and Wenqi Cao and Aziz Nasridinov and Bhuiyan, \{MD Zakirul Alam\} and Xuan Guo and Liang-Jie Zhang",

booktitle = "Big Data {\textendash} BigData 2020 - 9th International Conference, Held as Part of the Services Conference Federation, SCF 2020, Proceedings",

address = "גרמניה",

}

Levner, E, Kriheli, B, Benis, A, Ptuskin, A, Elalouf, A, Hovav, S & Ashkenazi, S 2020, Entropy-Based Approach to Efficient Cleaning of Big Data in Hierarchical Databases. in S Nepal, W Cao, A Nasridinov, MDZA Bhuiyan, X Guo & L-J Zhang (eds), Big Data – BigData 2020 - 9th International Conference, Held as Part of the Services Conference Federation, SCF 2020, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 12402 LNCS, Springer Science and Business Media Deutschland GmbH, pp. 3-12, 9th International Conference on Big Data, BigData 2020, held as part of the Services Conference Federation, SCF 2020, Honolulu, United States, 18/09/20. https://doi.org/10.1007/978-3-030-59612-5_1

Entropy-Based Approach to Efficient Cleaning of Big Data in Hierarchical Databases. / Levner, Eugene; Kriheli, Boris; Benis, Arriel et al.
Big Data – BigData 2020 - 9th International Conference, Held as Part of the Services Conference Federation, SCF 2020, Proceedings. ed. / Surya Nepal; Wenqi Cao; Aziz Nasridinov; MD Zakirul Alam Bhuiyan; Xuan Guo; Liang-Jie Zhang. Springer Science and Business Media Deutschland GmbH, 2020. p. 3-12 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 12402 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Entropy-Based Approach to Efficient Cleaning of Big Data in Hierarchical Databases

AU - Levner, Eugene

AU - Kriheli, Boris

AU - Benis, Arriel

AU - Ptuskin, Alexander

AU - Elalouf, Amir

AU - Hovav, Sharon

AU - Ashkenazi, Shai

PY - 2020

Y1 - 2020

N2 - When databases are at risk of containing erroneous, redundant, or obsolete data, a cleaning procedure is used to detect, correct or remove such undesirable records. We propose a methodology for improving data cleaning efficiency in a large hierarchical database. The methodology relies on Shannon’s information entropy for measuring the amount of information stored in databases. This approach, which builds on previously-gathered statistical data regarding the prevalence of errors in the database, enables the decision maker to determine which components of the database are likely to have undergone more information loss, and thus to prioritize those components for cleaning. In particular, in cases where the cleaning process is iterative (from the root node down), the entropic approach produces a scientifically motivated stopping rule that determines the optimal (i.e. minimally required) number of tiers in the hierarchical database that need to be examined. This stopping rule defines a more streamlined representation of the database, in which less informative tiers are eliminated.

AB - When databases are at risk of containing erroneous, redundant, or obsolete data, a cleaning procedure is used to detect, correct or remove such undesirable records. We propose a methodology for improving data cleaning efficiency in a large hierarchical database. The methodology relies on Shannon’s information entropy for measuring the amount of information stored in databases. This approach, which builds on previously-gathered statistical data regarding the prevalence of errors in the database, enables the decision maker to determine which components of the database are likely to have undergone more information loss, and thus to prioritize those components for cleaning. In particular, in cases where the cleaning process is iterative (from the root node down), the entropic approach produces a scientifically motivated stopping rule that determines the optimal (i.e. minimally required) number of tiers in the hierarchical database that need to be examined. This stopping rule defines a more streamlined representation of the database, in which less informative tiers are eliminated.

KW - Data cleaning

KW - Entropy evaluation

KW - Entropy-based analytics

UR - https://www.scopus.com/pages/publications/85092079714

U2 - 10.1007/978-3-030-59612-5_1

DO - 10.1007/978-3-030-59612-5_1

M3 - ???researchoutput.researchoutputtypes.contributiontobookanthology.conference???

AN - SCOPUS:85092079714

SN - 9783030596118

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 3

EP - 12

BT - Big Data – BigData 2020 - 9th International Conference, Held as Part of the Services Conference Federation, SCF 2020, Proceedings

A2 - Nepal, Surya

A2 - Cao, Wenqi

A2 - Nasridinov, Aziz

A2 - Bhuiyan, MD Zakirul Alam

A2 - Guo, Xuan

A2 - Zhang, Liang-Jie

PB - Springer Science and Business Media Deutschland GmbH

T2 - 9th International Conference on Big Data, BigData 2020, held as part of the Services Conference Federation, SCF 2020

Y2 - 18 September 2020 through 20 September 2020

ER -

Levner E, Kriheli B, Benis A, Ptuskin A, Elalouf A, Hovav S et al. Entropy-Based Approach to Efficient Cleaning of Big Data in Hierarchical Databases. In Nepal S, Cao W, Nasridinov A, Bhuiyan MDZA, Guo X, Zhang LJ, editors, Big Data – BigData 2020 - 9th International Conference, Held as Part of the Services Conference Federation, SCF 2020, Proceedings. Springer Science and Business Media Deutschland GmbH. 2020. p. 3-12. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-030-59612-5_1

Entropy-Based Approach to Efficient Cleaning of Big Data in Hierarchical Databases

Abstract

Publication series

Conference

Bibliographical note

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this