A Randomized algorithm for constructing cross-feature tests from single feature tests

Guy Barash; Eitan Farchi

doi:10.1109/hldvt.2017.8167456

A Randomized algorithm for constructing cross-feature tests from single feature tests

Guy Barash
, Eitan Farchi

Western Digital
IBM

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

Memory protocol commands and firmware features for storage solutions may function correctly by themselves, but due to implicit cross-feature dependencies they may exhibit incorrect behavior when exercised in combination with other features or commands. How can we effciently use a set of standalone per-feature tests to test the cross-features dependencies? We define an algorithm that is based on greedy and randomized heuristics that can produce an optimized plan for cross-feature testing within a polynomial number of steps. The algorithm is designed to meet cross-feature coverage requirements and was used to find problems in memory chip firmware with good results. We were consistently able to produce a cross-feature test plan in no more than 5 hours given a set of more than 1400 per-feature test variants. We have successfully discovered bugs caused by incorrectly handled interrupts and errors in managing a shared resource, that would have been missed by the per-feature tests method.

Original language	English
Title of host publication	2017 IEEE International High Level Design Validation and Test Workshop, HLDVT 2017
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	1-8
Number of pages	8
ISBN (Electronic)	9781509039975
DOIs	https://doi.org/10.1109/hldvt.2017.8167456
State	Published - 5 Dec 2017
Externally published	Yes
Event	19th IEEE International High Level Design Validation and Test Workshop, HLDVT 2017 - Santa Cruz, United States Duration: 5 Oct 2017 → 6 Oct 2017

Publication series

Name	2017 IEEE International High Level Design Validation and Test Workshop, HLDVT 2017
Volume	2017-January

Conference

Conference	19th IEEE International High Level Design Validation and Test Workshop, HLDVT 2017
Country/Territory	United States
City	Santa Cruz
Period	5/10/17 → 6/10/17

Bibliographical note

Publisher Copyright:
© 2017 IEEE.

Keywords

Combinatorial interaction testing
Concurrent testing
Constraints
Greedy testing
Random testing

Access to Document

10.1109/hldvt.2017.8167456

Cite this

Barash, G., & Farchi, E. (2017). A Randomized algorithm for constructing cross-feature tests from single feature tests. In 2017 IEEE International High Level Design Validation and Test Workshop, HLDVT 2017 (pp. 1-8). (2017 IEEE International High Level Design Validation and Test Workshop, HLDVT 2017; Vol. 2017-January). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/hldvt.2017.8167456

@inproceedings{2ed152b1904d4949b03c41f1c9978025,

title = "A Randomized algorithm for constructing cross-feature tests from single feature tests",

abstract = "Memory protocol commands and firmware features for storage solutions may function correctly by themselves, but due to implicit cross-feature dependencies they may exhibit incorrect behavior when exercised in combination with other features or commands. How can we effciently use a set of standalone per-feature tests to test the cross-features dependencies? We define an algorithm that is based on greedy and randomized heuristics that can produce an optimized plan for cross-feature testing within a polynomial number of steps. The algorithm is designed to meet cross-feature coverage requirements and was used to find problems in memory chip firmware with good results. We were consistently able to produce a cross-feature test plan in no more than 5 hours given a set of more than 1400 per-feature test variants. We have successfully discovered bugs caused by incorrectly handled interrupts and errors in managing a shared resource, that would have been missed by the per-feature tests method.",

keywords = "Combinatorial interaction testing, Concurrent testing, Constraints, Greedy testing, Random testing",

author = "Guy Barash and Eitan Farchi",

note = "Publisher Copyright: {\textcopyright} 2017 IEEE.; 19th IEEE International High Level Design Validation and Test Workshop, HLDVT 2017 ; Conference date: 05-10-2017 Through 06-10-2017",

year = "2017",

month = dec,

day = "5",

doi = "10.1109/hldvt.2017.8167456",

language = "אנגלית",

series = "2017 IEEE International High Level Design Validation and Test Workshop, HLDVT 2017",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "1--8",

booktitle = "2017 IEEE International High Level Design Validation and Test Workshop, HLDVT 2017",

address = "ארצות הברית",

}

Barash, G & Farchi, E 2017, A Randomized algorithm for constructing cross-feature tests from single feature tests. in 2017 IEEE International High Level Design Validation and Test Workshop, HLDVT 2017. 2017 IEEE International High Level Design Validation and Test Workshop, HLDVT 2017, vol. 2017-January, Institute of Electrical and Electronics Engineers Inc., pp. 1-8, 19th IEEE International High Level Design Validation and Test Workshop, HLDVT 2017, Santa Cruz, United States, 5/10/17. https://doi.org/10.1109/hldvt.2017.8167456

A Randomized algorithm for constructing cross-feature tests from single feature tests. / Barash, Guy; Farchi, Eitan.
2017 IEEE International High Level Design Validation and Test Workshop, HLDVT 2017. Institute of Electrical and Electronics Engineers Inc., 2017. p. 1-8 (2017 IEEE International High Level Design Validation and Test Workshop, HLDVT 2017; Vol. 2017-January).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - A Randomized algorithm for constructing cross-feature tests from single feature tests

AU - Barash, Guy

AU - Farchi, Eitan

PY - 2017/12/5

Y1 - 2017/12/5

N2 - Memory protocol commands and firmware features for storage solutions may function correctly by themselves, but due to implicit cross-feature dependencies they may exhibit incorrect behavior when exercised in combination with other features or commands. How can we effciently use a set of standalone per-feature tests to test the cross-features dependencies? We define an algorithm that is based on greedy and randomized heuristics that can produce an optimized plan for cross-feature testing within a polynomial number of steps. The algorithm is designed to meet cross-feature coverage requirements and was used to find problems in memory chip firmware with good results. We were consistently able to produce a cross-feature test plan in no more than 5 hours given a set of more than 1400 per-feature test variants. We have successfully discovered bugs caused by incorrectly handled interrupts and errors in managing a shared resource, that would have been missed by the per-feature tests method.

AB - Memory protocol commands and firmware features for storage solutions may function correctly by themselves, but due to implicit cross-feature dependencies they may exhibit incorrect behavior when exercised in combination with other features or commands. How can we effciently use a set of standalone per-feature tests to test the cross-features dependencies? We define an algorithm that is based on greedy and randomized heuristics that can produce an optimized plan for cross-feature testing within a polynomial number of steps. The algorithm is designed to meet cross-feature coverage requirements and was used to find problems in memory chip firmware with good results. We were consistently able to produce a cross-feature test plan in no more than 5 hours given a set of more than 1400 per-feature test variants. We have successfully discovered bugs caused by incorrectly handled interrupts and errors in managing a shared resource, that would have been missed by the per-feature tests method.

KW - Combinatorial interaction testing

KW - Concurrent testing

KW - Constraints

KW - Greedy testing

KW - Random testing

UR - https://www.scopus.com/pages/publications/85043486453

U2 - 10.1109/hldvt.2017.8167456

DO - 10.1109/hldvt.2017.8167456

M3 - ???researchoutput.researchoutputtypes.contributiontobookanthology.conference???

AN - SCOPUS:85043486453

T3 - 2017 IEEE International High Level Design Validation and Test Workshop, HLDVT 2017

SP - 1

EP - 8

BT - 2017 IEEE International High Level Design Validation and Test Workshop, HLDVT 2017

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 19th IEEE International High Level Design Validation and Test Workshop, HLDVT 2017

Y2 - 5 October 2017 through 6 October 2017

ER -

Barash G, Farchi E. A Randomized algorithm for constructing cross-feature tests from single feature tests. In 2017 IEEE International High Level Design Validation and Test Workshop, HLDVT 2017. Institute of Electrical and Electronics Engineers Inc. 2017. p. 1-8. (2017 IEEE International High Level Design Validation and Test Workshop, HLDVT 2017). doi: 10.1109/hldvt.2017.8167456

A Randomized algorithm for constructing cross-feature tests from single feature tests

Abstract

Publication series

Conference

Bibliographical note

Keywords

Access to Document

Other files and links

Fingerprint

Cite this