Predicting Data Scientist Stuckness During the Development of Machine Learning Classifiers

Moshe Mash; Shoshana Oryol; Reid Simmons; Stephanie Rosenthal

doi:10.1109/vl/hcc53370.2022.9833124

Predicting Data Scientist Stuckness During the Development of Machine Learning Classifiers

Moshe Mash, Shoshana Oryol, Reid Simmons, Stephanie Rosenthal

Carnegie Mellon University

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

The success of data scientists in developing machine learning models is contingent on an iterative development process for detecting patterns in data, finding and extracting useful features, and maximizing their model's performance. However, it is often the case that they struggle during model development and become stuck and unable to make significant progress. We collected qualitative and quantitative data from the workflow of data scientists that allow us to learn from and examine such moments of stuckness. We used this data to develop a model for predicting stuckness based on real-time indicators, such as code artifacts, and then used the model to develop an innovative algorithm that determines precisely when a potential stuckness intervention should occur: as close as possible to the beginning of actual stuckness. Our algorithm's performance indicates the potential efficacy of predicting data scientist stuckness algorithmically under real-world circumstances and for real-world needs.

Original language	English
Title of host publication	Proceedings - 2022 IEEE Symposium on Visual Languages and Human-Centric Computing, VL/HCC 2022
Editors	Paolo Bottoni, Gennaro Costagliola, Michelle Brachman, Mark Minas
Publisher	IEEE Computer Society
ISBN (Electronic)	9781665442145
DOIs	https://doi.org/10.1109/vl/hcc53370.2022.9833124
State	Published - 2022
Externally published	Yes
Event	2022 IEEE Symposium on Visual Languages and Human-Centric Computing, VL/HCC 2022 - Rome, Italy Duration: 12 Sep 2022 → 16 Sep 2022

Publication series

Name	Proceedings of IEEE Symposium on Visual Languages and Human-Centric Computing, VL/HCC
Volume	2022-September
ISSN (Print)	1943-6092
ISSN (Electronic)	1943-6106

Conference

Conference	2022 IEEE Symposium on Visual Languages and Human-Centric Computing, VL/HCC 2022
Country/Territory	Italy
City	Rome
Period	12/09/22 → 16/09/22

Bibliographical note

Publisher Copyright:
© 2022 IEEE Computer Society. All rights reserved.

Funding

ACKNOWLEDGMENTS This research was funded by JPMorgan Chase & Co. Any views or opinions expressed herein are solely those of the authors listed, and may differ from the views and opinions expressed by JPMorgan Chase & Co. or its affiliates. This material is not a product of the Research Department of J.P. Morgan Securities LLC. This material should not be construed as an individual recommendation for any particular client and is not intended as a recommendation of particular securities, financial instruments or strategies for a particular client. This material does not constitute a solicitation or offer in any jurisdiction.

Funders	Funder number
JPMorgan Chase & Co

Access to Document

10.1109/vl/hcc53370.2022.9833124

Cite this

Mash, M., Oryol, S., Simmons, R., & Rosenthal, S. (2022). Predicting Data Scientist Stuckness During the Development of Machine Learning Classifiers. In P. Bottoni, G. Costagliola, M. Brachman, & M. Minas (Eds.), Proceedings - 2022 IEEE Symposium on Visual Languages and Human-Centric Computing, VL/HCC 2022 (Proceedings of IEEE Symposium on Visual Languages and Human-Centric Computing, VL/HCC; Vol. 2022-September). IEEE Computer Society. https://doi.org/10.1109/vl/hcc53370.2022.9833124

Mash, Moshe ; Oryol, Shoshana ; Simmons, Reid et al. / Predicting Data Scientist Stuckness During the Development of Machine Learning Classifiers. Proceedings - 2022 IEEE Symposium on Visual Languages and Human-Centric Computing, VL/HCC 2022. editor / Paolo Bottoni ; Gennaro Costagliola ; Michelle Brachman ; Mark Minas. IEEE Computer Society, 2022. (Proceedings of IEEE Symposium on Visual Languages and Human-Centric Computing, VL/HCC).

@inproceedings{633d43de3af047d4b0c34d570e113a17,

title = "Predicting Data Scientist Stuckness During the Development of Machine Learning Classifiers",

abstract = "The success of data scientists in developing machine learning models is contingent on an iterative development process for detecting patterns in data, finding and extracting useful features, and maximizing their model's performance. However, it is often the case that they struggle during model development and become stuck and unable to make significant progress. We collected qualitative and quantitative data from the workflow of data scientists that allow us to learn from and examine such moments of stuckness. We used this data to develop a model for predicting stuckness based on real-time indicators, such as code artifacts, and then used the model to develop an innovative algorithm that determines precisely when a potential stuckness intervention should occur: as close as possible to the beginning of actual stuckness. Our algorithm's performance indicates the potential efficacy of predicting data scientist stuckness algorithmically under real-world circumstances and for real-world needs.",

author = "Moshe Mash and Shoshana Oryol and Reid Simmons and Stephanie Rosenthal",

note = "Publisher Copyright: {\textcopyright} 2022 IEEE Computer Society. All rights reserved.; 2022 IEEE Symposium on Visual Languages and Human-Centric Computing, VL/HCC 2022 ; Conference date: 12-09-2022 Through 16-09-2022",

year = "2022",

doi = "10.1109/vl/hcc53370.2022.9833124",

language = "אנגלית",

series = "Proceedings of IEEE Symposium on Visual Languages and Human-Centric Computing, VL/HCC",

publisher = "IEEE Computer Society",

editor = "Paolo Bottoni and Gennaro Costagliola and Michelle Brachman and Mark Minas",

booktitle = "Proceedings - 2022 IEEE Symposium on Visual Languages and Human-Centric Computing, VL/HCC 2022",

address = "ארצות הברית",

}

Mash, M, Oryol, S, Simmons, R & Rosenthal, S 2022, Predicting Data Scientist Stuckness During the Development of Machine Learning Classifiers. in P Bottoni, G Costagliola, M Brachman & M Minas (eds), Proceedings - 2022 IEEE Symposium on Visual Languages and Human-Centric Computing, VL/HCC 2022. Proceedings of IEEE Symposium on Visual Languages and Human-Centric Computing, VL/HCC, vol. 2022-September, IEEE Computer Society, 2022 IEEE Symposium on Visual Languages and Human-Centric Computing, VL/HCC 2022, Rome, Italy, 12/09/22. https://doi.org/10.1109/vl/hcc53370.2022.9833124

Predicting Data Scientist Stuckness During the Development of Machine Learning Classifiers. / Mash, Moshe; Oryol, Shoshana; Simmons, Reid et al.
Proceedings - 2022 IEEE Symposium on Visual Languages and Human-Centric Computing, VL/HCC 2022. ed. / Paolo Bottoni; Gennaro Costagliola; Michelle Brachman; Mark Minas. IEEE Computer Society, 2022. (Proceedings of IEEE Symposium on Visual Languages and Human-Centric Computing, VL/HCC; Vol. 2022-September).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Predicting Data Scientist Stuckness During the Development of Machine Learning Classifiers

AU - Mash, Moshe

AU - Oryol, Shoshana

AU - Simmons, Reid

AU - Rosenthal, Stephanie

PY - 2022

Y1 - 2022

N2 - The success of data scientists in developing machine learning models is contingent on an iterative development process for detecting patterns in data, finding and extracting useful features, and maximizing their model's performance. However, it is often the case that they struggle during model development and become stuck and unable to make significant progress. We collected qualitative and quantitative data from the workflow of data scientists that allow us to learn from and examine such moments of stuckness. We used this data to develop a model for predicting stuckness based on real-time indicators, such as code artifacts, and then used the model to develop an innovative algorithm that determines precisely when a potential stuckness intervention should occur: as close as possible to the beginning of actual stuckness. Our algorithm's performance indicates the potential efficacy of predicting data scientist stuckness algorithmically under real-world circumstances and for real-world needs.

AB - The success of data scientists in developing machine learning models is contingent on an iterative development process for detecting patterns in data, finding and extracting useful features, and maximizing their model's performance. However, it is often the case that they struggle during model development and become stuck and unable to make significant progress. We collected qualitative and quantitative data from the workflow of data scientists that allow us to learn from and examine such moments of stuckness. We used this data to develop a model for predicting stuckness based on real-time indicators, such as code artifacts, and then used the model to develop an innovative algorithm that determines precisely when a potential stuckness intervention should occur: as close as possible to the beginning of actual stuckness. Our algorithm's performance indicates the potential efficacy of predicting data scientist stuckness algorithmically under real-world circumstances and for real-world needs.

UR - http://www.scopus.com/inward/record.url?scp=85148713000&partnerID=8YFLogxK

U2 - 10.1109/vl/hcc53370.2022.9833124

DO - 10.1109/vl/hcc53370.2022.9833124

M3 - ???researchoutput.researchoutputtypes.contributiontobookanthology.conference???

AN - SCOPUS:85148713000

T3 - Proceedings of IEEE Symposium on Visual Languages and Human-Centric Computing, VL/HCC

BT - Proceedings - 2022 IEEE Symposium on Visual Languages and Human-Centric Computing, VL/HCC 2022

A2 - Bottoni, Paolo

A2 - Costagliola, Gennaro

A2 - Brachman, Michelle

A2 - Minas, Mark

PB - IEEE Computer Society

T2 - 2022 IEEE Symposium on Visual Languages and Human-Centric Computing, VL/HCC 2022

Y2 - 12 September 2022 through 16 September 2022

ER -

Mash M, Oryol S, Simmons R, Rosenthal S. Predicting Data Scientist Stuckness During the Development of Machine Learning Classifiers. In Bottoni P, Costagliola G, Brachman M, Minas M, editors, Proceedings - 2022 IEEE Symposium on Visual Languages and Human-Centric Computing, VL/HCC 2022. IEEE Computer Society. 2022. (Proceedings of IEEE Symposium on Visual Languages and Human-Centric Computing, VL/HCC). doi: 10.1109/vl/hcc53370.2022.9833124

Predicting Data Scientist Stuckness During the Development of Machine Learning Classifiers

Abstract

Publication series

Conference

Bibliographical note

Funding

Access to Document

Other files and links

Fingerprint

Cite this