Reproducibility in Management Science

With the help of more than 700 reviewers we assess the reproducibility of nearly 500 articlespublished in the journalManagement Sciencebefore and after the introduction of a new Data andCode Disclosure policy in 2019. When considering only articles for which data accessibility andhard- and software requirements were not an obstacle for reviewers, the results of more than 95%of articles under the new disclosure policy could be fully or largely computationally reproduced.However, for almost 29% of articles at least part of the dataset was not accessible for the reviewer.Considering all articles in our sample reduces the share of reproduced articles to 68%. Theintroduction of the disclosure policy increased reproducibility significantly, since only 12% of articlesaccepted before the introduction of the disclosure policy voluntarily provided replication materials,out of which 55% could be (largely) reproduced. Substantial heterogeneity in reproducibility ratesacross different fields is mainly driven by differences in dataset accessibility. Other reasons forunsuccessful reproduction attempts include missing code, unresolvable code errors, weak or missingdocumentation, but also soft- and hardware requirements and code complexity. Our findingshighlight the importance of journal code and data disclosure policies, and suggest potential avenuesfor enhancing their effectiveness.