The Challenges of Replicating Volatile Platform-Data Studies: Replicating Schatto-Eckrodt et al. (2020)

Abstract

Replication studies in computational communication science (CCS) play a vital role in upholding research validity, ensuring reliability, and promoting transparency. However, conducting such studies in CCS often proves challenging due to the data environments' dynamic nature and the complexities surrounding data and software sharing. To shed light on these challenges, we examine the replication process with CCS studies by computationally reproducing and replicating Schatto-Eckrodt et al.'s (2020) computational analysis of the X (formerly Twitter) debate about the term "gaming disorder" being added to the International Classification of Diseases 11. Our results indicate a reproduction success rate of 88.46% of the original findings. Replicating the analysis presents several obstacles, particularly in data access and availability. Five years after the original data collection, we were able to recollect only 55.08% of the initial sample, primarily due to user and platform activities, including account deletions, user suspensions, and privacy settings. Our reproduction and replication efforts revealed intricate challenges in conducting CCS research, particularly concerning data access and third-party platforms. To enhance replication in CCS, we emphasize the crucial role of data sharing, increased transparency, extensive documentation, and regulatory processes. Thus, our analysis underscores replications' critical role in enhancing CCS research validity and reliability.