TY - JOUR
T1 - Investigation and evaluation of randomized controlled trials for interventions involving artificial intelligence
AU - Wang, Jianjian
AU - Wu, Shouyuan
AU - Guo, Qiangqiang
AU - Lan, Hui
AU - Janne, Estill
AU - Wang, Ling
AU - Zhang, Juanjuan
AU - Wang, Qi
AU - Song, Yang
AU - Yang, Nan
AU - Luo, Xufei
AU - Zhou, Qi
AU - Shi, Qianling
AU - Yu, Xuan
AU - Ma, Yanfang
AU - Mathew, Joseph L.
AU - Ahn, Hyeong Sik
AU - Lee, Myeong Soo
AU - Chen, Yaolong
N1 - Publisher Copyright:
© 2021 Chinese Medical Association
PY - 2021/8
Y1 - 2021/8
N2 - Objective Complete and transparent reporting is of critical importance for randomized controlled trials (RCTs). The present study aimed to determine the reporting quality and methodological quality of RCTs for interventions involving artificial intelligence (AI) and their protocols. Methods We searched MEDLINE (via PubMed), Embase, Web of Science, CBMdisc, Wanfang Data, and CNKI from January 1, 2016, to November 11, 2020, to collect RCTs involving AI. We also extracted the protocol of each included RCT if it could be obtained. CONSORT-AI (Consolidated Standards of Reporting Trials–Artificial Intelligence) statement and Cochrane Collaboration's tool for assessing risk of bias (ROB) were used to evaluate the reporting quality and methodological quality, respectively, and SPIRIT-AI (The Standard Protocol Items: Recommendations for Interventional Trials–Artificial Intelligence) statement was used to evaluate the reporting quality of the protocols. The associations of the reporting rate of CONSORT-AI with the publication year, journal's impact factor (IF), number of authors, sample size, and first author's country were analyzed univariately using Pearson's chi-squared test, or Fisher's exact test if the expected values in any of the cells were below 5. The compliance of the retrieved protocols to SPIRIT-AI was presented descriptively. Results Overall, 29 RCTs and three protocols were considered eligible. The CONSORT-AI items “title and abstract” and “interpretation of results” were reported by all RCTs, with the items with the lowest reporting rates being “funding” (0), “implementation” (3.5%), and “harms” (3.5%). The risk of bias was high in 13 (44.8%) RCTs and not clear in 15 (51.7%) RCTs. Only one RCT (3.5%) had a low risk of bias. The compliance was not significantly different in terms of the publication year, journal's IF, number of authors, sample size, or first author's country. Ten of the 35 SPIRIT-AI items (funding, participant timeline, allocation concealment mechanism, implementation, data management, auditing, declaration of interests, access to data, informed consent materials and biological specimens) were not reported by any of the three protocols. Conclusions The reporting and methodological quality of RCTs involving AI need to be improved. Because of the limited availability of protocols, their quality could not be fully judged. Following the CONSORT-AI and SPIRIT-AI statements and with appropriate guidance on the risk of bias when designing and reporting AI-related RCTs can promote standardization and transparency.
AB - Objective Complete and transparent reporting is of critical importance for randomized controlled trials (RCTs). The present study aimed to determine the reporting quality and methodological quality of RCTs for interventions involving artificial intelligence (AI) and their protocols. Methods We searched MEDLINE (via PubMed), Embase, Web of Science, CBMdisc, Wanfang Data, and CNKI from January 1, 2016, to November 11, 2020, to collect RCTs involving AI. We also extracted the protocol of each included RCT if it could be obtained. CONSORT-AI (Consolidated Standards of Reporting Trials–Artificial Intelligence) statement and Cochrane Collaboration's tool for assessing risk of bias (ROB) were used to evaluate the reporting quality and methodological quality, respectively, and SPIRIT-AI (The Standard Protocol Items: Recommendations for Interventional Trials–Artificial Intelligence) statement was used to evaluate the reporting quality of the protocols. The associations of the reporting rate of CONSORT-AI with the publication year, journal's impact factor (IF), number of authors, sample size, and first author's country were analyzed univariately using Pearson's chi-squared test, or Fisher's exact test if the expected values in any of the cells were below 5. The compliance of the retrieved protocols to SPIRIT-AI was presented descriptively. Results Overall, 29 RCTs and three protocols were considered eligible. The CONSORT-AI items “title and abstract” and “interpretation of results” were reported by all RCTs, with the items with the lowest reporting rates being “funding” (0), “implementation” (3.5%), and “harms” (3.5%). The risk of bias was high in 13 (44.8%) RCTs and not clear in 15 (51.7%) RCTs. Only one RCT (3.5%) had a low risk of bias. The compliance was not significantly different in terms of the publication year, journal's IF, number of authors, sample size, or first author's country. Ten of the 35 SPIRIT-AI items (funding, participant timeline, allocation concealment mechanism, implementation, data management, auditing, declaration of interests, access to data, informed consent materials and biological specimens) were not reported by any of the three protocols. Conclusions The reporting and methodological quality of RCTs involving AI need to be improved. Because of the limited availability of protocols, their quality could not be fully judged. Following the CONSORT-AI and SPIRIT-AI statements and with appropriate guidance on the risk of bias when designing and reporting AI-related RCTs can promote standardization and transparency.
KW - Artificial intelligence
KW - Methodological quality
KW - Randomized controlled trials
KW - Reporting quality
UR - http://www.scopus.com/inward/record.url?scp=85134446068&partnerID=8YFLogxK
U2 - 10.1016/j.imed.2021.04.006
DO - 10.1016/j.imed.2021.04.006
M3 - Article
AN - SCOPUS:85134446068
SN - 2096-9376
VL - 1
SP - 61
EP - 69
JO - Intelligent Medicine
JF - Intelligent Medicine
IS - 2
ER -