A Multi-Stage Deep Learning Approach Incorporating Text-Image and Image-Image Comparisons for Cheapfake Detection

The advancement of multimedia and artificial intelligence (AI) technologies has dismantled the barriers of information sharing, yet it has also ushered in a double-edged sword: a surge in the spread of fake information. In this context, there is a growing need for research on the detection of ‘cheapfakes,’ which are low-cost fake media, known for their ease of creation. This paper proposes a multi-stage deep learning process designed to effectively detect the diverse and rapidly evolving nature of cheapfakes. A single-step deep learning model faces limitations in distinguishing various types of cheapfakes, necessitating the application of a complex deep learning model approach to detect subtle Out-of-Context (OOC) phenomena. This study employs models based on Bidirectional Encoder Representations from Transformers (BERT) and stable diffusion technologies to approach cheapfake detection. Through the ACM ICMR 2024 challenge, the performance of this model was evaluated on a real dataset, achieving an accuracy of 71.9% in Task 1, an improvement of 7% over previous methods, and an accuracy of 55.7% in Task 2. These results are expected to make a significant contribution to the development of strategies for creating and countering cheapfakes. Additionally, this research aims to contribute to the detection of OOC media misuse through this challenge.