Toward detecting advanced persistent threat using malicious non-executable files

Advanced Persistent Threat (APT) attacks vulnerable applications in client PC using social engineering and steals its information secretly. Most of APT controls the client's system by executing malicious code in non-executable file that the applications read and parse. In this paper, we propose a novel technique to detect a malicious non-executable file regardless of file format By regarding all nonexecutable files as byte sequences and executing the sequences forcibly from beginning to end, we target on detecting only an executable code in the byte sequences. Because it takes a long time to execute all bytes, we select suspicious bytes in a file using patterns of invalid instructions. We implement a tool to evaluate our idea using a debugger engine to change flow of execution freely. The experimental results show mat our idea is effective. Our idea can prevent APT by detecting malicious files in Honeynet or an email server beforehand.