ADAM: Web anomaly detection assistant based on feature matrix

Importance of web security cannot be overemphasized in the era of web-based economy. Although anomaly detection has long been considered a promising alternative to signature-based misuse detection technique, most studies to date used either small scale or artificially generated attack data. In this paper, based on security analysis applied on anonymous www.microsoft.com log of about 250GB, we propose Anomaly Feature Matrix (AFM) as an effective framework to characterize anomalies. Feature selection of AFM is based on the characteristics of well-known (e.g., DDoS) attacks as well as patterns of anomalous logs found in the Microsoft data. Independent security analysis performed on the same data by Microsoft security engineers concluded that 1) We did not miss any major attacks; and 2) AFM is a general enough framework to characterize likely web attacks. In order to assist AFM-based anomaly analysis in large organizations, we implemented an interactive and visual analysis tool named ADAM (Anomaly Detection Assistant based on feature Matrix). Integrated with mapping software such as Virtual Earth, ADAM enables efficient and focused security analysis on web logs.