On the efficacy of per-relation basis performance evaluation for PPI extraction and a high-precision rule-based approach

Junkyu Lee, Seongsoon Kim, Sunwon Lee, Kyubum Lee, Jaewoo Kang

Research output: Contribution to journalArticlepeer-review

7 Citations (Scopus)


Background: Most previous Protein Protein Interaction (PPI) studies evaluated their algorithms' performance based on «per-instance» precision and recall, in which the instances of an interaction relation were evaluated independently. However, we argue that this standard evaluation method should be revisited. In a large corpus, the same relation can be described in various different forms and, in practice, correctly identifying not all but a small subset of them would often suffice to detect the given interaction. Methods. In this regard, we propose a more pragmatic «per-relation» basis performance evaluation method instead of the conventional per-instance basis method. In the per-relation basis method, only a subset of a relation's instances needs to be correctly identified to make the relation positive. In this work, we also introduce a new high-precision rule-based PPI extraction algorithm. While virtually all current PPI extraction studies focus on improving F-score, aiming to balance the performance on both precision and recall, in many realistic scenarios involving large corpora, one can benefit more from a high-precision algorithm than a high-recall counterpart. Results: We show that our algorithm not only achieves better per-relation performance than previous solutions but also serves as a good complement to the existing PPI extraction tools. Our algorithm improves the performance of the existing tools through simple pipelining. Conclusion: The significance of this research can be found in that this research brought new perspective to the performance evaluation of PPI extraction studies, which we believe is more important in practice than existing evaluation criteria. Given the new evaluation perspective, we also showed the importance of a high-precision extraction tool and validated the efficacy of our rule-based system as the high-precision tool candidate.

Original languageEnglish
Article numberS7
JournalBMC Medical Informatics and Decision Making
Issue numberSUPPL1
Publication statusPublished - 2013

Bibliographical note

Funding Information:
This research was supported by the National Research Foundation of Korea (NRF) grant (2012R1A2A2A01014729) and the Next-Generation Information Computing Development Program through the NRF (2012M3C4A7033341) funded by the Ministry of Education, Science and Technology. This work is based on an earlier work: “High precision rule based PPI extraction and per-pair basis performance evaluation”, in Proceedings of the ACM Sixth International Workshop on Data and Text Mining in Biomedical Informatics, 2012 © ACM, 2012. http://doi.acm.org/10.1145/2390068.2390082

ASJC Scopus subject areas

  • Health Policy
  • Health Informatics
  • Computer Science Applications


Dive into the research topics of 'On the efficacy of per-relation basis performance evaluation for PPI extraction and a high-precision rule-based approach'. Together they form a unique fingerprint.

Cite this