Visual relationships provide higher-level information of objects and their relations in an image - this enables a semantic understanding of the scene and helps downstream applications. Given a set of localized objects in some training data, visual relationship detection seeks to detect the most likely 'relationship' between objects in a given image. While the specific objects may be well represented in training data, their relationships may still be infrequent. The empirical distribution obtained from seeing these relationships in a dataset does not model the underlying distribution well - a serious issue for most learning methods. In this work, we start from a simple multi-relational learning model, which in principle, offers a rich formalization for deriving a strong prior for learning visual relationships. While the inference problem for deriving the regularizer is challenging, our main technical contribution is to show how adapting recent results in numerical linear algebra lead to efficient algorithms for a factorization scheme that yields highly informative priors. The factorization provides sample size bounds for inference (under mild conditions) for the underlying [object, predicate, object] relationship learning task on its own and surprisingly outperforms (in some cases) existing methods even without utilizing visual features. Then, when integrated with an end-to-end architecture for visual relationship detection leveraging image data, we substantially improve the state-of-the-art.
|Title of host publication
|Proceedings - 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2018
|IEEE Computer Society
|Number of pages
|Published - 2018 Dec 14
|31st Meeting of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2018 - Salt Lake City, United States
Duration: 2018 Jun 18 → 2018 Jun 22
|Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition
|31st Meeting of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2018
|Salt Lake City
|18/6/18 → 18/6/22
Bibliographical notePublisher Copyright:
© 2018 IEEE.
ASJC Scopus subject areas
- Computer Vision and Pattern Recognition