Short-text topic modeling via non-negative matrix factorization enriched with local word-context correlations

Tian Shi; Kyeongpil Kang; Jaegul Choo; Chandan K. Reddy

doi:10.1145/3178876.3186009

Short-text topic modeling via non-negative matrix factorization enriched with local word-context correlations

Tian Shi, Kyeongpil Kang, Jaegul Choo, Chandan K. Reddy

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

129 Citations (Scopus)

Abstract

Being a prevalent form of social communications on the Internet, billions of short texts are generated everyday. Discovering knowledge from them has gained a lot of interest from both industry and academia. The short texts have a limited contextual information, and they are sparse, noisy and ambiguous, and hence, automatically learning topics from them remains an important challenge. To tackle this problem, in this paper, we propose a semantics-assisted non-negative matrix factorization (SeaNMF) model to discover topics for the short texts. It effectively incorporates the word-context semantic correlations into the model, where the semantic relationships between the words and their contexts are learned from the skip-gram view of the corpus. The SeaNMF model is solved using a block coordinate descent algorithm. We also develop a sparse variant of the SeaNMF model which can achieve a better model interpretability. Extensive quantitative evaluations on various real-world short text datasets demonstrate the superior performance of the proposed models over several other state-of-the-art methods in terms of topic coherence and classification accuracy. The qualitative semantic analysis demonstrates the interpretability of our models by discovering meaningful and consistent topics. With a simple formulation and the superior performance, SeaNMF can be an effective standard topic model for short texts.

Original language	English
Title of host publication	The Web Conference 2018 - Proceedings of the World Wide Web Conference, WWW 2018
Publisher	Association for Computing Machinery, Inc
Pages	1105-1114
Number of pages	10
ISBN (Electronic)	9781450356398
DOIs	https://doi.org/10.1145/3178876.3186009
Publication status	Published - 2018 Apr 10
Event	27th International World Wide Web, WWW 2018 - Lyon, France Duration: 2018 Apr 23 → 2018 Apr 27

Publication series

Name	The Web Conference 2018 - Proceedings of the World Wide Web Conference, WWW 2018

Conference

Conference	27th International World Wide Web, WWW 2018
Country/Territory	France
City	Lyon
Period	18/4/23 → 18/4/27

Bibliographical note

Funding Information:
As we can see from Fig. 2, all the graphs for the standard NMF model are very sparse. Some keywords with higher frequency in the corpus have lower degree which means that they are less correlated with the other words. For example, the frequency of ‘chicken’ is high, however, its most correlated words do not contain the other keywords and it is not in the most correlated word lists of the other keywords. In the standard topic modeling, these keywords might be viewed as noise. In Table 6, the keywords with degree less than two are colored in red. We can see that the topics obtained from the standard NMF model are noisy. On the other hand, we conduct the same experiments on our SeaNMF model. From Table 6 and Fig. 2, we can see that topics discovered by our SeaNMF model have less noisy words and the top keywords are more correlated. Therefore, these semantic analysis results demonstrate that the SeaNMF model can discover meaningful and consistent topics for short texts. 5 CONCLUSION In this paper, we introduce a semantics-assisted NMF (SeaNMF) model to discover topics for the short texts. The proposed model leverages the word-context semantic correlations in the training, which potentially overcomes the problem of lacking context that arises due to the data sparsity. The semantic correlations between the words and their contexts are learned from the skip-gram view of the corpus, which was demonstrated to be effective for revealing word semantic relationships. We use a block coordinate descent algorithm to solve our SeaNMF model. To achieve a better model interpretability, a sparse SeaNMF model is also developed. We compared the performance of our models with several other state-of-the-art methods on four real-world short text datasets. The quantitative evaluations demonstrate that our models outperform other methods with respect to widely used metrics such as the topic coherence and document classification accuracy. The parameter sensitivity results demonstrate the stability and consistency of the performance of our SeaNMF model. The qualitative results show that the topics discovered by SeaNMF are meaningful and their top keywords are more semantically correlated. Hence, we conclude that the proposed SeaNMF is an effective topic model for short texts. ACKNOWLEDGMENTS This work was supported in part by the National Science Foundation grants IIS-1619028, IIS-1707498 and IIS-1646881, and the National Research Foundation of Korea (NRF) grant funded by the Korean government (MSIP) (No. NRF-2016R1C1B2015924).

Funding Information:
Thiswork was supported in part by the National Science Foundation grants IIS-1619028, IIS-1707498 and IIS-1646881, and the National Research Foundation of Korea (NRF) grant funded by the Korean government (MSIP) (No. NRF-2016R1C1B2015924).

Publisher Copyright:
© 2018 IW3C2 (International World Wide Web Conference Committee), published under Creative Commons CC BY 4.0 License.

Keywords

Non-negative matrix factorization
Short texts
Topic modeling
Word embedding

ASJC Scopus subject areas

Computer Networks and Communications
Software

Access to Document

10.1145/3178876.3186009

Cite this

Shi, T., Kang, K., Choo, J., & Reddy, C. K. (2018). Short-text topic modeling via non-negative matrix factorization enriched with local word-context correlations. In The Web Conference 2018 - Proceedings of the World Wide Web Conference, WWW 2018 (pp. 1105-1114). (The Web Conference 2018 - Proceedings of the World Wide Web Conference, WWW 2018). Association for Computing Machinery, Inc. https://doi.org/10.1145/3178876.3186009

Short-text topic modeling via non-negative matrix factorization enriched with local word-context correlations. / Shi, Tian; Kang, Kyeongpil; Choo, Jaegul et al.
The Web Conference 2018 - Proceedings of the World Wide Web Conference, WWW 2018. Association for Computing Machinery, Inc, 2018. p. 1105-1114 (The Web Conference 2018 - Proceedings of the World Wide Web Conference, WWW 2018).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Shi, T, Kang, K, Choo, J & Reddy, CK 2018, Short-text topic modeling via non-negative matrix factorization enriched with local word-context correlations. in The Web Conference 2018 - Proceedings of the World Wide Web Conference, WWW 2018. The Web Conference 2018 - Proceedings of the World Wide Web Conference, WWW 2018, Association for Computing Machinery, Inc, pp. 1105-1114, 27th International World Wide Web, WWW 2018, Lyon, France, 18/4/23. https://doi.org/10.1145/3178876.3186009

Shi T, Kang K, Choo J, Reddy CK. Short-text topic modeling via non-negative matrix factorization enriched with local word-context correlations. In The Web Conference 2018 - Proceedings of the World Wide Web Conference, WWW 2018. Association for Computing Machinery, Inc. 2018. p. 1105-1114. (The Web Conference 2018 - Proceedings of the World Wide Web Conference, WWW 2018). doi: 10.1145/3178876.3186009

Shi, Tian ; Kang, Kyeongpil ; Choo, Jaegul et al. / Short-text topic modeling via non-negative matrix factorization enriched with local word-context correlations. The Web Conference 2018 - Proceedings of the World Wide Web Conference, WWW 2018. Association for Computing Machinery, Inc, 2018. pp. 1105-1114 (The Web Conference 2018 - Proceedings of the World Wide Web Conference, WWW 2018).

@inproceedings{5fcb17020f0d4e588475bbaf71489ecb,

title = "Short-text topic modeling via non-negative matrix factorization enriched with local word-context correlations",

abstract = "Being a prevalent form of social communications on the Internet, billions of short texts are generated everyday. Discovering knowledge from them has gained a lot of interest from both industry and academia. The short texts have a limited contextual information, and they are sparse, noisy and ambiguous, and hence, automatically learning topics from them remains an important challenge. To tackle this problem, in this paper, we propose a semantics-assisted non-negative matrix factorization (SeaNMF) model to discover topics for the short texts. It effectively incorporates the word-context semantic correlations into the model, where the semantic relationships between the words and their contexts are learned from the skip-gram view of the corpus. The SeaNMF model is solved using a block coordinate descent algorithm. We also develop a sparse variant of the SeaNMF model which can achieve a better model interpretability. Extensive quantitative evaluations on various real-world short text datasets demonstrate the superior performance of the proposed models over several other state-of-the-art methods in terms of topic coherence and classification accuracy. The qualitative semantic analysis demonstrates the interpretability of our models by discovering meaningful and consistent topics. With a simple formulation and the superior performance, SeaNMF can be an effective standard topic model for short texts.",

keywords = "Non-negative matrix factorization, Short texts, Topic modeling, Word embedding",

author = "Tian Shi and Kyeongpil Kang and Jaegul Choo and Reddy, {Chandan K.}",

note = "Funding Information: As we can see from Fig. 2, all the graphs for the standard NMF model are very sparse. Some keywords with higher frequency in the corpus have lower degree which means that they are less correlated with the other words. For example, the frequency of {\textquoteleft}chicken{\textquoteright} is high, however, its most correlated words do not contain the other keywords and it is not in the most correlated word lists of the other keywords. In the standard topic modeling, these keywords might be viewed as noise. In Table 6, the keywords with degree less than two are colored in red. We can see that the topics obtained from the standard NMF model are noisy. On the other hand, we conduct the same experiments on our SeaNMF model. From Table 6 and Fig. 2, we can see that topics discovered by our SeaNMF model have less noisy words and the top keywords are more correlated. Therefore, these semantic analysis results demonstrate that the SeaNMF model can discover meaningful and consistent topics for short texts. 5 CONCLUSION In this paper, we introduce a semantics-assisted NMF (SeaNMF) model to discover topics for the short texts. The proposed model leverages the word-context semantic correlations in the training, which potentially overcomes the problem of lacking context that arises due to the data sparsity. The semantic correlations between the words and their contexts are learned from the skip-gram view of the corpus, which was demonstrated to be effective for revealing word semantic relationships. We use a block coordinate descent algorithm to solve our SeaNMF model. To achieve a better model interpretability, a sparse SeaNMF model is also developed. We compared the performance of our models with several other state-of-the-art methods on four real-world short text datasets. The quantitative evaluations demonstrate that our models outperform other methods with respect to widely used metrics such as the topic coherence and document classification accuracy. The parameter sensitivity results demonstrate the stability and consistency of the performance of our SeaNMF model. The qualitative results show that the topics discovered by SeaNMF are meaningful and their top keywords are more semantically correlated. Hence, we conclude that the proposed SeaNMF is an effective topic model for short texts. ACKNOWLEDGMENTS This work was supported in part by the National Science Foundation grants IIS-1619028, IIS-1707498 and IIS-1646881, and the National Research Foundation of Korea (NRF) grant funded by the Korean government (MSIP) (No. NRF-2016R1C1B2015924). Funding Information: Thiswork was supported in part by the National Science Foundation grants IIS-1619028, IIS-1707498 and IIS-1646881, and the National Research Foundation of Korea (NRF) grant funded by the Korean government (MSIP) (No. NRF-2016R1C1B2015924). Publisher Copyright: {\textcopyright} 2018 IW3C2 (International World Wide Web Conference Committee), published under Creative Commons CC BY 4.0 License.; 27th International World Wide Web, WWW 2018 ; Conference date: 23-04-2018 Through 27-04-2018",

year = "2018",

month = apr,

day = "10",

doi = "10.1145/3178876.3186009",

language = "English",

series = "The Web Conference 2018 - Proceedings of the World Wide Web Conference, WWW 2018",

publisher = "Association for Computing Machinery, Inc",

pages = "1105--1114",

booktitle = "The Web Conference 2018 - Proceedings of the World Wide Web Conference, WWW 2018",

}

TY - GEN

T1 - Short-text topic modeling via non-negative matrix factorization enriched with local word-context correlations

AU - Shi, Tian

AU - Kang, Kyeongpil

AU - Choo, Jaegul

AU - Reddy, Chandan K.

N1 - Funding Information: As we can see from Fig. 2, all the graphs for the standard NMF model are very sparse. Some keywords with higher frequency in the corpus have lower degree which means that they are less correlated with the other words. For example, the frequency of ‘chicken’ is high, however, its most correlated words do not contain the other keywords and it is not in the most correlated word lists of the other keywords. In the standard topic modeling, these keywords might be viewed as noise. In Table 6, the keywords with degree less than two are colored in red. We can see that the topics obtained from the standard NMF model are noisy. On the other hand, we conduct the same experiments on our SeaNMF model. From Table 6 and Fig. 2, we can see that topics discovered by our SeaNMF model have less noisy words and the top keywords are more correlated. Therefore, these semantic analysis results demonstrate that the SeaNMF model can discover meaningful and consistent topics for short texts. 5 CONCLUSION In this paper, we introduce a semantics-assisted NMF (SeaNMF) model to discover topics for the short texts. The proposed model leverages the word-context semantic correlations in the training, which potentially overcomes the problem of lacking context that arises due to the data sparsity. The semantic correlations between the words and their contexts are learned from the skip-gram view of the corpus, which was demonstrated to be effective for revealing word semantic relationships. We use a block coordinate descent algorithm to solve our SeaNMF model. To achieve a better model interpretability, a sparse SeaNMF model is also developed. We compared the performance of our models with several other state-of-the-art methods on four real-world short text datasets. The quantitative evaluations demonstrate that our models outperform other methods with respect to widely used metrics such as the topic coherence and document classification accuracy. The parameter sensitivity results demonstrate the stability and consistency of the performance of our SeaNMF model. The qualitative results show that the topics discovered by SeaNMF are meaningful and their top keywords are more semantically correlated. Hence, we conclude that the proposed SeaNMF is an effective topic model for short texts. ACKNOWLEDGMENTS This work was supported in part by the National Science Foundation grants IIS-1619028, IIS-1707498 and IIS-1646881, and the National Research Foundation of Korea (NRF) grant funded by the Korean government (MSIP) (No. NRF-2016R1C1B2015924). Funding Information: Thiswork was supported in part by the National Science Foundation grants IIS-1619028, IIS-1707498 and IIS-1646881, and the National Research Foundation of Korea (NRF) grant funded by the Korean government (MSIP) (No. NRF-2016R1C1B2015924). Publisher Copyright: © 2018 IW3C2 (International World Wide Web Conference Committee), published under Creative Commons CC BY 4.0 License.

PY - 2018/4/10

Y1 - 2018/4/10

N2 - Being a prevalent form of social communications on the Internet, billions of short texts are generated everyday. Discovering knowledge from them has gained a lot of interest from both industry and academia. The short texts have a limited contextual information, and they are sparse, noisy and ambiguous, and hence, automatically learning topics from them remains an important challenge. To tackle this problem, in this paper, we propose a semantics-assisted non-negative matrix factorization (SeaNMF) model to discover topics for the short texts. It effectively incorporates the word-context semantic correlations into the model, where the semantic relationships between the words and their contexts are learned from the skip-gram view of the corpus. The SeaNMF model is solved using a block coordinate descent algorithm. We also develop a sparse variant of the SeaNMF model which can achieve a better model interpretability. Extensive quantitative evaluations on various real-world short text datasets demonstrate the superior performance of the proposed models over several other state-of-the-art methods in terms of topic coherence and classification accuracy. The qualitative semantic analysis demonstrates the interpretability of our models by discovering meaningful and consistent topics. With a simple formulation and the superior performance, SeaNMF can be an effective standard topic model for short texts.

AB - Being a prevalent form of social communications on the Internet, billions of short texts are generated everyday. Discovering knowledge from them has gained a lot of interest from both industry and academia. The short texts have a limited contextual information, and they are sparse, noisy and ambiguous, and hence, automatically learning topics from them remains an important challenge. To tackle this problem, in this paper, we propose a semantics-assisted non-negative matrix factorization (SeaNMF) model to discover topics for the short texts. It effectively incorporates the word-context semantic correlations into the model, where the semantic relationships between the words and their contexts are learned from the skip-gram view of the corpus. The SeaNMF model is solved using a block coordinate descent algorithm. We also develop a sparse variant of the SeaNMF model which can achieve a better model interpretability. Extensive quantitative evaluations on various real-world short text datasets demonstrate the superior performance of the proposed models over several other state-of-the-art methods in terms of topic coherence and classification accuracy. The qualitative semantic analysis demonstrates the interpretability of our models by discovering meaningful and consistent topics. With a simple formulation and the superior performance, SeaNMF can be an effective standard topic model for short texts.

KW - Non-negative matrix factorization

KW - Short texts

KW - Topic modeling

KW - Word embedding

UR - http://www.scopus.com/inward/record.url?scp=85052983968&partnerID=8YFLogxK

U2 - 10.1145/3178876.3186009

DO - 10.1145/3178876.3186009

M3 - Conference contribution

AN - SCOPUS:85052983968

T3 - The Web Conference 2018 - Proceedings of the World Wide Web Conference, WWW 2018

SP - 1105

EP - 1114

BT - The Web Conference 2018 - Proceedings of the World Wide Web Conference, WWW 2018

PB - Association for Computing Machinery, Inc

T2 - 27th International World Wide Web, WWW 2018

Y2 - 23 April 2018 through 27 April 2018

ER -

Short-text topic modeling via non-negative matrix factorization enriched with local word-context correlations

Abstract

Publication series

Conference

Bibliographical note

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this