download521 view759
twitter facebook

공공누리This item is licensed Korea Open Government License

Title
Empirical Comparison of Word Similarity Measures Based on Co-Occurrence, Context, and a Vector Space Model
Author(s)
Kadowaki, NatsukiKishida, Kazuaki
Publisher
Korea Institute of Science and Technology Information
Publication Year
2020-06-30
Abstract
Word similarity is often measured to enhance system performance in the information retrieval field and other related areas. This paper reports on an experimental comparison of values for word similarity measures that were computed based on 50 intentionally selected words from a Reuters corpus. There were three targets, including (1) co-occurrence-based similarity measures (for which a co-occurrence frequency is counted as the number of documents or sentences), (2) context-based distributional similarity measures obtained from a latent Dirichlet allocation (LDA), nonnegative matrix factorization (NMF), and Word2Vec algorithm, and (3) similarity measures computed from the tf-idf weights of each word according to a vector space model (VSM). Here, a Pearson correlation coefficient for a pair of VSM-based similarity measures and co-occurrence-based similarity measures according to the number of documents was highest. Group-average agglomerative hierarchical clustering was also applied to similarity matrices computed by individual measures. An evaluation of the cluster sets according to an answer set revealed that VSM- and LDA-based similarity measures performed best.
Keyword
Word similarity; Word clustering; Topic model; Word embedding
Journal Title
Journal of Information Science Theory and Practice;Volume 8 Issue 2
ISSN
2287-4577
Files in This Item:
Thumbnail 2020 JISTaP 8(2)-6-17.pdf468.56 kBDownload
Appears in Collections:
8. KISTI 간행물 > JISTaP > Vol. 8 - No. 2
Type
Serial
URI
https://repository.kisti.re.kr/handle/10580/15504
Export
RIS (EndNote)
XLS (Excel)
XML

Browse