Topic and Opinion Classification based Information Credibility Analysis on Twitter

Topic and Opinion Classification based Information Credibility Analysis on Twitter Yukino Ikegami Kenta Kawai Yoshimi Namihira Setsuo Tsuruta At SMC 2013 2013/10/16 1

Background and Motivation • False rumors often confuse people • Confirming reliability of rumors often requires a domain knowledge about the problem ➢Automatically Information credibility analysis 2013/10/16 2

Related Work (1) Using Web-page-dependent features • [Wassmer et al., 2005] – Use credentials of the site, advertisements and Web design • [Castillo et al. 2011] – Twitter-dependent features • E.g. number of followers – Twitter-independent features • E.g. number of !/? 2013/10/16 3

Related Work (2) Using textual features • Rumor information cloud system [Miyabe et al. 2011] – Confirm a rumor whether is truth or not by alerting information about a rumor – Find correcting information by SVM applying word ngrams model – The word n-gram model consists of words in front and back of the word “デマ” (“dema” is the abbreviation of “demagogic” in Japanese-English). • Dematter [Toriumi et al. 2012] – Assesse credibility by the percentage of alerting tweets about a rumor – Detect alerting tweets by keyword matching 2013/10/16 4

Topic and Opinion Classification based Information Credibility Analysis Tweet crawler Topic & opinion classifier Tweet opinion DB Twitter Tweet credibility calculator 2013/10/16 5

Topic classification • Classify tweet by topic model – Topic model: Latent Dirichlet Allocation (LDA) with Gibbs sampling [Griffiths, 2002] – Feature: content words (i.e.) noun, verb, adjective, adverb Topic1 Topic2 Topic3 Vegetable Measure Radioactive material Eat Amount of radiation In prefecture No problem Result Governor Leaf of tea Pool Fukushima 2013/10/16 6

Opinion Classification • Classify whether a tweet is positive opinion or negative one by a dictionary • Takamura’s semantic orientation dictionary [Takamura et al. 2006] – Contains word-positivity [-1, 1] pairs 2013/10/16 7

Information Credibility Assessment • Majority decision Negative All tweets Positive Tweets about the interest topic 2013/10/16 8

Evaluation • Dataset: 2960 tweets – Confirmed whether it is true or not by human • Criteria: Weighted kappa – Weight w is designed as follows: judging certainly false-information as certainly true or vice versa are critical error 2013/10/16 9

Result TABLE 1: Kappa of each conditions Fully Random method (All tweets) Our method (All tweets) Our method (Only Topic & Opinion correct) 0.003 0.604 0.616 • Landis’s kappa guideline: κ > 0.61 is substantial • Our method has the substantial effectiveness for assessing tweet credibility 2013/10/16 10

Conclusion • Topic and opinion classification based information analysis on Twitter – Topic model and sentiment analysis based majority decision • Evaluation shows it has substantial effect 2013/10/16 11

Future works • Weighting tweets by author’s expertise – people often determine whether information is trustworthy or not by author’s expertise • Applying online topic model – New topic and usage of existing words are created one after another • Excluding neutral tweets – No-sentiment tweets are useless on our method 2013/10/16 12

References • [Wassmer et al. 2015] M. Wassmer and C. Eastman, “Automatic evaluation of credibility on the Web,” ASIS&T 2005, 42(1), 2005. • [Castillo et al. 2011] Castillo, C., Mendoza, M., and Poblete, B. “Information credibility on twitter,” WWW 2011, pp. 675684, 2011. • [Miyabe et al. 2005] M. Miyabe, A. Umejima, A. Nadamoto and E. Aramaki, “Proposal of Rumor Information Cloud based on Rumor-Correction Information” (In Japanese), RRDS4-019, 2011. • [Toriumi et al. 2006] F. Toriumi, K. Shinoda, G. Kaneyama, “Accuracy Evaluation of Dema- gogue Detection System using Social Media” (In Japanese), IPSJ Digital Practice, 3.3, pp. 201-208, 2012. 2013/10/16 13