Detection of Football Spoilers on Twitter

1

SNSs SNSs are widely prevalent as a communication tool. 2

SNSs SNSs are widely prevalent as a communication tool. For example Message exchange, browsing a friend's doing now and so on. 3

Sharing Events in Real-time 4

Sharing Events in Real-time 5

Sharing Events in Real-time 6

Spoiler Problem Recording 7 I have to go to work!!

Spoiler Problem Going back to home... 8 I wonder what Sam is doing now.

Spoiler Problem Lando Calrissian is return!!!!! Spoiler!! Obi-Wan Kenobi is appear!!! Optimistic people want to hear the bad news first, while pessimists ask for the good. Realists just start drinking. I don’t have a dirty mind, I have a sexy imagination. 9

Spoiler Problem Not the best result we expected. Sports is very shocking. Bélgica 0-0 Japón Anything could happen. Tonight is the night. I’ll keep my fingers crossed for Belgian always look down Japanese. This is kalma. 10

Example(2) 11 By another interested event The festival seems interesting!

Example(2) They will easily encounter spoilers. 12

Example(3) 13 By push notifications @you My baby has come!!. Tapped!

Example(3) Spoiler... 14

Example(4) Recording 15 I'll go to bed now. VS 3:00 AM in Japan

Example(4) In the next morning, half asleep... 16

Example(4) Not the best result we expected. Spoiler!! Bélgica 0-0 Japón Anything could happen. Tonight is the night. I’ll keep my fingers crossed for Belgian always look down Japanese. This is kalma. 17

Example(5) Because of time difference 18

Example(5) An example of time difference spoiler. 19

Solution for Spoiler Problem People sometimes stay away from the Internet and choose a self imposed isolation. 20

Solution for Spoiler Problem 21 People sometimes stay away from the Internet and choose a self imposed isolation. It affects the maintenance of relationships with their friends.

Solution for Spoiler Problem 22 People sometimes stay away from the Internet and choose a self imposed isolation. We need automatic method for hiding spoilers. It affects the maintenance of relationships with their friends.

Related Researches Spoiler Alert [Boyd-Graber et al. 2013] targeted movie reviews and conducted an evaluation of machine learning approaches to find spoilers in social media posts. Spoiler detection in TV program tweets [Jeon et al. 2015] proposed a method of detecting spoilers in comments on Twitter about television programs. 23

Related Researches Spoiler Alert [Boyd-Graber et al. 2013] targeted movie reviews and conducted an evaluation of machine learning approaches to find spoilers in social media posts. How to detect sports spoilers on SNSs with high accuracy was Spoiler detection in TVinsufficient program tweets [Jeon et al. 2015] proposed a method of detecting spoilers in comments on Twitter about television programs. 24

[Our goal in this work] Examining methods for detecting football spoilers with high accuracy on Twitter. 25

Contents 1. Generating spoiler dataset 2. Experiment of spoiler detection 26

Generating Spoiler Dataset 27 We collect tweets of 9 games by the Japan national football team. Match Score Day 2015 Women’s World Cup “Japan vs. England” JPN 2 – 1 ENG 07/01/15 2015 Women’s World Cup “Japan vs. United States” JPN 2 – 5 USA 07/05/15 2015 EAFF East Asian Cup “Japan vs. South Korea” JPN 1 – 1 KOR 08/05/15 2015 Women’s EAFF East Asian Cup “Japan vs. China” JPN 2 – 0 CHN 08/08/15 2015 EAFF East Asian Cup “Japan vs. China” JPN 1 – 1 CHN 08/09/15 World Cup Qualifiers “Japan vs. Cambodia” JPN 3 – 0 KHM 09/03/15 World Cup Qualifiers “Japan vs. Afghanistan” JPN 6 – 0 AFG 09/08/15 Friendlies “Japan vs. Iran” JPN 1 – 1 IRI 10/13/15 World Cup Qualifiers “Japan vs. Singapore” JPN 3 – 0 SIN 11/12/15

Generating Spoiler Dataset 5 labelers helped us label the tweets. Evaluation system 28

Generating Spoiler Dataset 5 labelers helped us label the tweets. Evaluation system 29

Generating Spoiler Dataset 30 5 labelers helped us label the tweets. 1000 tweets × 9 games × 5 labelers = 45000 data Evaluation system

Some Examples of The Dataset 31 We defined spoiler’s as tweets labeled more than half number. We got 1651 spoiler tweets. Tweet Elapsed time Label “Ooooh! Kagawa scores a goal！！！” 20 spoiler “Already allowed two goals (´Д` )” 0 spoiler “Now kick off” 0 non-spoiler “Hmm. A missed pass is no good” 20 non-spoiler “I saw a sweeping victory for the first time in a very long time” 120 spoiler

Some Examples of The Dataset 32 We defined spoiler’s as tweets labeled more than half number. We got 1651 spoiler tweets. Tweet “Ooooh! Kagawa scores a goal！！！” Words themselves Elapsed timelike spoiler... Label are 20 spoiler “Already allowed two goals (´Д` )” 0 spoiler “Now kick off” 0 non-spoiler “Hmm. A missed pass is no good” 20 non-spoiler “I saw a sweeping victory for the first time in a very long time” 120 spoiler

Terms Used Frequently in Spoiler Winning Terms 33 Losing TF-IDF Terms Tying TF-IDF Terms TF-IDF [Player] 0.742 [Player] 0.531 [Player] 0.627 [Team] 0.442 [Team] 0.475 [Team] 0.552 0.261 Break the deadlock 0.238 Goal 0.226 [Num]th points 0.173 Allowing goals 0.238 Tie 0.201 [Num]points 0.131 Parry 0.224 Match 0.151 [Num] 0.117 Second 0.112 [Num]-[Num] 0.136 Win 0.106 Score 0.112 End 0.125 Match 0.099 Too 0.112 National 0.110 Goal ※[Player]: player names, [Team]: team names, [Num]: number

Terms Used Frequently in Spoiler Winning Terms 34 Losing TF-IDF Terms Tying TF-IDF Terms TF-IDF [Player] 0.742 [Player] 0.531 [Player] 0.627 [Team] 0.442 [Team] 0.475 [Team] 0.552 0.261 Break the deadlock 0.238 Goal 0.226 [Num]th points 0.173 Allowing goals 0.238 Tie 0.201 [Num]points 0.131 Parry 0.224 Match 0.151 [Num] 0.117 Second 0.112 [Num]-[Num] 0.136 Win 0.106 Score 0.112 End 0.125 Match 0.099 Too 0.112 National 0.110 Goal ※[Player]: player names, [Team]: team names, [Num]: number Some terms have the characteristics of spoiler directly.

Terms Used Frequently in Spoiler Winning Terms 35 Losing TF-IDF Terms Tying TF-IDF Terms TF-IDF [Player] 0.742 [Player] 0.531 [Player] 0.627 [Team] 0.442 [Team] 0.475 [Team] 0.552 0.261 Break the deadlock 0.238 Goal 0.226 [Num]th points 0.173 Allowing goals 0.238 Tie 0.201 [Num]points 0.131 Parry 0.224 Match 0.151 [Num] 0.117 Second 0.112 [Num]-[Num] 0.136 Win 0.106 Score 0.112 End 0.125 Match 0.099 Too 0.112 National 0.110 Goal ※[Player]: player names, [Team]: team names, [Num]: number

Terms Used Frequently in Spoiler Winning Terms 36 Losing TF-IDF Terms Tying TF-IDF Terms TF-IDF [Player] 0.742 [Player] 0.531 [Player] 0.627 [Team] 0.442 [Team] 0.475 [Team] 0.552 0.261 Break the deadlock 0.238 Goal 0.226 [Num]th points 0.173 Allowing goals 0.238 Tie 0.201 [Num]points 0.131 Parry 0.224 Match 0.151 [Num] 0.117 Second 0.112 [Num]-[Num] 0.136 Win 0.106 Score 0.112 End 0.125 Match 0.099 Too 0.112 National 0.110 Goal ※[Player]: player names, [Team]: team names, [Num]: number

Terms Used Frequently in Spoiler Winning Terms 37 Losing TF-IDF Terms Tying TF-IDF Terms TF-IDF [Player] 0.742 [Player] 0.531 [Player] 0.627 [Team] 0.442 [Team] 0.475 [Team] 0.552 0.261 Break the deadlock 0.238 Goal 0.226 [Num]th points 0.173 Allowing goals 0.238 Tie 0.201 [Num]points 0.131 Parry 0.224 Match 0.151 [Num] 0.117 Second 0.112 [Num]-[Num] 0.136 Win 0.106 Score 0.112 End 0.125 Match 0.099 Too 0.112 National 0.110 Goal The terms of spoilers differed depending on the game status.

Experiment of Spoiler Detection 38 We examined methods for detecting spoilers with high accuracy. Compared methods 3 word based methods Evaluation points Precision, Recall, F-measure

Our Proposed method(SVM with Status of Match) 39 The method uses different learning models of SVM depending on game status. Kick-off 0-1 0-2 Game 1: 1-0 Game 2: 1-0 Game 3: 1-1 Full time

Our Proposed method(SVM with Status of Match) 40 The method uses different learning models of SVM depending on game status. Kick-off 0-1 0-2 Game 1: 1-0 Game 2: Winning model 1-1 1-0 Game 3: Winning model Full time

Our Proposed method(SVM with Status of Match) 41 The method uses different learning models of SVM depending on game status. Kick-off 0-1 Game 1: 0-2 Losing model Full time Losing model 1-0 Game 2: Winning model 1-1 1-0 Game 3: Winning model

Our Proposed method(SVM with Status of Match) 42 The method uses different learning models of SVM depending on game status. 0-1 Kick-off Game 1: 0-2 Losing model Tying model Full time Losing model 1-0 Game 2: Tying model 1-1 1-0 Game 3: Tying model Winning model Winning model Tying model

Our Proposed method(SVM with Status of Match) 43 The method uses different learning models of SVM depending on game status. 0-1 Kick-off Game 1: 0-2 Losing model Tying model Full time Losing model 1-0 Game 2: Tying model Winning model 1-1 1-0 Game 3: Target Game: Tying model Tying model Winning model Detect Tying model 1-0 Winning model Detect （A system monitors game status)

Methods Used in Experiment 44 3 word-based methods (player names, team names, number are normalized) Pattern matching (from Jeon’s research) [Jeon et al. 2015] Tweets that containing spoiler keywords were judged as spoilers Spoiler keywords were terms with 0.100 or higher of a TF-IDF value SVM (from Jeon’s research) [Jeon et al. 2015] The SVM model detected spoilers Features for SVM were generated using a Bag of Words of each tweet SVM with Status of Match (Our proposed method) The SVM model detected spoilers Features for SVM were generated using the Bag of Words of each tweet

Bag of Words 45 We conducted word division After it, we selected noun, verb, adjective, adverb, interjection and generated the Bag of Words’ features For example Features for SVM of tweets about “Japan is winning” and “Fight! Japan!!” Tweet Japan win fight Japan is winning 1 1 0 Fight! Japan!! 1 0 1

How to calculate the result value Game 1 Game 2 Game 3 Game 4 Game 5 Game 6 Game 7 46 Game 8 Game 9

How to calculate the result value Game 1 Game 2 Game 3 Game 4 Game 5 Game 6 Game 7 47 Game 8 Generate spoiler keywords or SVM model Game 9

How to calculate the result value Game 1 Game 2 Game 3 Game 4 Game 5 Game 6 Game 7 48 Game 8 Detect spoiler keywords or SVM model Game 9

How to calculate the result value Game 1 Game 2 Game 3 Game 4 Game 5 Game 6 Game 7 49 Game 8 Detect spoiler keywords or SVM model Value 1 Game 9

How to calculate the result value Game 1 Game 2 Game 3 Game 4 Game 5 Game 6 Game 7 Generate spoiler keywords or SVM model Value 1 50 Game 8 Game 9

How to calculate the result value Game 1 Game 2 Game 3 Game 4 Game 5 Game 6 Game 7 Detect spoiler keywords or SVM model Value 1 51 Game 8 Game 9

How to calculate the result value Game 1 Game 2 Game 3 Game 4 Game 5 Game 6 Game 7 Detect spoiler keywords or SVM model Value 1 Value 2 52 Game 8 Game 9

How to calculate the result value Game 1 Value 1 Value 2 Game 2 Value 3 Game 3 Value 4 Game 4 Value 5 Game 5 Value 6 Game 6 Value 7 53 Game 7 Value 8 Value 9 Game 8 Game 9

How to calculate the result value Game 1 Value 1 Value 2 Game 2 Value 3 Game 3 Value 4 Game 4 Value 5 9 Game 5 Value 6 Game 6 Value 7 54 Game 7 Value 8 Value 9 Game 8 Game 9 The result = value

Result 55 Method Precision Recall F-measure Pattern matching 0.270 0.668 0.372 SVM 0.617 0.601 0.598 SVM with Status of Match 0.698 0.565 0.611

The evaluation points Precision = Recall = 𝒏𝒖𝒎𝒃𝒆𝒓 𝒐𝒇 𝒔𝒑𝒐𝒊𝒍𝒆𝒓′ 𝒔 𝒅𝒆𝒕𝒆𝒄𝒕𝒆𝒅 𝒄𝒐𝒓𝒓𝒆𝒄𝒕𝒍𝒚 𝒏𝒖𝒎𝒃𝒆𝒓 𝒐𝒇 𝒅𝒆𝒕𝒆𝒄𝒕𝒆𝒅 𝒕𝒘𝒆𝒆𝒕𝒔 𝒏𝒖𝒎𝒃𝒆𝒓 𝒐𝒇 𝒔𝒑𝒐𝒊𝒍𝒆𝒓′ 𝒔 𝒅𝒆𝒕𝒆𝒄𝒕𝒆𝒅 𝒄𝒐𝒓𝒓𝒆𝒄𝒕𝒍𝒚 𝒏𝒖𝒎𝒃𝒆𝒓 𝒐𝒇 𝒔𝒑𝒐𝒊𝒍𝒆𝒓 𝒕𝒘𝒆𝒆𝒕𝒔 F-measure = 2・(𝐩𝐫𝐞𝐜𝐢𝐬𝐢𝐨𝐧)・(𝐫𝐞𝐜𝐚𝐥𝐥) (𝐩𝐫𝐞𝐜𝐢𝐬𝐢𝐨𝐧)+(𝐫𝐞𝐜𝐚𝐥𝐥) 56

Result 57 Method Precision Recall F-measure Pattern matching 0.270 0.668 0.372 SVM 0.617 0.601 0.598 SVM with Status of Match 0.698 0.565 0.611

Result 58 Method Precision Recall F-measure Pattern matching 0.270 0.668 0.372 SVM 0.617 0.601 0.598 SVM with Status of Match 0.698 0.565 0.611

Result 59 Method Precision Recall F-measure Pattern matching 0.270 0.668 0.372 SVM 0.617 0.601 0.598 SVM with Status of Match 0.698 0.565 0.611 Pattern matching often detect a spoiler only from the player name. SVM related detect a spoiler not only from the player name. e.g.: “Kagawa’s missed pass is scary because of the heavy turf” Pattern matching: Spoiler SVM related: Non-spoiler

Result 60 Method Precision Recall F-measure 0.270 0.668 0.372 SVM 0.617 0.601 0.598 SVM with Status of Match 0.698 0.565 0.611 Pattern matching Pattern matching often detect a spoiler only from the player name. SVM related detect a spoiler not only from the player name. e.g.: “Kagawa’s missed pass is scary because of the heavy turf” Pattern matching: Spoiler SVM related: Non-spoiler

Result 61 Method Precision Recall F-measure Pattern matching 0.270 0.668 0.372 SVM 0.617 0.601 0.598 SVM with Status of Match 0.698 0.565 0.611

Result 62 Method Pattern matching Precision Recall F-measure 0.270 0.668 0.372 SVM 0.617 0.601 0.598 SVM with Status of Match 0.698 0.565 0.611

Result 63 Method Pattern matching Precision Recall F-measure 0.270 0.668 0.372 SVM 0.617 0.601 0.598 SVM with Status of Match 0.698 0.565 0.611 Mistakenly learned tweets by SVM were no longer learned for every time zone by SVM with Status of Match. e.g.: “I saw a national team match for the first time in a very long time!!” SVM: Spoiler SVM with Status of Match: Non-spoiler

Result 64 Method Pattern matching Precision Recall F-measure 0.270 0.668 0.372 SVM 0.617 0.601 0.598 SVM with Status of Match 0.698 0.565 0.611 Mistakenly learned tweets by SVM were no longer learned for every time zone by SVM with Status of Match. e.g.: “I saw a national team match for the first time in a very long time!!” Cause: Mistakenly learned tweets such as “I saw a sweeping victory for the first time in a very long time”

Result 65 Method Pattern matching Precision Recall F-measure 0.270 0.668 0.372 SVM 0.617 0.601 0.598 SVM with Status of Match 0.698 0.565 0.611 Mistakenly learned tweets by SVM were no longer learned for every time zone by SVM with Status of Match. Ex) “I saw a national team match for the first time in a very long time!!” The detection accuracy can be improved by considering Cause: Mistakenly learned of “I saw a sweeping victory thefirst statuses matches. for the time in a of very long time”

Result 66 Method Precision Recall F-measure Pattern matching 0.270 0.668 0.372 SVM 0.617 0.601 0.598 SVM with Status of Match 0.698 0.565 0.611

Result 67 Method Precision Recall F-measure Pattern matching 0.270 0.668 0.372 SVM 0.617 0.601 0.598 SVM with Status of Match 0.698 0.565 0.611 We don’t set up the standard for labeling. e.g.: “Nagatomo got a cramp!” was judged to be spoilers in the dataset

Generating new spoiler dataset A labeler predicts the game‘s result. （We defined more confident tweet is more crucial spoiler） I wonder if Haaril has an idea. [120 minutes elapsed] Confidence: Prediction: Win Draw Lose Not sure Japan is netted the rebound. [40 minutes elapsed] Confidence: Prediction: Win Draw Loooooooooong pass [20 minutes elapsed] Lose Not sure Choose spoilers ↓ Predict the results and it’s confidence 68

Result(new dataset) 69 We regarded tweets about 50 or more average confidence as spoiler and less than 50 average confidence as non-spoiler. Method Precision Recall F-measure SVM with Status of Match (old dataset) 0.698 0.565 0.611 SVM with Status of Match (new dataset) 0.831 0.880 0.852

Result(new dataset) 70 We regarded tweets about 50 or more average confidence as spoiler and less than 50 average confidence as non-spoiler. Method Precision Recall F-measure SVM with Status of Match (old dataset) 0.698 0.565 0.611 SVM with Status of Match (new dataset) 0.831 0.880 0.852

Result(new dataset) 71 We regarded tweets about 50 or more average confidence as spoiler and less than 50 average confidence as non-spoiler. Method Precision Recall F-measure SVM with judging only0.698 more crucial spoiler, Status ofIfMatch 0.565 0.611 (old it dataset) can be judged with more 0.8 accuracy. SVM with Status of Match (new dataset) 0.831 0.880 0.852

Application to real-world contents Normal digest Non-spoiler digest 72

Application to real-world contents Number of spoilers? 73 Non-spoiler digest eliminate eliminate time

Current Problem Precision was about 0.5 if keeping high recall. Precision-Recall curve 74

Current Problem 75 Precision was about 0.5 if keeping high recall. Precision-Recall curve Future task: We need to think other methods to design a high-recall model first and then create models realizing higher precision.

Findings 1. Terms themselves have the characteristics of spoiler and the terms of spoilers differed depending on the game status. 2. SVM with Status of Match method is the best about the detection accuracy. 3. If judging only more crucial spoiler, it can be judged with more 0.8 accuracy. 76

Summarize Examining methods for detecting spoilers with high accuracy on Twitter.  The detection accuracy can be improved by considering the statuses of matches.  If judging only more crucial spoiler, it can be judged with more 0.8 accuracy. [Future works]  Improvement of detection method.  Implementation of Twitter client. 77