Detecting Sports Spoiler Images on YouTube

2024.09.13 CollabTech 2024 Detecting Sports Spoiler Images on YouTube Yuichiro Kinoshita, Takumi Takaku, Satoshi Nakamura Meiji University, Japan 1

Which Team Won? 2

Spoiler Problems Spoilers: Information that reveals key details of content Spoilers negatively impact people’s experiences. • In movies: Reduce the desire to watch the film [Tsang+ 2009] • In comics: Decrease the interest in continuing the story [Maki+ 2018] • In sports: Diminish the enjoyment of watching the game [Shiratori+ 2018] This study focuses on spoilers in sports 3

Online Sports Viewing Many people prefer to watch matches in real time. Time differences or personal reasons often hinder live viewing. People use recordings or rebroadcasts. Time differences At 8 p.m. in Spain At 3 a.m. in Japan 4

Spoilers in Sports When watching a match later, there is a risk of encountering spoilers about the match outcome. Spoilers reduce tension and enjoyment [Shiratori+ 2018] W.Davis @spoilertweets・1m GOAL!! Spain2-1France #EURO2024 Spoilers on news sites (https://edition.cnn.com) Spoilers on Twitter (X) 5

Related Work on Sports Spoilers Blocking spoilers on the Web by masking the text[Nakamura+ 2012] Detecting Twitter (X) posts containing spoilers about baseball [Sasano+ 2019] and football [Jeon+ 2016] [Shiratori+ 2018] Previous studies have focused on text-based spoilers 6

Spoilers in Images The match outcome can be revealed through images. No method for preventing spoilers through images Displaying the final score Showing the standout player 7

Study Goal & Approach Study goal: Preventing image-based spoilers We verified the possibility of detecting spoiler images by ①Constructing a dataset of sports spoiler images ②Analyzing the characteristics of spoiler images ③Proposing and evaluating the performance of three methods for detecting spoiler images 8

Targeted Media Focused on YouTube thumbnails of sports highlight videos Why YouTube Thumbnails? Avoiding spoilers on YouTube is especially difficult due to its recommendation algorithm. 9

Encountering Spoilers on YouTube If you often watch FC Barcelona videos, YouTube is likely to recommend FC Barcelona videos in any situation. Encountering spoilers through the thumbnails of recommended videos Spoiler!! 10

Dataset Construction Focused on baseball, football, and basketball Collecting thumbnails from 13 YouTube channels using YouTube Data API Totaling 4,531 thumbnails (around 1,500 for each sport) 11

Definition of Spoiler Images The criteria for determining spoilers vary among individuals. There are various definitions of spoilers, but the definition varies depending on the study [Maki+ 2018] [Shiratori+ 2018] We defined spoiler images as images that enable the prediction of match outcomes based on the results of a preliminary annotation. 12

Levels of Spoilers The level of spoilers varies depending on the image. • Including the final score • Clearly revealing the outcome • Showing players only • Merely allowing speculation about the result 13

Establishing Three Spoiler Levels • Match outcomes cannot be predicted • Match outcomes can be somewhat predicted • Match outcomes are clearly predicted Annotating images by choosing one from three spoiler levels 14

Spoiler Label Annotation Each of the 3 annotators evaluated all 4,531 images. We determined an image as a spoiler when: • 2 or more annotators selected “Match outcomes can be somewhat predicted” • At least 1 annotator selected “Match outcomes are clearly predicted” 15

Annotation Results The label agreement rate: 0.78 The proportion of spoiler images: 0.24 Number of images Proportion of spoilers Baseball 1,506 0.19 Football 1,620 0.58 Basketball 1,405 0.20 Higher proportion compared to other sports 16

Spoiler and Non-Spoiler Images Spoiler images Non-spoiler images 17

Four Characteristics of Spoiler Images • The final match outcome is displayed • Players’ expressions include smiling or shouting • Players strike poses that express joy or excitement • Players from the same team gather to celebrate 18

Detection Methods of Spoiler Images Based on the characteristics of spoiler images, we propose 3 detection methods: • Image-Recognition Method • Vision-Direct Method • Vision-Text Method 19

Image-Recognition Method Leveraging the Google Cloud Vision API and YOLOv8 • OCR: Identifying scores • Emotion assessment: Detecting smiles or shouting • Pose assessment: Recognizing raising or spreading arms OCR Emotion assessment Pose assessment

Vision-Direct Method Leveraging the OpenAI Vision API (gpt-4-vision-preview) Using the API’s responses directly as the detection result Vision Spoiler / Non-Spoiler Prompt Please analyze this YouTube video thumbnail and determine if it is a spoiler or nonspoiler image. Define a spoiler image as one that reveals the outcome of an event, characterized by the presence of a score or result-related words, players exhibiting emotions of joy or triumph, such as smiling or cheering poses. If the image is a spoiler, respond with ‘Spoiler.’ If it is a non-spoiler, respond with ‘Non-spoiler.’ Do not output 21

Vision-Text Method Leveraging the OpenAI Vision API (gpt-4-vision-preview) Converting images to text and determining spoiler images through word matching with a spoiler dictionary Vision Word matching The image shows … Spoiler / Non-spoiler Prompt This image is a thumbnail for a YouTube video. Please describe this image. 22

Evaluation of Proposed Methods Evaluating the performance of our proposed methods using the constructed dataset Evaluation metrics • Recall • Precision • F1 score 23

Results Using the Entire Dataset Image-Recognition Recall Precision F1 score 0.90 0.39 0.55 Vision-Direct Vision-Text 0.72 0.75 0.74 0.80 0.76 0.78 Vision-Text > Vision-Direct > Image-Recognition (F1 score) Image-Recognition: The highest recall but lowest precision Determining most images as spoilers 24

Results for Each Sport (F1 Score Only) Image-Recognition Baseball Football Basketball 0.39 0.77 0.35 Vision-Direct Vision-Text 0.57 0.78 0.82 0.65 0.83 0.75 Vision-Text: Top F1 score in baseball & football Vision-Direct: Top F1 score in basketball F1 scores varied significantly across different sports. Football tended to have higher F1 score than other sports. 25

Variations in Results Across Sports Scoring frequency might have influenced the differences in results. Football Low scoring frequency The team that scored is likely to win Images capturing scoring scenes are likely considered spoilers Baseball & Basketball High scoring frequency Difficult to predict match results Need to consider the context in which the points were scored 26

Misdetection of Image-Recognition High recall and low precision Identifying most images as spoilers A very high number of false positives due to overly simplistic detection of poses expressing joy or excitement 27

Misdetection of Vision-Direct Seems to be an effective method (F1 score: 0.74) Configuring the prompt to output only detection results Unable to clarify the causes of misdetection False positive False negative 28

Misdetection of Vision-Text Appears to be the most effective method (F1 score: 0.78) Some false positives due to the reliance on word matching The image is showing a scene from a soccer match. …Their dynamic postures suggest they are either running or about to make contact with the ball. …The vivid imagery, team details, and action pose of the players are all aimed at capturing the excitement of the game. Example of output (false positive) 29

Future Work Improving the accuracy of the Vision-Text method Word matching outputs Evaluating the number and frequency of words from the spoiler dictionary in Expanding the dataset Increasing the variety of sports Collecting images from news sites 30

Summary Background: Spoilers through seeing YouTube thumbnails Purpose: Detecting sports spoiler images Methods: Image-Recognition, Vision-Direct, & Vision- Text Results: Vision-Text achieved the highest F1 score at 0.78 Future Work: Improving the accuracy of the Vision-Text Expanding the dataset 31