A Basic Study to Prevent Non-Earnest Responses in Web Surveys by Arranging the Order of Open-ended Questions

A Basic Study to Prevent Non-Earnest Responses in Web Surveys by Arranging the Order of Open-ended Questions Ikumi Yamazaki Kenichi Hatanaka, Satoshi Nakamura, Takanori Komatsu （Meiji University）

Question Types Open-ended question Close-ended question 2

Contribution We clarified the difference in the number of non-earnest responses. Group-First < Group-Last If you want to decrease the number of non-earnest responses in your survey, you should first locate the open-ended questions! 3

Background: Web survey Have you ever requested or responded to a Web survey? • Researchers often use Web surveys to collect responses on social research and to collect preliminary data for research purposes. • Researchers can gather many answers by Web survey more easily and quickly than paper-based surveys. • The number of people registering for crowdsourcing services rapidly increases because of easy participation and easy reward. 4

Background: Open-ended questions We focus on open-ended questions in Web surveys. • Researchers often use open-ended questions because they provide answers from various perspectives[1]. • Some respondents answer “nothing in particular” and random string such as “asdf;jk;” in the open-ended questions and gave non-earnest responses. [1] Reja, U., Manfreda, L. K., Hlebec, V., and Vehovar, V.: “Open-ended vs. Close-ended Questions in Web Questionnaires.” Adv Methodol Stats, 19(1), 159-177 (2003). 5

Background: Open-ended questions We focus on open-ended questions in Web surveys. These non-earnest responses in web surveys • Researchers often use open-ended questions because they provide [1]. answers from varioushave perspectives become a problem. • Some respondents answer “nothing in particular” and random string such as “asdf;jk;” in the open-ended questions and gave non-earnest responses. [1] Reja, U., Manfreda, L. K., Hlebec, V., and Vehovar, V.: “Open-ended vs. Close-ended Questions in Web Questionnaires.” Adv Methodol Stats, 19(1), 159-177 (2003). 6

Our research goal Reducing the number of non-earnest responses to open-ended questions in Web surveys. Why do people make NERs? The respondents become bored with answering and creating a response in the latter half of the survey. They may answer “nothing in particular” or “I don’t know” in the open-ended questions. 7

Related work Open-ended questions in the last half • The later the open-ended questions asked, the lower the number of interpretable responses[2]. • The later the question asked, the shorter the open-ended response text becomes than immediately after the question begins[3]. [2] Schmidt, K., Gummer, T., and Roßmann, J.: “Effects of Respondent and Survey Characteristics on the Response Quality of an Open-Ended Attitude Question in Web Surveys.” Methods, Data, Analyses, 14(1), 3-34 (2020). [3] Galesic, M., and Bošnjak, M.: “Effects of Questionnaire Length on Participation and Indicators of Response Quality in a Web Survey.” Public Opinion Quarterly, 73(2), 349- 360 (2009). 8

Related work Open-ended questions in the last half • The later the open-ended questions asked, the lower the number of interpretable Changing the position of open-ended questions [3]. responses has not shown to improve response quality. • The later the question asked, the shorter the open-ended response text becomes than immediately after the question begins[4]. [2] Schmidt, K., Gummer, T., and Roßmann, J.: “Effects of Respondent and Survey Characteristics on the Response Quality of an Open-Ended Attitude Question in Web Surveys.” Methods, Data, Analyses, 14(1), 3-34 (2020). [3] Galesic, M., and Bošnjak, M.: “Effects of Questionnaire Length on Participation and Indicators of Response Quality in a Web Survey.” Public Opinion Quarterly, 73(2), 349- 360 (2009). 9

Effect of NERs in the first If people faced the open-ended questions at an early stage of a survey… • They answer the questions more seriously because they would not feel bored. • People who are not the target of the survey will leave quickly. 10

Experiments Hypothesis People would give better answers to open-ended questions when they encounter them early in the process rather than when they encounter them later. To clarify the hypothesis, we conducted two experiments. 1. Survey about telework using Google Forms. 2. Survey about car driving using the survey system we implemented. 11

Experiments Hypothesis People would give better answers to open-ended questions when they encounter them early in the process rather than when they encounter them later. To clarify the hypothesis, we conducted two experiments. 1. Survey about telework using Google Forms. 2. Survey about car driving using the survey system we implemented. 12

Experiments with our system Are NERs affected by the position of the open-ended questions? • We implemented a survey system. • Divided participants into Group-FIRST and Group-LAST randomly. • Recording participants' behaviors, such as response times and unanswered questions. • We conducted the survey on Yahoo! Crowdsourcing[4] We collected answer behaviors from 1,000 respondents. [4] Yahoo!Crowdsourcing,https://crowdsourcing.yahoo.co.jp/ 13

Group-FIRST Group-LAST 14

Results: number of data and classification • We defined the non-earnest response. • An answer to the question did not accompany it. ex.) I don’t know, Nothing in particular. • An answer itself could not capture its meaning. ex.) I like drive, Safety. • We recruited two evaluators to classify the open-ended responses into earnest or non-earnest responses based on the definition. 15

Results: the percentage of NERs The percentage of NERs(%) 30 20 * ** 10 0 Q-e Q-f Q-g Q-h Question Group-FIRST Group-LAST N=979, *: p < 0.05, **: p < 0.01 • The number of NERs was lower in Group-FIRST in all questions than in Group-LAST. • We tested the difference in the proportions for each question and found a significant difference in two questions. 16

Results: the percentage of NERs The percentage of NERs(%) 30 20 * ** 10 0 Q-e Q-f Q-g Q-h Question Group-FIRST Group-LAST N=979, *: p < 0.05, **: p < 0.01 • The number of NERs was lower in Group-FIRST in all questions than in Group-LAST. • We tested the difference in the proportions for each question and found a significant difference in two questions. 17

Results: the percentage of NERs The percentage of NERs(%) 30 20 * ** 10 0 Q-e Q-f Q-g Q-h Question Group-FIRST Group-LAST N=979, *: p < 0.05, **: p < 0.01 • The number of NERs was lower in Group-FIRST in all questions than in Group-LAST. • We tested the difference in the proportions for each question and found a significant difference in two questions. 18

Results: the percentage of NERs The percentage of NERs(%) 30 20 * ** 10 0 Q-e Q-f Q-g Q-h Question Group-FIRST Group-LAST N=979, *: p < 0.05, **: p < 0.01 • The number of NERs was lower in Group-FIRST in all questions than in Group-LAST. • We tested the difference in the proportions for each question and found a significant difference in two questions. 19

Results: the percentage of NERs The percentage of NERs(%) 30 20 * ** 10 0 Q-e Q-f Q-g Q-h Question Group-FIRST Group-LAST N=979, *: p < 0.05, **: p < 0.01 • The number of NERs was lower in Group-FIRST in all questions than in Group-LAST. • We tested the difference in the proportions for each question and found a significant difference in two questions. 20

Results: the withdrawal rate 40 Withdrawal rate (%) 35 30 25 20 15 10 5 0 Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Q11 Q12 Q13 Q14 Q15 Q16 Q17 Group-FIRST Group-LAST Question number • Group-FIRST increase from Q1 to Q4 (open-ended questions). • There were few dropouts and most of the respondents answered all questions in Group-LAST. 21

Discussion • The percentage of NERs was lower in Group-FIRST. • Many people who left the Group-FIRST found to have left the Group-FIRST in the open-ended questions. Possibility of exclusion • People who left because they thought it was tiresome. • People who are trying to answer unserious. 22

Discussion: comparison by survey start time Anyone who responded to the survey as soon as it was published might be non-earnest respondents. We divided them into two groups. • Instant respondents who accessed our website within 10 minutes of the start of the survey. • Basic respondents who accessed the website more than 10 minutes after the start of the survey. 23

Discussion: comparison by survey start time The response time (seconds) in each group and respondent type. Group-FIRST Introduction +Q-e Q-f Q-g Q-h Group-FIRST and instant respondents 68.00 38.89 47.68 38.62 Group-FIRST and basic respondents 69.00 44.89 58.93 47.22 Group-LAST Q-e Q-f Q-g Q-h Group-LAST and instant respondents 35.89 41.56 50.53 39.50 Group-LAST and basic respondents 40.61 43.13 61.55 48.40 Basic respondents took longer to complete all open-ended questions than instant respondents in both groups. 24

Discussion: comparison by survey start time The response time (seconds) in each group and respondent type. Group-FIRST Introduction +Q-e Q-f Q-g Q-h Group-FIRST and instant respondents Instant respondents are 47.68seekers, 38.62 more68.00 likely to38.89 be money Group-FIRST basic respondents 69.00 58.93 whichandmay have resulted in shorter44.89 response times. 47.22 Group-LAST Q-e Q-f Q-g Q-h Group-LAST and instant respondents 35.89 41.56 50.53 39.50 Group-LAST and basic respondents 40.61 43.13 61.55 48.40 Basic respondents took longer to complete all open-ended questions than instant respondents in both groups. 25

Conclusion Background: Many non-earnest responses in web surveys. Purpose: Survey of open-ended questions placement to reduce non-earnest responses. Hypothesis: People would give better answers to open-ended questions when they encounter them early in the process. Results: By placing the open-ended questions first ・non-earnest responses was lower. ・the withdrawal rate increase from open-ended questions. 26