Visual Attention Model for Manipulating Human Attention by a Robot (ICRA2014)

Visual Attention Model for Manipulating Human Attention by a Robot Yusuke TAMURA1, Shiro YANO2, Hisashi OSUMI1 1 Chuo University, JAPAN 2 Ritsumeikan University, JAPAN 2014 IEEE International Conference on Robotics & Automation, Hong Kong, June 4, 2014

MOTIVATION Smooth interaction between humans... 2. Estimation/Prediction 1. Observation 3. Decision 2

MOTIVATION Smooth interaction between humans... 2. Estimation/Prediction 0. Elicitation 1. Observation 3. Decision 2

MOTIVATION Smooth interaction between humans... 2. Estimation/Prediction 0. Elicitation 1. Observation 3. Decision Manipulation of other’s attention is important for smooth interaction 2

MOTIVATION Smooth interaction between robot and human... 2. Estimation/Prediction 0. Elicitation 1. Observation 3. Decision 3

MOTIVATION Smooth interaction between robot and human... 2. Estimation/Prediction 0. Elicitation 1. Observation 3. Decision A robot has to understand how its behaviors affect human’s attention. 3

OBJECTIVE 4

OBJECTIVE 4

OBJECTIVE Development of a model of human attention for manipulating human attention by a robot (robot’s) behavior attention area environment 4

RELATED WORKS Computational visual attention model - Saliency map (Itti et al., 1998) - Saliency map + Face position (Cerf et al., 2008) - Saliency map + Face position/orientation (Ozeki et al., 2011) Input image Intensity Colors Orientation Motion Center-surround differences & normalization Across-scale combinations & normalization Saliency map 5

MANIPULATION OF ATTENTION Magicians are experts of manipulating human attention. 6

MANIPULATION OF ATTENTION Magicians are experts of manipulating human attention. Magic requires a method to achieve an effect. How the trick works What the spectator perceives 6

MANIPULATION OF ATTENTION Magicians are experts of manipulating human attention. Magic requires a method to achieve an effect. How the trick works What the spectator perceives Magicians manipulate spectators’ attention in order to prevent them from uncovering the methods by various skills. Eye gaze (Kuhn et al., 2009) Hand movement (Otero-Millan et al., 2011) Face, hands, and gaze (Akashi et al., 2013) 6

MODEL OF VISUAL ATTENTION Hypothesis: Human visual attention is influenced by visual saliency and others’ face, hands and gaze. 7

MODEL OF VISUAL ATTENTION Hypothesis: Human visual attention is influenced by visual saliency and others’ face, hands and gaze. Saliency map 7

MODEL OF VISUAL ATTENTION Hypothesis: Human visual attention is influenced by visual saliency and others’ face, hands and gaze. Saliency map Manipulation map 7

MODEL OF VISUAL ATTENTION Hypothesis: Human visual attention is influenced by visual saliency and others’ face, hands and gaze. Saliency map Manipulation map Attention map 7

SALIENCY MAP Visual saliency of input images Input image Intensity Colors (x2) Orientation (x4) Motion Center-surround differences & normalization Across-scale combinations & normalization Saliency map 8

MANIPULATION MAP Attention area (Akashi et al., 2013) Concentration Distraction Manipulation map describes - Position of face - Position of left/right hand - Direction of gaze 9

FACE & HAND xR Face-Hand map xF F (x) = X i 1 p exp 2⇡ |Si | ⇢ 1 (x 2 xi )T Si 1 (x xi ) xL 10

GAZE x d g xF Gaze map 1 G(x) = p 2⇡ exp d ✓ d(x) 2 d2 2 ◆ 11

MANIPULATION MAP + Face-hand map Gaze map Manipulation map 12

ATTENTION MAP Input&image Face1Hand&map Gaze&map Saliency&map Manipulation&map Attention map Attention&map 13

MAGIC VIDEO A magician performs “vanishing a ball” trick. method: a false transfer of the ball effect: vanish of the ball 1920 x 1080 14

SCENES OF THE VIDEO 1: Raise RH 2: Show the ball in RH 3: Raise LH 4: “Pass” the ball from RH to LH 5: Lower RH and raise LH 6: “Vanish” the ball in LH 8: Open LH 9: Lower LH 7: Change the orientation of LH 15

SCENES OF THE VIDEO 1: Raise RH 2: Show the ball in RH 3: Raise LH 4: “Pass” the ball from RH to LH 5: Lower RH and raise LH 6: “Vanish” the ball in LH 8: Open LH 9: Lower LH 7: Change the orientation of LH 15

SCENES OF THE VIDEO 1: Raise RH 2: Show the ball in RH 3: Raise LH 4: “Pass” the ball from RH to LH 5: Lower RH and raise LH 6: “Vanish” the ball in LH 8: Open LH 9: Lower LH 7: Change the orientation of LH 15

SCENES OF THE VIDEO 1: Raise RH 2: Show the ball in RH 3: Raise LH 4: “Pass” the ball from RH to LH 5: Lower RH and raise LH 6: “Vanish” the ball in LH 8: Open LH 9: Lower LH 7: Change the orientation of LH 15

SCENES OF THE VIDEO 1: Raise RH 2: Show the ball in RH 3: Raise LH 4: “Pass” the ball from RH to LH 5: Lower RH and raise LH 6: “Vanish” the ball in LH 8: Open LH 9: Lower LH 7: Change the orientation of LH 15

SCENES OF THE VIDEO 1: Raise RH 2: Show the ball in RH 3: Raise LH 4: “Pass” the ball from RH to LH 5: Lower RH and raise LH 6: “Vanish” the ball in LH 8: Open LH 9: Lower LH 7: Change the orientation of LH 15

SCENES OF THE VIDEO 1: Raise RH 2: Show the ball in RH 3: Raise LH 4: “Pass” the ball from RH to LH 5: Lower RH and raise LH 6: “Vanish” the ball in LH 8: Open LH 9: Lower LH 7: Change the orientation of LH 15

SCENES OF THE VIDEO 1: Raise RH 2: Show the ball in RH 3: Raise LH 4: “Pass” the ball from RH to LH 5: Lower RH and raise LH 6: “Vanish” the ball in LH 8: Open LH 9: Lower LH 7: Change the orientation of LH 16

SCENES OF THE VIDEO 1: Raise RH 2: Show the ball in RH 3: Raise LH 4: “Pass” the ball from RH to LH 5: Lower RH and raise LH 6: “Vanish” the ball in LH 8: Open LH 9: Lower LH 7: Change the orientation of LH 16

SCENES OF THE VIDEO 1: Raise RH 2: Show the ball in RH 3: Raise LH 4: “Pass” the ball from RH to LH 5: Lower RH and raise LH 6: “Vanish” the ball in LH 8: Open LH 9: Lower LH 7: Change the orientation of LH 16

SCENES OF THE VIDEO 1: Raise RH 2: Show the ball in RH 3: Raise LH 4: “Pass” the ball from RH to LH 5: Lower RH and raise LH 6: “Vanish” the ball in LH 8: Open LH 9: Lower LH 7: Change the orientation of LH 16

RESULT: SALIENCY MAP 17

RESULT: SALIENCY MAP 17

RESULT: MANIPULATION MAP 18

RESULT: MANIPULATION MAP 18

RESULT: PROPOSED MODEL 19

RESULT: PROPOSED MODEL 19

MEASUREMENT OF GAZE Measuring participants’ gaze points while watching the magic video 23-­‐inch  screen Participants: 5 healthy volunteers (21-24 yrs) Measurement: faceLAB4.2 (60Hz) cameras (FCB-­‐EX480B,  SONY) 20

EVALUATION INDEX gaze%point:%xg attention%value:% Attention mapA(xg ) evaluation%index:% A(xg ) A(x) saliency%value:% Saliency mapS(xg ) evaluation%index:% Attention'map attention value: Saliency'map saliency value: evaluation index: evaluation index: S(xg ) S(x) 21

RESULTS Saliency map Proposed model Gaze point Scene 1 2 3 4 22

RESULTS Saliency map Proposed model Gaze point Scene 1 2 3 4 22

RESULTS Saliency map Proposed model Gaze point Scene 1 2 3 4 22

RESULTS Saliency map Proposed model Gaze point Scene 5 6 7 8 23

RESULTS Saliency map Proposed model Gaze point Scene 5 6 7 8 23

RESULTS Saliency map Proposed model Gaze point Scene 5 6 7 8 23

Ratio to overall average EVALUATION 5 ** 4 3 2 1 0 Proposed Saliency ** p<0.01 The proposed model can better explain human visual attention than the original saliency map. 24

SUMMARY A model of human attention considering visual saliency and human face, hands and gaze was developed. The model can well explain human attention. FUTURE WORKS Developing an inverse model to calculate appropriate behaviors for any intended attention area. Implementing the model into a small humanoid robot. 25