【Unite Tokyo 2018】Unity for ディープ・ラーニング：ツールキット『ML-Agents』のご紹介

Unity for Deep Learning: ML-Agents Explained

Mike Geig Head of Global Evangelism Content

Let’s start with one important question...

Let’s start with one important question... Why program system to complete a specific task when you can design it to learn?

ML Training Environment Requirements Visual Complexity Physical Complexity Cognitive Complexity

The Unity Ecosystem

ML-Agents v0.3 ML-Agents v0.2 ML-Agents v0.1 Components ● Additional environments (two new continuous control Components ● Learning Environments environments, plus two ● Flexible training scenarios (single platforming environments) agent, simultaneous single agent, ● Curriculum Learning adversarial self-play, cooperative ● Broadcasting multi-agent, competitive multi- ● Flexible monitor agent, ecosystem ● Monitoring agent’s decision making ● Complex Visual observations Components ● Imitation Learning ● Multi-Brain training ● On-demand decision-making ● Memory-enhanced agents

How does it work?

Unity ML-Agents Workflow Create Environment Train Agents Embed Agents

Create Environment (Unity) Observe & Act Decide Coordinate

Unity ML-Agents Workflow Create Environment Train Agents Embed Agents

Training Methods Reinforcement Learning Imitation Learning ● Learn through rewards ● Learn through demonstrations ● Trial-and-error ● No rewards necessary ● Super-speed simulation ● Real-time interaction ● Agent becomes “optimal” at task ● Agent becomes “human-like” at task

Unity ML-Agents Workflow Create Environment Train Agents Embed Agents

Embed Agents (Unity) ● Simply import a .bytes file (trained brain) into Unity project ● Set corresponding brain component to “Internal” mode. ● Support for Mac, Windows, Linux, iOS, and Android.

Let’s see it in action!

Learning Scenarios

Twelve Agents, One Brain, Independent Rewards Goal Balance ball as long as possible Observations Actions Platform rotation, ball position and rotation Platform rotation (in x and z) Rewards Bonus for keeping ball up

Two Agents, One Brain, Cooperative Rewards Goal Keep ball up as long as possible Observations Positions and velocities of racket and ball Actions Forward, backward, and upward movement Rewards +0.1 when sent over net by agent -0.1 when ball falls because of agent

Four Agents, Multi-Brain, Competitive Rewards Striker Goal Get the ball into the opponents goal Goalie Goal Defend own goal from opponents Observations Local ray-cast perception on nearby objects Actions Movement and rotation in x, z plane Striker Rewards +1 when its team scores goal -0.1 when opponent scores goal Goalie Rewards -1 when opponent scores goal +0.1 when its team scores goal

Multi-Stage Soccer Training Offense Train one brain with positive reward for ball entering opponents goal Defense Train one brain with negative reward for ball entering their goal Combined Train both brains together to play against opponent team

Learning Methods

Curriculum Learning

Easy Curriculum Learning ● Bootstrap learning of difficult task with simpler task ● Utilize custom reset parameters ● Change environment task based on reward or fixed progress Difficult

Imitation Learning

Imitation Learning Collect demonstrations from a teacher Learn policy via imitation

ML-Agents v0.3 ML-Agents v0.2 ML-Agents v0.1 Components ● Additional environments (two new continuous control Components ● Learning Environments environments, plus two ● Flexible training scenarios (single platforming environments) agent, simultaneous single agent, ● Curriculum Learning adversarial self-play, cooperative ● Broadcasting multi-agent, competitive multi- ● Flexible monitor agent, ecosystem ● Monitoring agent’s decision making ● Complex Visual observations Components ● Imitation Learning ● Multi-Brain training ● On-demand decision-making ● Memory-enhanced agents

We are hiring!

Get it Now github.com/Unity-Technologies/ml-agents Contact us https://unity3d.ai [email protected]

Thank you! Mike Geig [email protected] @MikeGeig