Autoware Architecture Proposal

2019. 2020.03 .17 Version 1.0 Autoware Architecture Proposal Architecture proposal for AWF by Tier IV Inc.

． ． ． ● ● ． ．

Here is subtitle 01 1 Requirement Why we need a new architecture

Why we need a new architecture Why? Problem ● It’s diffcult to improve Autoware.AI capabilities ● ● No concrete architecture design A lot of technical debt ○ Tight coupling between modules ○ Unclear responsibility of modules Tier IV's proposal ● ● ● Define a layered architecture Clarify the role of each module Simplify the interface between modules 4 4

Here is subtitle 01 2 Requirement Considered use cases

Considered use cases Example use cases that were considered during architecture design: Module Use cases Sensing ● 360-degree sensing by the camera-LiDAR fusion Perception ● Recognition of dynamic objects and traffic lights Localization ● Robust Localization using multiple data sources ● ● ● Route planning, dynamic planning based on vector map (not only waypoint following) Automatic parking Object avoidance ● High control performance on many kinds of vehicle-controllers Planning Control Features that are not considered yet (for the sake of development speed) ● Real-time processing ● HMI / Fail safe / Redundant system / State monitoring system / etc… Will consider these items at AWF WGs 6 6

Demo video Module Contents Whole Scenario demo Sensing+Perception 360° FOV, Prediction Localization Robustness of localization, Return from error Planning Lane change, Obstacle avoid, Parking Control Slow brake (normal stop), Rapid brake (emergency stop) 7 7

• https://youtu.be/kn2bIU_g0oY
• https://youtu.be/whzx-2RkVBA
• https://youtu.be/uhhMIxe1zxQ
• https://youtu.be/ydPxWB2jVnM
• https://youtu.be/fY9BmwcD9G4
• https://youtu.be/SCIceXW9sqM
• https://youtu.be/s_4fBDixFJc
• https://youtu.be/e9R0F0ZJbWE
• https://youtu.be/sWKjTArXckM
• https://youtu.be/M2vW3SkLreY

Here is subtitle 01 3 Requirement Architecture overview Layered architecture High level introduction of each module

DRAFT: https://github.com/tier4/Autoware_architecture_proposal Autoware Planning Perception Scenario Dynamic object Detection Prediction Tracking Perception Surrounding environment Scenario selector Mission Traffic light Detection Trajectory Planning Lane following Control Parking etc. Classifier Current pose/velocity Localization Sensor data Map data Map Sensing steering/velocity feedback Vehicle sensor Vehicle interface Vehicle command Map Sensor Vehicle 9

• https://github.com/tier4/Autoware_architecture_proposal

DRAFT: https://github.com/tier4/Autoware_architecture_proposal Autoware Planning Perception Scenario Dynamic object Detection Prediction Tracking Surrounding environment Scenario selector Trajectory Mission Traffic light Detection Control Lane following Driving Parking etc. Classifier Current pose/velocity Localization Sensor data Map data Map Sensing steering/velocity feedback Vehicle sensor Vehicle interface Vehicle command Map Sensor Vehicle 10

• https://github.com/tier4/Autoware_architecture_proposal

DRAFT: https://github.com/tier4/Autoware_architecture_proposal Autoware Planning Perception Scenario Dynamic object Detection Prediction Tracking Surrounding environment Scenario selector Trajectory Mission Traffic light Detection Control Lane following Parking etc. Classifier Current pose/velocity Localization Sensor data Map data Map Sensing steering/velocity feedback Vehicle sensor Vehicle interface Vehicle command Map Sensor Vehicle 11

• https://github.com/tier4/Autoware_architecture_proposal

[Architecture] Sensing Design Doc Role : Conversion of sensing data to ROS message Localization Perception Planning Sensing /sensing/lidar/ no_ground/pointcloud /sensing/gnss/[name]/ pose_with_covariance /sensing/camera/ [name]/image_raw /sensing/imu/[name]/imu_raw /sensing/lidar/pointcloud Preprocessor IMU preprocessor Pointcloud preprocessor Camera preprocessor GNSS preprocessor ROS drivers (mainly from upstream) Driver IMU driver GNSS driver Examples of preprocessors ● Distortion correction ● Outlier filter ● Concat filter ● Ground filter Camera driver LiDAR driver 12

• https://github.com/tier4/Autoware_architecture_proposal/blob/master/Sensing/Sensing.md

[Architecture] Map Design Doc Role : Distribute static environment information to other modules Localization Perception Planning Map library Map library Map library Map is distributed as a single message, and accessed through library API Map Pointcloud map loader Vector map loader Conversion to on-memory data structure Coordinate transformation Pointcloud map Vector map 13

• https://github.com/tier4/Autoware_architecture_proposal/blob/master/Map/Map.md

[Architecture] Localization Design Doc Role : Integration of each sensor data and estimation of self-pose and self-twist Support multiple localization methods based on different sensors(LiDAR/Camera/GNSS) Fuse pose and twist (e.g.EKF) Localization PoseWith Covariance.msg GNSS LiDAR (Camera) Pose estimator Pose Estimator tf (map to base_link) Pose twist fusion filter TwistWith Covariance.msg CAN IMU twist Twist estimator Pose Estimator Merge sensor inputs 14

• https://github.com/tier4/Autoware_architecture_proposal/blob/master/Localization/Localization.md

[Architecture] Perception Design Doc Role : Dynamic object recognition and traffic light recognition Dynamic object’s information including: ● ● ● ● Perception Dynamic object Sensor data (e.g. Camera, LiDAR) Detection Tracking Prediction ID Semantic (label, confidence) Shape State (pose, velocity, acceleration, predicted path, etc) DynamicObjectArray.msg Map Traffic light Sensor data (e.g. Camera, LiDAR) Map Detection Classifier TrafficLightStateArray.msg Traffic light’s information including: ● State (red, green, yellow, etc) Could be customized for unique traffic light types. 15 15

• https://github.com/tier4/Autoware_architecture_proposal/blob/master/Perception/Perception.md

[Architecture] Planning Design Doc Role : From a calculation of the route to the goal to drive trajectory generation Selects appropriate module depending on situation Planning Route.msg Dynamic objects Scenario planning Trajectory.msg Scenario selector Traffic lights Select Obstacles tf(map to base_link) Map Mission planning Parking On road Route planning from vector map 16 etc. Moduled scenarios allow: ● distributed development ● Easier parameter tuning per situation

• https://github.com/tier4/Autoware_architecture_proposal/blob/master/Planning/Planning.md

[Architecture] Control Design Doc Role : Generation of control signals to drive a vehicle following trajectories Control Trajectory.msg VehicleStatus.msg Generate signals for trajectory following ControlCommand Stamped.msg Trajectory follower Steer Vehicle angle velocity VehicleCmd.msg (autoware.ai) ----------------------------------------header steer_cmd -header -steer accel_cmd -header -accel brake_cmd -header -brake gear twist_cmd -header -linear -angular ctrl_cmd -linear velocity -linear acceleration -steering angle Remove redundant output 17 Final gateway for commands to vehicle: ● engage ● acceleration limits ● etc Vehicle cmd gate Engage Vehicle Command.msg RemoteCmd VehicleCommand.msg (proposed architecture) ----------------------------------------header control -steering angle -velocity -steering angle velocity -acceleration shift -data Support low level control

• https://github.com/tier4/Autoware_architecture_proposal/blob/master/Control/Control.md

Here is subtitle 01 4 Requirement Contribution steps to Autoware community

Contribution steps to Autoware community Tier IV Architecture proposal Documentation writing Clean up sample impl Today's presentation AWF (Including Tier IV) Design additional modules(mainly related to Safety) Architecture validation in real-environment End of December Develop cutting edge features Feedback design and sample impl Detailed doc Sample impl Architecture discussion and refinement at Discourse and GitLab Autonomous Valet Parking Feb. Mar. Apr. Next ODD development Design and implement new architecture Architecture clean up Technical debt clean up Documentation clean up Mai. Jun. Jul. 19 Aug. Sep. Oct. Nov. Dec. Jan.

Here is subtitle 01 5 Requirement Conclusion

Conclusion ● We proposed a new architecture of Autoware ● Tier IV will contribute our achievements to Autoware community ○ Detailed design documents ○ Reference source code ● We would like to discuss the new architecture at WGs ○ Any feedback is welcome ● We will make an effort to implement the new architecture in Autoware.Auto 21 21

Appendix 22 22

Appendix - Table of Contents ● ● Design alternative Module implementation ○ Implementation details ○ Achevements 23 23

Appendix - Table of Contents ● ● Design alternative Module implementation ○ Implementation details ○ Achevements 24 24

[Design alternative] Planning (1/2) Pre-study of the planning architecture Type BOSS type (Junior type) CMU Dr. paper type Victor Tango type Structure Mission Mission Mission Apollo type Behavior Motion Trajectory Behavior Motion Scenario Mission ︙ Scenario Pros. Cons. ・Structure is intuitive and easy to understand. ・Easy to handle separately. ・Behavior has to make a conservative decision in order to reduce discrepancies with a motion’s decision. ・It solves and optimizes both behavior and motion simultaneously to overcome a cons. of the BOSS type. ・Difficult to improve if any issue was found. ・Optimization of the whole autonomous driving functions is not realistic.. ・Behavior and motion are tightly coupled to overcome a cons. of the BOSS type. ・Easy to improve if any issue was found ・Difficult to take a new research result into the modules. ・Scenario is one layer upper concept. ・We can adopt prefered types of the planning structure according to each scenario. ・In the scenario, cons. such as things described above are still remained. ・If scenarios increased, switching mechanism can be complicated. 25 25

[Design alternative] Planning (2/2) Finally, we adopted Apollo + Boss type On road scenario Behavior Mission Lane following scenario : BOSS type ● It’s clear and convenient to develop as OSS Motion Whole module : Apollo type ● New scenario can be added as needed Parking scenario Free space Future challenge ● In lane following scenario, behavior still makes a conservative decision. And an aggressive driving like changing lane by entering into crowded lane cannot be executed. ︙ ** scenario 26 26

Appendix - Table of Contents ● ● Design alternative Module implementation ○ Implementation details ○ Achevements 27 27

TF Tree map base_link Vehicle specified setting (Here is an example of Tier IV’s implementation) gnss_* aipxx velodyne_* The origin of MGRS coordinate The point projected vertically downwards from the center point of the rear wheel axle to the ground. The bottom of the velodyne_top camera_* imu_* 28 28

[Sensing] Components Localization Perception Planning Sensing /sensing/lidar/ no_ground/pointcloud /sensing/gnss/[name]/ pose_with_covariance /sensing/lidar/pointcloud MGRS convertion Points preprocessor /sensing/camera/ [name]/image_raw /sensing/imu/[name]/imu_raw /sensing/gnss/[name]/nav_sat_fix IMU driver GNSS driver Camera driver LiDAR driver 29

[Sensing] Feature ● Distortion correction ○ Corrects a distortion of the pointclouds due to an observation time gap ⇨ Accuracy of the localization is improved ● Ground filter ○ Removes a ground from pointclouds (.ai has same feature) ● Outlier filter ○ Removes outliers by ring-base method ⇨ Reduce impacts of leaves and insects ● Concat filter ○ Integrates some pointclouds ○ Reduces an impact when a part of sensors stop ○ Corrects time gaps between each LiDAR with odometry ⇨ Accuracy of the localization is improved 30 30

[Localization] Components Localization /sensing/gnss/[name]/pose_with_covariance Pose Twist Fusion Filter Pose Estimator Pose Initializer Sensing /sensing/lidar/pointcloud Sensing Crop Measurement Range /tf DownSampler NDT Scan Matcher /map/pointcloud Loader EKF map Twist Estimator IMU-VehicleTwist fuser /localization/fusion_localizer/t wist /sensing/imu/[name]/imu_raw IMU-GNSS-VehicleTwist fuser Sensing /vehicle/status/twist_with_covar iance diagnostic Localizer Diagnostics diagnostic Safety Diagnostic Safety Node Node Localization System Diagnostic Node /localization/diagnostic 31 31

[Localization] Feature ● Pose initializer ○ Automatic initial self position estimation by GNSS + Monte-Carlo method ● NDT scan matcher ○ Uses an estimated value of EKF as an initial position of the scan matching ⇨ If scan matching was failed, localization can be returned ○ Performance and accuracy are improved (Open-MP implementation, accuracy improvement of the initial position, improvement of the gradient method, distortion correction of pointclouds, and etc.) ● Scan matching failure judgement ○ Monitors statuses of scan matching based on a score ○ If score is lower than a threshold, an estimated result isn’t output ● EKF localization ○ Integrates the estimated self position of the scan matching and the velocity of CAN＋IMU ○ If scan matching broke down, the vehicle can drive a certain distance with odometry only ● IMU Vehicle-twist fusion ○ Uses a translation velocity of CAN and yaw rate of IMU ⇨ Accuracy of odometry is improved ● Localizer Diagnostics ○ Monitors a status of a whole localization module (Temporary implementation) ※ Only NDT is implemented in Pose Estimator now. In future, other estimators like a white line recognition based one will be added. 32 32

[Perception] Components Dynamic Object Detection sensing/lidar/pointcloud Tracking PointCloud Segmentation perception/detection/objects Image Image Detection Image Detection Detection perception/tracking/objects perception/prediction/objects Map Based Prediction Multi Object Tracking Shape Estimation Fusion sensing/camera/*/image_raw Prediction sensing/camera/*/camera_info /map/semantic Loader Map /map/semantic Loader Traffic Light Map Sensing Planning TrafficLightRoiArray Classification /perception/traffic_light/rois Detection /sensing/camera/[name]/camera_info Map based detection Fine detection Classifier /planning/mission_planning/route 33 33

[Perception] Feature Dynamic Object ● Point Cloud Segmentation : Splits pointclouds into object clusters ○ Becomes faster by algorithm improvement (It can work in real-time on CPU) ● Image Detection : Detects objects on an image as ROI ○ Becomes to work on 40FPS on Jetson AGX by Tensor RT and int8 quantization ● Fusion : Matches the result of Pointclouds Segmentation and the ROI of the image detection result ○ Matching accuracy is improved by using IoU ● Shape Estimation : Geometrically approximates the whole size of the objects by Point Cloud Segmentation ○ Shape estimations by each object class ○ Accuracy improvement by changing fitting algorithm of Bounding Box ● Multi-Object Tracking : Assigns IDs based on time series data, estimates the velocity and the acceleration, and removes outliers ○ Performance improvement by changing a tracking model by class labeling ○ Data association considering the class label and the size ● Map Based Prediction : Predicts moving path of the objects with the lane information in a map ○ Infers objects’ intent of the behavior and estimates each reliability of the predicted moving paths. Traffic Light ● Map Based Detection : Extracts an ROI of the traffic light in camera image based on the self position and map data ○ Takes errors of a self position, a calibration and a hardware vibration into consideration ● Fine Detection ○ Extracts the ROI of the traffic light more precisely with a learner ● Classifier : Recognizes a state of the traffic light by a color information in the image ○ Reduces detection errors by fine noise removement process 34 34

[Planning] Components On road Mission Mission planner Behavior Lane change planner Route.msg /planning/mission_planning/route Motion Behavior velocity planner Motion path planner Obstacle stop planner Path.msg /planning/scenario_planning/scenarios/ lane_following/behavior_planning/path Jerk filter Trajectory.msg /planning/scenario_planning/scenarios/ lane_following/motion_planning/trajectory Parking freespace Costmap generator Freespace planner Scenario selector Trajectory.msg /planning/scenario_planning/trajectory Trajectory.msg /planning/scenario_planning/scenarios/ parking/freespace_planner/trajectory OccupancyGrid.msg /planning/scenario_planning/scenarios/parking/ freespace_planner/costmap_generator/occupancy_grid 35 35

[Planning] Feature ● Scenario selector ○ Chooses and executes “lane_following” or “parking” scenario according to the information of the mission ● Mission planner ○ Automatically searches the route from a self position to a goal via certain pass through points (by using lanelet function) ○ Automatically generates a route by using a map information ● Lane change ○ Judges a situation which needs lane change (Drive route limitation / street parking avoidance) ○ Judges whether a collision with a dynamic will happen obstacle and if a lane change is possible, it will be executed ● Behavior Velocity Planner ○ Plans a velocity based on traffic rules ○ Supports intersections, crosswalks, back-side check when turning right/left, stop with traffic lights and temporary stop line ○ Uses results of the perception (dynamic object information) ● Obstacle avoidance (motion path planner) ○ Plans a path to avoid the obstacle while driving (A* + QP) ○ Generate a smooth path considering a drivable area and the obstacle ● Obstacle stop (obstacle stop planner) ○ Judges whether a collision with obstacle pointcoulds considering the vehicle shape, and fill a stop velocity ● Jerk filter ○ Smoothens the velocity under the limitation of max speed, acceleration and lateral acceleration (QP) ○ Enables to switch rapid/slow acceleration and deceleration plans 36 36

[Control] Components ● ● Longitudinal Controller ○ Calculates a target velocity and a target acceleration by PID ○ Supports a gradient correction and start/stop at a slope ○ Supports a smoothly stop Control Trajectory Follower Longitudinal Controller Lateral Controller Trajectory ○ Calculates a steering angle and an angular velocity (pure pursuit or MPC) ● LatLon Coupler ○ Integrates the longitudinal and lateral control command values ● Vehicle Cmd Gate ○ Switches the some command values like “remote” and “auto” ○ Attaches a limitation for the control command ■ longitudinal/lateral acceleration, longitudinal/lateral jerk /control/longitudinal/ control_cmd LatLon Coupler /control/control_cmd /control/vehicle_cmd Vehicle CmdGate Lateral Controller /control/lateral/ control_cmd RemoteCmd /autoware/engage ・In this implementation, longitudinal and lateral controls are separately culcurated ・In future implementation, these might be culcurated collectively 37 37

[Vehicle Interface] Components ● ● Accel Map Converter ○ Converts a target acceleration to a vehicle specific accel/brake pedal value by Accel Map ○ If the vehicle_interface supports a velocity/acceleration command, this converter isn’t necessary Vehicle Interface /control/vehicle_cmd Vehicle Interface (vehicle specific) ○ Interface between Autoware and a vehicle ○ Communicates control signals ○ Obtains the vehicle information Accel Map Converter /vehicle/pedal_cmd /control/vehicle_cmd /vehicle/turn_signal_cmd To Vehicle /vehicle/status/twist Vehicle_interface /vehicle/status/steering /vehicle/status/turn_signal /vehicle/status/shift 38 38

Other features ● Web Controller ○ ● Through this controller, we can engage and specify the max velocity and also use “Go Home” button, “lane change OK” button and sensor Hz monitor Autoware State Monitor ○ State monitoring (Initializing / WaitForEngage / Driving / …) 39 39

Appendix - Table of Contents ● ● Design alternative Module implementation ○ Implementation details ○ Achevements 40 40

Achievements (1/2) ● ● Automatic initial self pose estimation Robustness of self pose estimation improvement ● CAN / imu fusion, EKF feedback, Gradient method algorithm improvement, pointclouds distortion correction, estimation speed improvement by Open-MP implementation ● ● ● ● ● ● ● ● 3D objects class recognition and shape estimation Performance improvement of dynamic objects tracking Moving path prediction of dynamic objects by using map information 360-degree sensing by the camera-LiDAR fusion Automatic lane change planning & decision Object avoidance (with lane change/ in the same lane) Intersection support (consider the oncoming vehicles and crossing vehicles) Person recognition at pedestrian crossing 41

Achievements (2/2) ● ● ● Performance improvement of the traffic light recognition Blind spot check when turning right/left Route generation from current position to a destination (via specified passing points) ○ ● ● ● ● ● ● With this feature, “Go Home button” can be implemented Automatic parking Slope support Slow / rapid brake planning support Performance improvement of the vehicle control Automatic deceleration at curve Collision judgement considering a vehicle shape 42