Sensing & Perception 技術概要と性能課題

Sensing & Perception 技術概要と性能課題 株式会社ティアフォー 大里 章人

TIER IV Speaker 経歴 ~ 2016 .3 東京理科大学大学院 理工学研究科機械工学専攻 修士課程修了 ~ 2017.7 ソニー株式会社 深層強化学習ベースの自律移動ロボットの研究開発に従事 2017.7~ Akihito Ohsato Architect 株式会社ティアフォー 4人目の正社員として入社 Autoware 開発および複数の自動車両開発プロジェクトに携わった後 Sensing/Perception チームリーダーを経て 現在は自動運転システム開発におけるアーキテクトを担当

TIER IV TIER IV 1. System Overview 2. Sensing Technology 3. Perception Technology 4. Issues

System Overview

TIER IV What is Autoware? Autowareは、LinuxとROSをベースとした、世界初のオープン ソース⾏動運転ソフトウェアで、⾏動運転を設計する上で必要な 全ての機能を有している。 ティアフォーはAutoware開発をその設立当初からリードし、安心 ・安全な自動運転の実装のために数々の実地⾏⾏を世界中のさ まざまな環境下で⾏っている。 AutowareはThe Autoware Foundation (AWF)の登録商標 30+ 20+ 500+ VEHICLES COUNTRIES COMPANIES

Perception Planning Scenario Dynamic Object Detection Tracking Scenario Selector Prediction Control Mission Lane Driving Traffic Light Detection Parking Etc. Classifier Localization Sensing Sensors Map Data Vehicle Interface Vehicle

TIER IV

• http://www.youtube.com/watch?v=GjxEvmbtDPA

Sensing Technology

Perception Planning Scenario Dynamic Object Detection Tracking Scenario Selector Prediction Control Mission Lane Driving Traffic Light Detection Parking Etc. Classifier Localization Sensing Sensors Map Data Vehicle Interface Vehicle

Perception Planning Scenario Dynamic Object Detection Tracking Scenario Selector Prediction Control Mission Lane Driving Traffic Light Detection Parking Etc. Classifier Localization Sensing Sensors Map Data Vehicle Interface Vehicle

TIER IV Robo-taxi Sensors JPNTaxi model Velo d yne VLS -128 u-blox F9P TamagawaSeiki TAG300N (GNSS / Localization) (IMU/ Localization) AP Velodyne VLS-128 Alpha Prime (LiDAR / 360° Detection) Velo d 6 LP-1 ne V dy Velo yne V LP-1 6 TIER IV C1 85deg + TIER IV C2 62 deg TIER IV C1 85 deg x 6 LiDAR Configuration (FRONT) Velodyne VLP-16 Velo d (LiDAR / Side/Rear Detection) yne VLS -128 AP Camera Configuration (TOP) Velo d yne Continental ARS408-21 Continental ARS408 VLP -16 TIER IV C1 + TIER IV C2 w/ Jetson AGX (RADAR / Long-range Detection) LiDAR Configuration (SIDE) (Camera / Traffic Light Recognition) TIER IV C1 w/ JetsonAGX (Camera / Object Recognition)

TIER IV Shuttle-bus Sensors J6 model Septentrio mosaic x5 (GNSS / Localization) HES AI P anda r 40P TamagawaSeiki TAG300N (IMU/ Localization) x4 HESAI Pandar QT x4 HESAI Pandar40P (LiDAR / Long-range Detection) TIER IV C1 w/ JetsonAGX (Camera / Object Recognition) HESAI PandarQT (LiDAR / Short-range Detection) LiDAR Configuration (SIDE view) TIER IV C1 + TIER IV C2 w/ Jetson AGX (Camera / Traffic Light Recognition) Continental ARS408-21 (RADAR / Long-range Detection) TIER IV C1 85deg + TIER IV C2 62 deg TIER IV C1 85 deg x6 Continental ARS408 x6 RADAR Configuration (TOP view) Camera Configuration (TOP view)

TIER IV Sensor/ECU system (example) Camera Perception Pipeline Object Recognition Camera #1 Object Recognition Camera #2 Object Recognition Camera #3 Object Recognition Camera #4 Object Recognition Camera #5 Object Recognition Camera #6 Traffic Light Recognition Camera #1 Traffic Light Recognition Camera #2 LiDARs RADARs GMSL2 Perception ECU #1 (Jetson Xavier) Ethernet Perception ECU #2 (Jetson Xavier) Network Switch Ethernet Perception ECU #3 (Jetson Xavier) Perception ECU #4 (Jetson Xavier) LiDAR Perception & Fusion Pipeline Ethernet CAN Network Switch CAN Hub Main ECU (x86 + Discrete GPUs) CAN VCU

TIER IV Sensor Calibration & Synchronization Sensor Calibration (tier4/CalibrationTools) Perception Components Sensor Synchronization (tier4/sensor_trigger)

• https://github.com/tier4/CalibrationTools
• https://github.com/tier4/sensor_trigger

TIER IV

Perception Technology

Perception Planning Scenario Dynamic Object Detection Tracking Scenario Selector Prediction Control Mission Lane Driving Traffic Light Detection Parking Etc. Classifier Localization Sensing Sensors Map Data Vehicle Interface Vehicle

TIER IV ML-based Object Detection 2D Object Detection Sensor Fusion Camera based → YOLO, SSD… 3D Object Detection Camera based → CenterNet, SMOKE… LiDAR based → PointPillars, CenterPoint… Camera-LiDAR based → Frustum PointNet… Detected Object ● Class label ● 3D Pose ● 3D Shape ● Velocity ● …

TIER IV Object Recognition Pipeline (example) Perception / Detection Layer Sensing layer Object Recognition Cameras Pre-processing YOLO X LiDAR-Camera BoundingBox Fusion LiDAR Clustering LiDARs Pre-processing RADARs Pre-processing Perception / Tracking Layer Shape Estimation Detection by Tracker Centerpoint Merger RADAR Fusion Multi Object Tracker

TIER IV Camera-LiDAR Fusion 1. BoundingBox level Fusion LiDAR-Camera BoundingBox Fusion Easy to Deploy YOLO X LiDAR Clustering Github - autowarefoundation/autoware.universe image_projection_based_fusion 2. Pointpainting YOLO X Pointpainting LiDAR Pointcloud Centerpoint Vora, Sourabh, et al. "Pointpainting: Sequential fusion for 3d object detection." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020. Performance 3. Sensor level Fusion Transformer based Fusion Camera Image 2D Backbone LiDAR Pointcloud 3D Backbone Bai, Xuyang, et al. "Transfusion: Robust lidar-camera fusion for 3d object detection with transformers." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.

• https://github.com/autowarefoundation/autoware.universe/tree/main/perception/image_projection_based_fusion

Perception Planning Scenario Dynamic Object Detection Tracking Scenario Selector Prediction Control Mission Lane Driving Traffic Light Detection Parking Etc. Classifier Localization Sensing Sensors Map Data Vehicle Interface Vehicle

TIER IV Traffic Light Recognition Pipeline (example) Perception / Traffic Light Recognition Layer Sensing layer Traffic Light Recognition Camera (Wide) Pre-processing Traffic Light Recognition Camera (Narrow) Pre-processing Map-based Detector YOLO X (Fine Detector) MobileNet V2 (Classifier) Map / TF Merger Map-based Detector YOLO X (Fine Detector) MobileNet V2 (Classifier)

Sensing & Perception Issues

TIER IV Computation Cost ↕ Performance

TIER IV 1. Huge Computation Cost :( YOLO X x8 process 10 Hz @ Jetson Xavier x 4 Centerpoint x1 process 10 Hz @ NVIDIA Quadro RTX 5000

TIER IV 1. Huge Computation Cost CPU resource analysis @ MainECU (Xeon E-2278GE 8core/16threads) LiDAR pre-processing + Perception components > 50 % Power consumption analysis (w/o Sub/Logging System) x3 Main ECU ECU (- GPU) ~ 95 [W] GPU ~ 50 [W] x1 Perception ECU Object Recog x3 ~ 90 [W] Traffic Light Recog x1 ~ 20 [W] Total ≒ 255 [W]

TIER IV 2. More Performance :( Long-range Detection Case Usecase: Obstacle Stop YOLOX-tiny FP16 (608x960 input) Usecase : Intersection e.g. 幼児ターゲット H = 70 cm Φ = 16 cm ↑FP, FN YOLOX-sPlusOpt Int8 (960x960 input) 自車速度 [km/h] 60 緩減速 [m] 160.56 準急減速 [m] 急減速 [m] 自車が緩加速 75.81 54.4 自車が準急加速 他車速度 [km/h] 60 他車が減速しないケース [m] 182.96 他車が途中から緩減速 [m] 165.05 他車が減速しないケース [m] 114.81 他車が途中から緩減速 [m] 111.92

TIER IV 2. More Performance :( TLR camera upgrade C1 (2.5 MP) → C2 (5.4 MP) Low-latency Detection Case Usecase: Traffic Light Recognition 自車速度 [km/h] 60 緩減速 [m] 160.56 急減速1 [m] 75.81 急減速2 [m] 54.4 Sensing ~ Recognition Delay < 150 [ms]

物体検出 DNN の高効率化で 課題解決をめざす 29

TIER IV CONTACT US https://tier4.jp/ Thanks Again !

• https://tier4.jp/