Strands Physics / Tools English Edition

Strands Physics / Tools ©CAPCOM 1

Agenda 1. Strand simulation framework 2. Tools and animation pipeline 3. Future improvement The agenda of my talk will mainly focus on how strand hair simulation works in Resident Evil 4 and what could improve in the future. I will also mention some of the work on strands tools and animation pipelines. ©CAPCOM 2 2

Simulation Workflow Ornatrix Alembic (*.abc) • Binding Connect strand to mesh Binding Mesh (*.fbx) • Simulating Guiding strands only Strand Asset Binding Asset • Guiding Broadcast guiding strands to all strands Simulation First, Let me introduce the strand hair simulation workflow. Artists use ornatrix to build the strands in Maya, then export the data as alembic files. The files are then imported into RE ENGINE. The alembic strands are bound to mesh. Mesh deformation and object collision gives strand constraints. Constraints is solved by using a method called Position based dynamics. 3 After that, a guiding system is used to blend simulated strands to all strands which saves on cost. Based on the framework I will break down this talk into three parts: - The binding part will introduce how strand bind to meshes. - The simulating part will introduce the method used through simulation. - The guiding part will introduce how simulation results interpolate and how to build a weight system for guiding. ©CAPCOM 3

Simulation Workflow This asset is used for testing during development First I will show some simulation results. This asset is used for checking if the simulation works correctly during development. 4 After testing this asset, and making sure everything works fine, we start to test Resident Evil 4 assets. ©CAPCOM 4

Simulation Use Cases from Resident Evil 4 These are the simulation use cases from Resident Evil 4. 5 ©CAPCOM 5

Simulation Use Cases from Resident Evil 4 These are additional simulation use cases from Resident Evil 4 If strand hair configuration is on, hair simulation is used during Resident Evil 4‘s gameplay and in cutscenes by default. 6 ©CAPCOM 6

Binding Bind Strand to Mesh 1. Implement SAH-BVH. 2. CPU ray tracing from strand root to mesh triangle. Ingo Wald “On fast Construction of SAH-based Bounding Volume Hierarchies” *1 Let's go back to the first topic: "How strands bind to meshes. Strands are composed of multiple line segments. Basically, rays are traced from line segments, then are checked to see if they intersect with meshes. 7 If they intersect, the hit triangle information is recorded. Then CPU based SAH-BVH is implemented, because not every computer supports hardware ray tracing. The performance is acceptable for practical use cases, because the scene is not that complicated, only bound mesh and binding strands exist for the intersection test. ©CAPCOM 7

Binding Result Iterative traces from two directions. Takes minimum distance to root position. Barycentric UV Mesh indices A, B, C • Trace direction depends on if strand intersects with the head. • If the hit fails, iterative traces it again. • Take minimum distance hit. Invalid all properties There are several cases that traces could fail. If the rays do not intersect with the mesh in the first trace, the rays will be traced iteratively from two directions. Taking the minimum distance to strand root position. 8 If all traces fail, an invalid flag is recorded as the result. If the mesh is intersected, mesh indices and uv coordinates are written to the bind asset. ©CAPCOM 8

Strand Binding Update Mesh Deformation Deform transformation Blend Shape Skinning Rest Deformed Mesh Rest Mesh Rotation / Translation Offset Update All Vertex in Strand Curve Deformed For runtime logics, comparing a deformed mesh and rest mesh's transformation. Compute the transform matrix from every triangle in a deformed mesh. 9 Update the matrix through all strand vertices and keep their transformation consistent with the deformed mesh. ©CAPCOM 9

Implementation of XPBD Miles Macklin, Matthias Muller, Nuttapong Chentanez “XPBD: Position-Based Simulation of Compliant Constrained Dynamics” *2 The next step is to simulate the strands. Basically, our approach is based on extended position-based dynamics, abbreviated as "XPBD." XPBD is a method that directly manipulates the vertex positions of object meshes instead of forces or impulses. 10 It solves a set of constraints on the positions by using an iterative projection scheme, where the positions satisfy the constraints of each iteration. XPBD improves position-based dynamics by introducing a compliance parameter that controls the stiffness of the constraints. ©CAPCOM 10

Orientation Deformation Improved • Bring angular constraints through quaternions • Angular constraints solved the same as position constraints through XPBD • Expandable for twist and shear deformation Kugelstadt, Tassilo and Schömer, Elmar “Position and Orientation Based Cosserat Rods” *3 Orientation deformation-based logic is implemented based on the paper, "Position and Orientation Based Cosserat Rods." This paper presents a novel method for simulating elastic rods within the PBD framework. 11 It proposes a new type of constraints to couple orientations which are represented by quaternions. Strain measures are used directly as constraints in PBD. It also shows that the method is fast and robust, and could handle large deformation situations with few artifacts. ©CAPCOM 11

Implementation Velocity Field Advect Delta Time Substep Substep Substep Linear Interpolation final deformed status Collision Constraint XPBD Solver Step Interpolation Substep Wind Guide Interpolation Advect Gravity Let me talk about concrete implementation. Basically, like other PBD methods did. 12 First, external force is applied through advect steps, including air drag force and gravity. Collision is solved after that through strand vertex. XPBD constraint projection is iteratively applied to solve the system. Finally, guide interpolation is applied for propagating guiding strands to all strands. Every frame is then separated into sub-steps. It needs to be noted that when solving sub-steps, the current and previous status need to be updated to avoid frame instability through simulation. ©CAPCOM 12

Strain Limiting • Sometimes simulation results are not as good as the artist expected for large motions. • Limit deformation gradient to prevent a large deformation explosion. Jerry Hsu, Tongtong Wang, Zherong Pan, Xifeng Gao, Cem Yuksel, Kui Wu “Sag-Free Initialization for Strand-Based Hybrid Hair Simulation” Sometimes simulation results are not as good as the artist expected for large motions. Deformation gradient through every sub-step is clamped to a max value to relieve this problem. But it still doesn't solve physically accurate. 13 For handling cutscenes in Resident Evil 4, initial frame instability also existed through teleport movement of the character. A reset position flag is used to address unexpected deformation when warming up the first frame. Another potential problem is that the first frame hair style changes from what artists designed due to lacking friction or internal forces between hairs. Recent research from Tencent Lightspeed Studio solved this problem. The information can be found in their paper titled, "Sag-Free Initialization for Strand-Based Hybrid Hair Simulation." Please read about the details there. ©CAPCOM 13

Guiding Original Guiding • Account for 1%-5% of total strands • Depends on... 1. Strand shape similarity 2. Strand position distance About guiding interpolation. For improving performance, only 1% to 5% strand budget is used through simulation. Which is called guiding strand. 14 The algorithm for deciding which strand could be the guiding one depends on its shape similarity and the position distance between strand roots. ©CAPCOM 14

Guiding Shape similarity Position distance • Shape similarity is calculated multiple times, using dynamic programming to cache the result. K means iteration • 1 – 3 guiding strands are used to give weight to a normal strand. Assign weight All strand shapes are encoded into a large group. The first step is to use a cluster method to aggregate similar strands. 15 After that, finding the mean value through cluster, the chosen candidate is used as a guiding strand for the strands around it. Then the other strands are assigned weight from the guiding strand. 1 – 3 guiding strands are used to give weight to a normal strand. ©CAPCOM 15

Strand Models Strand count Luis Leon Ashley Ada 25,978 23,115 21,291 38,751 Vertices count Luis Leon Ashley Ada 1,298,900 1,386,900 2,129,100 1,923,550 About performance: Resident Evil 4 Artists created, on average, 20 thousand strands per hair model. Which includes an average of 2 million vertices. 16 The image shows the strand counts and the vertices count of the main characters from Resident Evil 4. ©CAPCOM 16

LOD Factor Rendering all strands • 60 ms-70 ms Rendering only 30% of strands • 35 ms Scene 35.3312ms Still doesn't work well. After shading optimization is finished, a 4-character scene runs at about 60-70 ms, per frame. A method which cuts off strand count is used to reduce the performance load. 17 But even if only 30% of the strands are used in a scene, it still doesn't achieve the performance request for PS5. This is a scene from Resident Evil 4 where Luis, Leon, and Ashley are on screen at the same time. ©CAPCOM 17

LOD Factor Comparison (PS5) Strand count before Luis Leon Ashley Ada 25,978 23,115 21,291 38,751 Scene 60~70ms Strand count after (original * 0.3) Luis Leon Ashley Ada 7,793 6,934 6,387 11,625 Scene 35.3312ms Let me explain in more detail. You can see from the image that reducing the strand count from an average of 20 thousand strands to 7 thousand strands, 18 the performance improved from 60 ms to 35 ms per frame. But still, it is not good enough for the title's objective. We know that humans have about 100 thousand hair strands. If the strand count is reduced too much, the strands will either become thicker or bald spots will appear. ©CAPCOM 18

Vertex LOD Average strand length Expected vertex count • Difficulty is smoothly interpolating the deformation between different LODs Interpolation vertex through steps (Length / vertex count) • Average of 16 vertexes are used for Resident Evil 4 Vertex LOD Jiayi Eris Zhang, Jérémie Dumas, Yun (Raymond) Fei, Alec Jacobson, Doug L. James, Danny M. Kaufman “Progressive Simulation for Cloth Quasistatics” So reducing the strand count could work if the head is far from the camera, If not in that case, it could raise the risk of causing artifacts. Even if you cut 70% of the hair strands, it’s still not enough to reach real-time performance. 19 Another approach is provided for the LOD system to reduce the vertex count. First, the average strand length is calculated. Expectation vertex count is also given from parameters. After that, the vertex is interpolated through the average length divided by the expected vertex count. One of the problems with this approach is it is difficult to smoothly interpolate the deformation between different LODs. One idea is to use the same model to calculate the simulation, and interpolate the consistent simulation result through all LODs. ...Which will sometimes cause unnecessary budget. Recently Eris Zhang did some work for the progressive LOD interpolation through cloth simulation. If you are interested, please check her work for details. ©CAPCOM 19

Vertex LOD Before (35 ms) This result is based on the PS5. • 15 ms saved total 1. 13 ms saved for rendering 2. 2 ms saved for simulating After (20 ms) After the vertex LOD is applied, 15 ms is saved totally. 13 ms is saved for rendering budget, mostly from shading. 2 ms is saved from the simulation because vertices are a burden for the simulation. 20 Rest and guiding position update is unavoidable through simulation. ©CAPCOM 20

Vertex LOD Visual Comparison Original (1,386,900) Vertex LOD (290,052) The image shows a comparison between the original result and the LOD result. 80% of vertices are cut through LOD progress. 21 You can see the LOD result is much smoother. The artist accepted this result as one of the solutions to accelerate the performance. But, when reviewing this method, smooth artifacts could be improved by adjusting the LOD interpolation algorithm. The results could be improved in the future. ©CAPCOM 21

Vertex LOD Comparison Strand Vertex before Luis Leon Ashley Ada 1,298,900 * 0.3 1,386,900 * 0.3 2,129,100 * 0.3 1,923,550 * 0.3 Scene 35.3312ms Strand Vertex after Luis Leon Ashley Ada 316,252 * 0.3 290,052 * 0.3 289,904 * 0.3 264,960 * 0.3 Scene 20.0729ms The final results are displayed on this slide. In addition to these adjustments, dual scattering optimization was also used in the game scene to further improve performance, and to 22 achieve title performance objectives. ©CAPCOM 22

Simulation Performance Function Time Initialize Less than 0.1 ms Root Binding Less than 0.1 ms Strand Binding 0.2 – 0.4 ms Simulation 0.4 ms Interpolation 0.1 ms Guiding 0.2 – 0.4 ms Total 1 ms • PS5 Performance • Deformation binding and Guiding performance depend on vertex counts • After moving to Async compute, less than 1 ms per character in Resident Evil 4 Let's discuss simulation performance for PS5. Generally speaking, 1 ms per character in Resident Evil 4 is not bad, but still could improved. 23 It depends on the vertex count in most cases, half of the time, it’s using it for strand position binding and strand guiding interpolation. When reviewing, the binding progress might merge to final guiding progress, which could save 0.1-0.2 ms. Simulation may also be improved through the latest techniques. ©CAPCOM 23

Animation framework Strand Alembic (*.abc) Rig Info (*.abc) Strand Asset Puppet (*.fbx) Animation Except for simulation, strand animation framework is also provided. Strand and rig information is generated in Maya and exported as a strand asset. 24 Puppet mesh is used to save joint and animation info. Chain simulation is what was used in the PS4 generation simulation of Resident Evil 4, so it could also be used through this approach as well. ©CAPCOM 24

Maya custom attribute Maya alembic plugin Joint Name List Joint Matrix List Strand Joint Index List Strand Weight List • Auto weighted binding is tried, but failed • Weight controlled by artist in Maya RE ENGINE importer Joint matrix and strand weight is imported from Maya. At first artists propose to bind joint guides to strands through a strand auto-weight system. It doesn't work because the joint that the artist provided does not totally match what the strand looks like. Some artifacts cannot be removed through binding a different mesh to a strand. And after discussing it with the artist, the weight is then controlled in Maya. Then the weight data is saved as an alembic file and imported into RE ENGINE. 25 After that, strands bind weight to the puppet mesh. Animation or CPU based simulation could also work through this approach. But still, it is not recommended to use this approach because the visual quality is not good enough for both animation and CPU based simulation. With the development of newer hardware, a more accurate approach should be used through real-time simulation. ©CAPCOM 25

ADMM ADMM (Alternating Direction Method of Multipliers) based decomposition implicit time integration through DER GPU Optimization • Global tridiagonal solve • Local elasticity solve Gilles Daviet “Simple and Scalable Frictional Contacts for Thin Nodal Objects” *4 Gilles Daviet “Interactive Hair Simulation on the GPU using ADMM” *5 Daviet presents a novel method for simulating frictional contacts between thin nodal objects, using the "Alternating Direction Method of Multipliers" or "ADMM" for short. 26 This algorithm solves convex optimization problems by breaking them into smaller pieces, each of which are then easier to handle. It is based on the augmented Lagrangian method, which replaces a constrained optimization problem by an unconstrained one with a penalty term and a Lagrange multiplier term. The algorithm splits the frictional contact dynamics problem into two sub-problems that are much easier to solve individually. Recently the algorithm was also optimized through a local-global solver that largely improved the performance in GPU. This is one of future directions for providing a more accurate simulation model in real-time games. ©CAPCOM 26

Hybrid MPM Advantages 1. Support Intersecting with other materials 2. Improved collision solution Disadvantages 1. MPM optimization Xuchen Han, Theodore F. Gast, Qi Guo, Stephanie Wang, Chenfanfu Jiang, Joseph Teran “A Hybrid Material Point Method for Frictional Contact with Diverse Materials” *6 On the other hand, another group uses FEM and MPM to accelerate the speed and accuracy. Which named Hybrid Lagrangian Material Point Method (MPM) for simulating hair. This method utilizes a Lagrangian mesh for contact and a Eulerian grid for material deformation. 27 The Lagrangian mesh is used to compute the contact forces and impulses, and to transfer them to the grid. The Eulerian grid is used to compute the material deformation and stress, and to update the particle positions and velocities. The paper also demonstrates the effectiveness and versatility of the method through various examples involving frictional contact with diverse materials such as snow. ©CAPCOM 27

Conclusion Strand simulation system 1. Binding 2. Simulating 3. Guiding Strand animation system Future simulation improvement In conclusion, this talk introduced how strand physics work in RE ENGINE from binding to simulating, to guiding progress. The talk also briefly explained strand animation progress in RE ENGINE. And the direction of future strand simulation efforts. ©CAPCOM 28 28

Thank you for listening. Quotation Reference 1. Ingo Wald IEEE "On fast Construction of SAH-based Bounding Volume Hierarchies" October 8, 2007 https://www.sci.utah.edu/~wald/Publications/2007/ParallelBVHBuild/fastbuild.pdf(8/24/2023) ⚫ 2. Miles Macklin, Matthias Muller, Nuttapong Chentanez MIG '16 "XPBD: Position-Based Simulation of Compliant Constrained Dynamics" October 10, 2016 https://matthias-research.github.io/pages/publications/XPBD.pdf(8/24/2023) 3. Kugelstadt, Tassilo and Schömer, Elmar SIGGRAPH "Position and Orientation Based Cosserat Rods" 2016 https://animation.rwth-aachen.de/media/papers/2016-SCA-Cosserat-Rods.pdf(8/24/2023) 4. Gilles Daviet ACM Transactions on GraphicsVolume 39 "Simple and Scalable Frictional Contacts for Thin Nodal Objects" August 12, 2020 http://gdaviet.fr/files/hardContacts.pdf(8/24/2023) 5. Gilles Daviet SIGGRAPH "Interactive Hair Simulation on the GPU using ADMM" August 6, 2023 https://d1qx31qr3h6wln.cloudfront.net/publications/Interactive%20Hair%20Simulation%20on%20 the%20GPU%20Using%20ADMM.pdf(8/24/2023) 6. Xuchen Han, Theodore F. Gast, Qi Guo, Stephanie Wang, Chenfanfu Jiang, Joseph Teran Proceedings of the ACM on Computer Graphics and Interactive Techniques, Volume 2"A Hybrid Material Point Method for Frictional Contact with Diverse Materials" July 26, 2019 https://qig.github.io/Hybrid_MPM.pdf(8/24/2023) Jerry Hsu, Tongtong Wang, Zherong Pan, Xifeng Gao, Cem Yuksel, Kui Wu "Sag-Free Initialization for Strand-Based Hybrid Hair Simulation" July 26, 2023 https://graphics.cs.utah.edu/research/projects/sag-freehair/sig23_sagfree-hair.pdf ⚫ Jiayi Eris Zhang, Jérémie Dumas, Yun (Raymond) Fei, Alec Jacobson, Doug L. James, Danny M. Kaufman SIGGRAPH Asia 2022 "Progressive Simulation for Cloth Quasistatics" 2022 https://pcs-sim.github.io/pcs-main.pdf ⚫ Chenfanfu Jiang, Craig Schroeder, Joseph Teran, Alexey Stomakhin, Andrew Selle SIGGRAPH 2016 "The Material Point Method for Simulating Continuum Materials" May, 2016 https://www.math.ucla.edu/~cffjiang/research/mpmcourse/mpm course.pdf That's all of my presentation, thank you for listening. 29 ©CAPCOM 29

• https://www.sci.utah.edu/~wald/Publications/2007/ParallelBVHBuild/fastbuild.pdf(8/24/2023)
• https://matthias-research.github.io/pages/publications/XPBD.pdf(8/24/2023)
• https://animation.rwth-aachen.de/media/papers/2016-SCA-Cosserat-Rods.pdf(8/24/2023)
• http://gdaviet.fr/files/hardContacts.pdf(8/24/2023)
• https://d1qx31qr3h6wln.cloudfront.net/publications/Interactive Hair Simulation on
• https://qig.github.io/Hybrid_MPM.pdf(8/24/2023)
• https://graphics.cs.utah.edu/research/projects/sag-free-
• https://pcs-sim.github.io/pcs-main.pdf
• https://www.math.ucla.edu/~cffjiang/research/mpmcourse/mpm