Stylized Rendering Insights from Japan (Unreal Fest Gold Coast 2023)
55.8K Views
December 01, 23
スライド概要
Stylized Rendering Insights from Japan
Nori Shinoyama (篠山範明)
There are so many great games in Japan using stylized rendering made with Unreal Engine. In this presentation, we will briefly introduce and outline how different kinds of stylization can be achieved. We will also share tips on combining stylized rendering with new UE5 features such as Nanite and Lumen. You will learn what you need to know when creating unique stylized graphics in UE5.
Unreal Engineを開発・提供しているエピック ゲームズ ジャパンによる公式アカウントです。 勉強会や配信などで行った講演資料を公開しています。 公式サイトはこちら https://www.unrealengine.com/ja/
関連スライド
各ページのテキスト
Stylized Rendering Insights from Japan Nori Shinoyama Epic Games Japan, Developer Relations, Lead
There are many examples of Stylized rendering using Unreal Engine in Japan.
Game
Anime
No Guns Life: Cyclone Graphics turns comics into animation with real-time rendering
Real-time rendered animation matches traditional methods for quality on PreCure
Comic / Manga
Manga artist Inio Asano creates real-time backdrops in Unreal Engine
Through a lot of support of stylized rendering on UE4… Epic Games Japan has supported a lot of games using stylized rendering ● I want to make this kind of unique graphics, but how do I make it with UE? ● Do I need to modify the engine? ● Performance! Performance! Performance!
What to talk about today ● Basic techs of stylized Rendering ● What are the main concerns of many Japanese professionals when using stylized rendering ● What to expect when using stylized rendering with the new features of UE5
What not to talk about today ● Things related to platforms such as PS, Switch, XBox, etc. ● Design and other art-related topics ● Non-realtime rendering use-cases such as comics or movies ● Mainly focus on games
A japanese word I want you to remember in this session ダウンロード
A japanese word I want you to remember in this session ダウンロード = download
Who I am? ● Nori Shinoyama ● ● Epic Games Japan, Developer Relations, Lead ● Twitter: @tempkinder ● International Sake Sommelier
Batten Girls - OiSa -
Basic Elements of Stylized Rendering
Basic Elements of Stylized Rendering Post Processing Outline Shading ©2019 Bandai Namco Online Inc. ©2019 Bandai Namco Studios Inc.
Outline
Outline: two basic methods Inverted Hull Post Processing Image courtesy of Tango Gameworks and Bethesda Softworks aaaaaa
Outline method: Inverted Hull ● Preparing a mesh for outlines, then superimposing it on the original mesh ● ● Extruding each vertex of the outline mesh to each normal direction By erasing front surfaces of the outline meshes, the outlines can appear. outline
Outline method: Inverted Hull ● What titles use it? ● Many titles reproducing comics and anime characters ● Advantage ● Can be adjusted for each characters ● Outline width / color ● Unique expressions ● Tips ● Some outlines are written directly on the textures, and used in combination with other outline methods Image courtesy of Tango Gameworks and Bethesda Softworks
Outline method: Inverted Hull Make front faces transparent Extrusion in the normal direction for the thickness of the outlines
Outline method: Inverted Hull (UE4) ● In UE4, you had to have an outline mesh and synchronize animations Two SkeletalMesh Components in a BP AnimBP for Outline Mesh Need to copy the original character animation to the outline mesh
Outline method: Inverted Hull (UE5) ● Translucent Overlay Material feature was added since UE5.1 ● ● You don’t need to make a custom BP and an animation BP anymore for the inverted hull method You just need to assign the material for Outline to “Overlay Material” in MeshComponent ! ● … Translucent ???
Tips: Translucent Overlay Material ● Can also be used for Opaque / Masked Materials ● ● ● The function is just to render the same mesh with the overlay material It is not limited to translucency (But you might need to WPO / PDO to avoid depth fight)
Outline method: Inverted Hull (Caveat) ● GPU cost ● ● Vertex Shader Cost ● Computation of double the number of vertices Pixel Shader Cost ● Opacity (Mask) need to be calculated on all surfaces of the outline mesh
Outline method: Inverted Hull (Caveat) ● If you don’t want to use a Twosinged node in order to reduce the Pixel Shader cost ● Option 1: Using a model with inverted normals for the outline mesh ● Can be rendered using the same method as UE4’s inverted hull ● Unfortunately, Reversing skeletal mesh normals isn’t possible on UE and must be done on another DDC tool such as Maya ● Option 2: Setting up front face culling for outline meshes ● It only allows back faces of the outline meshes to be drawn and the front faces are discarded in rasterization ● Static mesh component has Reverse Culling option ● Unfortunately, skeletal meshes does not have Reverse Culling setting ● You may need to create your own Mesh Component (and SkeletalMeshSceneProxy, etc.) or modify the Engine code for Reverse Culling of skeletal mesh
Outline method: Post Process
Marketplace ● There are many post processes for outlines in the Marketplace! Try them out!
Edge Detection: Basic Algorithm ● Detect edges from the difference with neighboring pixels in the source images ● ● ● G-Buffers can be used as a source image There are a variety of edge detection filters ● Sobel Filter / Prewitt filter / Laplacian filter / RobertsCross filter… Complicated or special edge detection filters are hardly used for games (as far as I can see) ● Seems to be mainly due to processing load, not due to implementation difficulties -1 0 +1 -1 -1 -1 +1 0 -2 0 +2 -1 +8 -1 0 -1 -1 0 +1 -1 -1 -1 Sobel Source Images Color? Depth? Normal? Laplacian Filtering 0 +1 -1 0 Roberts Cross Edges
Case Study: Valkyrie Elysium [UNREAL FEST WEST '22] Graphics and Optimizations Tips of Valkyrie Elysium
Case Study: Valkyrie Elysium ● Source Images for Character Outlines ● ● ● ● Normal Boundary with Background Depth Material ID? ● By modifying their Engine [UNREAL FEST WEST '22] Graphics and Optimizations Tips of Valkyrie Elysium
Case Study: Valkyrie Elysium ● Source images for BG Outlines ● ● Depth Shading Model [UNREAL FEST WEST '22] Graphics and Optimizations Tips of Valkyrie Elysium
Outline method: Post Process (Caveat) ● Post-process for outlines tends to be heavy on the GPU … Many source textures -1 0 +1 -2 0 +2 -1 0 +1 A lot of texture reads Can be a very complex and heavy loaded postprocess!! Lots of code to accommodate various edge cases
Outline: two basic methods Inverted Hull Post Processing Image courtesy of Tango Gameworks and Bethesda Softworks
Stylized Shading
Stylized Shading: Basic Idea ● In order to achieve unique stylized shading, you need to modify or override the existing UE’s shading system ● LUTs (Look Up Tables) are widely used for stylized rendering
Stylized Shading: Basic Idea Ｑ. How should the stylized shading be implemented? Material Lighting (Shading Model) Post Process
Stylized Shading: Basic Idea Using Materials? ● Method ● By passing lights’ information into materials via Material Parameter Collection(MPC) ● Pros ● Artists can precisely customize materials for each object ● Cons ● Material Might not get any benefit from the various lighting and rendering systems of UE ● Shading, Shadowing, etc. Lighting (Shading Model) Post Process
Stylized Shading: Basic Idea Using Post Process? ● Method ● Using various data in G-Buffer ● Pros ● Can easily change the whole scene ● Cons ● Material Difficult to adjust shading of each object or character Lighting (Shading Model) Post Process
Stylized Shading: Basic Idea Ｑ. How should the unique shading be implemented? A. Many games are using modified engines a. Custom G-buffer layout and/or additional buffers b. Custom shading model c. Custom post process (using the result from the above customizations) Material Lighting (Shading Model) Post Process
Case Study: THE IDOLM@STER STARLIT SEASON [UNREAL FEST EXTREME '22 SUMMER] Graphics Implementation and Optimizations in "THE IDOLM@STER STARLIT SEASON"
Case Study: THE IDOLM@STER STARLIT SEASON [UNREAL FEST EXTREME '22 SUMMER] Graphics Implementation and Optimizations in "THE IDOLM@STER STARLIT SEASON"
Case Study: THE IDOLM@STER STARLIT SEASON ● G-Buffer Customization ● Shading Model Customization [UNREAL FEST EXTREME '22 SUMMER] Graphics Implementation and Optimizations in "THE IDOLM@STER STARLIT SEASON"
Case Study: THE IDOLM@STER STARLIT SEASON [UNREAL FEST EXTREME '22 SUMMER] Graphics Implementation and Optimizations in "THE IDOLM@STER STARLIT SEASON"
Case Study: Hi-Fi RUSH Image courtesy of Tango Gameworks and Bethesda Softworks Hi-Fi RUSH" Development Case Study Graphics and Optimization Introduction | UNREAL FEST 2023 TOKYO
Case Study: Hi-Fi RUSH Hi-Fi RUSH" Development Case Study Graphics and Optimization Introduction | UNREAL FEST 2023 TOKYO
Other tips for stylized shading ● Each game has various implementations for its own shading. ● Case Study ● Shin-megami tensei ● Blue Protocol ● No More Heroes 3
Case Study: Shin Megami Tensei V CGWORLD: 原画イラストの再現を目指した個性的なキャラクターと悪魔たち『真・女神転生V』
Case Study: Shin Megami Tensei V CGWORLD: 原画イラストの再現を目指した個性的なキャラクターと悪魔たち『真・女神転生V』
Case Study: Shin Megami Tensei V ● Shadow Mask CGWORLD: 原画イラストの再現を目指した個性的なキャラクターと悪魔たち『真・女神転生V』
Case Study: Shin Megami Tensei V ● Unique hair anisotropic reflex (a.k.a. angel's ring)
Case Study: Shin Megami Tensei V ● Normal Transfer ● ● ● Another way to achieve unique shading Left: original normal Right: applied a normal map
Case Study: Shin Megami Tensei V ● CGWORLD translated the article into English for this presentation!!!!!! ● Unique characters and demons aimed at reproducing the original illustrations "Shin Megami Tensei V"
Case Study: Blue Protocol
Case Study: Blue Protocol ©2019 Bandai Namco Online Inc. ©2019 Bandai Namco Studios Inc.
Case Study: NO MORE HEROES 3 ©Marvelous Inc. Developed by Grasshopper Manufacture Inc.
Case Study: NO MORE HEROES 3 ● Fresnel node for pseudo-rim lighting and edge enhancement
Case Study: NO MORE HEROES 3 ©Marvelous Inc. Developed by Grasshopper Manufacture Inc.
Stylized Post Processing
Custom Post Process ● We’ve already seen some post processes in the Outline and Shading sections ● In addition to those, there are some interesting filters that will greatly change the image ● ● ● Postarization ● Bilateral Filter ● Kuwahara Filter Hatching etc
Kuwahara Filter
Hatching
Custom Post Process ● But, honestly, most of the game devs I’d asked told us… ● ● They’d implemented and tried many filters in the early stages of their development But they didn’t end up with most of the filters in their final products ● Reason: no one-fits-all solution! ● ● ● ● Difficult or very time-consuming to adjust for different situations Can reduce visibility Can affect the design of icons, characters, fonts, etc. Constant GPU cost ● Nevertheless, here are some examples of titles that partially utilize stylized post processes
Case Study: NARUTO TO BORUTO Shinobi Striker
Case Study: NARUTO TO BORUTO Shinobi Striker You can see the hatching effect in the shadows at the feet from “NARUTO TO BORUTO Shinobi Striker” Official Site
Case Study: Hi-Fi RUSH Image courtesy of Tango Gameworks and Bethesda Softworks Hi-Fi RUSH" Development Case Study Graphics and Optimization Introduction | UNREAL FEST 2023 TOKYO
Hi-Fi RUSH Added hatching effect into Ambient Occlusion pass Hi-Fi RUSH" Development Case Study Graphics and Optimization Introduction | UNREAL FEST 2023 TOKYO
Case Study: Valkyrie Elysium [UNREAL FEST WEST '22] Graphics and Optimizations Tips of Valkyrie Elysium
Case Study: Valkyrie Elysium Custom post-processing to add various colors to shadow areas [UNREAL FEST WEST '22] Graphics and Optimizations Tips of Valkyrie Elysium
Case Study: Valkyrie Elysium Custom post-processing to add various colors to shadow areas [UNREAL FEST WEST '22] Graphics and Optimizations Tips of Valkyrie Elysium
Tips: Custom Post Process ● Default post-processes of UE can be used with Stylized Rendering
Case Study: Demon Slayer Kimetsu no Yaiba The Hinokami Chronicles
Case Study: Demon Slayer Kimetsu no Yaiba The Hinokami Chronicles CyberConnect2 Effects Development Case Study (Unreal Fest Japan 2022) They talked about Demon Slayer Kimetsu no Yaiba The Hinokami Chronicles
Case Study: Demon Slayer Kimetsu no Yaiba The Hinokami Chronicles Capture the sense of movement and speed from japanese animation with high-quality graphics UE’s default depth of field and bloom are used without any modifications CyberConnect2 Effects Development Case Study (Unreal Fest Japan 2022)
Is it difficult to use stylized post processes that significantly affects the entire screen? As I said, such kinds of filters are rarely used in the final products However, there are great demands for them and many people are trying them! Today, I would like to share with you an great example from the Japanese community.
3Dnchu’s Stylized Post Processing ● This project is available! ● ● https://yamato3d.gumroad.com/l/StylizedPP-UE5 https://www.youtube.com/watch?v=NJjwrQialp8&t =1s ● Who 3Dnchu is? ● ● ● ● ● 3D人 - 3Dnchu twitter: @ymt3d Professional game developer runs one of the most popular news site about computer graphics in Japan ● https://3dnchu.com/ great guy who also shares his techniques with the public, such as this stylized PP Youtube Project
Insights from Japan
To make the graphics more unique… ● Prefer the combination of simple methods to complicated and highly technical algorithms ● Artist-driven, one by one and steadily built. ● The 2% visual error is more noticeable than the rest of 98% stable graphics ● A lot of work you need to mitigate the 2%
Engine Customization ● a lot of devs customized rendering code in their engine to fit their graphics requirements ● Where do they customize? ● ● ● ● Mesh Component G-buffer Shading Model Complicated Multi Pass Post Processing
Performance and Optimization ● Stylized rendering may make the picture look simpler than Realistic rendering, But it can require a lot of GPU resources ● Many materials and post-processes created in an artist-driven manner ● Be sure to profile regularly from the early stages of development! ● Ideally, have a graphics engineer regularly check your scenes
Golden Rule: profiling first, optimization second ● There is a lot of information on optimization in the world ● Polycount! Texture Resolution! Shader Complexity!! ● But they may not be effective for your projects ● Basically, always profile before doing any work that reduces the quality of the object or takes up the artist's time! ● MUST regularly profile your game on your target platforms ● Profiling and optimization in the end of dev cycle is not kawaii
Stylized Rendering with UE5 Rendering Features
Stylized Rendering with UE5 Rendering Features ● UE5 Rendering Features ● ● ● Nanite Lumen Temporal Super Resolution
Nanite with stylized rendering? ● Of course, you can use it! ● Caveats ● ● Translucency Overlay Material not yet supported by Nanite Nanite doesn’t support vertex colors. You cannot put any special data into each vertex of Nanite meshes. So you also cannot store per instance data into vertex color Nanite：ON Nanite：OFF (with Overlay Material)
Nanite with stylized rendering? ● Q. Our graphics is highly stylized so we don’t need to have high poly meshes. Still Nanite is needed? ● A. If you want to use VSM or Lumen, as many objects as possible should be Nanite from a perspective of GPU performance
Lumen with Stylized Rendering? Lumen: Off Lumen: On Need to make sure if Lumen fits the expression you want to achieve But you also need to understand the general caveats of Lumen
Lumen with Stylized Rendering? You need to make sure whether your stylized materials works in the Lumen Scene or not
Lumen with Stylized Rendering? It is very difficult to do lighting with Emissive only. Here is example of strong delay during camera switching.
TSR with Stylized Rendering? ● Temporal Super Resolution is UE5’s powerful anti aliasing and upscaling technology ● There may be a few problems on the combination of TSR and Stylized Rendering
TSR with Stylized Rendering? Strong Blur when the camera moved
TSR with Stylized Rendering? ● If such a blur occurs in UE5.1 or later ● ● Try to set the Console variable r.TSR.ShadingRejection.Flickering to 0 It may not be a complete improvement, but it will get better to some extent
TSR with Stylized Rendering? Noise texture scrolling Ghosting… Texture Animation Ghosting…
TSR with Stylized Rendering? I posted an article about this TSR issue on the Epic Developer Community An English version will be available...
Stylized Rendering Insights from Japan Nori Shinoyama Epic Games Japan, Developer Relations, Lead
Summary ● The well-known old techs are still used for stylized rendering, However, they can be extended and combined to meet the requirements of the game. ● There is a lot of information on Stylized Rendering in Japan Please take a look if you are interested! ● Understand the pros and cons of UE5's new features and use them to express yourself uniquely!
Thank you for listening! OiSa! https://www.youtube.com/watch?v=m3w2Lw--G0s