Improving Development Iterations: A Deep Dive into Enhanced Feature

Unreal Fest Gold Coast 2024 Improving Development Iterations: A Deep Dive into Enhanced Feature Ken Kuwano Software Engineer, Developer Relations Epic Games Japan

Note: ● This session will cover the content of UE5.4 (May 2024) ● This session will introduces features that are still under development and may be updated in the future

Summary You will be able to do… ● ● ● ● Gain a better understanding of the features that can improve your development iterations Accelerate routine iterations such as cooking and packaging by applying them to your projects Check it quickly on the target device and save time to adjust the look and feel Consider yourself the best solution for your own project

Why improve development iterations?

Recent Trends Large projects Number of assets increases Increased map size Increased synchronization and load times Storage requirements ● ● Remote-work facilitates Difficult to get to the office Work the same way from anywhere Cloud computing Leverage build, storage, and virtualization environments WorldPartition, Nanite, Lumen, etc… in UE5 Our development environment is getting better, even better content is required

Customer Feedback Packaging takes a long time, so verification on actual equipment is time consuming Project synchronization takes a long time from the beginning Packaging is very long, but I don't know why Cook process takes a long time, but I don't know how to shorten it Long loading times in the editor, can't load assets in the background Even if I change one file, it takes a long time before I can check it on the actual device There is a long waiting time when executing in PIE Increased number of files takes time to transfer files to the actual device for confirmation It is difficult to build for all platforms with multiple build configurations Insufficient disk space because of increasing number of files to handle Long shader compilation time if don not have DDC Tight disk space, including project resources The editor is slow to start up and open maps Slow project sync due to the large number of files OFPA files Long time to build projects, both compiling and linking

Creating better products require further improvement of development iterations

Agenda ● Sync ● Build ● Cook ● Load

● Agenda Sync ○ Virtual Asset ○ Unreal Cloud DDC ● Build ● Cook ● Load

Virtual Assets Virtual Asset Structured Data Production Ready for UE5.3 ● ● ● ● ● Splitting assets into two pieces of data reduce disk space and increase synchronization efficiency The larger project size, the faster and more efficiently data can be synchronized between members Faster synchronization process by reducing the number of synchronizations Works well for projects that use a lot of textures and meshes On Ancient Game, first sync saves x50 disk space Perforce 11000101010 00010101010 0101 FA1C238E //Game/Content Asset .uasset Bulk Data Perforce FA1C238E - payload 1101010101001 0101010100101 1100101010101 10010101111010 1010101010101 0100101010100 110010101010 //Payloads DDC

Asset Structure Virtual Aaaset Texture Asset Texture Asset Structured Data (.uasset) Properties and other data published in the editor Components required as an asset (Required Data) .uasset Bulk Data (.upayload) Pixel-by-pixel rendering data and other data for display Components required for texture (Additional Data) By separating and managing assets, which are smaller data required for updating, according to their use

Virtual Assets Operation Flow //Game/Content //Payloads download //Payloads check out pull //Payloads check in Structured Data (.uasset) Bulk Data (.upayload) Sync PIE Edit Submit

Notes: Virtual Assets ● ● ● ● ● ● ● Network connection required, effective in high-speed network Use Perforce for Source Control Apply to new project, but be careful applying to projects under development Effectiveness varies depending on the project Even if you "only virtualize huge assets", you can benefit from it Less likely to benefit if not dealing with large size assets Small scale, small number of people, and single development are especially difficult to benefit from

Region: US - East Region: EU - West Accelerate Data Synchs Unreal Cloud DDC Production Ready for UE5.4 Bring your data closer to distributed teams • General content addressable storage • Active & passive region replication • Auto-connect to closest region • OIDC login & authentication • Don’t build what has already been built! Region Replication Region: Asia Perforce Sync Cloud DDC Studio Shared DDC • Source and docker image on Github • Production tested on AWS • Instructions available for Azure Local DDC Local DDC Local DDC Remote Users

• https://github.com/EpicGames/UnrealEngine
• https://github.com/MicrosoftDocs/azure-gaming-docs/blob/docs/azure-gaming-pr/unreal-cloud-ddc/quickstart-github.md

UnrealCloudDDC document https://github.com/EpicGames/UnrealEngine/tree/5.4/Engine/Source/Programs/UnrealCloudDDC

• https://github.com/EpicGames/UnrealEngine/tree/5.4/Engine/Source/Programs/UnrealCloudDDC

DDC ● ● ● ● ● Function to cache derived data (real data, etc.) of textures and meshes Accelerates access to large amounts of data by caching them in advance Cache is checked when each asset is loaded, and if it is not there, it is created If there is one, the cache is used to expand the data File access to LocalDDC is faster than SharedDDC

Saved Data in DDC Static Mesh Optimized vertex buffers and index buffers UV, LOD resources, Nanite resources, etc... Skeletal Mesh LOD resources Texture Output of compression tools (DXT binary) Material Compiled shadermap Shader bitecode Audio Streaming audio resources

Agenda ● Sync ● Build ○ Unreal Build Accelerator ● Cook ● Load

Unreal Build Accelerator (UBA) Beta for UE5.4 ● UBA is a local and distributed compilation solution for C++ Compilation used in conjunction with Unreal Build Tool and/or Horde’s Remote Build (compute task) system ● Horde is the coordinator to assign helper resources. UBA transmits local data, executes the remote task and returns the results ● ● Win64 platform + C++ compilation job Recommended for deployment infrastructure on the same physical and private network UBA

Horde Documentation https://github.com/EpicGames/UnrealEngine/blob/5.4/Engine/Source/Programs/Horde/Docs/Tutorials/RemoteCompilation.md

• https://github.com/EpicGames/UnrealEngine/blob/5.4/Engine/Source/Programs/Horde/Docs/Tutorials/RemoteCompilation.md

Distribution Build Steps Horde Server Build Machine UBA Local PC Horde Agent UBA Remote PC

Distribution Build Steps Horde Server 1) UBT request a Compute Task to the Horde Server Build Machine Local PC Remote PC

Distribution Build Steps 2) Horde Server allocates helpers to the requesting job (new or existing) Horde Server Build Machine Local PC Horde Agent Remote PC

Distribution Build Steps Build Machine UBA Local PC Horde Agent 3) UBA on Compile Agent uploads itself to Helper Agents Remote PC

4) UBA Helpers connect to UBA on the Compile Agent to assist with the Compile operation Distribution Build Steps Build Machine UBA Local PC UBA Remote PC

UBA Visualizer

Tips https://forums.unrealengine.com/t/unrealbuildaccelerator-uba/1826434

• https://forums.unrealengine.com/t/unrealbuildaccelerator-uba/1826434

Agenda ● Sync ● Build ● Cook ● ○ Multiprocess Cooking ○ Iterative Cooking ○ Incremental Cooking Load

Cooking in packaging flow Build Cook Load Assets LoadPackage Stage Package Save Assets Serialize Archive

Cooking in packaging flow Build Cook Stage .uasset .uasset .uexp .ubulk Package Archive UAsset Files Package Header data, load first file when the package is loaded Export Files Package archive data, load package import/export data when deploying Bulk Files Stores the actual data of the asset, loads platform-specific data, map data, etc. when needed

Multi-Process Cook Production Ready for UE5.4 Faster, parallel cooks • Leverage available CPU cores and memory resources by directing the cook process across multiple workers • Set the following parameter in Editor.ini [CookSettings] CookProcessCount=4 Or, set the cooking argument -CookProcessCount=4 Cook Director Cook Worker 1 Cook Worker 2 Cook Worker 3 Cook Worker 4

Cooking CitySample with 16 workers Cook Director Cook Worker 1 Cook Worker 2 Cook Worker 3 Cook Worker 4

Multiprocess cook architecture LoadPackage InQueue UnrealEditor-Cmd LoadPackage ForCooking Cook Director WorkerServer0 LoadPackage WorkerClient0 UnrealEditor-Cmd WorkerServer1 WorkerClient1 Load Serialize Cooking UnrealEditor-Cmd WorkerServer2 WorkerClient2 UnrealEditor-Cmd WorkerServer3 Create, Tick, Assign Launch, Snd, Rcv WorkerClient3

Multiprocess cook effectiveness (CitySample)

Cook Iterative Cooking Collect Files ● ● ● ● Iterative-cooking is an alternative to full-cooking Cooking with -iterative arguments Iterative-cooking is parameterized by the results of a previous cook Verify if there are cooked assets in /Saved/Cooked/ directory Requests exists? N Y Cooked Asset already exists? N Cooking Saving AssetRegistry Y

Iterative Cooking x14.0 faster Cooking time reduced from 815 to 58 s (12m37s saved)

Incremental Cooking ● Future optimization aims to incorporate incremental changes to config settings and code, minimizing redundant data re-cooking ● Limited handling of scenarios like partial map file changes, occasional false positives in cooking list ● Future focus on "incremental cooking" to reduce false positives in verification while maintaining iterative guarantees TBD: Iterative Cooking Loading Flow

Agenda ● Sync ● Build ● Cook ● Load ○ Zen Store ○ Zen Storage ○ Zen Loader

Zen Serise ● Zen Store ○ Feature to store cache data ● Zen Storage ○ Location of the cache data ○ Stored here if Zen Store is enabled ● Unreal Editor Zen Server Caching Zen Server ○ Application that manages the cache ○ Supports writing to and reading from Zen Storage Zen Storage Zen Store ● Zen Loader, Zen Cooker, Zen Snapshot, Zen Streaming, etc…

Zen Server ● ● ● Manage multiple data outside of editor Used to manage DDC locally Set the following parameter in Engine.ini [Zen] AutoLaunch=true https://github.com/EpicGames/zen

• https://github.com/EpicGames/zen

ZenServer sibling process launched by UnrealEditor UnrealEditor ● UnrealEditor can launch and manage lifetime of a local instance of ZenServer ● Hosts multiple distinct storage services ● Separate codebase with support for all our editor platforms Zen Server

Zen Storage as Local DDC Production Ready for UE5.4 DDC ● Replace use of local filesystem DDC ● Deduplicate DDC contents across workspaces on a machine ● Gives us more control over data caching than when using the filesystem directly UnrealEditor Resource ZenServe r DDC data

Zen Storage as Shared DDC Experimental for UE5.4 ● Replace use of network file shares as DDC storage ● Provide consistency across editor platforms DDC UnrealEditor Resource ZenServe r UnrealEditor UnrealEditor DDC data

Zen as Cooked Output Beta for UE5.4 UnrealEditor Cooked Assets Assets ZenServe r Target Device Cooked Assets Data ● Avoid writing to large numbers of loose files when cooking ● Opens the door to efficient network data streaming to target platforms

Comparison applying Zen Store (on PS5) Launch Application (s) Load Map (s) x1.8 faster x3.1 faster

Zen Loader ● ● ● ● New asset loading system with efficient File I/O and less blocking Shorter time to start and manipulate the Editor More efficient asset loading and faster map loading Set the following parameters in Engine.ini Game Thread Comp lete Wait Load Request Async Loading Thread2 Load IoService Wait Comp lete Read Seek/Read Data .ucas .uasset [/Script/Engine.EditorStreamingSettings] s.ZenLoaderEnabled=True s.AsyncLoadingThreadEnabled=True

Loading WireframeMaterial with Zen Loader is disabled (30.6ms) Blocking load with GameThread

Loading WireframeMaterial with Zen Loader is enabled (12.2ms: approx. x2.5 faster) Asynchronous DDC access

Testing for Zen Loader Launch Editor(s) x1.1 faster Open Map(s) x1.8 faster Start PIE(s) x3.1 faster

Conclusion ● ● ● ● Gain a better understanding of the features that can improve your development iterations Accelerate routine iterations such as cooks and packaging by applying them to your projects Check it quickly on the target device and save time to adjust the look and feel Consider yourself the best solution for your own project

Thank you!