Press Button, Drink Coffee : An Overview of UE4 build pipeline and maintenance【CEDEC 2019】

「Press Button, Drink Coffee」 An overview of UE4 build pipeline and maintenance

Introduction Axel Riffard Epic Games Japan Support Engineer ● French 🇫🇷 ● Twitter : @AxRiff ● Hobbies : Whisky, Cats and Disneyland ● Currently Playing: Dragon Quest XI, COD Mobile, The Witcher III (Switch) #UE4 | @UNREALENGINE

FIRST OF ALL … #UE4 | @UNREALENGINE

HELL YEAH POLAND ! #UE4 | @UNREALENGINE

Ruiner ! Frostpunk ! Dead Island ! Dying Light ! The Vanishing of Ethan Carter ! And … #UE4 | @UNREALENGINE

Geralt of Rivia ! <3 #UE4 | @UNREALENGINE

Ok, back on topic #UE4 | @UNREALENGINE

#UE4 | @UNREALENGINE

Disclaimer There is no such thing as THE perfect pipeline. Rely on experience ! In this session, I will only talk to you about mine and give you a few best practices. But donʼt trust me, trust yourself ! #UE4 | @UNREALENGINE

Agenda ● CI : What is it ? ● UnrealAutomationTool ● Building the Engine and your Project ● The other stuff ● The Unreal Way #UE4 | @UNREALENGINE

Agenda ● CI : What is it ? ● UnrealAutomationTool ● Building the Engine and your Project ● The other stuff ● The Unreal Way #UE4 | @UNREALENGINE

#UE4 | @UNREALENGINE

Why ? Dev 1 Dev 2 Dev 3 #UE4 | @UNREALENGINE Versioning System

Consequences ● Local build breaks after updating ● Finding the bad commit wastes time ● Communication issues, fights in the team ● Non technical members are scared to commit things ● Building locally every time is a waste #UE4 | @UNREALENGINE

CI Software CI = Continuous Integration A combination of software and hardware allowing to merge the team source code and assets several times every day. #UE4 | @UNREALENGINE

Why ? Dev 1 Dev 2 Dev 3 #UE4 | @UNREALENGINE Versioning System

Software #UE4 | @UNREALENGINE

CI Software In 2019, there is no bad choice. I have worked with both TeamCity and Jenkins, and was satisfied with both of them #UE4 | @UNREALENGINE

CI Software Jenkins #UE4 | @UNREALENGINE TeamCity

Jenkins #UE4 | @UNREALENGINE

Jenkins Merits︓ ● Free Open Source ● Plenty of documentation and plugins ● Huge Community Drawbacks︓ ● Commercial Support exists but is complicated ● Some plugins are unreliable #UE4 | @UNREALENGINE

TeamCity #UE4 | @UNREALENGINE

TeamCity Merits : ● Building pipelines is crazy fast ● Technical support is great Drawbacks : ● Limitations on the Free version #UE4 | @UNREALENGINE

CI Software Every advice can be applied to TeamCity, or any CI solution. Also, totally unrelated, but this cat looks like Jenkins mascot, so Iʼm going to use him instead. #UE4 | @UNREALENGINE

Hardware #UE4 | @UNREALENGINE

Build Machine Master → Slave → #UE4 | @UNREALENGINE Manages the tasks. Does the tasks.

Build Machine DEV PC Commit Versioning Versioning Poll CI Server Master Slave 1 #UE4 | @UNREALENGINE Slave 2 Slack Distribute Slave 3 Company Server

Build Machine – How many Prototyping Stage : ● Just the Master Production Stage : ● One Master ● One slave for every release platform #UE4 | @UNREALENGINE

Build Machine - OS Master︓Win64 or Linux Slave︓Win64（for platforms SDK), MacOS for Apple Platforms #UE4 | @UNREALENGINE

Build Machine - Windows Storage︓SSD Memory ︓32GB or more CPU︓i7 / i9 #UE4 | @UNREALENGINE

Build Machine - MacOS No C++ and Shader compiling distribution on Mac, So get as many cores as you can for your CPU. If itʼs just for a build machine, no need for GPU. High spec Mac Mini is good enough. #UE4 | @UNREALENGINE

Agenda ● Continuous Integration : What is it ? ● UnrealAutomationTool ● Building the Engine and your Project ● The other stuff ● The Unreal Way #UE4 | @UNREALENGINE

Building in UE4 ● ● ● ● Engine and Editor Game sources Game Assets (lights, shaders, etc.) Tools #UE4 | @UNREALENGINE

Build Commands Find all the command lines here https://github.com/AxRiff/UnrealEngineJenkins #UE4 | @UNREALENGINE

UnrealAutomationTool UnrealAutomationTool is a complete system to build and test your games inside of Unreal Engine UE4 is mainly written in C++, but UnrealAutomationTool is made in C# #UE4 | @UNREALENGINE

UnrealAutomationTool Depending on your company size, type of project, workflow, etc., you can use different parts of UnrealAutomationTool ● BuildGraph Custom Script ● BuildGraph InstalledBuild Script ● BuildCookRun Command Line Tool #UE4 | @UNREALENGINE

BuildGraph #UE4 | @UNREALENGINE

BuildGraph BuildGraph is a script based build automation system inside of UnrealAutomationTool ● XML ● By Epic for Epic ● Original approach #UE4 | @UNREALENGINE

BuildGraph Youʼve got plenty of sample codes in Engine/Build/Graph/Examples/ #UE4 | @UNREALENGINE

BuildGraph Merits : ● Reduce dependency on CI framework. ● Work on it locally -> No more Jenkins master ● XML Based : easy to read and write ● Platform Agnostic ● Can be shared between projects Drawbacks : ● Steep learning curve at first #UE4 | @UNREALENGINE

BuildGraph ● Agents: Group of nodes being executed on the same build machine ● Nodes: Sequence of Tasks ● Tasks: Build Process Step (Updating, Compiling, etc) #UE4 | @UNREALENGINE

BuildGraph <Agent Name="Default Agent" Type="CompileWin64"> <!-- Update the engine version files --> <Node Name="Update Version Files"> <SetVersion Change="$(Change)" Branch="$(EscapedBranch)" Licensee="$(Licensee)" If="$(Versioned)"/> </Node> <!-- Compile the tool executables --> <Node Name="Compile Tools Win64" Requires="Update Version Files" Produces="#ToolBinaries"> <Compile Target="UnrealHeaderTool" Platform="Win64" Configuration="Development" Tag="#ToolBinaries"/> </Node> <!-- Compile the editor executable --> <Node Name="Compile $(EditorTarget) Win64" Requires="Compile Tools Win64" Produces="#EditorBinaries"> <Compile Target="$(EditorTarget)" Platform="Win64" Configuration="Development" Tag="#EditorBinaries"/> </Node> <!-- Compile the game targets --> <Property Name="GameBinaries" Value=""/> <ForEach Name="GameTarget" Values="$(GameTargets)"> <ForEach Name="TargetPlatform" Values="$(TargetPlatforms)"> <Node Name="Compile $(GameTarget) $(TargetPlatform)" Requires="Compile Tools Win64" Produces="#GameBinaries_$(GameTarget)_$(TargetPlatform)"> <Compile Target="$(GameTarget)" Platform="$(TargetPlatform)" Configuration="Development" Tag="#GameBinaries_$(GameTarget)_$(TargetPlatform)"/> </Node> <Property Name="GameBinaries" Value="$(GameBinaries)#GameBinaries_$(GameTarget)_$(TargetPlatform);"/> </ForEach> </ForEach> </Agent> #UE4 | @UNREALENGINE

BuildGraph <Agent Name="Default Agent" Type="CompileWin64"> <!-- Update the engine version files --> <Node Name="Update Version Files"> <SetVersion Change="$(Change)" Branch="$(EscapedBranch)" Licensee="$(Licensee)" If="$(Versioned)"/> </Node> <!-- Compile the tool executables --> <Node Name="Compile Tools Win64" Requires="Update Version Files" Produces="#ToolBinaries"> <Compile Target="UnrealHeaderTool" Platform="Win64" Configuration="Development" Tag="#ToolBinaries"/> </Node> <Agent Name="Default Agent" Type="CompileWin64"> <!-- Compile the editor executable --> <Node Name="Compile $(EditorTarget) Win64" Requires="Compile Tools Win64" Produces="#EditorBinaries"> <Compile Target="$(EditorTarget)" Platform="Win64" Configuration="Development" Tag="#EditorBinaries"/> </Node> <!-- Compile the game targets --> <Property Name="GameBinaries" Value=""/> <ForEach Name="GameTarget" Values="$(GameTargets)"> <ForEach Name="TargetPlatform" Values="$(TargetPlatforms)"> <Node Name="Compile $(GameTarget) $(TargetPlatform)" Requires="Compile Tools Win64" Produces="#GameBinaries_$(GameTarget)_$(TargetPlatform)"> <Compile Target="$(GameTarget)" Platform="$(TargetPlatform)" Configuration="Development" Tag="#GameBinaries_$(GameTarget)_$(TargetPlatform)"/> </Node> <Property Name="GameBinaries" Value="$(GameBinaries)#GameBinaries_$(GameTarget)_$(TargetPlatform);"/> </ForEach> </ForEach> </Agent> #UE4 | @UNREALENGINE

BuildGraph <Agent Name="Default Agent" Type="CompileWin64"> <!-- Update the engine version files --> <Node Name="Update Version Files"> <SetVersion Change="$(Change)" Branch="$(EscapedBranch)" Licensee="$(Licensee)" If="$(Versioned)"/> </Node> <!-- Compile the tool executables --> <Node Name="Compile Tools Win64"version Requires="Update <!-Update the engine filesVersion --> Files" Produces="#ToolBinaries"> <Compile Target="UnrealHeaderTool" Platform="Win64" Configuration="Development" Tag="#ToolBinaries"/> </Node> <Node Name="Update Version Files"> <SetVersion Change="$(Change)" Branch="$(EscapedBranch)" <!-- Compile the editor executable --> <Node Name="Compile $(EditorTarget) Win64" Requires="Compile Tools Win64" Produces="#EditorBinaries"> Licensee="$(Licensee)" If="$(Versioned)"/> <Compile Target="$(EditorTarget)" Platform="Win64" Configuration="Development" Tag="#EditorBinaries"/> </Node> </Node> <!-- Compile the game targets --> <Property Name="GameBinaries" Value=""/> <ForEach Name="GameTarget" Values="$(GameTargets)"> <ForEach Name="TargetPlatform" Values="$(TargetPlatforms)"> <Node Name="Compile $(GameTarget) $(TargetPlatform)" Requires="Compile Tools Win64" Produces="#GameBinaries_$(GameTarget)_$(TargetPlatform)"> <Compile Target="$(GameTarget)" Platform="$(TargetPlatform)" Configuration="Development" Tag="#GameBinaries_$(GameTarget)_$(TargetPlatform)"/> </Node> <Property Name="GameBinaries" Value="$(GameBinaries)#GameBinaries_$(GameTarget)_$(TargetPlatform);"/> </ForEach> </ForEach> </Agent> #UE4 | @UNREALENGINE

BuildGraph <Agent Name="Default Agent" Type="CompileWin64"> <!-- Update the engine version files --> <Node Name="Update Version Files"> <SetVersion Change="$(Change)" Branch="$(EscapedBranch)" Licensee="$(Licensee)" If="$(Versioned)"/> </Node> <!-- Compile the tool executables --> <Node Name="Compile Tools Win64" Requires="Update Version Files" Produces="#ToolBinaries"> <Compile Target="UnrealHeaderTool" Platform="Win64" Configuration="Development" Tag="#ToolBinaries"/> </Node> <!-Compile the tool executables --> <!-- Compile the editor executable --> <Node Name="Compile $(EditorTarget) Win64" Requires="Compile Win64" Produces="#EditorBinaries"> <Node Name="Compile Tools Win64"Tools Requires="Update Version Files" <Compile Target="$(EditorTarget)" Platform="Win64" Configuration="Development" Tag="#EditorBinaries"/> </Node> Produces="#ToolBinaries"> <!-- Compile<Compile the game targets --> Target="UnrealHeaderTool" Platform="Win64" <Property Name="GameBinaries" Value=""/> <ForEach Name="GameTarget" Values="$(GameTargets)"> Configuration="Development" Tag="#ToolBinaries"/> <ForEach Name="TargetPlatform" Values="$(TargetPlatforms)"> <Node Name="Compile $(GameTarget) $(TargetPlatform)" Requires="Compile Tools Win64" </Node> Produces="#GameBinaries_$(GameTarget)_$(TargetPlatform)"> </ForEach> </Agent> #UE4 | @UNREALENGINE </ForEach> <Compile Target="$(GameTarget)" Platform="$(TargetPlatform)" Configuration="Development" Tag="#GameBinaries_$(GameTarget)_$(TargetPlatform)"/> </Node> <Property Name="GameBinaries" Value="$(GameBinaries)#GameBinaries_$(GameTarget)_$(TargetPlatform);"/>

BuildGraph <Agent Name="Default Agent" Type="CompileWin64"> <!-- Update the engine version files --> <Node Name="Update Version Files"> <SetVersion Change="$(Change)" Branch="$(EscapedBranch)" Licensee="$(Licensee)" If="$(Versioned)"/> </Node> <!-- Compile the tool executables --> <Node Name="Compile Tools Win64" Requires="Update Version Files" Produces="#ToolBinaries"> <Compile Target="UnrealHeaderTool" Platform="Win64" Configuration="Development" Tag="#ToolBinaries"/> </Node> <!-- Compile the editor executable --> <Node Name="Compile $(EditorTarget) Win64" Requires="Compile Tools Win64" Produces="#EditorBinaries"> <Compile Target="$(EditorTarget)" Platform="Win64" Configuration="Development" Tag="#EditorBinaries"/> </Node> <!--Compile Compile the game --> executable --> <!-thetargets editor <Property Name="GameBinaries" Value=""/> <Node Name="Compile $(EditorTarget) Win64" Requires="Compile Tools Win64" <ForEach Name="GameTarget" Values="$(GameTargets)"> <ForEach Name="TargetPlatform" Values="$(TargetPlatforms)"> Produces="#EditorBinaries"> <Node Name="Compile $(GameTarget) $(TargetPlatform)" Requires="Compile Tools Win64" Produces="#GameBinaries_$(GameTarget)_$(TargetPlatform)"> <Compile Target="$(EditorTarget)" Platform="Win64" <Compile Target="$(GameTarget)" Platform="$(TargetPlatform)" Configuration="Development" Configuration="Development" Tag="#EditorBinaries"/> Tag="#GameBinaries_$(GameTarget)_$(TargetPlatform)"/> </Node> </Node> <Property Name="GameBinaries" Value="$(GameBinaries)#GameBinaries_$(GameTarget)_$(TargetPlatform);"/> </ForEach> </Agent> #UE4 | @UNREALENGINE </ForEach>

BuildGraph <!-- Compile the game targets --> <Property Name="GameBinaries" Value=""/> <ForEach Name="GameTarget" Values="$(GameTargets)"> <ForEach Name="TargetPlatform" Values="$(TargetPlatforms)"> <Node <!-- Compile the tool executables --> Name="Compile $(GameTarget) $(TargetPlatform)" <Node Name="Compile Tools Win64" Requires="Update Version Files" Produces="#ToolBinaries"> Requires="Compile Tools Win64" <Compile Target="UnrealHeaderTool" Platform="Win64" Configuration="Development" Tag="#ToolBinaries"/> </Node>Produces="#GameBinaries_$(GameTarget)_$(TargetPlatform)"> <Compile Target="$(GameTarget)" Platform="$(TargetPlatform)" <!-- Compile the editor executable --> <Node Name="Compile $(EditorTarget) Win64" Requires="Compile Tools Win64" Produces="#EditorBinaries"> Configuration="Development" <Compile Target="$(EditorTarget)" Platform="Win64" Configuration="Development" Tag="#EditorBinaries"/> </Node> Tag="#GameBinaries_$(GameTarget)_$(TargetPlatform)"/> <!-- Compile the game targets --> </Node> <Property Name="GameBinaries" Value=""/> <Property Name="GameBinaries" <ForEach Name="GameTarget" Values="$(GameTargets)"> <ForEach Name="TargetPlatform" Values="$(TargetPlatforms)"> Value="$(GameBinaries)#GameBinaries_$(GameTarget)_$(TargetPlatform);"/> <Node Name="Compile $(GameTarget) $(TargetPlatform)" Requires="Compile Tools Win64" Produces="#GameBinaries_$(GameTarget)_$(TargetPlatform)"> </ForEach> <Compile Target="$(GameTarget)" Platform="$(TargetPlatform)" Configuration="Development" Tag="#GameBinaries_$(GameTarget)_$(TargetPlatform)"/> </ForEach> <Agent Name="Default Agent" Type="CompileWin64"> <!-- Update the engine version files --> <Node Name="Update Version Files"> <SetVersion Change="$(Change)" Branch="$(EscapedBranch)" Licensee="$(Licensee)" If="$(Versioned)"/> </Node> </ForEach> </Agent> #UE4 | @UNREALENGINE </ForEach> </Node> <Property Name="GameBinaries" Value="$(GameBinaries)#GameBinaries_$(GameTarget)_$(TargetPlatform);"/>

InstalledBuild #UE4 | @UNREALENGINE

InstalledBuild ● Think of it as your own Unreal Launcher ● Created by Epic to deploy a non editable engine Engine/Build/InstalledEngineBuild.xml ● Easy to use for Designers and Artists ● Engine build only. Canʼt use it to build a project. #UE4 | @UNREALENGINE

InstalledBuild Installed Build mains steps 1. 2. 3. 4. 5. 6. 7. 8. 9. Update Version File Compile UnrealHeaderTool Win64 Compile UE4Editor Win64 Compile UE4Game Win64 Compile UE4Game (Platform) Build Tools Win64 Build Tools CS Make Feature Packs Build DDC Win64 #UE4 | @UNREALENGINE

InstalledBuild Engine/Build/BatchFiles/RunUAT.bat BuildGraph -target”Make Installed Build Win64” -script=/Engine/Build/InstalledEngineBuild.xml -clean #UE4 | @UNREALENGINE

InstalledBuild Engine/Build/BatchFiles/RunUAT.bat BuildGraph -target”Make Installed Build Win64” -script=/Engine/Build/InstalledEngineBuild.xml -clean #UE4 | @UNREALENGINE

InstalledBuild Engine/Build/BatchFiles/RunUAT.bat BuildGraph -target”Make Installed Build Win64” -script=/Engine/Build/InstalledEngineBuild.xml -clean #UE4 | @UNREALENGINE

InstalledBuild Engine/Build/BatchFiles/RunUAT.bat BuildGraph -target”Make Installed Build Win64” -script=/Engine/Build/InstalledEngineBuild.xml -clean #UE4 | @UNREALENGINE

BuildCookRun #UE4 | @UNREALENGINE

BuildCookRun #UE4 | @UNREALENGINE

BuildCookRun ● Script located at /Engine/Build/BatchFiles/ ● From build to package, do anything ● Highly customizable ● Command line based : nostalgic feeling ! <3 #UE4 | @UNREALENGINE

BuildCookRun, BuildGraph, InstalledBuild ● Depending on the situation, the best choice will differ ● I think adaptability is the most important thing in game dev, so I personally use BuildCookRun most of the time. #UE4 | @UNREALENGINE

Agenda ● CI : What is it ? ● UnrealAutomationTool ● Building the Engine and your Project ● The other stuff ● The Unreal Way #UE4 | @UNREALENGINE

Command Lines #UE4 | @UNREALENGINE

Engine DLL Build If you donʼt edit the Engine a lot, trigger it manually If you extend the Engine a lot, do it once a day. Morning or Lunch Time is preferred #UE4 | @UNREALENGINE

Build Machine Commit to Source Control Jenkins Update Build Notify #UE4 | @UNREALENGINE

Build Command %ENGINE_ROOT%/Engine/Build/BatchFiles/RunUAT.bat BuildCookRun project=Samples/StarterContent/StarterContent.uproject platform=%ENGINE_PLATFORM_TARGET% clientconfig=%ENGINE_COMPILATION_TARGET% -build ENGINE_ROOT=EngineRoot ENGINE_COMPILATION_TARGET=Development ENGINE_PLATFORM_TARGET=Win64 Yup, thatʼs a little hard to read #UE4 | @UNREALENGINE

Build Command Engine/Build/BatchFiles/RunUAT.bat BuildCookRun -project=Samples/StarterContent/StarterContent.uproject -platform=Win64 -clientconfig=Development -build #UE4 | @UNREALENGINE

Build Command Engine/Build/BatchFiles/RunUAT.bat BuildCookRun -project=Samples/StarterContent/StarterContent.uproject -platform=Win64 -clientconfig=Development -build You need a uproject to use RunUAT. You can use a sample project or blank one ! #UE4 | @UNREALENGINE

Build Command Engine/Build/BatchFiles/RunUAT.bat BuildCookRun -project=Samples/StarterContent/StarterContent.uproject -platform=Win64 -clientconfig=Development -build #UE4 | @UNREALENGINE

Build Command Engine/Build/BatchFiles/RunUAT.bat BuildCookRun -project=Samples/StarterContent/StarterContent.uproject -platform=Win64 -clientconfig=Development -build #UE4 | @UNREALENGINE

Build Command Engine/Build/BatchFiles/RunUAT.bat BuildCookRun -project=Samples/StarterContent/StarterContent.uproject -platform=Win64 -clientconfig=Development -build #UE4 | @UNREALENGINE

Project DLL build Poll every 10 minutes to build Commit to Source Control Jenkins Update Build Notify #UE4 | @UNREALENGINE

Build Command %ENGINE_ROOT%/Engine/Build/BatchFiles/RunUAT.bat BuildCookRun -project= project=%GAME_SUB_PATH%/%GAME_NAME%.uproject platform=%ENGINE_PLATFORM_TARGET% clientconfig=%ENGINE_COMPILATION_TARGET% -build ENGINE_ROOT=EngineRoot ENGINE_COMPILATION_TARGET=Development ENGINE_PLATFORM_TARGET=Win64 GAME_SUB_PATH=C:/GamePath GAME_NAME=GameName #UE4 | @UNREALENGINE

Build Command Engine/Build/BatchFiles/RunUAT.bat BuildCookRun -project=C:/PathToTheGame/GameName.uproject -platform=Win64 -clientconfig=Development -build #UE4 | @UNREALENGINE

#UE4 | @UNREALENGINE

Packaging #UE4 | @UNREALENGINE

Nightly Build ● Well, itʼs build every night. ● Development, Test（for profiling), Shipping for every release platform. Time doesnʼt matter, so clean build #UE4 | @UNREALENGINE

Nightly Build Flow Copy to Shared Folder Jenkins Update Build Zip Notify #UE4 | @UNREALENGINE

Build Command %ENGINE_ROOT%/Engine/Build/BatchFiles/RunUAT.bat BuildCookRun project=%GAME_SUB_PATH%/%GAME_NAME%.uproject -noP4 platform=%ENGINE_PLATFORM_TARGET% clientconfig=%ENGINE_COMPILATION_TARGET% -cook -allmaps -build -stage -pak -archive -archivedirectory=%TEMP_DIR% ENGINE_ROOT=PathToYourEngine GAME_SUB_PATH=PathToYourProject GAME_NAME=UprojectFileWithoutTheExtension ENGINE_COMPILATION_TARGET=Development|Shipping|etc ENGINE_PLATFORM_TARGET=Win64|etc #UE4 | @UNREALENGINE

Build Command Engine/Build/BatchFiles/RunUAT.bat BuildCookRun -project=GamePath/GameName.uproject -noP4 -platform=Win64 -clientconfig=Development -cook -allmaps -build -stage -pak -archive -archivedirectory=C:/TempDirectory/ #UE4 | @UNREALENGINE

Build Command Engine/Build/BatchFiles/RunUAT.bat BuildCookRun -project=GamePath/GameName.uproject -noP4 -platform=Win64 -clientconfig=Development -cook -allmaps -build -stage -pak -archive -archivedirectory=C:/TempDirectory/ #UE4 | @UNREALENGINE

Build Command Engine/Build/BatchFiles/RunUAT.bat BuildCookRun -project=GamePath/GameName.uproject -noP4 -platform=Win64 -clientconfig=Development -cook -allmaps -build -stage -pak -archive -archivedirectory=C:/TempDirectory/ #UE4 | @UNREALENGINE

Build Command Engine/Build/BatchFiles/RunUAT.bat BuildCookRun -project=GamePath/GameName.uproject -noP4 -platform=Win64 -clientconfig=Development -cook -allmaps -build -stage -pak -archive -archivedirectory=C:/TempDirectory/ #UE4 | @UNREALENGINE

Build Command Engine/Build/BatchFiles/RunUAT.bat BuildCookRun -project=GamePath/GameName.uproject -noP4 -platform=Win64 -clientconfig=Development -cook -allmaps -build -stage -pak -archive -archivedirectory=C:/TempDirectory/ #UE4 | @UNREALENGINE

Build Command Engine/Build/BatchFiles/RunUAT.bat BuildCookRun -project=GamePath/GameName.uproject -noP4 -platform=Win64 -clientconfig=Development -cook -allmaps -build -stage -pak -archive -archivedirectory=C:/TempDirectory/ #UE4 | @UNREALENGINE

Build Command Engine/Build/BatchFiles/RunUAT.bat BuildCookRun -project=GamePath/GameName.uproject -noP4 -platform=Win64 -clientconfig=Development -cook -allmaps -build -stage -pak -archive -archivedirectory=C:/TempDirectory/ #UE4 | @UNREALENGINE

Build Command Engine/Build/BatchFiles/RunUAT.bat BuildCookRun -project=GamePath/GameName.uproject -noP4 -platform=Win64 -clientconfig=Development -cook -allmaps -build -stage -pak -archive -archivedirectory=C:/TempDirectory/ #UE4 | @UNREALENGINE

Build Command Engine/Build/BatchFiles/RunUAT.bat BuildCookRun -project=GamePath/GameName.uproject -noP4 -platform=Win64 -clientconfig=Development -cook -allmaps -build -stage -pak -archive -archivedirectory=C:/TempDirectory/ #UE4 | @UNREALENGINE

Build Command Engine/Build/BatchFiles/RunUAT.bat BuildCookRun -project=GamePath/GameName.uproject -noP4 -platform=Win64 -clientconfig=Development -cook -allmaps -build Argument to zip all the assets in one “pak” file -stage -pak -archive -archivedirectory=C:/TempDirectory/ #UE4 | @UNREALENGINE

Build Command Engine/Build/BatchFiles/RunUAT.bat BuildCookRun -project=GamePath/GameName.uproject -noP4 -platform=Win64 -clientconfig=Development -cook -allmaps -build -stage -pak -archive -archivedirectory=C:/TempDirectory/ #UE4 | @UNREALENGINE

Build Command Engine/Build/BatchFiles/RunUAT.bat BuildCookRun -project=GamePath/GameName.uproject -noP4 -platform=Win64 -clientconfig=Development -cook -allmaps -build -stage -pak -archive -archivedirectory=C:/TempDirectory/ #UE4 | @UNREALENGINE

Build Command robocopy %ARCHIVE_DIR%/latest %ARCHIVE_DIR%/%ENGINE_PLATF ORM_TARGET%/archive /E /IS /MOV set PATH=%PATH%;C:/Program Files/7-Zip/ 7z.exe a -tzip -mmt mx1 %ARCHIVE_DIR%/latest/%BUILD_NUMBER%.zip %TEMP_DIR%/ exit 0 #UE4 | @UNREALENGINE

Build Command Wait a sec !! #UE4 | @UNREALENGINE

Build Command EA DICE Studios has open sourced EACopy, a faster alternative to robocopy ! Give it a try !!! https://github.com/electronicarts/EACopy Twitter : @honk_dice #UE4 | @UNREALENGINE

#UE4 | @UNREALENGINE

Incremental Build ● Manual Trigger ● For those “Hey mate, I need to show the build in 40 minutes to our investor / boss / publisher, do smth plz lolwut” ● Contrary to the nightly build, faster is better so … ● Use -partialgc ● Donʼt make it clean #UE4 | @UNREALENGINE

Incremental Build Flow Send to DevKit Jenkins Update Build Notify For DevKit deployment, check your platform But you might get an idea /Engine/Source/Programs/AutomationTool/PlatformName/PlatformNamePlatfo rm.Automation.cs #UE4 | @UNREALENGINE

Build Command Engine/Build/BatchFiles/RunUAT.bat BuildCookRun -project=GamePath/GameName.uproject -noP4 -platform=Win64 -clientconfig=Development -cook -allmaps Instead of cleaning the memory when you run out of it, it will free it -build up at the end of the packaging process -stage -pak -archive -partialgc -archivedirectory=C:/TempDirectory/ #UE4 | @UNREALENGINE

#UE4 | @UNREALENGINE

Jenkins Plugins #UE4 | @UNREALENGINE

Plugins Build Pipeline Plugin https://wiki.jenkins.io/display/JENKINS/Build+Pipeline+Plugin #UE4 | @UNREALENGINE

Plugins Jenkins Slack Plugin https://github.com/jenkinsci/slack-plugin #UE4 | @UNREALENGINE

Plugins Retaliation https://github.com/codedance/Retaliation #UE4 | @UNREALENGINE

Agenda ● CI : What is it ? ● UnrealAutomationTool ● Building the Engine and your Project ● The other stuff ● The Unreal Way #UE4 | @UNREALENGINE

Other CI Jobs #UE4 | @UNREALENGINE

Commandlet Command Line Applet is a command program to be run inside of Unreal Engine environment. #UE4 | @UNREALENGINE

Shared DDC Derived Data Cache (DDC) is a feature to keep assets that were prepared for every platform. To make it simple, itʼs an asset caching feature. #UE4 | @UNREALENGINE

Shared DDC Dev PC 1 Build Assets on 1st launch #UE4 | @UNREALENGINE Copy Built Asset Shared Folder

Shared DDC Dev PC 2 Dev PC 3 Shared Folder Dev PC 4 #UE4 | @UNREALENGINE Copy Built Asset Dev PC 5

Shared DDC Dev PC 4 Modify / Add Asset #UE4 | @UNREALENGINE Copy Built Asset Shared Folder

Shared DDC https://docs.unrealengine.com/enUS/Engine/Basics/DerivedDataCache/index.html BaseEngine.ini [InstalledDerivedDataBackendGraph] MinimumDaysToKeepFile=7 Root=(Type=KeyLength, Length=120, Inner=AsyncPut) AsyncPut=(Type=AsyncPut, Inner=Hierarchy) Hierarchy=(Type=Hierarchical, Inner=Boot, Inner=Pak, Inner=EnginePak, Inner=Local, Inner=Shared) Boot=(Type=Boot, Filename=%GAMEDIR%DerivedDataCache/Boot.ddc, MaxCacheSize=256) Local=(Type=FileSystem, ReadOnly=false, Clean=false, Flush=false, PurgeTransient=true, DeleteUnused=true, UnusedFileAge=17, FoldersToClean=-1, Path=../../../Engine/DerivedDataCache) Shared=(Type=FileSystem, ReadOnly=false, Clean=false, Flush=false, DeleteUnused=true, UnusedFileAge=19, FoldersToClean=-1, Path=//mystudio.net/DDC, EnvPathOverride=UE-SharedDataCachePath) AltShared=(Type=FileSystem, ReadOnly=true, Clean=false, Flush=false, DeleteUnused=true, UnusedFileAge=19, FoldersToClean=-1, Path=//mystudio.net/DDC2, EnvPathOverride=UE-SharedDataCachePath2) Pak=(Type=ReadPak, Filename=%GAMEDIR%DerivedDataCache/DDC.ddp) EnginePak=(Type=ReadPak, Filename=../../../Engine/DerivedDataCache/DDC.ddp) #UE4 | @UNREALENGINE

Shared DDC Argument Type Comment ReadOnly Bool Make it false when Shared Clean Bool Delete files that went over UnusedFileAge limit Flush Bool Delete all DDC DeleteUnused Bool Il false, never deletes old assets (overrides UnusedFileAge and Clean) UnusedFileAge Int Limit of days after which an asset will be deleted, Fortnite sets it to 10. FoldersToClean Int Max limit of folders to be flushed each time. ０ or -1 means no limit. #UE4 | @UNREALENGINE

Shared DDC
// Flush and Clean argument DerivedDataBackends.cpp:553
if( bFlush )
{
IFileManager::Get().DeleteDirectory( *(Path / TEXT(“”)), false, true );
}
else if( bClean )
{
DeleteOldFiles( *Path );
}

/** Delete the old files in a directory **/

void DeleteOldFiles(const TCHAR* Directory)
{
float MinimumDaysToKeepFile = 7;
GConfig->GetFloat( *GraphName, TEXT("MinimumDaysToKeepFile"), MinimumDaysToKeepFile, GEngineIni );
check(MinimumDaysToKeepFile > 0.0f); // sanity

//@todo
}

#UE4 | @UNREALENGINE

Shared DDC //@todo #UE4 | @UNREALENGINE

Shared DDC del /s /q c:/ServerShareDDCPath/* UE4Editor-cmd.exe ProjectName –run=DerivedDataCache -fill #UE4 | @UNREALENGINE

Lighting Build ● All lights except the ones calculated in real time are to be built. ● Of course, Epic is providing the tools for it. But … #UE4 | @UNREALENGINE

Lighting Build #UE4 | @UNREALENGINE

Lighting Build Itʼs SUPER heavy and slow Let your CI pipe do it ! #UE4 | @UNREALENGINE

Lighting Build UE4Editor-cmd.exe ProjectName -run=resavepackages -buildlighting -allowcommandletrendering #UE4 | @UNREALENGINE

Lighting Build There are times you want to check the lights in the editor / game In this case, build locally, and Swarm is your new BFF #UE4 | @UNREALENGINE

Lighting Build - Swarm With Swarm, you can distribute the task of building lights between the computers of your team. You will need a Coordinator. Your Jenkins master should do the trick. https://wiki.unrealengine.com/Swarm_Agent_Trou bleshooting #UE4 | @UNREALENGINE

Lighting Build - Swarm Engine/Binaries/DotNET/SwarmCoordinator.exe Engine/Binaries/DotNET/SwarmAgent.exe To add it at startup, Create a shortcut the the Agent and copy it here. %APPDATA%/Microsoft/Windows/Start Menu/Programs/Startup #UE4 | @UNREALENGINE

Lighting Build - Swarm #UE4 | @UNREALENGINE

Redirectors ● When you move or rename an asset in the editor, a Redirector is created to know the ”new address” ● Super convenient feature but it has a tendency to bug, so you will need to clean them up frequently ● Once a month is fine, but do it once a week in the critical phases of the project (alpha, beta, gold, patch release) https://docs.unrealengine.com/en-US/Engine/Basics/Redirectors/index.html #UE4 | @UNREALENGINE

Redirectors UE4Editor-cmd.exe ProjectName -run=fixupredirects #UE4 | @UNREALENGINE

Agenda ● CI : What is it ? ● UnrealAutomationTool ● Building the Engine and your Project ● The other stuff ● The Unreal Way #UE4 | @UNREALENGINE

The Unreal Way : Fortnite #UE4 | @UNREALENGINE

Fortnite ● Only engineers have Visual Studio ● We are building with BuildGraph managed by ElectricCommander ● We are heavily using UnrealGameSync #UE4 | @UNREALENGINE

Iteration Iteration can seem complicated without Visual Studio. But actually, itʼs pretty manageable. ● Heavily rely on PIE for testing ● Leverage your CI software properly #UE4 | @UNREALENGINE

Testing a Level on Device So letʼs say your artist has just changed a characterʼs hair color and wants to see how it looks on the release platform screen before committing… First, create an incremental build task on your CI software, and commit the binaries to Project/Binaries/PlatformName #UE4 | @UNREALENGINE

Custom Launch Profile Then in the editor, click on the triangle on the right side of of “Launch”, scroll down and create a new Custom Launch Profile Stored in /Engine/Programs/UnrealFrontend/Profiles In the profile, untick the Build step, and then you can deploy the currently edited level on the machine connected to the computer, even without Visual Studio. Magic !! #UE4 | @UNREALENGINE

Build Machine Jobs What I think a UE4 build pipeline should have : ● Engine Build ● Project Build ● Nightly Package ● Incremental Package ● Incremental Package for the Device Launcher ● Lightmaps Build ● Fixup Redirectors Command ● SharedDDC Cleanup Command ● SharedDDC Fillup Command Any suggestion on what to add ? Tell me in Q&A !!! #UE4 | @UNREALENGINE

Versioning ● Epic uses Perforce. If you want to be as close as possible to our workstyle, you know what to do ● SVN might be a little slow, but it is very reliable and still heavily used ● GIT is free, modern design choices and is very used for mobile and social games (in Japan !) SVN and Perforce have Editor support ! GIT too, but itʼs still experimental ! #UE4 | @UNREALENGINE

UnrealGameSync Made by Epic for Epic. Heavily used on Fortnite Itʼs Perforce only, and it allows non technical staff not to have to deal with P4V #UE4 | @UNREALENGINE

UnrealGameSync #UE4 | @UNREALENGINE

UnrealGameSync Sync Get Engine P4 Server Local Builds (Binary Engine) Build Machine #UE4 | @UNREALENGINE

UnrealGameSync Get the latest version anytime on GitHub and Perforce. UGS dev is unrelated to UE4, so always get the latest version even if youʼve locked your engine version ! https://docs.unrealengine.com/latest/en/Programming/Deployment/Unre alGameSync/Reference/ #UE4 | @UNREALENGINE

Links https://github.com/gpageot/JenkinsUE4 Twitter : @GregoryPAGEOT https://github.com/AxRiff/UnrealEngineJenkins Twitter : @AxRiff https://github.com/electronicarts/EACopy Twitter : @honk_dice Manga by @shiba_zushi, Technical Artist at Epic Games Japan. #UE4 | @UNREALENGINE

Merci ! #UE4 | @UNREALENGINE