Automated Checks to Ensure Stable VFX Behavior

Automated Checks to Ensure Stable VFX Behavior I would like to talk about our "automatic checking" efforts to ensure stable operation of VFX. ©CAPCOM 1

Agenda • Implementation Background • Automatic Check Flow • Difference Checking by Image Comparison • Results Here's today's agenda: • • • • How the automatic checks were introduced. The automatic check flow. “Difference Check by Image Comparison“ (One of the automatic checks) The results of implementing these automatic checks. ©CAPCOM 2 2

Implementation Background First, let's look at how we introduced this system. 3 ©CAPCOM 3

Bugs and errors are sometimes inadvertently introduced during refactoring, or when adding new features. When you implement a new feature or do some refactoring. There are times when a defect is accidentally introduced. 4 ©CAPCOM 4

Implementation Background Bug added Bug added Support cost: Medium Bugs that change behavior significantly are relatively easy to notice and get reported quickly. Support cost: High Bugs that only cause slight changes mostly go unnoticed and only get reported much later. These images are an extreme example, but... Bugs that cause a major change in behavior, such as an effect rotated 90 degrees, are quickly reported and can be 5 dealt with easily enough. However, bugs that cause minor changes, such as the left and right sides being reversed, or bugs in functions that are used infrequently, tend to go unnoticed. You get a report much later from the title side, saying that the behavior has changed, or that the behavior is strange. If a report is received after a long period of time, it may be necessary to check the code and behavior with older versions of the engine, or to separate the process so as not to break the behavior of assets that have already been created and rely on the bug's behavior. This was a cause of reduced development efficiency. ©CAPCOM 5

Implementation Background Checking the relevant VFX can prevent this, but... Because VFX have a wide variety of functions it is costly and impractical to manually check each effect one by one. Perform checks automatically at implementation time, to try to reduce costs and improve stability. These problems can be prevented, though, by checking the affected VFX at the time of implementation. However, because of the wide variety of VFX, it is necessary to manually check a large number of VFX one by one if the affected area is large. 6 This is costly and impractical, and there is a high possibility of missing something. Therefore, by automatically checking for problems and behavior changes at the time of implementation, it is possible to avoid the need for human intervention. We decided to introduce those automatic checks in order to improve stability and reduce the cost of manual checks and subsequent followup. ©CAPCOM 6

Automatic Check Flow Next, we will introduce one of the automatic checks, "Difference Checking by Image Comparison." 7 ©CAPCOM 7

Automatic Check Flow commit & push Suppose you have implemented something, committed it, and pushed. 8 ©CAPCOM 8

Automatic Check Flow commit & push Execute CI commit & push [GitLab] Merge Request [GitLab] Merge into Master [GitLab] Merge Request Pipeline (CI) executes according to rules. The CI is activated when it detects that a push has been made. RE ENGINE development uses GitLab and merge by merge request. The checker checks the merge request, and if there is no problem, it is merged into master. 9 In merge requests, CI can be executed by a function called merge request pipeline. We set a rule that if the change is for VFX only, it will be subjected to checks. ©CAPCOM 9

Automatic Check Flow commit & push Build and spin up engine Execute CI commit & push [GitLab] Merge Request [GitLab] Merge into Master [GitLab] Merge Request Pipeline (CI) executes according to rules. The CI task checks out the branch with the changes, builds it, and starts the engine. 10 ©CAPCOM 10

Automatic Check Flow Repeat until all relevant VFX have been played back. commit & push Build and spin up engine Execute CI commit & push [GitLab] Merge Request Display the effect Take a screen capture every few frames [GitLab] Merge into Master Compare with the correct image and check the difference Create the correct image Execute CI [GitLab] Merge Request Pipeline (CI) executes according to rules. In the engine... Playback the VFX and take screen captures of each frame, then compare against images of correct behavior prepared in advance. This flow is repeated until all VFX to be checked have been run. 11 The correct image is created separately by CI, which I'll go into detail about later. ©CAPCOM 11

Automatic Check Flow Repeat until all relevant VFX have been played back. commit & push Build and spin up engine Execute CI commit & push [GitLab] Merge Request [GitLab] Merge into Master [GitLab] Merge Request Pipeline (CI) executes according to rules. Display the effect Take a screen capture every few frames New checks can be added ＋ Fine control is possible due to checks running in-engine. Compare with the correct image and check the difference Create the correct image Execute CI This area can be changed and checks can be added later. Each check process is performed in the engine itself. 12 So it is possible to control the check process in detail, such as playing back an effect and advancing it one frame at a time. This allows for a wide variety of checks. ©CAPCOM 12

Automatic Check Flow Repeat until all relevant VFX have been played back. commit & push Build and spin up engine Execute CI commit & push [GitLab] Merge Request [GitLab] Merge into Master [GitLab] Merge Request Pipeline (CI) executes according to rules. Display the effect Take a screen capture every few frames New checks can be added ＋ Fine control is possible due to checks running in-engine. Compare with the correct image and check the difference Throw error on crash or if check fails Create the correct image Execute CI If a crash occurs during the check, or if even one difference is detected, the merge request is put into an error state to prevent problematic changes from being made. 13 That is the flow of the "Difference Check by Image Comparison." ©CAPCOM 13

Difference Checking by Image Comparison Next, I would like to talk about the "Difference Check by Image Comparison" itself. First, I would like to show you what kind of differences can be detected by this check. 14 ©CAPCOM 14

Difference Checking by Image Comparison Correct After feature added Here is the image as the correct image and after a feature was added. At a quick glance, they look the same, however, there is a degree of discrepancy that cannot be seen without careful comparison. 15 In reality, the images are moving or have randomness due to random numbers. If checked by the human eye, you will never notice it. If you run these two images through a difference check... ©CAPCOM 15

Difference Checking by Image Comparison It's able to notice changes in behavior that cannot be caught by the human eye. Correct Differences detected; error thrown After feature added Differences are detected in this way and treated as errors. This example is one taken from our actual game development. 16 The behavior was changed by deleting necessary code during a refactor, and the difference was caught by the difference check. Even a difference such as this, which is not noticeable to the human eye, can drastically change the appearance of an effect, depending on the situation. This check helps prevent unintentional changes in processing. ©CAPCOM 16

Difference Checking by Image Comparison Required components to make it possible 1. Deterministic design 2. Preparation of correct images 3. Image comparison processing 4. Assets for checking The following is an introduction to each of the four things that were necessary to achieve the difference check. 17 ©CAPCOM 17

Difference Checking by Image Comparison 1 Deterministic design The image must be the same when the behavior or screen is captured no matter how many times it is replayed to match the correct image. Fixed frame rate Reproducible random states for particles Frame-byframe playback control The same behavior must always result in the same captured image. The first: "deterministic design." To match the correct image, the behavior of the effect must be the same no matter how many times it is replayed. 18 Therefore, we fix both the frame rate and any random numbers that affect the particles. By controlling the playback of VFX on a frame-by-frame basis, it is possible to set the capture timing to every few frames. This ensures the same appearance and the same captured image are achieved every time. ©CAPCOM 18

Difference Checking by Image Comparison 1 Deterministic design The most problematic was the "GPUParticle" ・Interlock, etc., causes discrepancies ・Changing algorithm just for checking will invalidate the check itself The goal is to check the behavior. Performance can be ignored Adopting a 1Particle/1Dispatch approach allows for checking without changing the processing involved in the behavior GPUParticle was quite a challenge in creating this structure. Because the updating of particles runs in parallel, if Interlocked or other methods were used, the generated particles19would be out of alignment and it would not work as it was. In addition, if you change the process to check the particles, it doesn't really count as a check anymore. Since the purpose of this check is to check for differences in behavior, performance issues can be ignored. Therefore, by using 1Particle/1Dispatch, it is possible to check the same behavior even if it is replayed many times without changing the processing related to the behavior. ©CAPCOM 19

Difference Checking by Image Comparison 2 Preparation of correct images ・Execute via CI nightly to generate correct image ・ In order to ensure exact behavior, reuse the same checking process, capture several of its results and send them to server. The correct image is how the behavior looks when working as intended. Only send those VFX that need new images generated Prevent changes in appearance caused by other factors, guaranteeing prior behavior and appearance forever. Second: “Preparation of correct images.” Have CI run every day late at night, using the same fixed framerates and random numbers concept as when checking. 20 Playback the effect in frame by frame, capturing once every few frames to create the correct images. The correct image is uploaded to a location where it can be accessed from a server or other PC where the CI is executed. The correct images must be the images of the intended behavior. Only regenerate images for those VFX that have changed since last time, or those that don‘t have images on file yet. This prevents other factors from changing the look and feel of the correct image, and ensures that the old behavior and look is maintained. ©CAPCOM 20

Difference Checking by Image Comparison 3 Image comparison processing The biggest problem: Captured images are not an exact match. ・Checking by exact match is not possible. ・Checking by similarity ratio means tuning the ratio and settings for each effect, which is difficult. An accurate, easily adjustable detection method. Check with edge detection by OpenCV ・Edge detection after shrinking and blurring. ・If the edge is larger than the specified size, an error is generated. Third: "Image Comparison Processing." The biggest problem here was that captured images are not exactly the same; although we try to ensure the same behavior 21each time, there is always a discrepancy of a few pixels. Therefore, the check by exact match cannot be used. It proved difficult to tune the settings and ratios for similarity ratio checking, and was ultimately not practical. Therefore, as a method that can detect normally and can be adjusted to some extent, we adopted a check using OpenCV edge detection. Edge detection is performed after blurring or shrinking to reduce errors, If there are edges larger than the specified size, an error is thrown. It is a simple check, but we have never had any false positives or negatives, and we have been able to detect even minor differences in behavior. ©CAPCOM 21

Difference Checking by Image Comparison 4 Assets for checking Problem #1: Title assets are huge and not well-designed for checking. ・These checks cannot be executed every time a push is made because of the time required. ・VFX associated with tiny particles, and those that rely on in-game elements cannot be checked properly. A quick and effective way to check Create check-only assets But there's a problem with this too... The fourth and final one: "Assets for checking." If we use the assets from the title, the number of assets to be checked will be huge. It takes a long time to check the assets, so it's not possible to do it every time you push. In addition, some title assets are not suitable for checking. 22 So, as a quick and effective checking method, we decided to create an asset dedicated to checking. However, there was a problem with this as well. ©CAPCOM 22

Difference Checking by Image Comparison 4 Assets for checking Problem #2: It's hard to create assets dedicated to checking. ・The cost of creating assets is high due to the wide variety of functions. ・There are so many assets to deal with that it quickly becomes a bottleneck. Low-cost ways to increase assets Reuse of assets used during implementation Use of assets for checking modifications and new features and gradually expand the scope of checks. It is simply too costly to create dedicated checking assets. It takes a significant amount of time to create an asset that covers a wide variety of functions. 23 We have decided to use assets for checking modifications and assets for checking new functions as a low-cost way to increase the number of assets. New features are very useful as checks because they are prone to defects. In addition, since assets for checking are often created together with various functions, the scope of checking can be expanded. The number of assets has been gradually increased and now covers about 200 assets (about several hundred patterns). All of them are checked in about 15 minutes. That's it for the introduction to difference checking by image comparison. ©CAPCOM 23

Other Automatic Checks In addition to, "Difference Checking by Image Comparison," various other checks have been introduced. ・Can the asset be opened and saved successfully? ・Can the shaders used by the effect be converted correctly? ・Can effect assets be previewed properly? We have mainly talked about the "Difference Check by Image Comparison" check, but there are other checks available that can improve stability and reduce checking costs as well. 24 ©CAPCOM 24

Results of the introduction of automatic checks What effect did implementing these checks have? 25 ©CAPCOM 25

Results of the introduction of automatic checks The cost of labor has decreased. 0 min. Operation check Approx. 30 min. First, the cost of labor was reduced. If a simple operation check is done manually, it takes about 30 minutes to build, start the engine, and complete the check. 26 However, if the checks can be done automatically, it doesn't take any time at all, which contributes to improved development efficiency. By detecting defects in advance, the number of people checking manually can be reduced, which also reduces the burden on those staff. ©CAPCOM 26

Results of the introduction of automatic checks The amount of bugs that were reported decreased. Number of bugs corrected (1 year) Before introduction: 242 After introduction: 212 Number of bugs caught before merging by automated checks: 47 Next, the number of bug fixes reported by the title side also decreased. In the year since the introduction of the automatic check, we have detected 47 crashes or differences in behavior and 27prevented them before merging. As a result, the number of bug fixes has decreased by approximately 10%. This has led to a reduction in support and response costs. However, the number of bugs is still large. In the future, we will continue to check using title assets that can cover a variety of functions. We are also planning to implement a system that can automatically generate and check assets. Once implemented, we can spend more time on research and adding new features. ©CAPCOM 27

Results of the introduction of automatic checks Easier to refactor Number of refactorings performed in a year: After introduction: Before introduction: 75 38 The introduction of automated checks not only reduced costs, it also created the benefit of making it easier to engage in refactoring. Large-scale refactoring, where the scope of impact is hard to predict, are not a problem. 28 The automated checks cover a wide range of functions, so we don't have to worry. The engine developers get a sense of security that the automatic checks will cover a wide range of functions. This has led to increased motivation for refactoring and action. A comparison of one year before and after the automatic checks shows the following: The number of refactorings, both major and minor, has doubled. This also helps to make the engine better. These are the results of the introduction of automated checks. I hope this information will be of some help. ©CAPCOM 28

That is all. Thank you for your attention. 29 ©CAPCOM 29