Los Pingheros - News - UpdateMonitor

DevBlog #4: Streamlining Game Development with GitLab CI/CD

[h3]Hey Pingheros![/h3]

In today’s game development, production time optimization is key. In any software development company, the time spent by developers on compiling code, packaging assets, testing applications, and deploying a release can rapidly become a nightmare. I have personally worked on projects where the entire pipeline, from the moment a line of code is changed to the moment the application is tested internally, takes up to 3 hours.

This is a nightmare.

On multiple levels, it is a huge problem for the company that encounters this issue:

Production: It is quite evident, LOST MONEY = LOST HOURS x MAN HOURS COST x NUMBER OF DEVELOPERS x DAYS OF DEVELOPMENT.

Morale: Probably the worst part of waiting is not the waiting itself, but the constant focus switching that it generates. The longer a developer waits, the more they will look for other subjects to handle in the meantime, causing continuous back and forth between tasks that ultimately leads to mistakes in the original task of the developer.

Confusion: The bigger the project, the more, in each phase of the development, specificities are added. Let’s see a common scenario:

An engine programmer adds a PRAGMA DEFINE for a specific platform at a given time. Some hours later, an AI developer starts to compile the project, and the first developer had no time to notify or forgot to add some info in the common documentation.

Of course, the build instantly fails, and the developer has to ask for information internally.

Why should they even care about compiling software if their task is to animate a beautiful crocodile?

Fortunately, we do not have to reinvent the wheel, and four letters have the solution to all our problems: CI/CD.

CI/CD stands for Continuous Integration/Continuous Deployment and is the combination of practices adopted by software development companies to avoid these kinds of problems.

If you are developing any kind of application and you are not using CI/CD, it’s like eating pasta without olive oil. You could… but why the hell???

More seriously, at Hectiq, when we started to develop Los Pingheros, we tackled and integrated a CI/CD pipeline from week 3. Although it requires 0.5 to 2 man-days per month to maintain, it is probably one of the most practical achievements of the Hectiq team.

There is no single way of setting up a CI/CD pipeline, and most of the time it depends on the tech stack used by your company.

Let’s see how we defined our CI/CD, keeping in mind that we use Unity as a game engine and Git as a version control system (link to previous article).

Choosing the right DevOps

As I had no intention to spend money on a DevOps platform, I looked for a more affordable solution.

On the table, I didn’t have too many choices:

Jenkins: Completely open source, and I used it on multiple previous projects. It works like a charm for a video game project with an 8-person team.

GitLab CI: A standard for software development, though less so for the video game industry.

As we saw in our previous article, we had already installed GitLab to manage our Git project on an internal server. Of course, it is an on-premises version.

As I’m lazy, I try not to over-complicate things.

This means that even though I was comfortable with Jenkins, I wasn’t happy adding another tool to our tech stack right next to GitLab.

We would have ended up with GitLab as a Git repository manager and Jenkins as a DevOps platform.

It can work; however, as Los Pingheros does not aim to find the next cure for cancer, I estimated that a simple CI pipeline built with a powerful tool like GitLab would do the job.

It was definitely the right call so far.

[h2] Setting Up the CI/CD Pipeline [/h2]

Let’s dive deep into our CI/CD configuration. Keep in mind that this was my first GitLab CI/CD pipeline, and I learned on the job. Although it works very well for Hectiq’s use case, an expert user will surely find optimizations and sometimes a better way to express my intentions.

[h3] Initial Setup: [/h3]

To integrate GitLab CI for Los Pingheros, we started by configuring the .gitlab-ci.yml file within our GitLab repository. This file defines the pipeline stages, including build and deploy. Periodically, on specific tag, and manually the CI pipeline, automating the build. This setup allowed us to streamline our development workflow, ensuring that code changes are automatically validated and ready for deployment, catching error earlier and improving efficiency.

[h3] The Pipe: [/h3]


stages:
  - setup
  - photon_build
  - unity_build
  - deploy
  - upload
  - release

variables:
  BUILD_TYPE:
    value: PREDEFINED
    options: 
      - "PREDEFINED"
      - "release"
      - "development"
    description: "The build type to use."
  STEAM_BRANCH:
    value: PREDEFINED
    options: 
      - "PREDEFINED"
      - "beta_staging"
      - "development"
      - "staging"
    description: "The steam branch to use."

  # not exposed to the user ui, but overridable
  SERVER_CONFIG: "Test"
    # description: "The server config to use. See NetServicesUtils.cs for available configs."
    # value: 
  GIT_CLEAN_FLAGS: none # avoid gitlab runner to delete untracked files, like "Library" folder
  GIT_FETCH_EXTRA_FLAGS: --tags
  GIT_DEPTH: 1
  GIT_SUBMODULE_STRATEGY: recursive
  UNITY_HUB: C:\Program Files\Unity\Hub\Editor\
  UNITY_ACTIVATION_FILE: ./Unity_v$UNITY_VERSION.alf
  UNITY_DIR: ./LosPingos # this needs to be an absolute path. Defaults to the root of your tree.
  # You can expose this in Unity via Application.version
  VERSION_NUMBER_VAR: $CI_COMMIT_REF_SLUG-$CI_PIPELINE_ID-$CI_JOB_ID
  VERSION_BUILD_VAR: $CI_PIPELINE_IID
  PACKAGE_REGISTRY_URL: ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/generic/${PRODUCT_NAME}/${CI_COMMIT_SHORT_SHA}
  PACKAGE_REGISTRY_URL_BUILD : $PACKAGE_REGISTRY_URL/$PRODUCT_NAME.zip
  MAJOR_VERSION: 0 # Major version number, this should never change, unless you are releasing a new game. It can only be changed here.
  ENV_FILE: "ci_environment.ini"

include:
  - local: '/ci/rules/default-config-ci.yml'
  - local: '/ci/rules/beta-config-ci.yml'
    rules:
      - if: $CI_COMMIT_BRANCH == 'beta'
  - local: '/ci/rules/release-config-ci.yml'
    rules:
      - if: $CI_COMMIT_BRANCH == 'release'
  - local: '/ci/rules/dev-config-ci.yml'
    rules:
      - if: $CI_COMMIT_BRANCH == 'dev'
  - local: '/ci/rules/tag-config-ci.yml'
    rules:
      - if: $CI_COMMIT_TAG != null

default:
 #default global settings
  tags:
    - general

setup_pipe:
   stage: setup
   hooks:
      pre_get_sources_script:
      - ssh-keyscan -t rsa  gitlab.local.hectiq >> ~/.ssh/known_hosts
   script:
     - powershell -ExecutionPolicy Bypass -File .\ci\version.ps1
   artifacts:
    reports:
      dotenv: ci_environment.env

photon_build:
  stage: photon_build
  script:
    - nuget restore .\quantum_code\quantum_code.sln -ConfigFile ".\quantum_code\NuGet.Config"
    - powershell -ExecutionPolicy Bypass -File .\ci\photon_build.ps1
  artifacts:
    paths:
      - ci_environment.ini
    expire_in: 1 week 


.unity_build: &unity_build
  stage: unity_build
  # <  script:
    - powershell -ExecutionPolicy Bypass -File .\ci\unity_build.ps1
  artifacts:
    paths:
      - builds/
      - ci_environment.ini
    expire_in: 1 week 


build-StandaloneWindows64:
  <  variables:
    BUILD_TARGET: StandaloneWindows64
  rules:
    - if: '$DEPLOY_PLATFORM == "steam"'
      allow_failure: false

build-Switch:
  <  variables:
    BUILD_TARGET: Switch
  rules:
    - if: '$DEPLOY_PLATFORM == "switch"'
      allow_failure: false


deploy_on_steam:
  stage: deploy
  variables:
    GIT_STRATEGY: none
    BUILD_TARGET: StandaloneWindows64
  script:
    - powershell -ExecutionPolicy Bypass -File .\ci\deploy_steam.ps1
  rules:
    - if: '$DEPLOY_PLATFORM == "steam"'
      allow_failure: false
  artifacts:
    paths:
      - ci_environment.ini
    expire_in: 1 week 

deploy_on_switch:
  stage: deploy
  variables:
    GIT_STRATEGY: none
    BUILD_TARGET: Switch
  script:
    - powershell -ExecutionPolicy Bypass -File .\ci\deploy_switch.ps1
  rules:
    - if: '$DEPLOY_PLATFORM == "switch"'
      allow_failure: false

workflow:
  rules:
    - if: '$CI_PIPELINE_SOURCE == "web"'
      when: always
    - if: '$CI_PIPELINE_SOURCE == "schedule"'
      when: always
    - if: $CI_COMMIT_TAG
      when: always
    - when: never

[h2] The Setup [/h2]

Our CI/CD pipeline for Los Pingheros is structured into six key stages: setup, photon_build, unity_build, deploy, upload, and release. The .gitlab-ci.yml file orchestrates these stages, defining specific tasks for each.

Setup: Prepares the environment, including setting up necessary SSH keys and environment variables.
Photon Build: Handles building our multiplayer logic using Photon.
Unity Build: Compiles the game for different platforms (e.g., Windows, Switch).
Deploy: Automates deployment to platforms like Steam or Switch.
Upload Stage: Uploads the build artifacts to a GitLab Packages and registry section.
Release Stage: Finalizes the release process by making a snapshot of the uploaded artifacts in the registry section and the project code.

[h2] Integration Note 0: [/h2]

As with every CI/CD pipeline, GitLab allows me to configure everything with a certain degree of precision.

Each stage of the pipeline can be really complex and is mostly handled by a PowerShell script that communicates and retrieves all variables from GitLab CI and executes a job with the given variables.

At some point we had to simplify the pipeline, as it was too complex and was generating confusion and unexpected CI fails due to the multiple variables given to the user.

Each variable was exposed, and the user could input unexpected values that led to mistakes.

We wanted to leave the possibility for expert users to override specific variables, while hiding them from non-programmers who mostly just want to “click a button” and wait for their project to be available on Steam.

Again, simplifying was the proper solution.


variables:
  BUILD_TYPE:
    value: PREDEFINED
    options: 
      - "PREDEFINED"
      - "release"
      - "development"
    description: "The build type to use."
  STEAM_BRANCH:
    value: PREDEFINED
    options: 
      - "PREDEFINED"
      - "beta_staging"
      - "development"
      - "staging"
    description: "The steam branch to use."
    
	 # not exposed to the user ui, but overridable
  SERVER_CONFIG: "Test"

Any user now entering the CI page has access to this view being able to change only 2 variables:

If an expert user wants to change a hidden value, it is still possible:

[h2] Integration Note 1: [/h2]

Last trick we used is to define a yaml file to include based on specific rules.

Each ruleset is based on a specific branch.

Here an example of how those files are configured:


#tag-config-ci.yml
variables:
  REF_BUILD_TYPE: release
  REF_STEAM_BRANCH: beta_staging
  DEPLOY_PLATFORM: steam
  UNITY_VERSION: 2022.3.40f1
  PRODUCT_NAME: "Los Pingheros Playtest"
  STEAM_TARGET_APP_ID: "[YOURID]"
  PLAYFAB_TITLE_ID: "[YOURID]"
  PLAYFAB_SECRET_KEY: "[YOURKEY]"
  PHOTON_APP_ID: "[YOURPHOTONID]"
  INCLUDE_PDB: 'false'
  DEFINE_SYMBOLS: 'PINGOS_RELEASE'

upload:
  stage: upload
  variables:
    GIT_STRATEGY: none
  image: curlimages/curl:latest
  rules:
    - if: '$CI_COMMIT_TAG'
      when: always
      allow_failure: false
  script:
    - powershell -ExecutionPolicy Bypass -File .\ci\upload.ps1

release:
  stage: release
  variables:
    GIT_STRATEGY: none
  rules:
    - if: '$CI_COMMIT_TAG'
      when: always
      allow_failure: false
  script:
    - powershell -ExecutionPolicy Bypass -File .\ci\release.ps1

[h2] Example: [/h2]

With this approach not only, you will be able to define different variables for a different target, but you will also be able to add specific stages for a configuration.

In the previous image, a developer tags a commit, it means that it is a special version, and that requires the CI to run the stage of upload and release.

[h2] Integration Note 2: [/h2]

I’m not going to detail all the stages. However, I will include the Deploy to Steam stage as a reference.


."$PSScriptRoot\common_utility.ps1"

$ErrorActionPreference = "Stop"
Set-StrictMode -Version Latest

$CI_COMMIT_REF_NAME = $env:CI_COMMIT_REF_NAME
$STEAMWORKS_SDK = $env:STEAMWORKS_SDK
$BUILD_TARGET = $env:BUILD_TARGET
$UNITY_DIR = $env:UNITY_DIR
$STEAMWORKS_USER = $env:STEAMWORKS_USER
$STEAMWORKS_PASSWORD = $env:STEAMWORKS_PASSWORD
$STEAM_MAIN_APPID = $env:STEAM_MAIN_APPID
$CI_COMMIT_SHORT_SHA = $env:CI_COMMIT_SHORT_SHA
$DISCORD_WEBHOOK_URL = $env:DISCORD_WEBHOOK_URL
$STEAM_PLAYTEST_APPID = $env:STEAM_PLAYTEST_APPID 
$INCLUDE_PDB = $env:INCLUDE_PDB
$ENV_FILE = $env:ENV_FILE

. "$PSScriptRoot\override_variables.ps1"

$debug_build = $false
if ($debug_build -eq $true) 
{
    $BUILD_TARGET = "StandaloneWindows64"
    $UNITY_DIR = "C:\Projects\lospingos\LosPingos"
    $CI_COMMIT_SHORT_SHA = "1234567"
    $STEAM_BRANCH = 'beta_staging'
    $STEAM_MAIN_APPID = 'AAAAAAA'
    $STEAM_PLAYTEST_APPID = 'BBBBBBB'
    $CI_COMMIT_REF_NAME = 'beta'
    $STEAMWORKS_SDK = 'C:\steamworks_sdk'
    $STEAMWORKS_USER = 'HectiqAdmin'
    $STEAMWORKS_PASSWORD ='****************'
    $CI_COMMIT_SHORT_SHA = "CCCCCCC"
    $DISCORD_WEBHOOK_URL = "*************"
    $ENV_FILE = "ci_environment.ini"
    $INCLUDE_PDB = $true
}

Write-Host "Deploy STEAM on BRANCH $STEAM_BRANCH"
Set-IniFileValue $ENV_FILE GENERAL STEAM_BRANCH $STEAM_BRANCH

$steamworkSDKExists = Test-Path $STEAMWORKS_SDK

if ($steamworkSDKExists) {
    Write-Host "The STEAMWORKS_SDK environment variable is $STEAMWORKS_SDK"
} else {
    throw "The STEAMWORKS_SDK environment variable does not exist."
}

$steamScriptsPath = "$STEAMWORKS_SDK\tools\ContentBuilder\scripts"
Remove-FolderContents -FolderPath $steamScriptsPath

$steamScriptsPath = "$STEAMWORKS_SDK\tools\ContentBuilder\content"
Remove-FolderContents -FolderPath $steamScriptsPath

Write-Host "Building for $BUILD_TARGET"
Write-output "UNITY_DIR path: $UNITY_DIR"
$BUILD_PATH = (Join-Path $UNITY_DIR "/../builds/$BUILD_TARGET/") 
$STEAM_SCRIPTS_PATH = (Join-Path $UNITY_DIR "../ci/steam/")
Write-output "Build path: $BUILD_PATH"
$ASOLUTE_BUILD_PATH = $BUILD_PATH | Resolve-Path
$ABSOLUTE_UNITY_DIR = $UNITY_DIR | Resolve-Path
$ABSOLUTE_STEAM_SCRIPTS_PATH = $STEAM_SCRIPTS_PATH | Resolve-Path
Write-output "ABSOLUTE BUILD PATH: $ASOLUTE_BUILD_PATH"
Write-output "ABSOLUTE UNITY DIR: $ABSOLUTE_UNITY_DIR"
Write-output "ABSOLUTE STEAM SCRIPTS PATH: $ABSOLUTE_STEAM_SCRIPTS_PATH"

if ($STEAM_BRANCH -eq "beta" -or $STEAM_BRANCH -eq "staging" -or $STEAM_BRANCH -eq "beta_staging" -and $INCLUDE_PDB -eq $false) 
{
    Write-Output  "Copying build folder to Steam content folder excluding debug folders..."
    Copy-FolderContents -SourceFolder $ASOLUTE_BUILD_PATH -DestinationFolder "$STEAMWORKS_SDK\tools\ContentBuilder\content" -ExcludeFolders "*_BurstDebugInformation_DoNotShip", "*_BackUpThisFolder_ButDontShipItWithYourGame"  -ExcludeRootFiles "*.pdb"
}
else
{
    Write-Output  "Copying build folder to Steam content folder with PDB files..."
    Copy-FolderContents -SourceFolder $ASOLUTE_BUILD_PATH -DestinationFolder "$STEAMWORKS_SDK\tools\ContentBuilder\content"
}
Copy-FolderContents -SourceFolder $ABSOLUTE_STEAM_SCRIPTS_PATH -DestinationFolder "$STEAMWORKS_SDK\tools\ContentBuilder\scripts"



Get-ChildItem -Path "$STEAMWORKS_SDK\tools\ContentBuilder\content" -Force | Format-Table
if (-not (Get-ChildItem -Path "$STEAMWORKS_SDK\tools\ContentBuilder\content")) {
  throw "Build folder is empty"
}



$steamcmd = "$STEAMWORKS_SDK\tools\ContentBuilder\builder\steamcmd.exe"
$steamCmdArgs = "+login $STEAMWORKS_USER $STEAMWORKS_PASSWORD +run_app_build `"..\\scripts\\app_${STEAM_BRANCH}_${STEAM_MAIN_APPID}.vdf`" +quit"
$steamCmdArgsDebug = $steamCmdArgs -replace $STEAMWORKS_PASSWORD, "*********"
Write-Host "SteamCMD Path: $steamcmd"
Write-Host "SteamCMD Args: $steamCmdArgsDebug"

# Execute SteamCMD and capture the output
$steamCmdOutput = Start-Process -FilePath $steamcmd -ArgumentList $steamCmdArgs -RedirectStandardOutput "steam_deploy_output.log" -RedirectStandardError "steam_deploy_error.log" -NoNewWindow -Wait


if($STEAM_BRANCH -eq "beta_staging")
{
    Write-Host "syncronizing this branch with Playtest branch $STEAM_BRANCH"

    $steamCmdArgs = "+login $STEAMWORKS_USER $STEAMWORKS_PASSWORD +run_app_build `"..\\scripts\\app_${STEAM_BRANCH}_${STEAM_PLAYTEST_APPID}.vdf`" +quit"
    $steamCmdArgsDebug = $steamCmdArgs -replace $STEAMWORKS_PASSWORD, "*********"
    Write-Host "SteamCMD Path: $steamcmd"
    Write-Host "SteamCMD Args: $steamCmdArgsDebug"

    # Execute SteamCMD and capture the output
    $steamCmdOutput = Start-Process -FilePath $steamcmd -ArgumentList $steamCmdArgs -RedirectStandardOutput "steam_deploy_output.log" -RedirectStandardError "steam_deploy_error.log" -NoNewWindow -Wait
}

$steamCmdOutput = Get-Content "steam_deploy_output.log"  # Read the output from the file
$steamCmdErrors = Get-Content "steam_deploy_error.log"  # Read the output from the file

Write-Host "Steam CMD Output:"
$steamCmdOutput | ForEach-Object { Write-Host $_ }

Write-Host "Steam CMD Errors:"
Write-Host $steamCmdErrors

# Further processing here
# For example, checking for specific output indicating success or failure
if ($steamCmdOutput -like "*Success*") {
    Write-Host "Deploy and submission successful."
    $Message = "Deployed Git ref: $($CI_COMMIT_REF_NAME) to Steam branch: $($STEAM_BRANCH) with SSHA: $CI_COMMIT_SHORT_SHA"
    SendDiscordMessage -WebhookUrl $DISCORD_WEBHOOK_URL -Message $Message
} else {
    Write-Host ""
    throw "Deploy failed."
}

[h2] Deployment and Delivery [/h2]

With GitLab CI, we've improved our development process, eliminating the need for local builds. Our focus is on maintaining a pipeline that completes within 20 minutes, ensuring efficiency. Integration with Discord allows our team to receive instant notifications on build status—whether it's a failure or success—facilitating quick action.

Each build is tagged with a name that combines the latest branch tag and a shortened Git commit SHA. This naming convention is optimized for platforms like Nintendo Switch, which have strict character limits for version names. By standardizing this across all platforms, we maintain consistency and avoid potential issues.

If a build fails, the development team can immediately click a link to review and address the error. On success, the build is deployed to the Steam product betas section according to predefined rules. Importantly, we never automatically update the official branch; this final push to the main Steam branch is always done manually to ensure quality control.

This CI/CD pipeline not only supports our current needs but also positions us for future growth. As Los Pingheros expands, this system will easily scale, enabling us to deliver regular updates and DLCs confidently. Our approach ensures that we can meet increasing demand without compromising on quality or speed.

[h2] Challenges and Lessons Learned [/h2]

Initially, we tried to define our entire CI/CD pipeline from day one, but we quickly learned that this approach was flawed. The pipeline evolved every 2-3 months as new ideas and requirements emerged—things we hadn’t anticipated at the start. This taught us that building a CI/CD pipeline for a video game like Los Pingheros requires a "baby-steps" approach. It was only three months ago, during our first public playtest, that we finally designed a streamlined pipeline with simple, effective rules that truly met our needs.

[h2] Conclusion [/h2]

Incorporating a CI/CD pipeline has been transformative for our development process at Hectiq, particularly with Los Pingheros. By leveraging GitLab CI, we've eliminated the inefficiencies of local builds and minimized the time from code changes to deployment, ensuring that our pipeline runs smoothly within 20 minutes. This approach has not only optimized our workflow but given us the confidence to handle updates and expansions in the future. The evolution of our pipeline taught us the importance of flexibility and simplicity, ensuring that our setup is robust yet adaptable to future needs. As Los Pingheros grows, this pipeline will continue to support our goals, allowing us to scale efficiently and maintain high-quality delivery for our players.

[h2] Next Steps [/h2]

We are planning to add an automated testing phase to our pipeline to further improve quality assurance. This brings up an interesting question: how do you handle testing in video game development? We’d love to hear your thoughts.

Additionally, we’re working on integrating Notion and OpenAI APIs to automatically generate player-facing changelogs based on our internal commits. What do you think of this idea? Your feedback could help us refine this feature.

Finally, if your company is looking to enhance its CI/CD processes or needs expertise in game development pipelines, we’re available to collaborate and share our knowledge. Let’s work together to build something great!

Luca Pierabella - Hectiq

Los Pingheros News

DevNews #5 - New tutorial, challenges, and objectives!

DevNews #4 - New progression system, rewards, and night maps are coming!

DevBlog #4: Streamlining Game Development with GitLab CI/CD

Join Steam Playtest now!

DevNews #3 - New legendary skin, new VFX, and bots are getting smarter!