DevBlog #1: Hectiq & VCS: why we opted for GIT LFS 🐧
[h3]Hey Pingheros![/h3]
In the fast-paced world of game development, choosing the right Version Control System (VCS) is crucial for managing complex projects efficiently. With numerous assets, large binary files, and a need for seamless collaboration among distributed teams, game developers require a robust and flexible VCS. Among the various options available, Perforce, Git, and SVN (Subversion) are three of the most prominent choices. Each has its strengths and weaknesses, but selecting the best fit can significantly impact a project's success.
At Hectiq, our flagship project, Los Pingheros, a unique multiplayer snowball fight game set in a vibrant Mexico, draped in a unique snowy mantle, demands a VCS that can handle its complexity and scale. In this article, we'll briefly explore the key features and limitations of Perforce, Git, and SVN, and explain why we decided to adopt Git with Git Large File Storage (LFS) for our game development needs. Our goal is to provide insight into the decision-making process and highlight the advantages that Git with Git LFS brings to our workflow.
[h2]Perforce[/h2]
[h3]Main Strengths and Common Use in Game Development:[/h3]
[h3]Key Limitations:[/h3]
[h2]Git[/h2]
[h3]Main Strengths, Particularly with Git LFS:[/h3]
[h3]How It Handles Large Files Effectively:[/h3]
[h3]Key Limitations:[/h3]
[h2]SVN (Subversion)[/h2]
[h3]Main Strengths:[/h3]
[h3]Key Limitations:[/h3]
Understanding where your VCS will be hosted is critical.
[h3]Perforce Hosting:[/h3]
[h3]Git Hosting:[/h3]
[h3]SVN Setup:[/h3]
All three solution can be installed on a local server, but this approach has drawbacks, including maintenance time, backup, and security concerns.
[h2]Distributed vs. Centralized[/h2]
[h2]Financial Costs[/h2]
[h2]Ease of Use[/h2]
In our project repository, among multiple folders, two particularly critical ones require special attention:
While the UnityProject folder doesn't necessitate specific handling as its files are universally needed across all development teams, the ArtSource (soon to be renamed to AssetsSource) presents unique challenges. This folder houses all the artistic assets and localization data, amounting to over 100 GB of files.
Why Download What You Don't Need?
For a programmer who doesn't need access to texture sources such as Substance files or PSDs, it's impractical to clone this entire folder. This situation conflicts with Git's principle that users should have access to the entire project history locally. The implications include:
To mitigate these issues, we developed a script that allows users to clone the project while excluding unnecessary folders, significantly reducing the time and space required:
[h2]Example Script to Optimize Cloning[/h2]
[h2]Adaptive Access for Specific Needs[/h2]
During development, a technical artist might need temporary access to the Character Rig Sources. To address this, we created a tool that allows team members to "opt-in" to access resources from other departments as needed.
This tool leverages the git config command to dynamically include or exclude folders from LFS tracking, facilitating efficient data management tailored to current needs:
Here's an illustration of the tool in action, demonstrating how developers can selectively sync data relevant to their specific tasks.
By integrating smart repository management practices and tools, we significantly streamline the development process, minimizing overhead and ensuring that all team members have exactly the resources they need, exactly when they need them.
The primary reason we chose Git with Git LFS at Hectiq is our strong internal IT skill set. We easily set up a GitLab solution on our internal server, spending no more than five man-days over an entire year on infrastructure setup.
We have an internal server accessible via a VPN through WireGuard. All software on the server is installed using a Docker Compose file, simplifying maintenance. The server has a dedicated SSD for storing Git LFS files.
While we initially doubted the scalability of this system, we have successfully scaled to 10 users with minimal additional costs or IT time. This setup has saved us thousands of euros, but we strongly advise against this approach without skilled IT workforce, as it can divert focus from the main project development to infrastructure maintenance.
The last aspect that definitively convinced us to adopt Git is the easy integration with a robust DevOps platform like GitLab…This on his own is an entire subject and we will deep dive into it in future articles.
In conclusion, the decision to choose Git with Git LFS over Perforce and SVN was driven by our need for a cost-effective, flexible, and scalable solution that fits our remote-friendly team structure. The strong support and integration capabilities of Git, combined with our internal IT expertise, made it the best choice for Hectiq's game development needs. Our flagship project, Los Pingheros, a thrilling snowball fight brawler that brings the lively spirit of Mexico to a snowy battleground, benefits immensely from this setup, ensuring that we can manage its complexity and scale effectively. Using Git with Git LFS, we are well-equipped to maintain efficient, collaborative, and high-quality game development.
What is your opinion on the subject? Share your feedback and experience with us, and if you have further questions or needs do not hesitate to reach us. We are always ready to discuss further insight on this matter.
In the fast-paced world of game development, choosing the right Version Control System (VCS) is crucial for managing complex projects efficiently. With numerous assets, large binary files, and a need for seamless collaboration among distributed teams, game developers require a robust and flexible VCS. Among the various options available, Perforce, Git, and SVN (Subversion) are three of the most prominent choices. Each has its strengths and weaknesses, but selecting the best fit can significantly impact a project's success.
At Hectiq, our flagship project, Los Pingheros, a unique multiplayer snowball fight game set in a vibrant Mexico, draped in a unique snowy mantle, demands a VCS that can handle its complexity and scale. In this article, we'll briefly explore the key features and limitations of Perforce, Git, and SVN, and explain why we decided to adopt Git with Git Large File Storage (LFS) for our game development needs. Our goal is to provide insight into the decision-making process and highlight the advantages that Git with Git LFS brings to our workflow.
Quick Overview of Each VCS
[h2]Perforce[/h2]
[h3]Main Strengths and Common Use in Game Development:[/h3]
- Centralized Version Control System: Perforce uses a centralized model, which is familiar and well-mastered by the Hectiq team.
- Handling Large Binary Files: Perforce excels at managing large binary files, a common necessity in game development.
- On-Premises Installation: Perforce can be installed on our own servers, providing control over our data.
- Intuitive Tools and UI: The user interface is highly intuitive, making it easier for our team, particularly artists, to use.
[h3]Key Limitations:[/h3]
- Cost: Perforce is expensive, costing $495 per user annually. As a startup, Hectiq aims to reduce expenses and dependencies on costly solutions.
- Scaling Costs: While scaling is technically easy, the financial burden increases significantly with a growing team.
- Branch Management: Working with branch is definitively not straight forward as with GIT
[h2]Git[/h2]
[h3]Main Strengths, Particularly with Git LFS:[/h3]
- Distributed System: Git is a distributed VCS, allowing for greater flexibility in workflows.
- Efficient Storage: For non-binary files, Git stores only the differences from previous versions, optimizing storage.
- Developer Preference: Git is the preferred solution among programmers due to its powerful features and flexibility.
- Open Source: Git is free to use, which is crucial for a cost-conscious startup.
- Integration with DevOps Platforms: Git can seamlessly integrate with platforms like GitLab, enhancing our CI/CD processes.
- Industry Standard: Git is widely adopted in the tech industry and continuously improved, with many solutions built on top of it, such as GitOps.
[h3]How It Handles Large Files Effectively:[/h3]
- Git LFS: Large binary files are managed through Git LFS, which stores these files in a specific location on the server and maintains pointers to them. This approach simplifies the handling of large assets in game development.
[h3]Key Limitations:[/h3]
- Git is easy to learn, hard to master
- Artist tends to struggle handling conflicts
[h2]SVN (Subversion)[/h2]
[h3]Main Strengths:[/h3]
- Centralized System: SVN is an older, centralized VCS, which some teams might find familiar.
- Open Source: Like Git, SVN is free to use.
- Handling Large Files: SVN manages large files effectively.
[h3]Key Limitations:[/h3]
- Lack of Popularity: SVN is not the preferred VCS in the industry today. Opting for a less popular system can result in fewer available tools, community support, and resources.
- Potential Drawbacks: Choosing a less popular VCS can lead to challenges in finding support and integrating with modern development tools.
Server Costs
Understanding where your VCS will be hosted is critical.
[h3]Perforce Hosting:[/h3]
- Using Perforce's server is straightforward but comes at a higher cost.
- Perforce can be installed on your own server, and it has a GUI tool to manage it
[h3]Git Hosting:[/h3]
- Hosting a code repository on GitHub or GitLab is often free, but the costs for storing LFS files can be substantial.
- Git can be installed on a local server too; in our team we use a docker instance of Gitlab on-premises
[h3]SVN Setup:[/h3]
- Setting up SVN requires a specific IT skill set and ongoing maintenance, similar to Git and Perforce.
All three solution can be installed on a local server, but this approach has drawbacks, including maintenance time, backup, and security concerns.
Key Comparison Points
[h2]Distributed vs. Centralized[/h2]
- Distributed: As a young, remote-friendly team, Hectiq has half of its team working remotely, the distributed approach allows us to work and change branches without being constantly connected.
- Centralized: A centralized approach requires an internet connection to access the VCS, which can be limiting for remote teams.
[h2]Financial Costs[/h2]
- Cost Optimization: The video game industry is financially challenging, so cost optimization is crucial.
- Perforce Costs: Perforce is free for up to five users, but costs increase significantly beyond that. A team of 20 would incur a yearly expense of $10K.
- Scaling Considerations: Hectiq aims to grow beyond a small indie developer, so we consider long-term costs in our decision-making.
[h2]Ease of Use[/h2]
- Perforce: Easier for artists and less technical team members to use.
- Git: More complex but powerful, with tools like SourceTree simplifying the experience. Git LFS adds commands that could be better integrated but are manageable with our IT skill set.
GIT and Artists: Structuring Repositories for Efficiency
In our project repository, among multiple folders, two particularly critical ones require special attention:
- ArtSource
- UnityProject
While the UnityProject folder doesn't necessitate specific handling as its files are universally needed across all development teams, the ArtSource (soon to be renamed to AssetsSource) presents unique challenges. This folder houses all the artistic assets and localization data, amounting to over 100 GB of files.
Why Download What You Don't Need?
For a programmer who doesn't need access to texture sources such as Substance files or PSDs, it's impractical to clone this entire folder. This situation conflicts with Git's principle that users should have access to the entire project history locally. The implications include:
- Increased clone and pull times for all users.
- Expanded storage requirements on local machines.
To mitigate these issues, we developed a script that allows users to clone the project while excluding unnecessary folders, significantly reducing the time and space required:
[h2]Example Script to Optimize Cloning[/h2]
batchCopy code
@echo off
setlocal
rem Define variables
set REPO_SSH=ssh://git@[LOCALDOMAIN:PORT]/hectiq/myproject.git
set CLONE_DIR="C:\Repository\MyProject"
set CERTIFICATE_DIR="C:\local.hectiq.crt"
rem Initialize the repository
cd %CLONE_DIR%
git init
rem Configure Git to exclude certain paths
git config --local lfs.fetchexclude "artsource,artsource_techart"
git config --local http.sslCAInfo %CERTIFICATE_DIR%
rem Add and fetch from remote
git remote add origin %REPO_SSH%
git fetch origin
rem Checkout development branch
git checkout dev
rem Update submodules
git submodule update --init --recursive
rem Open directory in Explorer
explorer %CLONE_DIR%
pause
endlocal
[h2]Adaptive Access for Specific Needs[/h2]
During development, a technical artist might need temporary access to the Character Rig Sources. To address this, we created a tool that allows team members to "opt-in" to access resources from other departments as needed.
This tool leverages the git config command to dynamically include or exclude folders from LFS tracking, facilitating efficient data management tailored to current needs:
git config --local lfs.fetchexclude "folderToExclude1,folderToExclude2"
Here's an illustration of the tool in action, demonstrating how developers can selectively sync data relevant to their specific tasks.
By integrating smart repository management practices and tools, we significantly streamline the development process, minimizing overhead and ensuring that all team members have exactly the resources they need, exactly when they need them.
Why We Chose Git with Git LFS
The primary reason we chose Git with Git LFS at Hectiq is our strong internal IT skill set. We easily set up a GitLab solution on our internal server, spending no more than five man-days over an entire year on infrastructure setup.
We have an internal server accessible via a VPN through WireGuard. All software on the server is installed using a Docker Compose file, simplifying maintenance. The server has a dedicated SSD for storing Git LFS files.
While we initially doubted the scalability of this system, we have successfully scaled to 10 users with minimal additional costs or IT time. This setup has saved us thousands of euros, but we strongly advise against this approach without skilled IT workforce, as it can divert focus from the main project development to infrastructure maintenance.
The last aspect that definitively convinced us to adopt Git is the easy integration with a robust DevOps platform like GitLab…This on his own is an entire subject and we will deep dive into it in future articles.
Conclusion
In conclusion, the decision to choose Git with Git LFS over Perforce and SVN was driven by our need for a cost-effective, flexible, and scalable solution that fits our remote-friendly team structure. The strong support and integration capabilities of Git, combined with our internal IT expertise, made it the best choice for Hectiq's game development needs. Our flagship project, Los Pingheros, a thrilling snowball fight brawler that brings the lively spirit of Mexico to a snowy battleground, benefits immensely from this setup, ensuring that we can manage its complexity and scale effectively. Using Git with Git LFS, we are well-equipped to maintain efficient, collaborative, and high-quality game development.
Share your thoughts
What is your opinion on the subject? Share your feedback and experience with us, and if you have further questions or needs do not hesitate to reach us. We are always ready to discuss further insight on this matter.