Ever since
AUTOMATON and
2BRO ignited interest for Nevergrind Online in Japan, I have been working hard to support the massive surge in concurrent users. I have never supported a product with this many concurrent users before! In our earliest days, we struggled with 250 users. Over time we have made incremental improvements to improve the architecture and net code that makes Nevergrind Online work.
My vision is to keep all players connected in one giant world. But the dream met reality and I was forced to resort to shards as a quick fix to support the massive surge. Shards worked very well, but it broke up friends, guilds, and undermined the excitement of being in one massive socially connected world full of people that you built relationships with over time. Behind the scenes, I was hard at work re-designing a key technical bottleneck that would help us reach the goal of supporting over a thousand players at once. Sharding worked. It helped us reach an all-time peak of 1,184 concurrent players. But I knew it was not Nevergrind Online's final form. I had to persevere and make it better!
Throughout the last week, I continued to improve the servers. It was not simply a matter of throwing more money at the problem. When you reach this scale, it requires meticulous attention to detail to the server configuration, limitations, and strengths of various hardware and software. To be more specific—the servers that route our game data were changed from Apache HTTP Server to Nginx. Additionally, I must have read dozens of articles on server configuration for supporting maximum TCP connections. This challenge is like an onion where several layers of software can all constrain how your server handles TCP connections. When your server limits TCP connections, it results in your users dealing with login queues. At peak hours, the servers had enough processing power to route the game data, but, at peak hours, players could not login due to no available TCP connections on the server. This is not an unusual problem. In fact it's quite common! Not just in the gaming industry, but in telecommunications as well. I can't imagine the havoc
EverQuest created in 1999 when its success broke the internet.
Nevergrind Online had a rocky week around peak hours and it cost us a few negative reviews due to causing disconnects and login queues. It is unfortunate, but we earned those negative reviews. Players should expect an online game to work most of the time and this was a rough week full of stumbling and bumbling toward the end zone. I even had to fix the idle disconnect logic on the client which was
incorrectly causing disconnects mid-stream for
NGC_the_Twitch. Thankfully, he was much more patient than most streamers would be. I was able to quickly resolve the issue within minutes, enabling him to finish the stream! You don't want your peak hour chart to look like this:
We survived, but you definitely don't want streamers to deal with technical problems on stream. In the subsequent days, I continued making improvements and it seemed that I had finally configured everything the way that it needed to be! As of last night, we finally went over 1,000 players in one world without a single technical issue the entire night! Look how smooth and lovely that curve looks! Not a single bump the entire night!
Of course, there are no guarantees that another challenge won't emerge in the weeks ahead, but this is a fantastic milestone that I will never forget! With server challenges taking up less of my time, you can expect even more features and updates in the weeks ahead! Stay tuned!