EVE Online News

Tireless Capsuleers,

No Downtime starts today!

As you may have seen in the recent development blog, No Downtime starts today after 11:00 UTC. That means, if everything goes to plan, that on 9 September EVE Online will be entering uncharted territory yet again past the 24 hour mark.

The ultimate goal is for EVE to have no downtimes at all. The first step has been to shorten downtime, which is quickly approaching its asymptotic value of around 3 minutes. Then, the goal is to have fewer and fewer downtimes, eventually eliminating it altogether. This is a long-term goal and all of EVE’s technological advances are aimed towards that.

Just like last time, your help is required to flag everything that seems suspect or weird, letting us know how the game performs for you when you do the things you normally do.

There are a number of reasons why running another one of these experiments is beneficial:

• Improvements which have been made since the last experiment can be validated
• Things that may have regressed will be flagged
• Memory usage will be observed
• Verification that anything new is not making any downtime assumptions

Ultimately, an experiment such as this is dedicated to the Capsuleers of New Eden who hail from countries such as New Zealand, Japan, Korea, and Australia. As EVE Online has entered and launched in Korea and Japan over the past year, the activity in this timezone has increased, and not disturbing players during their peak hours is important.

For those that remember the video from 2019’s No Downtime, a whole lotta effort and power is needed for such a heroic stunt. Find out more about how that is possible in the No Downtime Vol 2 video.

[previewyoutube][/previewyoutube]

PLAY EVE AND ENTER THE UNKNOWN

Every day I think about downtime, wait for it to pass, and record the result.

The last time I wrote this (did you read that dev blog? It's really good, I promise), Tranquility's auto-reboot on weekends was approximately 4 minutes and 20-40 seconds, just enough for a quick cup of tea. Today's auto-reboot, a year and a half later, was 3 minutes and 34 seconds - just enough for a quicker cup of tea (I measured it while writing this dev blog). Given the improvements we have made since 2019, an auto reboot downtime of 3 minutes and 30-40 seconds is pretty normal these days.

I could focus on this improvement of 50-60 seconds, a 22.5% improvement between 3 Dec 2019 and 3 July 2021, and predict the end of downtime on 19 December 2026 with this super-scientific graph, but the reality is more complicated than that.

There is a (soft) lower bound of approximately 3 minutes given the three different activities during downtime - shutdown, database jobs, startup - which last approximately 1 minute each, unless fundamental changes are made, and the most fundamental one is still to not have any downtime at all. Downtime will not become much less than 160-200 seconds; instead there must first be fewer downtimes and then none at all. Nevertheless, I wanted to start this blog with a concrete example of improvements made in downtime reduction since last time. And now another no-downtime experiment is being planned for September 9.

The purpose of this second no-downtime experiment is at least four-fold:

• Verify the fixes made for the issues discovered in the previous experiment in the live production environment
• Verify that no other code/features have regressed since last time and in general look for further issues
• Observe memory usage
• Verify that our technology platform (which you will hear more about later) is not making any downtime assumptions

So what did we discover last time, I hear you ask?

First and foremost we discovered reliance on downtime as an event to mark the beginning of a daily cycle, and a reliance on a daily startup, such as structures not finishing 24+ hour timers and corporations not joining Faction Warfare. We fixed all those issues that we found, and those you reported to us. Now we want to verify them further (of course they have been tested but our test environments don't have Tranquility's scale) and look for more such issues.

We also observed time desynchronization (which we fixed), and significant memory usage (which we improved somewhat).

The time desynchronization was a known issue, but last time we were observing whether players noticed at the end of day #2. The target for time desynchronization is a maximum of ±0.5 seconds. But with newer hardware, we had been observing an end-of-run desynchronization of 2.25 seconds and - predictably - 4.5 seconds at the end of day #2 in the first no-downtime experiment in 2019.



Players started to notice once the desynchronization was above 3 seconds, mostly by noting what felt like module lag or delay when their client and the node hosting their solar system disagreed significantly about when modules were cycling. Time desynchronization is now normally within ±1/100 of a second, well within the maximum of ±0.5 seconds.

Tranquility has always been memory hungry. For better performance, then, we have always opted for pre-computing values & processing data and storing the results for later reference rather than re-computing those values again later. As an example, the Brain in a Box and Dogma Rewrite projects in 2015 were all about computing and storing skills and their effects (i.e., the characters' brains) and transferring the computed results between solar systems instead of re-computing the brains on each entry to a new solar system. We also never clean up any memory, as the cluster node memory is reset every day anyway, which is a reliance on a daily reboot (note: we of course don't clear our DB cache memory or our Redis cache memory, but the main simulation cache memory is cleared in the reboot in each downtime).

The most memory-hungry nodes in the Tranquility cluster, the Character Services nodes that store those brains I mentioned above (among other things), were at 75% memory pressure at the end of day #2 last time, which is just below our operating tolerance of 80%. We might be able to run Tranquility for 3 days (and perhaps 7 hours more) if we were to run the cluster to a "first-node-at 100% memory usage" state, given those 2019 numbers. In 2019, the day #1 memory pressure was at 55%, but these days it is around 35% and so we want to rebase our observations.

No-downtime is a long-term goal and all our technological advances aim towards that. We have been working for a few years now on a micro-service and message bus technology platform for EVE, and started using that platform for a number of features. We now want to observe how that ecosystem holds up with no downtime of the primary game cluster, making sure no assumptions have been made about a daily downtime.

See you on Thursday, 9 September, as the second No Downtime experiment commences. And, just like last time, there will be a video coming soon. A whole lotta effort and power is needed for such a heroic stunt.

Ambitious Capsuleers,

Join your trusted allies in heavy combat during a new, special Caldari Union Day Proving Grounds event, battling side-by-side aboard powerful battlecruisers as you prove the unbreakable strength of your bonds, evoking the unifying collective drive of the Caldari people!

This event begins at 11:00 UTC on 3 September, and runs until 11:00 UTC on 7 September.

The ruleset for the event is as follows:
- Participants receive 20m ISK each time they enter this Proving Ground event
- Only the Drake, Ferox, and Naga may enter
- Only one local repair module can be fitted per ship
- No shield boost amplifiers, rechargers, power relays, flux coils, or purger rigs allowed
- Sensor dampeners cannot be fitted
- Modules are restricted to meta level 5 and below
- No pirate implants allowed

Gather your friends, fit your battlecruisers, and grab your Proving Ground filaments (found in Abyssal Deadspace or on the ingame market) as you prepare for intense team combat, battling in tandem to climb the Proving Grounds leaderboard, achieving glory and earning special rewards!

PLAY EVE NOW

Self-reliant capsuleers,

As the Foundation quadrant draws to a close it's time for the Caldari to celebrate Union Day.

As with the other factions, the State seeks contributions of capsuleer propaganda that celebrate the unity and resilience of Caldari! To be in the running to win some cool prizes you must submit a piece of propaganda for review. Some of the topics it could address are:

• celebrate the rich history of the Caldari people
  
• denigrate the the enemies of the State - i.e. the Gallente, Minmatar, Guristas or Triglavians
  
• highlight a famous location in the State
  
• recognize a notable Caldari citizen and their accomplishments (this includes both player and non-player characters)

Or something else not listed here - as long as it's putting the Caldari in the spotlight we want to see it! You can find some examples of the excellent work produced by capsuleers for the previous Foundation quadrant contests in these threads 1|2|3 on the official forums.

[h2]PRIZES[/h2]

Submissions will be pored over by a crack team of CCP developers with impeccable taste and the chosen winners will receive the following:

[h3]1st place[/h3]

• Your choice of any Caldari ship model (Condor, Naga or Scorpion) offered by our friends at Mixed Dimensions
• A set of brand new apparel: the State Commander's Dress Jacket and Pants (men's and women's)
  
• Golem Octopus Orange Flight SKIN

[h3]2nd place[/h3]

• 1100 PLEX
  
• State Commander's Dress Jacket and Pants (men's and women's)
  
• Raven Octopus Orange Flight SKIN

[h3]3rd place[/h3]

• 500 PLEX
  
• State Commander's Dress Jacket and Pants (men's and women's)
  
• Drake Octopus Orange Flight SKIN

All winners and any distinguished entries chosen as honorable mentions by the jury will also receive a medal from the Caldari State to display with pride on your character sheets.

[h2]HOW TO ENTER[/h2]

• In order for your work to be considered it must be submitted in this thread on the EVE Online forums.
  
• The contest starts now and the cutoff for submissions is Monday 13 September at 11:00 UTC. Winners will be notified soon after!
  
• All entries must abide by the EVE Online Terms of Service
• Although no longer required for entry, please do continue to share your propaganda on EVE-related social media, such as Twitter with the #tweetfleet hashtag.

[h2]Livestream quiz[/h2]

Also be sure to to tune in to CCP TV at 18:00 UTC on Monday 6 September for the final installment in our series of Foundation quadrant livestream trivia! There'll be 3 quizzes of 15 Caldari-flavoured questions with Omega, apparel, SKINs and medals up for grabs. Set a reminder and then tune in to test your Caldari knowledge for a shot at some prizes!

Patriotic Capsuleers,

Caldari State Workers Union Day is on 4 September, and celebrations run now through to 8 September! Union Day is an important event in the calendar of the Caldari State and is mandated by the Chief Executive Panel (CEP) throughout all State-controlled regions. Representing the strength of the Union and the workers who make up the populace, this day also serves as a convenient time for the eight largest megacorporations to hold their annual Industrial Planning Summit.Capsuleer celebrations this year will feature daily gifts, new lore monuments, mining blitz sites, and plenty more. There are even deals on celebratory SKINs including the temporary return of the popular State Police SKIN, now on more ships than ever before!

For the latest on Caldari matters, stay tuned for a highly anticipated address, in the form of a video, from the newly selected leader of the State, Akimaka Saraki, which is expected to be delivered within the next few weeks.

[h2]Daily Gifts and Challenges[/h2]
There’s lots to get involved in for Caldari Union Day. By logging in each day of the event you’ll receive rewards ranging from SKINs to pilot clothing and boosters. There’s even up to 25,000 free Skill Points available for Omega players.

You can also get a Caldari Union Day Expert System, giving you everything you need to experience the life of a loyal Caldari pilot. Don’t forget that upgrading at any point during the event will unlock any Omega rewards you might have missed!

As with previous events, there will be a warp speed bonus across Caldari space during the celebrations. This is great for travelling around New Eden, but even better if you’re looking to visit the Titan parades. The following corporations will be showcasing the might of their private militaries so be sure to pay them a visit and fire off some fireworks to receive a reward.

• Nugoeihuvi Corporation – Josameto
• Lai Dai Corporation – Airkio
• Sukuuvestaa Corporation - Saisio
• Kalaakiota Corporation – Sobaseki
• Wiyrkomi Corporation – Annaro
• Ishukone Corporation – Malkalen
• Hyasyoda Corporation - Suroken
• CBD Corporation – Muvolailen

[h2]New Lore Monuments[/h2]
Two new lore monuments are being constructed in time for Caldari Union Day. One such is the Landfall Kutuoto Miru Orbital Center in New Caldari. The facility is dedicated to the teachings of Kutuoto Miru, the Caldari philosophy that seeks to align the mind and body. Loosely translated to "mind tool," Kutuoto Miru encompasses a range of disciplines designed to help practitioners maximize their focus, equanimity, and mental fortitude.

You can also visit the Urlen II Provist Riots Memorial to pay your respects to the lives of State citizens lost at the hands of terrorists in YC110. Anti-Gallente riots took place across Caldari space following a State assault on the contested world of Caldari Prime. While most incidents resulted in relatively few casualties, the riots in the underground industrial colonies of Urlen II resulted in the deaths of at least 1,000 State citizens, the vast majority of whom were members of the ethnic Gallente communities on that world.



[h2]State Military Stockpile Mining Blitz[/h2]
For Capsuleers interested in a mining expedition, Mining Blitz sites return across the constellations of Umamon, Okomon, and Santenpaa. Visit these sites to support the State Military Stockpile, earning LP to spend on SKINs and more. Three new ships are being added to the list of SKINs available this year: the Griffin, Corax, and Ferox.



[h2]2v2 Caldari Battlecruiser Proving Ground[/h2]
Show your prowess with missiles and blasters in a 2v2 T1 battlecruiser Proving Ground. This PvP event comes with a repeatable challenge refunding you 20m ISK, helping cover your losses while you battle your way to the top of the leaderboards in a Drake, Ferox, or Naga.

The rules for this Proving Ground include a limit on repair modules allowed, with only one local repair module permitted (either shield or armor). The following modules are also banned:

• Sensor Dampeners
• Shield Boost Amplifiers
• Shield Rechargers
• Shield Power Relays
• Shield Flux Coils
• Purger Rigs

Stand your ground and blast your opponents away, Capsuleers. For the State!



[h2]CCPTV Stream and Propaganda Contest[/h2]
CCPTV will be hosting another of the popular empire-themed quiz streams soon so don’t miss out! Study your State lore over in the Fiction Portal and see if you can claim one of the top spots on the quiz leaderboard. Winning Capsuleers will receive a unique in-game Caldari Commander uniform, in-game medals, and even several months of Omega!

For the creative capitalists out there be sure to take part in the Caldari propaganda contest. We will share more information in the next couple of days!

[h2]PLEX Deals and State Police SKINs[/h2]
It wouldn’t be Union Day without some slick deals and new Caldari SKINs! This year you’re getting a free Union Day SKIN bundled in with select PLEX packages:

• 240 PLEX + Free Corax SKIN
• 500 PLEX + Free Drake SKIN
• 2,860 + Free Naga & Rokh SKINs

Take your new PLEX to the in-game New Eden Store and you’ll find a brand new set of ships returning with the ever-popular State Police SKIN. For just 110 PLEX each you can chase down criminals with SKINs for the Crow, Hawk, Manticore, Eagle, and Ferox. And for 250 PLEX you’ll get a State Police SKIN for the ultimate Caldari subcapital ship, the Golem!

If you’d rather flaunt your Caldari pride in a different shade, there will be more deals on offer with 20% off the majestic Kimotoro Eskeitan SKIN line in the New Eden store as it returns for a limited time! Most SKINs are available for just 88 PLEX with capital SKINs going for 312 PLEX.

Don’t miss out on the fun for Caldari Union Day. Log in, claim your rewards, and join your fellow Capsuleers in celebrating this important event!

FOR THE STATE!