EVE Online News

Noble Citizens of New Eden,

It is once again the time of year when we set out to assemble the next Council of Stellar Management, which will closely collaborate with CCP Games in the ongoing development of EVE Online. The democratic elections we are about to embark on are a unique tool within the gaming industry, providing unparalleled access to developers and embodying the tremendous value we see in shaping the future of the game hand-in-hand with the players who have shaped the world and history of EVE into what it is today. We cannot express enough how deeply appreciative we are for the dedication and passion the EVE community has demonstrated in the game’s nearly nineteen years, and for every invaluable player who is ready and willing to offer their time and work with us to continue improving the world’s largest living work of science-fiction!

Though we unfortunately were not able to welcome the council to Reykjavik this term for the yearly CSM summit, thanks to the flexibility and willingness of everyone involved, we managed to work around it and remained highly productive. Out of necessity, the world evolved – and we don’t want to stay too far behind. While we will always strive to host at least one summit per term, we don’t feel that the ability to attend one should be a requirement to run for the CSM as members can contribute and add value throughout the entire term’s length. For that reason, the CSM 17 elections will be more accessible than before by being open to players who might not want or be able to travel to Iceland.
Another improvement we are introducing this year is the 72-hour application average processing time. In the past, due to the way we announce confirmed candidates for the first time after the submission period is over, we noticed situations where players were disappointed when their candidate did not pass our thorough internal review process. Our hope here is to give you some buffer to re-evaluate your options or have time to find another candidate to put forward in the case of organized player groups.

CSM TERMS COME INTO EFFECT

Serving on the CSM is a feat, and even the most valuable members need to take a break to recharge their capacitors. Following the announcement of a three-year limit at the start of the CSM 15 Elections, we will have several current members go on cooldown for a year at the end of their term in June. We want to take this moment to thank them for their contributions and wish them an excellent year enjoying spaceships without the duties of a public CSM representative.

Members on cooldown this election:

• Innominate
• Merkelchen
• Gobbins
• Vily

TIMELINE

All new Capsuleers who might not be familiar with the function and operations of the CSM are encouraged to check out this page for more information before diving into the details below.

• 2-16 May - Submissions Open
• 9-20 May - Application Processing
• 25 May - Live candidate announcement
• 25 May-5 June - Campaign period
• 7-14 June - Voting period
• 16 June - CSM17 Announcement

APPLICATION REQUIREMENTS

Important disclaimer: incorrectly filed or incomplete submissions will be rejected. The rules are unambiguous, but we are happy to answer any questions if you are unsure about the process. You can send us an email at [email protected].

To be eligible for the CSM, applicants must meet the following requirements:

• Your account must be older than 60 days at the time candidacy applications close.
• Characters on both Alpha and Omega accounts are eligible to run.
• You must have a history of honoring the EULA and the Terms of Service. We will screen every applicant thoroughly, and EULA/TOS violations on your record can result in a rejected application.
• If you are unsure if you qualify, we strongly encourage you to apply early, as applications are now processed in 72 hours.
• AFTER you receive confirmation that you have been greenlit as a candidate, you must create a campaign thread in the CSM section on EVE Online forums covering the following questions:
• Your EVE Online story.
• Your areas of expertise. In which areas of the game do you feel you are the most knowledgeable? What qualities set you apart from other candidates?
• Why are you applying for the CSM?
• What can players expect from you?
• You are free to add more information and manage your campaign thread as you see fit, but the above four questions are the bare minimum required from everyone.
• NOTE: Campaign threads for candidates that have not been approved might be removed to avoid confusion.
• Your account must have updated and correct information at the time of your application. This includes your real-life name, correct date of birth, and the same email you use to submit your application. To view and edit this information go to the account management website .
• Candidates must be 18 years of age. If the legal adult age in your home country (the age at which you have the legal capacity to enter into a contract) is higher than 18, that number applies instead.
• As an applicant, you must consent to provide your personal details to CCP, including your real name and a copy of your passport or national ID card for those that do not plan to attend the CSM summit in person. CCP needs to affirm your real-life identity for NDA contract purposes.
• If you do not currently have a valid passport, we will accept a picture/scan of a valid and approved passport application accompanied by a picture of your national ID card.
• You must consent to share your country of origin with the EVE community and display it on the EVE Online website. You will not be required to share any other personal information.
• If you are running as an "alt" and controlling a character with a reputation in the EVE universe, CCP may require you to run under that identity at its full discretion.

All personal data will be handled in accordance with European data protection and processing laws.

HOW TO APPLY

There are three mandatory steps Capsuleers interested in running for the CSM 17 must take:

1. 
   Fill in the application form between 2-16 May. Beware session time-outs that can refresh the page, and make sure to save your text in a document before. If you don't get confirmation, re-enter the information and try again.
   
2. 
   Submit your document scans/pictures to [email protected] in high quality with the subject "CSM 17 application".
   
3. 
   Create a forum thread in the CSM section on EVE Online forums. This step must be completed after you receive confirmation that your application has been both processed and approved.
   

Remember that applications cannot be edited after you submit them, so triple-check to see if the information entered is correct! We will be sending an email to all applicants, first to confirm that we received their application and second after the internal screening process is complete.

We warmly look forward to receiving your applications and following details of the many campaigns to come; we at CCP feel that we are far stronger and more effective at developing and maintaining the unique world of EVE side-by-side with the CSM. By banding together with common purpose and resolve, and by communicating with each other in a straightforward manner, we can realize our shared ambitions for the intrepid and limitless future of EVE Online.

o7

Strategic Capsuleers,

The Abyssal Proving Grounds are opening once more, and this time 1v1 strategic cruiser combat is the name of the game.

Running for 24 hours from 23 - 24 April, this Proving Grounds clash asks you go head-to-head and flex your combat skills. It’s also the perfect chance to get a feel for a newly revamped classic hull.

As part of the Road to Fanfest updates, the beloved, once mighty Proteus cruiser has been given a boost to its power and capacity, updating its performance to meet the needs of modern pilots. With that in mind, the next Abyssal Proving Grounds event will focus on 1v1 action for strategic cruisers, including the revised Proteus.

To level up the excitement and drama of this round, the following arena effects will be in place throughout the event:
- The benefits gained from overheating the following module groups will be doubled: tackle modules, propulsion modules, repair modules, resistance modules, energy warfare modules, turrets, and launchers
- Competing Capsuleers will receive 150m ISK for each match where they deal at least 7,000 damage to their opponent

The event is open to Legion, Tengu, Loki, and – of course – Proteus hulls. During this Proving Ground clash, the arena environment will contain large numbers of Bonus Nexus objects that will explode and apply weapon and mobility buffs when approached by Capsuleer ships.

Meanwhile, sensor dampeners, weapon disruptors, shield boost amplifiers, shield rechargers, shield power relays, shield flux coils, and core defence field purgers are banned for this Proving Grounds round. Additionally, ships entering this 1v1 event are only permitted to have a maximum of one local repair module fitted (either shield or armor).

Pilots that join this event will be restricted to Meta 5 modules.

To take part in this strategic cruiser-focused Abyssal Proving Grounds round, grab some Proving Filaments and head to the Proving Grounds between 23 April and 24 April. And as ever, the Proving Grounds leaderboards will allow you to monitor Capsuleer performance throughout the event.

With millions of market transactions and hundreds of thousands of explosions every day, keeping New Eden running smoothly – and still setting records after nearly 20 years – is a monumental task. Like our players, we’re constantly on the hunt to push boundaries. In the last two years, we’ve made significant changes to the hardware behind Tranquility, resulting in almost daily records for EVE downtime (among other things). But to see where we are, it’s important to look at where we’ve been.

Back in 2005, only 2 years after launch, EVE was already setting records as a standout single-shard universe. New Eden was generating some 1,250 database calls per second which added up to over 60 million per day! As more players came online, the servers struggled to keep up. Game features were taking over 20 seconds to load, making everyday tasks in New Eden such as warping across the galaxy, completing market transactions at the new major market hub Jita 4-4, and upgrading your ship far more challenging.

The solution to ending this suffering came in the form of a Texas Memory Systems magic box that stored data on RAM chips. It didn’t really make sense, cost a boat load of money, and promised the world. We were sold!

Within days of putting the new database servers and storage into production, we were breaking PCU records and EVE pilots were enjoying a much smoother gaming experience. Forums lit up with much happier Capsuleers, and suddenly the server could handle massive fleet battles of... 100v100 with ease. Oh, how times have changed!

Fast forward to 2009 and New Eden was still growing. Not only were there more players, but wormholes were discovered for the first time. The ecosystem was demanding more storage and the accumulation of space junk was wearing on our current storage.

Being so pleased with our last tech adventure, we looked again to Texas Memory Systems for the solution. This time we went a slightly different route and picked their RamSan 500 with a massive two terabyte configuration! Now the entire EVE Database could be stored on an SSD, which was almost unheard of at the time!

Texas Memory Systems were just as excited as us and we managed to convert their president, Woody Hutsell, into a proud member of the Gallente Federation.

[previewyoutube][/previewyoutube]

Then 2011 rolled around and, while the storage is ok, it became clear that our Database Servers themselves were starting to become our main bottleneck and throttling the expanding universe of EVE. Denizens of New Eden were met with the dreaded three-letter word: lag. Solving this was a multi-pronged attack which saw the introduction of Time Dilation and further optimizations. But while exploring the Database Servers themselves we thought – why think short term? Let’s future proof all the things!

What happened next was, at the time, one of the most comprehensive upgrades Tranquility and gaming had ever seen: HyperThreading, aka the jaw-dropping CPU upgrade that took TQ by storm.

This monumental update quadrupled the RAM (128GB DDR2 to 512 DDR3), overhauled and upgraded the network and storage systems (which tripled transfer speeds), and increased storage from 2TB of storage up to 11.5TB! You can read more about it in this news item we released.

The war on Lag is never-ending, and in 2015 we decided to improve things even further with the launch of the TQ Tech III upgrade as well as physically moved datacenters. For the first time, our database servers were in the form of a flex node, Lenovo x880’s to be exact. This meant upping the RAM to 768 GB per node and using super-fast 3.2 GHz e7-8893 v3 CPU’s.

The storage array got a boost as well, moving in new SVC primary controllers and v5000’s to help consolidate all the storage together. We added 9x 800GB SSDs and 83x 1.2TB 10k RPM SAS disks. This new hardware allowed us to slowly migrate to the new data center with minimal downtime and lag, ensuring it wouldn’t disrupt players.

In 2016, not so long after the Tech III upgrade, we found that the 4-core CPUs were just not giving us what we needed. While the CPUs were extremely fast, they just didn’t provide SQL with enough worker threads to satisfy the incoming requests. We needed more cores for such a high level of database activity.We had a solution, but it was both complex and fraught with danger! Just our style...

We had four monster x880 nodes, two for the primary game cluster and two for another database cluster that hosts the other databases. This secondary cluster was effectively idling and what we needed to do became clear: stack two of the flex nodes together and join them together. Imagine this like when the Decepticon Constructicons merge to form the great Devastator!

This brought the database server back to 32 total cores (4x CPUs with 4/8 hyper-threaded cores each) and a whooping 1.5 TB of RAM! Bigger fights, more market transactions, and more action across New Eden.

The Operations team gave a presentation at Fanfest in 2016: A Day in the life of Operations. They showed actual footage of this merge happening that you can see here:

[previewyoutube][/previewyoutube]

Things ran well for the next four years with one exception. We had a re-occurring problem related to SQL and NUMA Nodes which caused server crashes, inconsistency, and difficulty in troubleshooting. The way SQL Server side is designed, and because we now had four NUMA nodes, we would end up in situations where one NUMA Node (or one CPU) was maxed out and the others were idling. This would lead to slow startups or no startups at all. We eventually traced the issue to a small subset of the 15,000 open database connections and how EVE was using them.There are two main ways to improve things: more hardware or development time. Or sometimes, and in this case, it’s providing hardware that helps devs fix issues in the software!

So mid 2019 rolls around and this NUMA business was starting to have an actual impact outside of database failovers – leading to more of what we call ‘cold starts’ which is when we bring the cluster online with an empty database memory cache. We don’t like to see these because it makes our server vulnerable- and EVE is one mean S.O.B on such a defenseless database server.

2019 is also when we discovered the mighty AMD EPYC 7742, which was a new hardware system that would help us solve these recurring issues once and for all. We were fortunately up with AMD at a gaming conference, and to our surprise, they were excited to work with us! Well, it’s not all that surprising, internet spaceships are serious business after all.

Before long we were racking the most powerful database server known to CCP! A 2U server with 2 TB of memory and a single CPU rocking 128 cores running at 2.25GHz and capable of boosting to 3.4 GHz! Now that’s a lot of horsepower under the hood.

We ran this server in production for several months to get a feel for it (who doesn’t test in production?) and it was an absolute monster! While it ended up as two NUMA nodes (Windows currently cannot support more than 64 hardware threads per NUMA Node) it didn’t matter as this ate everything we threw at it and asked for more! All our NUMA issues were handled with ease. We just had one problem: the cost.

We tried out other hardware, but nothing came close to our AMD server. To make everything work, we reviewed the data and narrowed down our criteria: two sockets and less than 128 cores, but more than 32 to allow for growth and overhead. Meeting in the middle seemed best so 64 cores would be our target. Now, how much memory can we fit in the box?!

Jump Jump Jump – it’s mid 2020 and allegedly someone was buying up all the large memory modules to power their cloud services. We decided that we either had to wait or settle on smaller chips. So, we decide to settle. The boxes have 32 memory slots and if you fill them all with 128GB chips you should see something like this.



But, is 4 TB of memory even settling!? Nestled snuggly amongst all that memory are twin Intel Gold 6346 V3 CPU’s running at 3.6GHz, giving us a total of 64 cores across two NUMA Nodes. The two Practically-Tesla-Plaid-Priced 2U Servers are wonderous machines with double the CPU cores, more than double the RAM, double the disk access speed, and they are current-gen hardware in comparison to our previous 5-year-old tech.

We decided that they needed some way to connect to the new SAN as well - so the HBA speed should get bumped as well! Wait, hang on... New SAN? Who said anything about a new SAN?!

To ensure we can give the best possible experience to both players and devs, we went all in! We replaced racks worth of slow SSDs, hundreds of 10k SAS disks, and what seemed like a never-ending number of controllers. In their place went a super slick, insanely speedy box of 9.6 TB NVMe modules! (Two actually, we need high availability after all). This SAN upgrade was not only raw storage but also impacted how we connect to it. We jumped from 16 Gb/s fiber switches to 32 Gb/s! This cut our maintenance job duration in half. Backups are now done in just over 30 minutes.

And funnily enough, it turns out that IBM bought Texas Memory Instruments some time ago. So, while we didn’t go out looking for another RamSan, the universe had aligned to make it so. Flash System 7200 is the name and making databases faster is the game!







Here are a few images of the previous generation of SAN gear we removed.









It has now been several months since we put the new database servers into production, and we’ve also broken multiple shutdown/startup records - The new hardware is performing wonderfully and in no small part to the amazingly smart EVE Devs that fixed the NUMA issues just weeks before these machines finally landed in our Data Center!

Thanks for coming along with us on this historic journey through nearly 20 years of EVE’s database hardware evolution. We hope you enjoyed stepping back in time with us as much as we have. We can’t wait to see what database changes we have in store next! MySQL? NoSQL? YourSQL? Whatever it is, it’ll be here helping to ensure that internet spaceships stay serious business!

PS: No hamsters were harmed before, during, or after this blog post. Previous generation workers are always retired with the honor and respect they deserve, to that big (non-cloud-based-server) farm in the sky.

Ferocious Capsuleers,

The Hunt may be almost over, but there’s still time to make your mark and pick up one of the Guristas Hunter Packs. Choose the Outlaw Hunter Pack to receive 30 days of Omega time, 50,000 Skill Points, and among other goodies, a Core Ship Operations Expert System. Use it to get a taste of your potential and instantly gain access to the ‘Magic-14’ skills for a limited time!

For more veteran Capsuleers, pick the Podhunter Pack for a package of 500 PLEX, 100k Skill Points, and more bundled alongside four new Ace of Podhunters SKINs. The SKINs are usable on the Coercer, Cormorant, Catalyst, and Thrasher – perfect for taking on The Hunt with.

Be sure to act fast because you can only get these deals until the end of The Hunt on 19 April!


BUY NOW

o7 fellow Capsuleers,

We are back with another hit of the Community Beat!

In the last episode, we covered some of the cool projects in the community – We heard some of the super cool stats to come out of the Stay Frosty Event, we saw some of the creative projects players were performing to help raise funds for Plex4Good, the Easter Egg Hunt planned later this month and our latest partner spotlights.

CAKE! Not just any cake, not even only an EVE cake, but a MINING CAKE! When we saw this epic birthday cake, we had to share it with everyone else. From the straw mining lasers to the exploding chocolate asteroids, to the Ic(e)cing crystals, we loved every bit of it.. except us not being able to eat any!


EVE Onlines sandbox has a plethora of people filling multiple roles and a whole host of skills, some trainable, some, a little more creative. Returning for the ninth time, Lunarisse Aspenstar is hosting another “New Eden Capsuleer’s Writing Contest”. Running until May 22nd, the competition allows those with a more literary background to flourish and share that creative energy with the rest of the community. With no minimum length (and a maximum of 15,000 words) the contest is split into 4 different categories depending on where your skills align, which you can read more about in the thread.

Even if you aren’t a big writer, but love reading material written within the Eve Universe, all the entries get posted onto the main thread so you can read to your heart’s content!
Playing with your preferred choices of coloured bricks has been a pastime for people all over the world. Some of us love to have a set of instructions out and build something that perfectly matches the design, but others look at a block as the foundation of something to build something new, unique, and incredible. OleenaNatiras over on Twitter took inspiration from Triglavians aesthetic and not only designed a new type of ship but also built it.. from scratch! It’s even got some sweet decals on it! The attention to detail is mind-blowing.
Back towards the end of 2010 and the start of 2011, there was a competition set up called “Create a Starship” and it gave rise to some incredible submissions from the community. Ships like the Tornado were born from the works of art that were submitted during that competition.

As time passes, ships occasionally come under review for reimagining and redesigning, roles change and occasionally the desire for more wingy bits or Vertical/Horizontal supremacy takes over. We get excited when we see players taking one of the ship designs currently seen in the universe and sharing their redesign on it. Artemis Pachet posted their redesigned catalyst over on Reddit and we thought it was cool! (Honestly, I was happy that they kept the kick-stand wingy bit in!)
zKillboard (or zKill for short) has been around as a community service for an incredibly long time, not only that, but it’s probably one of the most accessed 3rd party services. When a ship is lost in New Eden a Killmail is generated, for the victim and in the case of PvP, for the person to land the final shot. This information can be taken out of Eve and published on zKill for others to be able to check out.

It’s a good educational tool to learn with as well. Figuring out the reason you died was in fact because you were using a wrong module somewhere (totally never done that). It's also great for intelligence gathering, gaining information about the person attacking you, like, where they might be located (for revenge purposes of course) or, if they fight solo or with a gang!

Big shoutout to Squizz Caphinator for his continued work, development and operation of it!

We'd love to hear from YOU, head on over to the Eve Online Forums and share anything that you're enjoying doing or making, we'd love to see it! With Fanfest coming we know you all love to bring things, why not share some of them!