EVE is a community-driven massively multiplayer online (MMO) game, set in a world of galactic proportions. This online universe is governed by a hyper-capitalistic economy where space flight is the path to all commerce, communication, and conflict. Your mission is to establish yourself as a major competitor, trusted by your friends, feared by your enemies. To accomplish this, your principle tools -- apart from an impressive array of sophisticated equipment, customizable space ships and in-game corporations -- will be your natural business acumen, social skills, Machiavellian thinking and cunning combat strategies.

Set tens of thousands of years in the future, EVE Online is a breathtaking journey to the stars, to an immersive experience filled with adventure, riches, danger and glory. With nearly a quarter of a million subscribers worldwide inhabiting the same virtual universe, EVE features a vast player-run economy where your greatest asset is the starship, designed to accommodate your specific needs, skills and ambitions. EVE offers professions ranging from commodities trader to mercenary, industrial entrepreneur to pirate, mining engineer to battle fleet commander or any combination of these and much more. From brokering business deals to waging war, you will have access to a diverse array of sophisticated tools and interfaces to forge your own destiny in EVE.

CCP Games announced a new technology platform for its spacefaring MMO EVE Online that frees up server space and advances the game’s technological capabilities. The platform, Quasar, allows the game’s server, Tranquility, to handle larger player battles than ever before.

Quasar uses technology to move in-game systems, likely the recently released Skill Plans, to separate databases for the first time, operating outside of Tranquility. Moving these systems allows for more actions to take place at a faster pace, making record-setting battles bigger and smoother.

Introducing Quasar

The last time the networking layer was fundamentally changed was in 2011 with the introduction of CCP’s IOCP implementation, “CarbonIO”, which eventually became the foundation of the infamous time dilation. Originating as some scribbles under the heading “Project Sanguine”, reasoning began about the problem space in which CarbonIO lived. Every optimization in EVE comes down to careful negotiation with Python’s Global Interpreter Lock (GIL). Simply, Python can only do one thing at a time. EVE’s adoption of Stackless Python, implementation of IOCP through StacklessIO then CarbonIO, and cooperative design around time dilation is all to maintain the favorite illusion: New Eden breathes. What if the GIL didn’t have to be courted for every idea that arose? How can the hardware industry’s explosion in core counts over individual processor clock speed be taken advantage of?

There have been many experiments in this regard which are tangential to Project Sanguine, with the most public one being EVE: Aether Wars. The goal there wasn’t to fundamentally change the communication model of EVE Online, but instead change the simulation model. In contrast, Project Sanguine targeted the boring bits which represent EVE’s dense feature set. Simulating nearly 9,000 players in the same space could be faster if New Eden didn’t have to worry about everything else on its to-do list. So Project Sanguine landed on two goals: dodge the GIL and clear the table for moar lasers.

The first form of Project Sanguine emerged with ESI and the first iteration of EVE Portal in late 2016. Through these projects, a new paradigm was established within the server architecture of EVE Online: a message bus. From this new escape hatch, the bottlenecks associated with the GIL were rediscovered, but with a clearer picture of their expensive manifestations: message routing, serialization, and transmission. If one ship fires one laser in the middle of 1000 ships, that’s 1000 messages which need to be sent immediately all over the globe. The simulation must address that message to 1000 destinations as a copy (message routing), convert that data to a wire format (serialization), and then send the data over the wire (transmission). In most cases, CarbonIO has been addressing each of those issues, but within the custody of the GIL. CarbonIO has served EVE Online well for quite some time, but much has changed on the turbulent seas of the internets since 2011.

After seeing the patterns evolve in this new ecosystem, it became clear that a more standardized protocol was needed if this paradigm were ever to be exploited. With the integration of gRPC it became possible to combine the message routing capabilities of the message bus with the lightning fast serialization of protocol buffers (gRPC’s message standard). It is still necessary to schedule data with the GIL for transmission, but this is now buffered at a higher level on a separate thread. This means all transmission, serialization, and message routing happens outside the GIL except the memory copy that has to happen in-between. It cannot get much faster than that.

A firehose was now attached to New Eden, but where does it all go? When the building of ESI began, so did the adoption of more cloud native technologies such as Kubernetes, and as the need for simple concurrency primitives to digest this information started to be seen, a greater move into Go was made. With these technologies accumulating into an ecosystem of their own, work started on building out features to take advantage of the new ability to work with New Eden with modern standards. You’ve seen many of them.

The first was the Activity Tracker. It attaches itself to the firehose and monitors New Eden’s respiration to keep track of all your exploits. There’s also a variation of that with Opportunities which attempts to predict the trajectory of a Capsuleer and highlight more interesting parts of New Eden. The message bus has also been used to power the Abyssal Proving Grounds leaderboards. A massive amount of work has gone into providing the development teams with an ecosystem to harness the power of a messaging architecture with these features. However, each of these features represents a gap in capabilities: the desktop client.

Until the release of Skill Plans, each feature has “smuggled” data into Tranquility through CarbonIO. This is no longer the case as skill plan operations are not only communicated through gRPC but never touch Tranquility, or its database.

Why is bypassing Tranquility and its database so important? To really understand that, one must talk about the failures. Part of the journey has led to many new techniques and tools in which to view New Eden. One concept is distributed tracing using a new favorite toy: Honeycomb.io. (More about that journey here.) Armed with all the new shiny toys, it was clear exactly what was happening with Skill Plans as it was released into the wild.

It could also be seen in general that the performance was ok with a lot of room for improvement.

Then the following morning a chaos monkey appeared and started to harass the hamsters.

Yeah, that’s 500k milliseconds, AKA 8 minutes and 20 seconds, to send a message. The details will require some Fanfest beer and a marker board. Here are the important parts of this fail state: Tranquility didn’t go down, thousands of players weren’t disconnected, and most players weren’t affected (you can see a majority of the messages are still packed in the very bottom of the graph). This is because we don’t communicate at all with Tranquility in the traditional sense. No CarbonIO to the traditional proxies which then go to the server node and then the database. Instead, Tranquility focuses on what’s more important, and the EVE desktop client is communicating via gRPC into the new ecosystem where the Skill Plan service lives with its own database.

As you might remember recently, a volcano was drained to test No Downtime for Tranquility. One of the most powerful characteristics of the new ecosystem: No Downtime. There wasn’t a need to restart Tranquility to rescue Skill Plans. There was no need to deploy a patch to the server or the desktop client. This is a peek into the journey of Project Sanguine becoming EVE’s new technology platform: Quasar. The feeling was that now was the right point in time to give it a name so it could be more easily comprehended and referenced, as well as give you more insight into what has been going on recently with EVE’s technological advancements and how they align with setting the game up for a thriving third decade. It’s also serendipitous that this quadrant name is Gateway as it heralds the direct usage of the gRPC gateways.

Also, space words man. They just sound good.

What’s next?

Work continues on clearing the table by refurbishing many old services which provides two vectors of momentum: build up more foundational capabilities for Quasar in terms of manipulating more than just Skill Plans in the universe, and normalize ancient systems to pave the way for faster iteration.

What does that mean for the average Capsuleer? More opportunities to expose more powerful features across more mediums.

The idea of publishing simulation data through Quasar has also been toyed with…but that might take a minute.