This post describes some of the changes done to the gameplay layer of the networking engine in order to improve performance.
Let's start with some background. The game basically works with 6 networkable types: vehicles, soldiers, projectiles, cameras, vehicle spawners and soldier spawners. This is only part of the truth as vehicles can consist of multiple network objects depending on vehicle type and its moving parts (turrets, flaps, springs, etc). Each part is represented with its own networked object. It's also worth pointing out that only a fraction of the projectiles are networked. As long as the client and server can simulate the projectile independent of each other there is no need to network anything. This is true for "normal" projectiles that travels in straight lines, detonating on impact. Examples of networked projectiles are grenades, TV-guided missiles and heat seeking missiles. We also have a third category, stickies, which are used for all kinds of things, but for this post the most important are medic boxes, ammo crates, and claymores, i.e. deployables.
Traditionally in Battlefield games, the selection of weapons available to the player was controlled by kits. Each kit contained a fixed set of weapons. This made it easy to calculate memory and network usage. But as one of the driving forces in the business model used in Battlefield Play4Free is to sell weapons, the kit contents is no longer a finite collection of weapons as the store is constantly updated and all weapons in the catalog are possible to be used. So instead of a handful weapons we could very well be looking at hundreds of weapons and gadgets.
From the game perspective this means that it needs to know about all weapons and any network objects they use. This gives us challenge number 1: How to deal with an ever growing collection of weapons? This is however outside of this post as it will deal with another challenge.
If challenge 1 is about dealing with possible weapons, challenge 2 is about the amount of active network objects. Design wise Battlefield Play4Free have increased the number of networked objects compared to BF2/142 by adding more vehicles to the maps (at the time of writing this we have approx 50% more vehicles added to Oman than the original). The game also allows more medic boxes, ammo crates and other gadgets to exist simultaneously in the world compared to BF2/142 adding more objects needed to be networked.
These design changes don't manifest themselves as an increase in bandwidth as we do a good job in selecting, compressing and predicting network data. The hit is taken in CPU performance, especially on the server side. This post isn't about the networking engine as such, but it's important to know that the server holds a database containing all active network objects aka ghosts for each player. So if we have 180 active network objects, the server needs to serialize approx 5700 ghosts for 32 players, i.e. every single network object might exist as 32 ghosts, which makes us want to keep the number of network objects down to a minimum. It's the serialization process that is time consuming. Unlike a game client where we can pretty much utilize the full CPU capacity, the server needs to be as lightweight as possible to allow as many simultaneous game servers to run on a host as possible. The more game servers we can run, the less we need to spend on hosting costs. For a free game this is important.
Challenge 2 have normally been handled by limiting the amount of vehicles allowed on a map and by limiting the amount of networked projectiles available on a map. And even though the initial thought in certain peoples heads was to do the same this time around I really wanted to improve the tech to cope with the design instead. It's not the DICE way to settle for the second best ;)
The solution was found by studying the characteristics of the ghosts, and as it turned out the majority of the objects the ghosts represents were simply non-moving objects. The reason for this is simple that many players grab a vehicle, drive to a capture point, jump out and starts fighting leaving the vehicle standing. Also vehicle spawners are often located at spawn points leaving many vehicle standing waiting for someone to turn up and start using them. Besides non-moving vehicles players throws a lot of medic boxes, ammo crates, claymores, etc on the ground as part of the gameplay, and they are even less mobile. Once they landed on the ground they won't ever move again.
So why are non-moving objects a problem? Before ghosts are to be serialized over the network, we put them through a scoping process that goes through all ghosts objects and sort them based on priority. The priority is based on the type of object it is, how far away they are from the active player, what speed they are moving with etc. This is done to assure that important objects near you are updated often enough to appear and behave as you expect. The scoping process also assures that all objects will be serialized over the network at some point. So even if a vehicle is standing still it's still serialized (even though we won't send much data as it isn't moving).
Enter culling. What if we could rule out the non-moving objects from the equation? That would save a lot of CPU time.
The first approach used to test the theory was to implement a very brute force backface culling mechanism, i.e non-moving network objects behind the player weren't networked. The good thing about the approach was that it was serverside only so a new server could be deploy to the production environment and be profiled using real players. The bad thing was the objects would pop in and out as the player rotated due to objects being removed on the client as the server stopped network them. Also shadows from objects behind the player would be removed. However the results clearly showed that less ghosts gave us the gain in performance we wanted (25-75% less CPU usage).
The second approach aimed att solving the object pop:ing was to reduce the frequency of which the non-moving objects would be serialized to an absolute minimum, i.e. they wouldn't be part of the expensive serialization process except to keep the objects alive and visible on the client. This fixed the popping side effects from the hard culling, but didn't give us the same small CPU usage as the first approach. Only approx 25% less CPU usage.
The third and final approach combines the two previous approaches picking the best of two worlds (with some tradeoffs in performance of course). The solution works by putting idle objects into three zones:
- Close by objects, i.e. objects close to the player (typically 50 meters). These are given super low priority. Just enough for them to be networked every once in a while to keep them visible and active.
- Objects behind the player. These are objects that are well behind the player (beyond 50 meters). These are culled away all together and stands for the majority of the performance gain as most idle objects are culled away.
- Soon to be visible objects. These are objects that are behind the player, but close to the culling line. These are given low priority. High enough for them to be visible but low enough to still not be updated at full rate.
The end result is a CPU gain similar to the the first approach. The big difference is that the CPU have longer "spikes" (maybe not spikes. more like hills) as more objects get serialized as players get into hot fighting zones on the map compared to the first approach. There are still some visual artifacts present on high latency connections, i.e. objects far away popping when twitching the mouse, but all in all the culling works really well. I also believe we can minimize the popping by fine tuning the zone boundaries.
And thats's where we are! Testing, tuning, refining until we feel it's ready to be deployed. But that's a completely different story.
Great work. Thanks for the info through a blog. I look forward to more, if there will be more.
ReplyDelete