200 players at a steady 30 Hz

In the last post we broke through the 127-player wall and promised the 200-player results. Here they are: the prototype server now holds a steady 30 Hz with 200 players and 5,000 physics props in the same space. That was our first milestone, and it took one round of performance work to get there. Before that round, the server needed twice as long for every frame.

The test

Every number in this post comes from one dedicated test server, measured the same way:

• The game server ran as a packaged build, limited to 4 CPU cores.
• The bots ran on the remaining cores of the same machine. They are headless game clients at 10 frames per second that wander around, jump and push props.
• The world is crowded on purpose: 5,000 physics props in an area of 240 × 240 m, and players spawning within about 300 × 300 m. Almost everyone is close to almost everything, which is the worst case for a server.
• We read the server's own metrics, one line per second, over the steady part of each run. Tick rate is server frames per second. Work is the time the server spends on one frame, not counting the time it waits for the next tick. At 30 Hz the budget is 33.3 ms per frame.
• In the tables, avg is the average over all seconds of a run, and p5 and p95 are the 5th and 95th percentiles: the tick rate stayed above its p5 in 95 % of all seconds, and the work stayed below its p95.

Half the work per frame

With 200 players, the old build needed 53.1 ms of work per frame. That is far over the budget, so the server could only manage 18.8 Hz. The new build needs 26.6 ms and holds 29.96 Hz.

What we changed

A profile at 160 and 200 players showed where the time went: mostly into deciding what each player needs to receive, and into sending it. The first round went after that from several sides. Every change has its own switch, so we could measure each one on its own.

• Props matter within 80 m. Before, every player received almost all 5,000 props. Now a prop is only relevant to players within 80 m of it.
• Distance tiers for players. Characters near you get updates at full priority. Further away they come less often: about a third as often at 60 m, a fifth at 100 m, a tenth beyond that.
• Fewer movement messages. Clients send at most 30 movement updates per second, and 10 while standing still.
• Push instead of poll. Iris, the engine's replication system, can be told when an object changed instead of checking every object on every frame. That mode was missing from our packaged server. Now it is built in.
• Less background chatter. Player ping is shared every 5 seconds instead of constantly, and debugging tools stay off on dedicated servers.

Prop relevance was the biggest single win. In a smaller test with 56 bots, it cut the work per frame by 13 % on its own, and all changes together cut it by 28 %. At 200 players, where the old build was overloaded, the gap grew to half. The distance tiers for players saved 8 % of the bandwidth and about 5 % of the work at 200 players. A cell-based variant of the prop filter saved another 3 % of work but cost 4 % more bandwidth, so it stays off.

Finding the limit

Then we kept adding bots.

The highest player count at which the tick rate stayed at 95 % of the target was 219. Before this round, the 30 Hz limit in the same kind of test was around 160 players, so that is 37 % more. Past 219, the server slows down for everyone. A run aimed at 300 bots stopped at 269, because the test machine itself ran out of CPU. Across all runs, no prop got lost and nothing crashed.

Faster ticks

30 Hz is a good rate for many games, but fast shooters and racing games want more. So we ran the same test with the server fixed at 60 Hz and at 120 Hz. The budget per frame shrinks with the rate: 16.7 ms at 60 Hz, 8.3 ms at 120 Hz.

We also checked that the server really sent updates at the higher rate and did not just simulate faster. With 40 bots at 120 Hz, each player received 106.9 kbps. With network sends held back to 30 Hz, it was 67.6 kbps.

When the server is full

What happens past the limit? Servers now protect their tick rate: when the work per frame gets close to the budget, the server turns new players away with “Server busy, please try again shortly”, and everyone who is already playing stays in.

We tested that with up to 300 bots and the protection switched on. The server started saying “Server busy” at 219 players and turned away 81 join attempts. During that wave the tick rate averaged 28.3 Hz, and afterwards it settled back at 29.9 Hz.

Turning players away still costs a little time, so the protection now starts at 90 % of the budget instead of 95 %.

How this compares

How often a server updates the world is one of the numbers players of online sandbox games care about most. Two popular platforms for roleplay servers publish theirs as settings. Here is where our measured rates sit next to them:

• The legacy version of FiveM sends at most 30 updates per second, or 40 with an extra setting. Our prototype held 30 per second with 219 players crowded into one small area, and with about 270 in a first test of the next round.
• The new Enhanced version of FiveM, in early access, sends 60 per second by default and can be set to 120. We held 60 per second with 160 players and 120 per second with 120 players, in the same crowded test.
• RAGE Multiplayer syncs every 40 ms by default, about 25 times per second. We held 30 with 219 players.

Their pages give the rate a server is set to, not how many players keep it. Ours is measured: every rate in this post was held by a full server on a worst-case world, with everyone close to everyone and 5,000 physics props around them.

What these numbers do not say

• The bots shared the machine. The server had its own 4 cores, but the bots ran on the same machine.
• Bots are cheaper than people. Our bots send at most 10 movement updates per second. Real players send more, and they bring scripts, vehicles and inventories, which this test did not have. Real players will cost more than these bots.
• One thread did the work. At 250 players the server used about one core: its main thread was busy 99.7 % of the time while the other cores it had stayed almost idle. In this round, the limit was a single thread, not the machine.
• Not tested yet: real players, and players spread out over a large map.

What's next

• More cores, more players. Spreading the work for each connection over several cores is the next big lever. A first test of that next round already looks good: on the same test server and in the same crowded world, the server held a steady 30 Hz with 250 players (29.9 Hz on average), used almost two cores instead of one, and kept 30 Hz up to about 270 players. The full numbers will be in the next post.
• Let the server pick its tick rate. A first version that chooses the rate from the player count and the load is in. It still has to prove itself in a measured run, and we will report the numbers here.
• A bigger world. The test world grows to 2 × 2 km with crowded hotspots, closer to a real game than one packed square.
• Real players. The game client is now playable with keyboard and mouse, which brings the first tests with people next to the bots closer.

The short version of every change is in the changelog.