Breaking the 127-player wall

Welcome to the GAMPEON dev blog. Every one to two days we will write up what changed in the engine and the platform around it: what we built, what we measured, and what broke along the way. The short version of every change lives in the changelog, which updates itself whenever we merge new work.

This first post covers the first stretch of engine work: a prototype server, our first load tests, and a wall we hit at exactly 127 players.

The first milestone

Before we build features, we want proof that the foundation holds. The first milestone is deliberately narrow: one dedicated server, 200 players and 5,000 physics-simulated props in the same space, at a stable 30 Hz. Every test run measures server tick rate, bandwidth per player, CPU load and desync between server and clients. Later runs will add a fifth number: how many cheaters the server catches.

Our pass marks:

The players are headless bots: real game clients without graphics or sound that join the server and move around on their own. They are crude stand-ins for people, but they produce real network traffic and real server load.

The prototype

The prototype is built on Unreal Engine 5.8 with Iris, the engine's newer replication system. Props are simple physics cubes simulated on the server. Every second, the server writes a line of metrics: tick rate, frame time, bandwidth per client, how many props are awake, and how many movements failed a plausibility check.

Two physics bugs showed up right away:

• Tunnelling. Fast props occasionally fell through the floor, because they travelled further in one physics step than the floor is thick. Continuous collision detection fixed that.
• Rocket pushes. Characters walking into props pushed them with far too much force and launched cubes across the map. Push force is now capped.

Making 5,000 props cheap

Most props, most of the time, are lying still. Two changes made them almost free for the server.

Dormancy. A prop that has been at rest for a second goes dormant and stops replicating until something touches it. With 5,000 props this saved about 3.5 ms of server time per frame.

A staggered relevance filter. For every player, the server decides which props are close enough to matter. Doing that for every player on every frame cost 3.5 ms. Our own grid filter spreads the work out: each player's view is refreshed every few frames, in turns. Filter time dropped to 0.9 ms.

First numbers

With those changes we ran the prototype on a dedicated test server, with bots connecting over the internet in steps of 32:

Once everyone has joined, each extra player costs roughly 0.13 ms of server time per frame, and each player downloads 160 to 190 kbps. Both are comfortably inside the targets.

One finding is less comfortable: joining is expensive. A new player has to receive the state of every prop around them, about 3,000 in our test map, and a big wave of players joining at once pushes the average frame time close to the 33 ms budget. Making joins cheaper is now near the top of our list.

We also chased a tick-rate ceiling of 22 Hz that had nothing to do with our code. Windows power saving was throttling the server process. Our launch script now switches that off.

The 127-player wall

Then we tried to go past 127 bots, and the server crashed. Every time, at exactly the 128th connection.

The cause: in Unreal Engine 5.8, Iris works with a fixed table of 128 connection IDs, which leaves room for 127 clients. Connection number 128 hit an internal assertion and took the whole server down. For a platform that wants big servers, a crash at a fixed number is the worst possible way to fail.

We fixed it in three layers:

1. A clean "server full". Before a player is let in, the server now checks the player limit and the connection limit, and turns extra players away with a clear message instead of crashing.
2. A handshake guard. Iris reserves its connection ID before that check runs, so a burst of simultaneous logins could still overflow the table. Our network driver now refuses new handshakes as soon as all IDs would be taken. Those clients time out, and the server keeps running.
3. An engine patch. We patched the engine so the number of Iris connection IDs is configurable. Our servers now use 512, and the player limit is set to 300.

What's next

• Past 127 for real. With the patch in place, the next test runs go to 136, 160 and then 200 bots on the test server.
• Cheaper joins. Spreading the initial prop state over several frames, so a wave of new players does not spike the frame time.
• Client physics, server checks. Right now the server simulates everything. Next, props that a player touches will be simulated by that player's client, while the server checks every movement for plausibility.
• Cheat bots. Bots that break the rules on purpose, so we can measure how many cheats the server catches and how often it wrongly flags an honest player.

We will post the 200-bot results here as soon as we have them.