Game performances

Measuring performances of an emulator is always a bit tricky. For box86 (and box64), I decided to use benchs of opensource softwares, and compare the results between native and i386 version. A few months ago, I did some benchs and wrote the results for the Fosdem. The article can still be found here. The result was that CPU intensive tasks were at around 50% of the native speed, and graphics intensive apps were close to 100% speed, because most work is done in the libs, so in native space because of the wrapping of the libs (the “twist of box86”). The conclusion was that a game should probably be somewhere in between.

Let’s answer that question: how does that translate to actual games. There are not a lot of AAA games that are opensource. Not actual ones at least, so the testing is a bit difficult. But in the past, some games went opensource. A major one is Quake3. It’s a highly optimized engine, that also contain a virtual machine with a JIT, and this game has an integrated benchs! So, using the fully opensource OpenArena, I can do some benchs on different architectures and see how fast box86 and box64 can be.

For that, I took the official OpenArena 0.8.8 build for i386 and x86_64, and I added an armhf and aarch64 build from the Mageia distribution. Then I take the bench setup from Phoronix, wrote a small script and I’m ready to bench!

I chose to bench using a fullscreen 1024×768 resolution, to avoid being bottlenecked too much by the GPU. The Phoronix script uses high detail, putting some pressure on the RK3399 board I’m using for this test. So I have 4 architectures: ARM64 (or aarch64), ARM32 (or armhf), i386 and x86_64.

Here are the rough results:

ARM64 version:
3398 frames 108.5 seconds 31.3 fps 16.0/31.9/68.0/6.7 ms

ARMHF version:
3398 frames 106.4 seconds 31.9 fps 15.0/31.3/67.0/6.6 ms

i386 version:
3398 frames 130.2 seconds 26.1 fps 17.0/38.3/157.0/11.0 ms

x86_64 version:
3398 frames 128.2 seconds 26.5 fps 16.0/37.7/168.0/9.9 ms

So:

FPS	Native	Emulated
32bits	31.9	26.1
64bits	31.3	26.5

The first thing that comes to mind is: the ARM64 version is NOT faster than the ARM32 one! Despite the legend that arm64 must be faster than 32bits version, here it’s not the case. While comparing binaries from the RPi repo might lead to this, that’s probably because the armhf binaries use conservative build option to stay compatible with the whole range of RPi, while those binaries comes from a distrib that is optimized for armv8, even for the 32bits build. So, while the ARMv8 ISA have many advantage over the ARMv7 (especially for 64bits integer handling!), it seems it’s not enough here to compensate for the 64bits address management…

The other analysis is, the speed of the emulated OpenArena is not that far from the native version! On 32bits, it’s 26.1 vs 31.9, so around 80% of native speed! It’s even better for the 64bits with 26.5 vs 31.3: almost 85% of native speed. And box64 is young, with some more space for optimisations!

I tried to run the benchmark on a PI/400, using the beta of the 64bits OS. But I ran in some issues: the 32bits package of libSDL1.2 has some weird dependencies that makes it impossible to install without removing critical components, and the OS also uses glibc 2.27, where the aarch64 version of OpenArena I use needs 2.29+. I cheated a bit and still got the 64bits version running. But on PI, at those settings, the game is completely GPU bound, giving just 9.3 fps for both ARM64 and x86_64 version. So, not a good test as-is for the PI.

So yeah, right between 50% and 100%, the emulated performances of OpenArena is at 80% (or 85% on 64bits). That’s pretty good! If some of you want to test, the package with the binary and the “bench.sh” script is available here (it will download the OpenArena binaries at the first launch).