Compression for Godot Games Using Web Workers and WebAssembly

Published 18 March 2020

Mikael Hernvall

2 minute read

Automatic compression

Reducing loading times improves user experience, which is why games uploaded at Gotm are automatically compressed.

In this post I'll talk about how Godot web games hosted on Gotm are compressed by up to 80% in the client's browser using Web Workers and WebAssembly.

The Game Pack

Godot games are encapsulated into a single file referred to as the pack. The pack is essentially the same as a .zip file of all the assets in a Godot project, minus the compression.

Simply compressing the entire pack with brotli or gzip would yield some size reduction. But we can do better than that! Image files and audio files have domain-specific compression algorithms that are much more efficient, e.g. webp and opus.

In order to compress specific files in the pack we must unpack it, compress whatever files we can, and finally repack it again.

Seems pretty straightforward, except...

Compression Is CPU-Intensive

Since the autocompression is performed by the client when uploading the pack, it's important that the process doesn't take too long, or it would detract from the user experience.

That's why the autocompressor is written in C++ compiled to WebAssembly, yielding near-native speeds. However, some larger games still take 10-15 minutes to compress. We have to do better!

Enter Web Workers. Web Workers are to web environments what processes are to native environments, and allow us to parallelize the compression and split the workload across multiple CPU cores. Since they don't normally share memory they have to communicate via asynchronous messaging à la IPC.

(Shared memory would've been more performant and easier to work with in this case, we would've much rather liked to do it that way. But since only Chrome supports it we'll have to do it the IPC way. ¯\_(ツ)_/¯)

Parallelizing the compression greatly reduces the compression time. But even if it only takes 5 minutes for larger games, that's still quite a long time for the user to wait every time they upload a new version of their pack. We can do better!

Reuse Previous Compressions

Since most files in the updated pack are probably unchanged, we shouldn't have to compress them again. After all, we already have them in a compressed format in the previously uploaded pack.

By hashing the files, we can efficiently compare the files in the new and old packs. If we find a match, we simply copy over the compressed file.

Now updated packs are compressed much faster even for very large games, since only a handful of files need to be processed.

Quality Concerns

Some game developers may be concerned if their game files' quality will suffer due to the compression.

When compressing audio files and image files, the autocompression uses a mix of lossless and lossy encodings depending on how the files are used in the game. This ensures that no quality degradation should be perceptible.