Bending Minecraft
challenges while realizing a persistent open world MMOG in Minecraft

StackTracing and the hidden GC-CPU usage

There is always a lot of talk about performance when talking about Minecraft servers. Mostly this involves different plugins and Minecraft/CraftBukkit/Spigot in general. There is always something you can do or change to make the server run a bit better.

In this topic were going to add another level to the common things: The Java garbage collection.

Interestingly you wont actually see the time the garbage collections "steals" from your process unless you actively look for it. Depending on your server and playercount you can possibly gain about 5-10% more CPU-Time for your process. Of course mostly this does absolutely not matter as your server is not running at its limit anyway.

Ok lets do a little step by step guide:
(Based on Java 8 and Ubuntu)

Get the time spent by your server on garbage collection

You can always do this step, no matter if your server is actually at its limit or not. First get the PID of your minecraft process. For example using top -n1 or htop you should see one or more "java" processes. You should do this when your server is populated, not at night when nobody is around anyway!

htop

Use the one for the server you want to inspect and type

jstat -gcutil 8243 1 1  


This command will give you some values for said process. For my process this currently would be:

minecraft@server:~$ jstat -gcutil 8243 1 1  
S0     S1     E      O      M     CCS    YGC     YGCT    FGC    FGCT     GCT  
0.00 100.00  57.14  83.77  95.71  88.78   2293   88.953     0    0.000   88.953  

Now you can ignore most of these values. But there are some simple rules:

  1. If you have something else than Zero on FGC or FGCT your process is not running good.
  2. GCT tells you the total time that the garbage collector stole from your java-process since you started it. This should never be more than 3-4% of the total runtime. If it is more you can try to optimize.
  3. If GCT is less than 3-4% of your total runtime - don't bother wasting your time here.
  4. The raw number of GCT does not mean anything. Only percentage of runtime/gctime will tell you something.

Now lets calculate how much time GC stole from my process. It started about 8 hours ago. Thats 28800 seconds. The GC in that time took 89 seconds.

89/28000 ~= 0,00318  


That is about 0,3% of the total time (granted today is thursday and the playercount is lower than normal). As a comparison with a high playercount and not optimized settings I had about 5-10% CPU time just for the GC. Now let's try to optimize. For this you have to interpret your own values.

Why FGC is always bad for games

FGC sands for the number of full GC events. This means:
1. Your process got paused
2. The garbage collector ran trough all of the memory at once
3. Cleaned up all it could at once
4. Allowed your process to run again

This is bad for games in general. A full garbage collection can take several seconds depending on the amount of memory your process uses. Everything will pause/lag and your players will definitely notice. This should never happen on a Minecraft process. Fix your settings (and/or plugins).

Fixing FGC and high GCT

There are several reasons why you could have FGC. It could be just not enough memory for the process or bad startup parameters. Additionally a high percentage of time used by GCT can have nearly the same reasons. For example giving your process only the minimal amount of memory to still be functional can drastically increase the GCT time.

Try to run your process with this command - but make sure you cange the -Xmx8G (Amount of memory the process may use) to an appropriate value.

java -XX:+UnlockExperimentalVMOptions -server -XX:+UseG1GC -XX:+AggressiveOpts -XX:+DoEscapeAnalysis -Xmx8G -XX:InitiatingHeapOccupancyPercent=45 -XX:G1HeapWastePercent=35 -XX:MaxGCPauseMillis=40 -XX:GCPauseIntervalMillis=100 -jar craftbukkit.jar

Lets shortly explain to most important values:

-XX:+UseG1GC
Tells java to use the new garbage collection. Especially for games and the goal of very short pauses this is needed.

-XX:G1HeapWastePercent=35
(Default value 10) Normally java only allows a process to waste 10% of memory for "old data". By allowing java to waste up to 35% of the given memory it has to to garbage collection runs less often and additionally will free up more memory in a run as there is more stuff that can be cleaned up in a smaller amount of memory. This can reduce your GCT a lot if you got the memory to spare! Do not go higher than 30-40% here. This can cause problems again and will probably even decrease overall performance (I tested the values from 5-65% in 5% steps over several days).

-XX:MaxGCPauseMillis=40
(default 200) Tells the GC to only use 40 milliseconds at a time to clean up garbage. This means your java process will be paused for 40 milliseconds at most at a time. This is less than a Minecraft tick.

-XX:GCPauseIntervalMillis=100
Tells the GC not to start a new garbage collection run too fast (only one GC run every 100 ms of process time).

-XX:InitiatingHeapOccupancyPercent=45
(default value 45) Tells java to clean up memory-regions that use more than 45%.

As you can see most of the performance gains of this setup result from the use of UseG1GC in addition to allowing the JVM to waste more memory (G1HeapWastePercent). But allowing the JVM to waste memory you have to make sure there is enough memory to be wasted! Allowing 35% G1HeapWastePercent with a high player count on a big server but only 3-4GB of memory will not be a good idea. Your server will not be able to clean the memory in time and be forced to run a FGC - the thing we want to avoid the most.

Optimizing the optimization
If you really want to squeeze out the best performance you will have to test yourself. Nobody can tell you the 'best values' for your server as every server is different. To do this you will need to add a shell/cronjob that writes out your FGC/GCT times to a file. When this is setup you start to change one value of your starting-parameters and try to compare the values after one or two days (same player count of course, don't compare a Sunday to a Thursday).

tl;dr: Use UseG1GC with G1HeapWastePercent and a lot of memory to save on garbage collection time. Most other recommendations are crap for Minecraft.