Bending Minecraft
challenges while realizing a persistent open world MMOG in Minecraft

Minecon Panels: Content for 'professionals' and MC-2025

The next Minecon is coming up and you can suggest topics for a panel. I personally always thought there was/is a lack of 'professional content'. Or at least I have never seen or heard of a panel/event that had a more technical/programming/code-design slogan (granted I was never there and only see what the internet provides). There are more than 140.000 minecraft servers running just for the pc version without realms and there is no panel for code-structure, optimization, general guidelines?

Even if we look at the reasons why Microsoft bought Minecraft it would make sense to me to have a discussion about the tools, techniques and design patterns that can improve the quality of projects in general. At least the overall goal should be to keep the modding community growing, as it is driving the whole thing.

Now realistically I don't expect something like this to happen at a Minecon. As far as i can tell from the YouTube-Videos and Twitch-Streams the general content is very...basic.

Still I wanted to make an effort and at least push this idea into their direction maybe this can achieve more than just a panel.
Well, what to do?

Send Mojang a video suggestion!
With some reasons why nurturing the 'professional' community can help

How to stand out?

There are probably a lot of suggestions and most of the panels are pre-planned. I´d argue only a handful panels are not yet decided. So we have to do something that stands out to even get noticed and in addition to that our goal here is hard to accomplish as a panel (or something similar) of this type was never (to my knowledge) at Minecon.

Let's stand out!

As we are talking about programming related stuff mostly, let's try to bring up some related topics to stand out with. The ideas I settled on were:

1. Stand out by fixing some of the most voted/problematic bugs in Minecraft. As I am not familiar with the client-code I will only focus on server-side problems.
2. Stand out by optimizing general design/code-problems in Minecraft. Again I will only focus on the server side.
3. Showcasing plugin technique/design that could potentially be interesting for the company itself.
4. (Boldly add a link to this post in the video and hope I did not screw up somewhere)

1. Fixing very influential bugs

The first thing that should come to mind here is MC-2025. It is a bug from 2012 and is the #1 most voted bug on the bugtracker.
I took it as a challenge for the video to fix this bug.

  • Fixing MC-2025 (and probably others: MC-2545,MC-9568,MC-23320)
    You know this one: Animals going out of fences. ୧༼ಠ益ಠ༽୨ After looking a little bit closer into this bug I realized that I could potentially fix this bug within a day or two (the remainder of it, several other problems were fixed over the years). Of course there is always a probability that its a bug that needs a lot of code-rewriting but from experience I had the impression this was not the case. In the end it took me about 4 hours. You can find my report here. This specific problem was even reported earlier but never used by Mojang. It seems to be the only thing missing to fix this problem for several MC-Versions now. Seems like I and Mojang should have read the comments more carefully to save some time.
  • Fixing MC-911/MC-80966
    You probably know this one too: lightning-bugs especially in the nether and on bigger buildings.
  • Fixing MC-10976 (and probably others)
    This bug took a lot of time to track down as it only occurs very rarely and in special situations. Anyhow this is one of the worst bugs still present in Minecraft as complete chunks of your map can get lost.

These problems and fixes are pretty interesting for a discussion:

  • MC-80966 is a very simple logical problem.
  • MC-10976 is very complex and a pain to track down.
  • MC-2025 the most famous one is a rather common but very interesting float/double rounding problem.

There is a lot to learn from these problems in general!

2. Optimizing the core functionality of MC

I´ll try to keep this short. Several core-concepts and design decisions from the early stages of Minecraft are not longer a good fit for the game. In retrospect this is always an easy thing to say - so let's add some actual practical optimizations to this:

  • TileEntities often tick needlessly. There are several TileEntities in Minecraft that do not actually need to tick. By simply not adding those to the TileEntity-tick-List we can save 50% or more of the iterations every tick.
  • ItemFrames are Entities and therefore pretty costly. But there is actually no reason to treat them as entities. They cant move, they have a block-position. They cant rotate freely: They pretty much act like a block. By adding a "ItemFrameManager" and completely removing them from the EntityTracker we can now add an unlimited amount of ItemFrames without worrying about the server performance.
  • The Grow-Algorithm ticks needlessly a lot of the time. This can be optimized by simply removing some blocks that were wrongfully marked as ticking. An example would be the cake-block. The intention here was probably to let the cake rot over time. There are about 10 to 25 more depending on how you want to handle some special cases.

3. Showcasing plugin features/technology

As the third and last point for this list let's look at plugin features Mojang/Minecraft might benefit from. This one is kind of tricky as we have to guess what path they are interested in for the game or a sequel.
Additionally there is a ton of plugins we could mention i'm gonna mention my own Img

  • Chunkli would be 'just another rehashed WorldEdit' if you don't actually try to understand it. But the idea of the plugin is not to just copy stuff from A to B. Its to build a API that allows admins, players and 3rd party plugins to transfer data from one place to another. A good example would be allowing a player to transfer a plot from one place to another (even to another server) without the risk of data loss or duplication. The probably most amazing thing/concept: Chunkli would enable you to create an 'online store' for Minecraft buildings! In the example below a secondary plugin uses the chunkli API to re-generate the gameworld without overwriting player plots (realtime; no speedup!).

Click to cycle (1)

  • FakeStuff allows you to build sub-block-structures (Models, Sculptures, Furniture, ...) without any client-modifications. You might have seen SethBlings Video about this. The main difference: FakeStuff does not need Commandblocks, and will not lag the server if you create several thousand or a million ArmorStands. And most important: You are at least 100x faster while building compared to commandblocks :).

Click to cycle (1)

But the actually interesting part from a professional point of view is not that it 'works' but how specifically things were achieved. Examples for the two plugins here would be:

  • Chunkli: Copying things from A to B is not that interesting. Transferring data with duplication-protection between servers and without any lags at runtime on the other hand is pretty interesting as it needs a completely different technology in the background.
  • FakeStuff: Creating Furniture/Sculptures is not that interesting. Having 100.000 ArmorStands in one world without any server lags is again something very different. That's when it gets interesting. :)

Summarized: I would love to see more ideas, more technology and more creative implementations - and I think others would love this too!

(*) I have no intention of making fun of anyone. The general idea of this post/video should be fun but I am definitely not trying to play anyone's work down or offend anyone.

StackTracing and the hidden GC-CPU usage

There is always a lot of talk about performance when talking about Minecraft servers. Mostly this involves different plugins and Minecraft/CraftBukkit/Spigot in general. There is always something you can do or change to make the server run a bit better.

In this topic were going to add another level to the common things: The Java garbage collection.

Interestingly you wont actually see the time the garbage collections "steals" from your process unless you actively look for it. Depending on your server and playercount you can possibly gain about 5-10% more CPU-Time for your process. Of course mostly this does absolutely not matter as your server is not running at its limit anyway.

Ok lets do a little step by step guide:
(Based on Java 8 and Ubuntu)

Get the time spent by your server on garbage collection

You can always do this step, no matter if your server is actually at its limit or not. First get the PID of your minecraft process. For example using top -n1 or htop you should see one or more "java" processes. You should do this when your server is populated, not at night when nobody is around anyway!


Use the one for the server you want to inspect and type

jstat -gcutil 8243 1 1  

This command will give you some values for said process. For my process this currently would be:

minecraft@server:~$ jstat -gcutil 8243 1 1  
S0     S1     E      O      M     CCS    YGC     YGCT    FGC    FGCT     GCT  
0.00 100.00  57.14  83.77  95.71  88.78   2293   88.953     0    0.000   88.953  

Now you can ignore most of these values. But there are some simple rules:

  1. If you have something else than Zero on FGC or FGCT your process is not running good.
  2. GCT tells you the total time that the garbage collector stole from your java-process since you started it. This should never be more than 3-4% of the total runtime. If it is more you can try to optimize.
  3. If GCT is less than 3-4% of your total runtime - don't bother wasting your time here.
  4. The raw number of GCT does not mean anything. Only percentage of runtime/gctime will tell you something.

Now lets calculate how much time GC stole from my process. It started about 8 hours ago. Thats 28800 seconds. The GC in that time took 89 seconds.

89/28000 ~= 0,00318  

That is about 0,3% of the total time (granted today is thursday and the playercount is lower than normal). As a comparison with a high playercount and not optimized settings I had about 5-10% CPU time just for the GC. Now let's try to optimize. For this you have to interpret your own values.

Why FGC is always bad for games

FGC sands for the number of full GC events. This means:
1. Your process got paused
2. The garbage collector ran trough all of the memory at once
3. Cleaned up all it could at once
4. Allowed your process to run again

This is bad for games in general. A full garbage collection can take several seconds depending on the amount of memory your process uses. Everything will pause/lag and your players will definitely notice. This should never happen on a Minecraft process. Fix your settings (and/or plugins).

Fixing FGC and high GCT

There are several reasons why you could have FGC. It could be just not enough memory for the process or bad startup parameters. Additionally a high percentage of time used by GCT can have nearly the same reasons. For example giving your process only the minimal amount of memory to still be functional can drastically increase the GCT time.

Try to run your process with this command - but make sure you cange the -Xmx8G (Amount of memory the process may use) to an appropriate value.

java -XX:+UnlockExperimentalVMOptions -server -XX:+UseG1GC -XX:+AggressiveOpts -XX:+DoEscapeAnalysis -Xmx8G -XX:InitiatingHeapOccupancyPercent=45 -XX:G1HeapWastePercent=35 -XX:MaxGCPauseMillis=40 -XX:GCPauseIntervalMillis=100 -jar craftbukkit.jar

Lets shortly explain to most important values:

Tells java to use the new garbage collection. Especially for games and the goal of very short pauses this is needed.

(Default value 10) Normally java only allows a process to waste 10% of memory for "old data". By allowing java to waste up to 35% of the given memory it has to to garbage collection runs less often and additionally will free up more memory in a run as there is more stuff that can be cleaned up in a smaller amount of memory. This can reduce your GCT a lot if you got the memory to spare! Do not go higher than 30-40% here. This can cause problems again and will probably even decrease overall performance (I tested the values from 5-65% in 5% steps over several days).

(default 200) Tells the GC to only use 40 milliseconds at a time to clean up garbage. This means your java process will be paused for 40 milliseconds at most at a time. This is less than a Minecraft tick.

Tells the GC not to start a new garbage collection run too fast (only one GC run every 100 ms of process time).

(default value 45) Tells java to clean up memory-regions that use more than 45%.

As you can see most of the performance gains of this setup result from the use of UseG1GC in addition to allowing the JVM to waste more memory (G1HeapWastePercent). But allowing the JVM to waste memory you have to make sure there is enough memory to be wasted! Allowing 35% G1HeapWastePercent with a high player count on a big server but only 3-4GB of memory will not be a good idea. Your server will not be able to clean the memory in time and be forced to run a FGC - the thing we want to avoid the most.

Optimizing the optimization
If you really want to squeeze out the best performance you will have to test yourself. Nobody can tell you the 'best values' for your server as every server is different. To do this you will need to add a shell/cronjob that writes out your FGC/GCT times to a file. When this is setup you start to change one value of your starting-parameters and try to compare the values after one or two days (same player count of course, don't compare a Sunday to a Thursday).

tl;dr: Use UseG1GC with G1HeapWastePercent and a lot of memory to save on garbage collection time. Most other recommendations are crap for Minecraft.

Minecraft 1.9: The Anti-Xray Problem

In this post we are going to look at the (simplified) data and protocol changes made from Minecraft 1.8 to Minecraft 1.9. Determine why previous Anti-Xray Code wont work anymore and build a new - event better Anti-XRay solution than before :)

Lets outline the changes:

Minecraft 1.8

  • Chunks are simple arrays you can easily parse.
  • Arrays contain the needed information directly
  • Chunks are being sent without any manipulation directly to the client.
  • Intercepting outgoing packages is a nice and easy way to manipulate data on the fly, as only minor changes are needed.

Minecraft 1.9 and forward

  • Chunks no longer are represented as simple arrays
  • A block-palette is used, the array only contains references to those palettes.
  • The system is pretty much exactly a dictionary-compression
  • Packet data is no longer easily accessible.
  • It would be possible for a chunk-section to contain no "stone" blocks. As we use stone to hide most ores we would need to add the stone-id to the palette and recode all data again on the fly. This would probably break our performance goals.
  • The array-size changes depending on the palette size. Adding/Changing blocks here is no longer trivial and can often result in in resizing/recreating the objects.

Here is an example of how block-changes are written to the new array types.

public void a(int i, int j) { // input is i= (x,y,z) and j = hashed block id for that chunk  
        Validate.inclusiveBetween(0L, (long) (this.d - 1), (long) i);
        Validate.inclusiveBetween(0L, this.c, (long) j);
        int k = i * this.b; // 4 bits or more per entry
        int l = k / 64;     // The long-position of the k entry
        int i1 = ((i + 1) * this.b - 1) / 64; // is (Z + ( 16 * Y ))
        int j1 = k % 64;

        // set the long with new data: convert the long data: first reset the bits by overwriting with zeros
        // and then add with blockHash 
        this.a[l] = this.a[l] & ~(this.c << j1) | ((long) j & this.c) << j1;

        if (l != i1) {
            int k1 = 64 - j1;
            int l1 = this.b - k1;

            this.a[i1] = this.a[i1] >>> l1 << l1 | ((long) j & this.c) >> k1;

As you can see these code parts are highly error prone and every time something here changes we have a lot of work to do. Even understanding these code parts takes up much time. I´d recommend not messing with the functionality of things like this unless you cannot find any other option (and keep in mind this will result in high workload on every update).

But we still want to mess with the code - we need Anti-XRay!
(If performance is not a problem use Orebfuscator)

Fortunately there is a pretty good way to implement Anti-XRay with this structure. And the best part: It is event more performant than all options previously used in 1.8!

In theory we can implement the code like this (strongly simplified):

  • Implement a derived classes of the ChunkSecion
  • Every time a Chunk is loaded not only create the normal chunk-data but create a obfuscated duplicate as well
  • Change the implementation of the Chunk-Packet. Send obfuscated data to users and normal data to admins.
  • Keep track of ore-locations for each ChunkSection and add them to the Service that gradually sends updates to the player depending on his position (e.g. ProximityHider but keep in mind this code has some flaws).
  • (Add some additional stuff like adjacency updating on block changes, neighbor updates on chunk loads etc.*)

*Sorry wont go into details on this one, this post is long enough already :)

Now let's highlight why this implementation is better than the one used in Minecraft 1.8:

  • We obfuscate the chunk on load. Generally every chunk that is being loaded will also be sent to at least one user. Normally a loaded chunk will be delivered to several players. The previous implementation would either re-encode the chunk-data every time it is sent to a player or at least re-check the cached data to make sure it is valid. The new implementation reduces the efforts of recalculations to a minium.
  • We generate the obfuscated data at the same time we load the data anyway. This means the memory-pages are already loaded in the L1,2,3 caches of the CPU and we do not load them multiple times like before.
  • Easy to upgrade. Our changes to the frequently changing minecraft-codes are very limited. Most of our work is simply deriving classes and adding new methods - not overwriting or re-implementing complex method flows.
  • Low memory usage.

At first the last point might seem wrong as we keep two versions of each chunk loaded. The memory usage should go up?
Yes and no.
If you only look on the actual occupied data at a given point: Yes we use more memory.

But memory-usage has more to it. Firstly the code-changes in Minecraft 1.9 reduce needed memory per chunk a bit. If you think further and account for unusable memory (waiting for GC) as well as the additional workload for the garbage collector with the old system:
No, we need less memory (related work).

To add some more information as to why this is true:
We can somewhat summarize the important parts of "memory" with:

  1. Actually occupied memory by needed data
  2. Occupied memory by no longer needed data (GC needed)
  3. Work to create new objects rather than re-using old ones
  4. Work to clean up no longer needed data (GC time & interruption frequency)

Memory is a pretty complex topic and reading guides online about "Optimizing the Java VM for Minecraft" often hand out plain wrong tips.
Specifically for Anti-XRay we want to re-use already existing objects in memory as they are pretty big and are needed often - but not short lived enough to die in the young generation.

Okay I think this is enough for this post. If you are interested in more information feel free to send me a message at kademlia[ä]

By the way for a survival server with a big playable area this is a good JVM startup at the current state of Minecraft (minimal loss of cpu-time from GC-Runs while not pausing for too long and never having to do a full GC). If you are running a minigame server don't bother changing the JVM/Java Garbace Collector at all.

java -XX:+UnlockExperimentalVMOptions -server -XX:+UseG1GC -XX:+AggressiveOpts -XX:+DoEscapeAnalysis -Xmx8G -XX:G1HeapRegionSize=32m -XX:InitiatingHeapOccupancyPercent=45 -XX:G1HeapWastePercent=35 -XX:MaxGCPauseMillis=40 -XX:GCPauseIntervalMillis=100 -jar craftbukkit.jar  

Requirement #2: Advanced Anti-Xray

From the previous post we know the basics that are needed to protect your server from item devaluation trough tools like xray. Let´s now have a look at the problems that will still occur in a gameworld that is not being reset every several month.

  • Finding special Structures is still easy

It does not really matter how extreme your basic Anti-Xray settings are setup. Finding special structures is still easy. Caves can be found by looking for water/lava-flow and it's easy to distinguish normal caves from mining-shafts by either looking for wooden blocks or comparing the tunnel-shapes. With this advantage you will most likely be the first to explore the caves and find the easy to access valuable stuff. The same goes for the rectangular spawner-rooms.

  • Valuable areas are easy to spot

Even if your server is build-protected and you need to buy a plot before farming xray will help a lot. You can distinguish valuable and non-valuable areas of the map from the surface. After buying a single plot and digging down you will be able to move trough all connected caves/mining-shafts and reveal spawners/ores.

  • Chest-Protection is annoying

Typically you replace chests and other valuable blocks in the lower parts of the map with stones. With this technique you can prevent xrayers from spotting mining-shafts and sapwner-rooms super easily. Unfortunately this is pretty annoying if you do not use xray. You'r storage-area looks pretty silly without chests and each time you walk around stones will convert to chests once again.

Building an advanced Anti-Xray

With these problems in mind lets look at the solution used on Kadcon. As always one of the most important goals is to stay very performant. Super specialized logic based hiding algorithms are not an option.

  • A complete copy of the world in its original state is held by the server
  • The Anti-Xray is set to only obfuscate data below Y=48, this saves cpu time and nearly everything valuable is below this height.
  • Parts of the world that are not yet owned by players are replaced with bedrock below Y=48. It is impossible to spot anything - not even an admin can see blocks here.
  • If a player buys a plot (part of the normal world) the blocks below Y=48 will be replaced with the actual blocks.

Lets have a look below the ground:

You can see in the video this method is fool-proof. No matter what permissions or clienthacks a person has: He wont see anything before investing into a plot.

Now lets look at the difference while using X-Ray.
You can click on the picture to toggle trough the Images

  • Image 1: X-Ray of the world with Bedrock invisble (admin mode; visible ores)
  • Image 2: X-Ray of the world with Bedrock visble (admin mode; visible ores)
  • Image 3: X-Ray disabled
  • Image 4: X-Ray without admin mode (missong valuable ores in the top left)

Click to cycle (1)

When looking at picture 4 you might think: Wow I can see so may ores! But what you see is actually just iron and coal - ores that are not really valuable on an populated long-running server. Not hiding those blocks is by design and saves a lot of memory/cpu on proximity-lookups.

You cannot tell from the pictures but the Anti-XRay actually behaves differently if a plot is owned by a player. Chest-Protections will be lowered. This is a small tradeoff between good protection and preventing the protection from being too annoying to the player.

Apart from the far better protection against cheaters this method actually keeps the game interesting for a longer period of time. If the underground was accessible everywhere from the start all good spots would be taken soon after your server started. With the bedrock-method you can always hope for something cool in the underground of your new plot :)

Requirement #1: Basic Anti-Xray

Building a McMMOG means having a persistent and fluent gameoworld in which players can highly interact with each other. If important resources like diamonds get worthless your project is ruined and the server will die as the gameplay no longer provides any challenges.

This post talks about the need of Anti-Xray. You cannot successfully run a server with the mentioned gameplay without it.

There are some obvious things we need to fix:

  • The World-Seed must stay a secret

If a user somehow knows your world-seed he can just re-generate the world locally and use world-manipulation tools to find all the important things (diamonds, spawners, structures).

  • As soon as you send data to the player it is known

Normally Minecraft sends all the needed world-data to a player as soon as he is in the vicinity. It does not matter if you can or cannot see normally see specific blocks. Expect every bit of data visible to a player as soon as it delivered. Meaning: As soon as you send the ChunkData around 0-16 (most diamonds are here) in the world the player will know about it instantly. How exactly does not matter. You cannot take any influence anymore past this point in time.

  • Xray is easy to install, use (and hide)

Everyone can install xray and everyone is going to if you do not punish it. With that said - even if you punish it severely a lot of people are still going to use it. Everyone thinks they can get away with it. Using simple plugins like XrayInformer can help a little bit but they have a huge drawback: Very high workload for your team (spying on your users to verify the results). And of course these tools can be tricked if you know how they work.

What is the minimal useful Anti-Xray?

To build a server with a working economy system you need at least a system that is able to:

  1. Hide the World-Seed
  2. Manipulate world-data that gets send to the players (remove ores)
  3. The ability to stream necessary block-updates on the fly when a player actually sees an ore/spawner/chest
  4. Be very performant on the Minecraft server thread
  5. Point 3 is generally only achievable by being multi threaded as well
  6. The advantage by using Xray must be at leas minimal (or better nearly neglectable)

4 and 5 are essential as Minecrafts code structure makes it nearly impossible to do any work in parallel. With 100+ players online you need every millisecond even with massive optimizations. There is nearly no room for heavy operations like world-data manipulation just for Anti-Xray.

How to implement Anti-Xray in 1.8 and earlier?

Minecraft 1.9 made some heavy changes to the way the world is saved and transmitted. I will present a good working solution in another post. To keep this short, as it is kinda obsolete now because of the 1.9 changes, this is how you would implement it:

  1. Hook the Netty outgoing queue of every player (or internal send-queue of mc)
  2. Listen to the Packets containing Chunk-Data
  3. Manipulate the data to either hide ores or replace common blocks with ores (see Orebfuscator for some visual examples)
  4. Keep a list of important structures (chests, visible ores) for every player
  5. regularly check the list of important structures and stream the actual block-data to the player if they are close enough

This first (and already pretty complex) solution will provide a basic protection against cheaters.