About a year ago, I started playing this online game, Mana World. It was pretty relaxing, and it even brought back memories of playing WOW or FFXI. The only thing is it has no real end game. The players fight in one zone from level 100-140 and it’s quite tedious. However, there are some quests that make leveling up fun.

For example, there’s a chest quest where you need 2000 keys (that’s right, 2k) and opening it at level 82 will give the player enough experience to instantly level up to 90. Also, it gives 1 of 2 possible rewards. One is really good, increasing the player’s accuracy. The other is a ring that is good for magic users. But, you want the first one. So, from about level 45 to 80+, you level up your character mostly in these dungeons where there are a bunch of spiders. These spiders drop the keys. There are other places to level, but you want to open the chest. (As a side note, at level 90, the player can do a 4-person quest that gives sweet rewards.)

A bit into leveling my first character, I found out that Mana World allows for automated leveling as long as you’re at your computer. And, it so happens that Mana World is open source (C++). So, as a software engineer, I had to see if I could automate the process of leveling from 45-80+.

So, I started out by looking up how to automate games in Python. I got tons of hits on youtube where people said something like, “I trained a neural net to beat so-and-so game!”, and they would show the outcome, but didn’t show how they did it, which fricking is anoying.

Eventually, I found this one link which is a series of tutorials that goes over Q-Learning and later it goes over Deep Q-Learning. Q Learning is a form of reinforcement learning, and Deep Q-Learning uses a deep neural network to augment the algorithm.

This was great! I had all of the tools I needed to automate the development process. And, since the Mana World client is open source, I could modify it directly to perform the movements requested by the neural network.

Q Learning basicaly performs random actions and uses a scoring mechanism to grade the choice taken. As it continues training, it relies less on random movements and starts using the ‘correct’ movements from past experience. The problem with this is that for the Q Learning algorithm to be effective, it needs to go through all possible circumstances many times (thousands) before it can effectively learn. Deep Q Learning uses a Deep Neural Network to help with the learning. After testing, it took far fewer circumstances for the Deep Neural Network to figure what to do. It can learn from previous situations, and apply that to new ones, whereas, just Deep Q Learning cannot do.

So, I my path was set. But wait… The tutorials are in Python and the client is written in C++. What did I do? Dealing with ANN stuff in C++ is no picnic and I’d have to shoehorn it into the game. To keep things separate, I decided to keep the ANN code in Python and communicate to it via gRPC (which was just a large pain to get working in C++). So, I had a Python app that sat at a gRPC endpoint and the game client would communicate with that endpoint, passing the player, mobs, and wall locations to the Python endpoint. Next, python would run the Deep Q algorithm and respond with the next movement location. I suppose this was the easy way out… But, work smarter, not harder.

I modified the code given from the tutorial to work for my situation (adding walls, multiple mobs at a time, etc) and generated the deep neural network.

Taking a step back, I didn’t create the deep neural network by doing this while the actual game was running. This would’ve taken multiple days to get enough data so the network was useful. And dying would be an issue because I’d need to run back to the dungeon from the death spawn location. It so happens that the levels are created in a level editor called Tiled and, being open source, this level data is readily available. So, I took a look at the schema of a tiled file and it was pretty simple. All I really needed was the location of the walls. So, I wrote a Python script to scrape the wall data out of the tiled file for the dungeon I was interested in. Next, I placed the player in a pre-determined start position on the map, and iteratively added mobs and placed them in different start positions. Then, I had the mobs move toward the player. I fed all of that information into the Deep Q network and ran it until the player got hit. Then, I reset the game, changing mob counts and positions , etc. I could run this millions of times in only a number of hours. Bingo!

After creating the deep neural network, I tested it in the actual game: and it worked! Woohoo! I now have 8 characters that are level 90+.

If there’s interest, I’ll make the code public on github. I just need to make certain that I don’t have any api keys in it. You see, I created a mini system in the game that would send me a text message if anybody sent me a chat message. This required a bit of AWS.

Cheers.