What are my plans for the future? I think I will continue writing about machine learning and AI, but I intend to expand the topic to other things related to broadly understood computer science. I will probably also write in English.

That's it, I hope I will manage to write regularly :)

While playing around with docker, I encountered a few problems and that's what this post will be about.

Volumes​

While trying to work with a simple project, I found that the container could not see the files, even though I had set the appropriate volumes in docker-compose.yml. Trying to fire up bash in the container with something like -v "$PWD/":/var/www/html set also failed. It turned out that the problem is Windows. Since I can't use Hyper-V because I don't have Windows 10 Professional I have to use Docker Toolbox. After two hours of struggle, it turned out that in docker the files are visible only if they are in any subdirectory c:\Users. Somehow VirtualBox works that way, they made it so. On the regular docker for Windows in the documentation, I saw that you can fix the issue in the GUI by making each partition accessible.

One thing puzzles me. Since you're using containers to have an isolated environment, maybe this should somehow apply to file systems? Because that way the isolation is not complete.

Symlinks​

The second issue is also related to Windows. After calling the command in the container:

npm install <package_name> --save-dev

NPM throws something like:

npm ERR! code EPROTO
npm ERR! errno -71
npm ERR! syscall symlink
npm ERR! EPROTO: protocol error, symlink '../uglify-js/bin/uglifyjs' -> 'var/www/html/node_modules/.bin/uglifyjs'

Well, yes, it is impossible to do symlinks in a container, because docker does not embrace it, despite the fact that on Windows it is possible to do symlinks (the so-called junction). The solution turned out to be adding the --no-bin-links switch to the command:

npm install <package_name> --save-dev --no-bin-links

What's worse is that this most likely won't work in other cases, so I'll have to find some other way.

Maybe I'll finally install the Linux? :-)

Project - development history​

Maybe you don't remember, but as part of the competition, I decided to develop a bot project for Starcraft 2 using reinforcement learning. It wasn't a very smart idea, because there was no API for creating bots at that time, and I naively assumed that Blizzard and DeepMind would be ready for this first quarter of 2017. They didn't make it, the API will be available only in the summer. I didn't know what to do, first I decided to create my own, simplified Starcraft in Phaser (a framework for writing games in JavaScript). I even wrote something there, it is in one repository (dloranc/simple-rts-and-rl-example), but it quickly became I discouraged. Too much work with this. I would have to write things like path-finding, and I wasn't happy about it. Ultimately, I abandoned this project, but it is possible that I will do something with it in the near future, a simple example with reinforcement learning.

Finally, I decided to take on Starcraft: Brood War, the predecessor of Starcraft 2. I put together quite quickly in Java, using BWAPI and BWMirror, a simple bot performing a very simple strategy known as 5 pool. I entered the bot into the SSCAIT tournament/ladder. Surprisingly, he does quite well on ladder considering his simplicity. His results can be viewed on the [results] page(http://sscaitournament.com/index.php?action=scores). This bot is located in the dloranc/five-pool-bot repository.

You can see the bot in action here:

After writing this bot, I planned to use RL, but it turned out that BWMirror is only 32-bit, so I couldn't use Java libraries because they are only 64-bit. I would have to transfer the bot from BWMirror to something else. I didn't feel like it too much, so I decided to switch to TorchCraft, a library that allows you to develop bots using machine learning. I still struggle with this today and it's not easy because this library has virtually no documentation.

In total, I created four repositories for the project. There were a total of 103 commits, which is rather average. I also regret that I didn't manage to do anything with machine learning.

I will continue to develop all projects, except maybe the one with pool 5, because it is completed and there is not much that can be added there.

The rest of the blog​

I'm even quite satisfied with the remaining posts. During the competition, I started a series of posts about reinforcement learning based on the book I was reading, Reinforcement Learning: An Introduction (Richard S. Sutton and Andrew G. Barto). All can be found under the tag Sutton & Barto and the repository with examples is dloranc/reinforcement-learning-an-introduction. I'm doing quite well, but I think I'm explaining it all a bit poorly. There is a lot of room for improvement here. The idea to start this series was good, earlier during the competition I had problems with selecting topics. I didn't have any ideas for posts, but now I just need to read the chapters from this book and it's good :)

What did the competition give me?​

Where to start... I learned to create posts relatively quickly. No wonder, the people wrote that a person becomes faster in writing by writing and it is true. However, I won't say that my posts are good, because as I wrote above, I have a problem with explaining things simply. I overcomplicate some things. I have a lot of training ahead of me here, I will have to look through my posts and think about what can be written better and simpler.

The second thing is that I managed to write quite regularly. I've always had a problem with this. Here I must admit that I took part in the first edition of "Daj Się Poznać" ("Let yourself be known"), the one in which there were not even a hundred participants. Then I dropped after two posts. It was a tragedy. However, I made it to the end of this year's edition and I consider it a success. Writing and developing the project was incredibly time-consuming, but it worked and I'm happy with it.

Have I made myself known?​

Heh, not really. It's true that there was some interest in my bot at the beginning and a few people even commented, but nothing has happened on my blog for a long time :) Analytics doesn't show much either. The most sessions I had were 38, about halfway through the competition. I was promoting something on Twitter, but the last month was poor. To tell you the truth, I thought it would be worse.

Acknowledgments​

Finally, I would like to thank the organizer Maciej Aniserowicz for organizing this competition. By the way, I apologize for my RSS :) I would also like to thank the readers of this blog (are there any?) for reading my stuff (comment more often, please!). I would also like to thank the other competition participants for sharing their knowledge and for conversations on Slack.

Thank you very much :)

In multi-armed bandit we need exploration to find the best action because the value of each action is uncertain. The value of the action changes when we perform the action from time to time and learn about the reward we receive. The more often a given action is selected, the more certain we are that the value of this action is correct. So far, however, we have not taken this rather intuitive observation into account in our calculations. The actions were selected randomly, without taking into account whether the action values ​​were closest to the best one or how certain the estimates were.

Let's recall how we chose the best action:

$A_t \doteq \underset{a}{\argmax}\> Q_t(a)$

Which corresponds to these lines of code:

argmax = np.argmax(self.rewards)
return argmax

From the choose_action method:

def choose_action(self):
    rand = np.random.uniform(0, 1)

    if rand > self.epsilon:
        # exploit
        argmax = np.argmax(self.rewards)
        return argmax
    else:
        # explore
        return randint(0, self.arms - 1)

Let's use the formula below:

$A_t \doteq \underset{a}{\argmax}\> \Bigg[Q_t(a) + c \sqrt{\frac{\log{t}}{N_t(a)}}\,\Bigg]$

Where $\log{t}$ is the natural logarithm (i.e. to the base of the $e$ constant), and $N_t(a)$ is the number of $a$ actions performed.

The square root part of the above formula measures the uncertainty in the estimate of the value of the stock $a$.

The code looks like this:

def choose_action(self):
    rand = np.random.uniform(0, 1)

    if rand > self.epsilon:
        # exploit
        ucb = self.rewards + \
            self.c * np.sqrt(np.log(self.t + 1) / (self.action_count + 1))

        return np.argmax(ucb)
    else:
        # explore
        return randint(0, self.arms - 1)

In the constructor I added c and t:

def __init__(self, arms, pulls, epsilon, c=0):
    self.t = 0
    self.c = c

c is a parameter that controls the degree of exploration.

I generated a table with possible values ​​for some $a$ action:

As can be seen from the values ​​in the table, with the passage of time $t$, the uncertainty value in the root generally decreases, but if the action was not selected, the uncertainty increases slightly.

Summary​

This method of selecting shares depending on uncertainty is abbreviated as UCB (upper confidence bound). This is a statistical method related to confidence intervals, or at least that's how I understand it. I barely remember anything about statistics, I was never good at it. UCB is quite a good method, but Sutton & Barto warn that it does not work well in non-stationary problems or in problems in which we are dealing with a large state space.

The code can be seen here.

Finally, some charts:

Pretty good, on average UCB is better than the version without it in terms of average rewards. This jump and drop at the beginning of the algorithm is interesting.

It's worse with optimal actions:

No wonder, exploration happens more often.

For one MAB run:

All the methods I have described so far depend on initial estimates of the value of $Q_1(a)$. This is especially visible when we calculate MAB with $\epsilon = 0$, i.e. without exploration, still selecting the best possible action (arm). In statistics, we call such methods biased. The bias disappears for methods with $\alpha$ of $\frac{1}{n}$ when each action is selected at least once. For constant $\alpha$, the bias does not disappear, it only decreases with time (subsequent iterations of the algorithm).

To get rid of the bias, we can use something like optimistic initial values. In the code of the examples I have written so far, the true values ​​of each action came from the normal distribution (mean 0, variance 1) and were set in the constructor:

self.true_reward = [np.random.randn() for _ in range(self.arms)]

By the way, I would like to remind you that getting the value for a given action looks like this:

def get_reward(self, action):
    return self.true_reward[action] + np.random.randn()

That's it for the reminder.

We can encourage the algorithm to explore at the beginning of its operation by setting very optimistic initial values. I chose the number 5 and it looks like this:

self.action_count = np.ones(self.arms)

self.rewards = np.ones(self.arms) * 5

And that's it, no more modifications to the code are needed. What happens when we run the script with such self.rewards values? Let's see this with an example with epsilon equal to zero:

self.rewards:
[ 5.  5.  5.  5.  5.  5.  5.  5.  5.  5.]

reward: 1.87411213524
[ 3.43705607  5.          5.          5.          5.          5.          5.
  1.          5.          5.        ]
reward: 0.504446069392
[ 3.43705607  2.75222303  5.          5.          5.          5.          5.
  1.          5.          5.        ]
reward: 0.374953664141
[ 3.43705607  2.75222303  2.68747683  5.          5.          5.          5.
  1.          5.          5.        ]
reward: -3.66371477543
[ 3.43705607  2.75222303  2.68747683  0.66814261  5.          5.          5.
  1.          5.          5.        ]
reward: -1.59327852639
[ 3.43705607  2.75222303  2.68747683  0.66814261  1.70336074  5.          5.
  1.          5.          5.        ]
reward: -2.41539982786
[ 3.43705607  2.75222303  2.68747683  0.66814261  1.70336074  1.29230009
  1.          5.          5.          5.        ]
reward: -0.641184621377
[ 3.43705607  2.75222303  2.68747683  0.66814261  1.70336074  1.29230009
  2.17940769  5.          5.          5.        ]
reward: -1.53850026836
[ 3.43705607  2.75222303  2.68747683  0.66814261  1.70336074  1.29230009
  2.17940769  1.73074987  5.          5.        ]

The rewards received are less than five and averaged, so even for an epsilon of zero, exploration occurs and the algorithm tries each action in turn. Let's see it graphically (epsilon equal to zero):

For comparison, without our optimization (epsilon is also zero):

The other charts, those jumps at the beginning are interesting. This is the result of our optimization:

Summary​

As you can see, this is a simple but quite effective way to encourage MAB with e-greedy strategy, even as it selects the best actions every time. However, this method only works at the beginning of the algorithm, during the first dozen or so iterations. In the next post I will deal with a way to improve exploration, the so-called Upper-Confidence-Bound method.

In the last post about the project I described how the maps look like and wrote how to create a basic script that connects to Starcraft and downloads the game state in a loop, or rather each subsequent logical frame games. I don't think I've written yet what a logical game frame is. The thing with logical frames is that graphics rendering is independent of calculations that change the game state. The frame rate is not constant and depends on the speed of your computer. The game state is calculated every interval. If you have played Starcraft, you probably know that you can set the game speed in the options. Changing the speed results in a change in the time between logical frame calculations. This is a fundamental difference, if we had constant 30 or 60 FPS, the issue would probably be solved differently.

Game state​

OK, time to check what we have available in terms of game status. First, let me remind you of the basic script from the previous note:

local hostname = "192.168.56.1"
local port = 11111

local tc = require 'torchcraft'

tc.micro_battles = true

tc:init(hostname, port)
local update = tc:connect(port)

local setup = {
    tc.command(tc.set_speed, 20),
    tc.command(tc.set_gui, 1),
    tc.command(tc.set_cmd_optim, 1),
}

tc:send({table.concat(setup, ':')})

while not tc.state.game_ended do
    update = tc:receive()

    -- code here
end
tc:close()

In TorchCraft, the game state changes every logical frame. You can access it by referring to tc.state. This variable is a table with the following keys:

--[[
    state will get its content updated from bwapi, it will have
    * map_data            : [torch.ByteTensor] 2D. 255 (-1) where not walkable
    * map_name            : [string] Name on the current map
    * img_mode            : [string] Image mode selected (can be empty, raw, compress)
    * lag_frames          : [int] number of frames from order to execution
    * frame_from_bwapi    : [int] game frame number as seen from BWAPI
    * game_ended          : [boolean] did the game end? (i.e. did the map end)
    * battle_just_ended   : [boolean] did the battle just end? (battle!=game)
    * waiting_for_restart : [boolean] are we waiting to restart a new battle?
    * battle_won          : [boolean] did we win the battle?
    * units_myself        : [table] w/ {unitIDs: unitStates} as {keys: values}
    * units_enemy         : [table] same as above, but for the enemy player
    * bullets             : [table] table with all bullets (position and type)
    * screen_position     : [table] Position of screen {x, y} in pixels. {0, 0} is top-left
]]

And not only that, there is also units_neutral (contains animals, minerals and gases) and probably a few others, but I would have to look for that in the TorchCraft code. I took this comment above from TorchCraft's code.

As you can see, the basic script already uses game_ended.

An interesting structure is map_data. This is the ByteTensor from Torch and contains information about the map, such as places that units cannot go to and the like. Very useful. For m5v5_c_far.scm the map size is 256x256 and can be obtained by:

local map = tc.state.map_data
print(map:size())

For us, the most interesting ones will be units_myself, units_enemy and bullets. Let's check what units_myself contains:

print(tc.state.units_myself)

{
  21 :
    {
      lifted : false
      pixel_size_x : 17
      detected : true
      gwcd : 0
      idle : false
      awrange : 16
      order : 6
      type : 0
      position :
        {
          1 : 83
          2 : 141
        }
      targetpos :
        {
          1 : 60
          2 : 150
        }
      energy : 0
      size : 1
      resource : 0
      gwdmgtype : 3
      pixel_y : 1128
      shieldArmor : 0
      awattack : 6
      playerId : 0
      visible : 1
      velocity :
        {
          1 : 0
          2 : 0
        }
      hp : 40
      awdmgtype : 3
      orders :
        {
          1 :
            {
              first_frame : 5
              target : -1
              type : 6
              targetpos :
                {
                  1 : 60
                  2 : 150
                }
            }
        }
      max_hp : 40
      target : -1
      armor : 0
      max_shield : 0
      maxcd : 15
      gwattack : 6
      shield : 0
      awcd : 0
      pixel_x : 664
      gwrange : 16
      pixel_size_y : 20
    }

    -- ...
}

A lot of data about the unit.

Commands​

It's time to give orders to units. Let's work through the while loop:

local give_orders = false

while not tc.state.game_ended do
    update = tc:receive()

    local actions = {}

    if give_orders == false then
        for uid, unit in pairs(tc.state.units_myself) do
            table.insert(actions,
                tc.command(
                    tc.command_unit_protected,
                    uid,
                    tc.cmd.Attack_Move,
                    -1,
                    103,
                    141
                )
            )
        end

        give_orders = true
    end

    tc:send({table.concat(actions, ':')})

    if tc.state.battle_just_ended or tc.state.waiting_for_restart then
        give_orders = false
    end
end

At the very beginning of the battle, the above code gives an order to attack the place where enemy units are located. To issue commands in a given frame, we create an actions table:

local actions = {}

Then we iterate over all our units:

for uid, unit in pairs(tc.state.units_myself) do
	-- ...
end

The pairs function returns keys and values ​​separately. They are assigned to the variables uid, unit.

We create commands by calling tc.command with the specified arguments:

tc.command(
    tc.command_unit_protected,
    uid,
    tc.cmd.Attack_Move,
    -1,
    103,
    141
)

The first argument, I have no idea what it does, the second is the unit id (each is unique), the third is the command, the fourth - I have no idea, the fifth and sixth are the x and y positions. We insert the commands into the actions table by using table.insert .

Time to send commands to Starcraft:

tc:send({table.concat(actions, ':')})

The above code merges the array into a string, with each command separated by a colon, and sends.

The above example also includes the give_orders variable. I added it to avoid command spamming. Spam causes Starcraft to run slowly and units, instead of attacking what is on the way, go to a given location regardless of anything.

Summary​

It's not difficult, it just took me some time to understand why giving the movement order doesn't work. It turned out that you need to add this magical -1 before the x and y positions. The operation of the presented code can be seen in the gif below.

In the next post I will try to either describe more things or finally use machine learning (Q-learning). We'll see if I have time.

Non-stationary problem​

In this post, I will discuss a particular type of multi-armed bandit (MAB) problem, which consists in the fact that for each one-armed bandit, the value of the rewards changes over time. This is the so-called non-stationary version of MAB. Until now, the value of rewards was obtained from a certain normal distribution with a certain mean and variance (the mean for each arm was selected randomly at the beginning in the constructor).

It looked something like this:

This is the so-called stationary process.

The non-stationary process looks like this and this is what we will deal with:

Solution​

Hmm, since the rewards we receive change over time, we'll need something like a weighted average.

One thing first, let's recall the formula from the previous entry:

$Q_{n+1} \doteq Q_n + \frac{1}{n}\Big[R_n - Q_n\Big]$

Let's transform it to:

$Q_{n+1} \doteq Q_n + \alpha\Big[R_n - Q_n\Big]$

Where $\alpha$ will be some parameter, which may have some constant value or the value of $\frac{1}{n}$ as before.

We transform:

Summary​

Sprawdźmy, czy to działa:

It works. That's it, it wasn't much. All we had to do was redo one equation again. I will only add that Sutton and Barto advise to use constant values ​​of $\alpha$ because they perform better due to the fact that they do not meet certain conditions for series convergence and this is desirable in non-stationary problems. More details can be found in the book.

The code is here. In the next entry I will deal with another minor optimization of multi-armed bandit.

This week I was going to write about Torch itself and how to create neural networks in it, but I decided that I would focus on the very basics of TorchCraft itself and its interaction with Starcraft. TorchCraft, unfortunately, has poor documentation and apart from the installation description, almost everything has to be figured out based on examples from the examples directory.

Maps​

Let's start with the fact that TorchCraft can be run in two modes: micro and normal. In normal mode, you can play on regular multiplayer maps. There are no surprises here. Micro maps do not contain buildings and two groups of units fight on them. Maps must be specially prepared for this mode. Four maps come with TorchCraft: dragoons_zealots.scm, m5v5_c_far.scm, sp_dragoons_zealots.scm, sp_m5v5_c_far.scm. These are actually two maps in two versions each. I have no idea what the difference is, those with the sp_ prefix maybe have something to do with single player?

Micro maps contain evenly spaced special, invisible units that reveal parts of the map (so-called Map Revealers), as well as Start Locations, i.e. points where units appear.

For these units to appear, so-called triggers are needed. They are created in the map editor in a special panel. Triggers allow you to do many different things, e.g. placing units on the map, controlling units, writing text on the screen, and in general - they allow you to control the game. Triggers contain two things: conditions - the conditions for which the trigger is fired, and actions - the actions triggered within a given trigger. You can also set separately for which player the trigger is activated.

Let's look at our micro maps:

We see that triggers at the very beginning of each game create several units for both players, and if necessary, units are sent to attack.

Ok, enough about the maps. Time to tackle the code.

The code​

First, let's write a minimal working example:

local hostname = "192.168.56.1"
local port = 11111

local tc = require 'torchcraft'

tc.micro_battles = true

tc:init(hostname, port)
local update = tc:connect(port)

while not tc.state.game_ended do
    update = tc:receive()

    -- code here
end

tc:close()

Let's analyze this example:

The first two lines are the host and port needed to connect to the host (Windows with Starcraft):

local hostname = "192.168.56.1"
local port = 11111

Then we load the TorchCraft library:

local tc = require 'torchcraft'

We switch TorchCraft to micro mode:

tc.micro_battles = true

We start the connection:

tc:init(hostname, port)
local update = tc:connect(port)

Then there is a loop in which we can give orders to units. In this example, we have not defined the loop termination conditions, so infinitely many battles will be played:

while not tc.state.game_ended do
    update = tc:receive()

    -- code here
end

At the very end we close the connection:

tc:close()

As you can see, it's very simple. But let's have some more fun. Let's add some configuration above the while loop:

local setup = {
    tc.command(tc.set_speed, 20),
    tc.command(tc.set_gui, 1),
    tc.command(tc.set_cmd_optim, 1),
}

tc:send({table.concat(setup, ':')})

These are three commands. set_speed is responsible for the speed of the game, if we set it to zero, the game will be very fast, because the time between the execution of logical frames will be practically zero. set_gui set to 1 means that you can see the action. Whereas set_cmd_optim is the command optimization used by BWAPI, if set BWAPI tries to reduce the number of actions by grouping and executing similar commands. You can set values ​​from 0 to 4 (documentation).

Let's see what is sent to TorchCraft:

print({table.concat(setup, ':')})

We will get:

{
  1 : "6,20:8,1:10,1"
}

We can see that commands are separated by colons and numeric command IDs are separated from values ​​by commas.

Summary​

Okay, that's it for now. Unfortunately, TorchCraft documentation is virtually non-existent, so I haven't been able to do much. For example, I haven't yet been able to figure out how to order a unit to move to another place without attacking anything along the way. Knowledge will have to be gained by reading code and constantly testing what works and what doesn't. It won't be easy :)

In last post I discussed the basic version of multi-armed bandit with $\epsilon$-greedy strategy. The presented algorithm has a small drawback, as it requires recording each reward and calculating the arithmetic mean of the rewards for a given action each time the best action is selected. Not only does the algorithm require memory for rewards, as many times as there are time steps, but each time it is necessary to choose the best action, a lot of unnecessary and quite time-consuming calculations take place. Let's imagine that we have to calculate the arithmetic mean of one million prizes. How long will it take? This can be solved better.

Optimization​

Let's recall what the current code looks like:

def __init__(self, arms, pulls, epsilon):
    # ...
    self.rewards = [[] for _ in xrange(self.arms)]

def get_means(self):
    means = np.zeros(self.arms)

    for index, action_rewards in zip(range(len(means)), self.rewards):
        if len(action_rewards) > 0:
            means[index] = sum(action_rewards) / len(action_rewards)

    return means

def save_reward(self, action, reward):
    self.rewards[action].append(reward)

You can see that it counts every time. You can separate the 'means' variable here as a class field, and write new action values ​​to it when a new reward arrives. However, then we are left with the matter of calculating the arithmetic mean itself, which is computationally and memory expensive if we were to calculate it for the new prize and all the old ones.

Ok, time for some math. Let's define the value of a stock as $Q_n$ after it has been selected $n - 1$ times:

And let's transform it into a better version:

As you can see, we no longer need to remember all the rewards. Just remember the last value of $Q_n$ and $n$ for each action.

The code​

In the constructor, we get rid of self.rewards = [[] for _ in xrange(self.arms)] in favor of self.rewards = np.zeros(self.arms) and add action_count (our $n$ to remember):

def __init__(self, arms, pulls, epsilon):
    self.action_count = np.zeros(self.arms)
    self.rewards = np.zeros(self.arms)

The save_reward function looks like this:

def save_reward(self, action, reward):
    # there is another reward, so we increase n by one
    self.action_count[action] += 1
    # calculate Q(A) = Q(A) + 1 / N(A)[R - Q(A)]
    self.rewards[action] = self.rewards[action] + \
        1. / self.action_count[action] * \
        (reward - self.rewards[action])

Summary​

It was very simple. The entire script is in repository along with the first one, which is not very optimal. After this code modification, it is worth comparing the execution times of both scripts 01_simple.py and 02_incremental.py from the repository. To do this you need to use the time command. Let's check, both scripts contain quite a time-consuming experiment with a large number of calculations.

$ time python 01_simple.py

real    2m10.297s
user    2m3.564s
sys     0m0.436s

$ time python 02_incremental.py

real    0m57.918s
user    0m57.268s
sys     0m0.304s

As you can see, the difference is significant. That's it for now, the next post will be about multi-armed bandit in the non-stationary version.

This whole week I was wondering what path to take for my bot project. I wanted to use Deeplearning4j, but this library conflicts with BWMirror, which requires Java 32-Bit. An alternative is to rewrite the entire project from BWMirror to JNIBWAPI. The second alternative is to rewrite the project to C++, which I am not very interested in because I don't feel good in this language. I mean, not in Java either, but writing in Java is easier to learn. However, I finally decided to take up TorchCraft. Of course, I lose the opportunity to participate in SSCAIT, but I think it will be better to finally do something related to reinforcement learning using a ready-made environment. If I had to do the same in Java, it would take me a lot of time.

Installation on Linux​

It took me a while to install Torch, Lua and TorchCraft. As it happens, when using the instructions on the project website something didn't work. After quite a long time of thinking about what to do to set everything up, I came to the conclusion that it is best to install Torch using the instructions from this website.

Going back to TorchCraft installation description instead of:

git clone git@github.com:torchcraft/torchcraft.git --recursive

You need to clone:

git clone https://github.com/TorchCraft/TorchCraft.git --recursive

Installation on Windows​

Here it was quite simple, because everything was limited to copying files. I used BVEnv.exe from the manual.

Tying everything together​

I installed everything. Time to launch it. On the Linux virtual machine, you need to run the following in the console:

th simple_exe.lua -t $server_ip

Instead of $server_ip you need to substitute the host IP. To do this, you need to use the ipconfig command in Windows and substitute one of the addresses that appear. There's probably nothing else you need to do in Linux.

It's worse on Windows. I ran BVEnv.exe and there's one thing I can't figure out, which is that after running this file, something like this is displayed:

Welcome to the Brood War TorchCraft Environment.
Compiled on Mar 18 2017, 07:48:54.
<Config Info>
  loaded: 1
  current path: C:/StarCraft/bwapi-data/torchcraft.ini
general
  port = 0
  log_path = C:/tc_data/torchcraft_log_cpp_port_
  display_log = 0
  img_mode = raw
  window_mode = windows
  window_mode_custom =
  img_save_path = C:/tc_data/output_
starcraft
  assume_on = 0
  launcher = injectory
  custom_launcher =

C:/StarCraft//TorchCraft/\BWEnv\bin\injectory.x86.exe --launch C:/StarCraft/\StarCraft.exe --inject C:/StarCraft/\bwapi-data\BWAPI.dll
While running command:
        00996F98
CreateProcess failed: 2
Connecting...

The path to Starcraft is incorrect, it is in a completely different place. I couldn't find out whether it is possible to pass any arguments to BWEnv.exe. The code shows that there is no such thing, so I will have to change and recompile this project.

Summary​

I didn't get much done this week. That's the life, I always have to be stopped by some stupid technical problems. I'll try to put it together in the near future. To make things more funny, something went wrong with the bot's Java project, and I didn't change anything in the code. When I try to compile the bot, I get the following error: Exception in thread "main" java.lang.UnsatisfiedLinkError: \{path\}\bwapi_bridge2_5.dll: Can't find dependent libraries, even though all DLLs are where they were and it worked before. How it's even possible?

As a consolation, I must say that my bot, despite its simplicity, is doing well on the SSCAIT ladder. I achieved a 60% winratio (39 wins, 42 losses), which in my opinion is a good result for such a simple bot.

Multi-armed bandit problem (or k-armed bandit problem) is one of the reinforcement learning problems, I don't know if it's the simplest one, but it allows for a relatively quick introduction to the subject and to become familiar with the basic concepts.

Let's imagine that we are in a casino and we are playing on a dozen or so one-armed bandits. Slot machines are different. For pulling the lever in some of them we can get a bigger reward than in others. The reward we can receive is from a certain probability distribution, sometimes larger, sometimes smaller. For now, we assume that the distribution does not change over time. Our goal is to find a slot machine with the highest accumulated prize value. We have to spend some time looking for a good machine, but we want to use the most optimal one as quickly as possible.

The above example with slot machines is, of course, quite artificial. Multi-armed bandit in practice can be used for things such as:

• clinical trials - to find the most effective experimental therapy and minimize the negative effects of these weaker therapies on patients.
• investment portfolio management - to find the best strategy for our investment portfolio.
• instead of A/B tests - to reduce conversion losses, in A/B tests traffic is divided equally, regardless of how each version of the website performs. In MAB the movement is gradually changing towards the best version.

Epsilon-greedy strategy​

Ok, let's start with the simplest strategy for solving MAB, which is the $\epsilon$-greedy strategy. What we are looking for is the so-called action value. Let us denote the action selected in step $t$ as $A_t$ and the corresponding reward as $R_t$. The value of the selected action, denoted as $q_\*(a)$, is the expected reward (average of rewards) when action $a$ is selected:

If we knew the value of each share, we could easily solve MAB, just choose always the share with the highest value. However, we do not know these values, we need to estimate them. We denote the estimated value of action $a$ at time $t$ as $Q_t(a) \approx q_*(a)$. At each time step $t$ there is at least one action that has the highest value. If we choose another, worse action, we explore. In the short term, we lose by performing this worse action, but we hope that in the long term we will accumulate a greater cumulative value of rewards if it turns out that one of these worse actions is better than the current best one. By accumulating knowledge about the value of rewards for each action, we become more and more convinced which action is the best by exploring and using the knowledge we have already acquired. An interesting issue is when and how often to explore and use the acquired knowledge. The simplest method uses a predetermined value of $\epsilon$. At each step, with probability $1 - \epsilon$, we choose the best action (knowledge exploitation phase, exploitation). We select the best action by simply calculating the simple arithmetic average of the accumulated prizes for each action separately and selecting the one with the highest value. However, with probability $\epsilon$, exploration takes place, during which we randomly select any action. Regardless of whether we explore or not, we save the received reward assigned to the selected action.

Time for the code​

I don't know if I described everything clearly above, but I think the code should clarify the matter more. The following example contains a Bandit class that executes $\epsilon$-greedy strategy for the three parameters given in the constructor. We can set the algorithm for how many actions, for how many time steps and for which epsilon. The constructor also initializes the true_reward variable with random values, and then, when the algorithm is executed, the rewards are returned with the true_reward value plus some noise (to make things easier). The whole essence of $\epsilon$-greedy strategy is contained in the choose_action method. There, exploration/exploitation of knowledge takes place with the help of epsilon.

The main code (under __main__) contains an example of a single run of the algorithm and, perhaps more interestingly, a comparison of three different values ​​of $\epsilon$ (0, 0.1, 0.01). I performed 2000 runs for each of them and averaged the results, so you can see in the chart below what the average rewards look like for a given epsilon depending on the number of iterations.

You can see that for a value of 0.1, the optimal action is found quickly, but will never be selected more than 91% of the time. For a value of 0.01, the optimal action is found more slowly, but in the long run it will achieve better average results.

Okay, here's the code:

'''
Multi-armed bandit with e-greedy strategy
With saving all rewards for each arm
'''

import numpy as np
import matplotlib.pyplot as plt
from random import randint
import random


class Bandit:
    def __init__(self, arms, pulls, epsilon):
        self.arms = arms
        self.pulls = pulls
        self.epsilon = epsilon
        self.history = []

        # random values from normal distribution
        self.true_reward = [np.random.randn() for _ in range(self.arms)]
        self.rewards = [[] for _ in xrange(self.arms)]

    def get_means(self):
        means = np.zeros(self.arms)

        for index, action_rewards in zip(range(len(means)), self.rewards):
            if len(action_rewards) > 0:
                means[index] = sum(action_rewards) / len(action_rewards)

        return means

    def choose_action(self):
        rand = np.random.uniform(0, 1)

        # select action with 1 - epsilon probability
        if rand > self.epsilon:
            # exploit
            means = self.get_means()  # compute all means
            argmax = np.argmax(means) # select arm with best estimated reward
            return argmax
        else:
            # explore
            return randint(0, len(self.rewards) - 1)

    def get_reward(self, action):
        return self.true_reward[action] + np.random.randn() # true reward with noise

    def save_reward(self, action, reward):
        self.rewards[action].append(reward)

    def run(self):
        for t in range(self.pulls):
            action = self.choose_action()
            reward = self.get_reward(action)
            self.save_reward(action, reward)

            self.history.append(reward)


if __name__ == '__main__':
    # example bandit
    bandit = Bandit(arms=10, pulls=2000, epsilon=0.01)
    bandit.run()

    for arm, reward, true_reward in zip(range(1, len(bandit.rewards) + 1),
                                        bandit.rewards, bandit.true_reward):
        pulls = len(reward)
        print "Arm {} pulls: {}, true reward: {}". \
            format(arm, pulls, true_reward)

    print "Best arm: {}".format(np.argmax(bandit.true_reward) + 1)

    # experiments
    pulls = 1000
    experiments = 2000

    epsilons = [0.01, 0.1, 0]

    mean_outcomes = [np.zeros(pulls) for _ in epsilons]

    for _ in range(experiments):
        for index, epsilon in zip(range(len(epsilons)), epsilons):
            bandit = Bandit(arms=10, pulls=pulls, epsilon=epsilon)
            bandit.run()
            mean_outcomes[index] += bandit.history

    for index, epsilon in zip(range(len(epsilons)), epsilons):
        mean_outcomes[index] /= experiments
        plt.plot(mean_outcomes[index], label="epsilon: " + str(epsilon))

    plt.ylabel("Average reward")
    plt.xlabel("Steps")
    plt.legend()
    plt.savefig('01_plot.png')

Sample output:

Arm 1	pulls: 3,		true reward: -0.903469191365
Arm 2	pulls: 3,		true reward: 0.365839293594
Arm 3	pulls: 3,		true reward: -0.854871239295
Arm 4	pulls: 2,		true reward: -0.445679248867
Arm 5	pulls: 94,		true reward: 1.0921733926
Arm 6	pulls: 3,		true reward: -0.123881634804
Arm 7	pulls: 0,		true reward: -0.928756860211
Arm 8	pulls: 5,		true reward: 0.860238065648
Arm 9	pulls: 1885,	true reward: 1.81443343678
Arm 10	pulls: 2,		true reward: 0.247351866388

It's not always rosy:

Arm 1 pulls: 1244, true reward: 0.865701931312
Arm 2 pulls: 1, true reward: -0.0986266557818
Arm 3 pulls: 2, true reward: -0.93574271516
Arm 4 pulls: 4, true reward: 0.273764997199
Arm 5 pulls: 2, true reward: -2.24599693076
Arm 6 pulls: 3, true reward: -0.0837555511977
Arm 7 pulls: 208, true reward: 0.679058209084
Arm 8 pulls: 2, true reward: -0.983193383821
Arm 9 pulls: 2, true reward: -0.0941783631123
Arm 10 pulls: 532, true reward: 1.78310799226

Summary​

I presented the simplest strategy for solving the multi-armed bandit problem known as $\epsilon$-greedy strategy. Of course, there is much more. You can imagine the value of $\epsilon$ decreasing over time, an algorithm with an exploration phase first and then using the acquired knowledge, and many, many other strategies. But more on that in subsequent posts.

So far, I have been testing my bot on my computer against the default bot created by Blizzard. It's time to test yourself against other bot creators. I know that instead of the standard bot, you can connect another one using Chaoslauncher - MultiInstance and editing bwapi.ini, but I preferred to see first what the registration procedure in SSCAIT looks like.

To register a bot in SSCAIT, in addition to providing various data, you need to generate a .jar file and insert it into the ZIP together with BWAPI.dll, the DLL responsible for connecting to Starcraft. Each bot can use a different version of BWAPI, so you need to include your DLL. Without thinking much, I generated artifact in IntelliJ IDEA, but after registration it turned out that my bot crashed six times and was blocked. I did something wrong and I didn't really know what. However, I noticed that in the results list there are links to download bots. I downloaded one written in Java, changed the extension from .jar to .zip and checked the file structure. It should look like this:

In particular, the META-INF directory containing the MANIFEST.MF file with a defined entrypoint is necessary, so that Java knows which class is the main one with the main method.

Manifest-Version: 1.0
Main-Class: dloranc.fivepoolbot.FivePoolBot

It took me a while, I'm sharing it for the future, because the tutorial on the SSCAIT website didn't contain much information about creating a working package.

As for my results in SSCAIT, after a few days I have 15 games won and 20 lost (I subtracted crashes). In my opinion, not bad for such a simple bot, especially since I was ahead of 15 bots in the scorelist.

Serious bug in the code from the tutorial​

While watching the replays, I noticed that my bot was behaving incorrectly. Sometimes, for unexplained reasons, he was unable to detect enemy buildings, which resulted in the attack going to the wrong location, which meant losing the game. After quite a lot of thinking, I managed to find out that the problem was not remembering the location of the enemy's buildings correctly. I even managed to visualize it. You can see this in the gif below:

Very interesting, because I copied the code for this solution from the tutorial on the SSCAIT website and assumed that everything worked. But no - some buildings did not have a red dot or it disappeared every frame, which resulted in the dot flashing on the screen.

A riddle for you: where is the error in the code below?

//always loop over all currently visible enemy units (even though this set is usually empty)
for (Unit u : game.enemy().getUnits()) {
	//if this unit is in fact a building
	if (u.getType().isBuilding()) {
		//check if we have it's position in memory and add it if we don't
		if (!enemyBuildingMemory.contains(u.getPosition())) enemyBuildingMemory.add(u.getPosition());
	}
}

//loop over all the positions that we remember
for (Position p : enemyBuildingMemory) {
	// compute the TilePosition corresponding to our remembered Position p
	TilePosition tileCorrespondingToP = new TilePosition(p.getX()/32 , p.getY()/32);

	//if that tile is currently visible to us...
	if (game.isVisible(tileCorrespondingToP)) {

		//loop over all the visible enemy buildings and find out if at least
		//one of them is still at that remembered position
		boolean buildingStillThere = false;
		for (Unit u : game.enemy().getUnits()) {
			if ((u.getType().isBuilding()) && (u.getPosition() == p)) {
				buildingStillThere = true;
				break;
			}
		}

		//if there is no more any building, remove that position from our memory
		if (buildingStillThere == false) {
			enemyBuildingMemory.remove(p);
			break;
		}
	}
}

It took me a lot of time, and the error was trivial and consisted in the following line:

if ((u.getType().isBuilding()) && (u.getPosition() == p)) {

I replaced it with (by the way I got rid of the excess brackets):

if (u.getType().isBuilding() && u.getPosition().equals(p)) {

Well, in Java, if we want to compare two objects, we use equals() because the == operator compares references to objects.

After the fix, nothing flashes:

I spent this week polishing up my Starcraft bot. I didn't want to take up reinforcement learning yet because I preferred to spend my time learning about BWAPI. The code can be viewed here in my repository dloranc/five-pool-bot.

Fixes​

In a previous post I wrote a to-do list. Some of them were successfully completed (these are crossed out below):

• Mineral gathering can be optimized according to this topic on TeamLiquid.
• The scout should be a created drone, not one drawn from the initial ones.
• If a drone intended for building intends to build, it should collect minerals if it has collected any before building. Now it's like that if she had any, they are lost.
• Scout may be more optimal, the drone should go to the bases that are closest to them.
• Sometimes when a drone is going to the last base on a large map and zerglings are produced, they move to random places on the map because they don't know where the opponent's base is. This can be improved by sending them to the base where the drone is heading, because then you know that this is the right base.
• Zerglings can fight better, you can apply priorities on what to attack first. It would also be useful to withdraw severely wounded units to regenerate.
• If the base and buildings around it are destroyed and the game is not over, it means that there is a building somewhere on the map that needs to be destroyed. Zerglings don't even search randomly, they just gather in one place.
• In general, it would be useful to write some class that allows you to give orders to units and cancel them when certain circumstances arise.

I especially had an interesting experience with bringing minerals back by drone before building the Spawning Pool. The sample logic should look something like this:

if (buildDrone.isCarryingMinerals()) {
	buildDrone.returnCargo();
} else {
	// ...

	buildDrone.build(UnitType.Zerg_Spawning_Pool, buildPosition);
}

But I found that sometimes it doesn't work. Looking at the BWAPI documentation of the returnCargo function, I noticed the following comment:

There is a small chance for a command to fail after it has been passed to Broodwar.

What's interesting is that at this point in the documentation this note is under almost every function. I wonder what causes the inability to perform a given action? And are you sure there wasn't some problem with my code?

I solved this problem differently, but not entirely optimally. It is possible that all drones will have minerals and the building algorithm must wait until one of them brings the minerals to the base. I don't know if such a small waste of time can sometimes be crucial to winning the game. I don't think so, but I like to have everything 100%.

Unit buildDrone = null;

for (Unit myUnit : self.getUnits()) {
    if (myUnit.getType() == UnitType.Zerg_Drone) {
        if (!myUnit.isCarryingMinerals() && myUnit.getID() != scoutDrone.getID()) {
            buildDrone = myUnit;
            break;
        }
    }
}

Drawing auxiliary figures​

BWAPI provides a dozen or so cool methods for drawing points, lines, squares, circles, ellipses, triangles, as well as for writing text on the screen.

Each method starts with the word draw and is called on an object of type Game. After the word draw we give what we want to draw (Text, Box, Triangle, Dot, Circle, Ellipse, Line), after which we can also add a word indicating how * BWAPI* is supposed to interpret the transmitted coordinates. If we want to draw a square on the map, we call the drawBoxMap method, if we want to draw on the screen, we call drawBoxScreen. The position (0, 0) will then be the upper left corner. We can also use drawBoxMouse, then the position (0, 0) will mean drawing a square where the cursor is.

drawBox(CoordinateType::Enum ctype, int left, int top, int right, int bottom, Color color, bool isSolid=false)
drawBoxMap(Position leftTop, Position rightBottom, Color color, bool isSolid=false)
drawBoxMouse(Position leftTop, Position rightBottom, Color color, bool isSolid=false)
drawBoxScreen(Position leftTop, Position rightBottom, Color color, bool isSolid=false)
// ...
drawLineMap(Position a, Position b, Color color)
drawLineMouse(Position a, Position b, Color color)
drawLineScreen(Position a, Position b, Color color)

I used this to graphically visualize the orders (in red in the screenshot below) that the units carry out. I have marked chokepoints on the map, i.e. narrow places that are important for strategic reasons (yellow). I also drew the zones where land units cannot move (in gray). The only question is why I had to multiply all the positions of the polygon points by 8? I also marked the place where the enemy's base is (in the screenshot in the header) with a circle. This was useful for me when debugging scouting.

Other useful things​

On starcraftai.com I discovered two ways to speed up the game. The first way is to completely turn off the sound engine, because normally even if we set the volume to zero, the functions that play sound and music will be called. Turning off the engine prevents these functions from being activated. To tell you the truth, I didn't notice the game speeding up much, but I turned it off just to be on the safe side.

[starcraft]
; Game sound engine = ON | OFF
sound = OFF

The second cool thing is the ability to turn off graphics rendering when the match starts. Well, almost, because from what I saw, sometimes it will render a frame.

game.setGUI(false);

I discovered that it is also possible to run Starcraft in headless mode (without rendering any graphics) on the command line. This can be done using the tscmoo/bwheadless project. I managed to compile it, but haven't tested it yet. It will definitely be useful. It's a pity that this project doesn't have at least README.md.

What's next?​

Now I will finally start reinforcement learning. My guess is that you can win most games, even with bots that have good micro, only with an early Zergling attack. With a perfect micro, especially Protoss may have big problems, because its basic unit, the Zealot, is very slow and is unable to keep up with the Zergligs. All I need is for my bot to focus on the buildings, avoiding the opponent as much as possible, and the game will be won. This is of course my hypothesis, we'll see how it comes out in the wash.

The question remains what to use. I tried TorchCraft, but it didn't solve my problems. TorchCraft uses the Lua language, and I would like to check my bot in SSCAIT, and there you can only report bots in Java and C++. Maybe combine what I've already written with deeplearning4j? I think I'll do that, I want to check how much my bot will be worth.

For some time now I have been trying to slowly read the book "Reinforcement Learning: An Introduction" by Richard S. Sutton and Andrew G. Barto. Someone asked for some good RL materials in one of my posts, so I'm sharing and recommending this book. This is supposedly a classic book in this field. In my opinion, it deserves this name, if one can say so after reading less than two chapters of this book.

To consolidate my knowledge, I decided to write code for the algorithms found in this book. There are sample codes on the website linked above, but I don't intend to use them (maybe if I get completely stuck on something). Repository (empty for now): dloranc/reinforcement-learning-an-introduction.

Reinforcement Learning - what is it?​

Reinforcement Learning is a certain way of solving some problems that cannot be solved in a simple way (analytically) or we do not have a good model. Something, we'll call it an agent, performs certain actions in some unknown environment that is in a certain state. For taking action, the agent receives a reinforcement signal. It can be positive (reward) or negative (punishment). By interacting with the environment, an agent learns a certain politics. The goal of reinforcement learning is to determine the optimal policy for which we will receive as many rewards as possible. Finding such a policy is not easy. Sometimes, for example, it is worth sacrificing rewards to gain in the long run.

It is a way of teaching inspired by the achievements of behavioral psychology. Animals learn about the world this way (humans are more complicated). For example, imagine a small dog learning to walk. Trying to learn this task, he performs various actions by trial and error. Thanks to his senses, he receives certain stimuli from the world and takes certain actions depending on them. If they don't fail, they will perform good actions that were successful more often and better (positive reinforcement). If it fails the floor, it will try not to take actions that would result in this result next time. And so on until he learns to walk.

In the table below I have provided examples of various agents, environments, etc. etc.:

But it doesn't matter what the tables are, let's see it in real projects. I've put together a playlist of some cool examples below:

I find this pancake-tossing robot particularly cute:

How does reinforcement learning differ from other methods?​

For example, what is the difference between supervised learning and reinforcement learning? In supervised learning, we receive some data and it is already pre-classified. Based on them, the algorithm must learn how to distinguish them (classify) or how to predict a certain value based on them (regression). We don't have anything for starters in reinforcement learning. We need to find a solution based on performing actions in the environment and on rewards/punishments. In addition, there is the problem of exploring and using knowledge acquired through interaction with the environment. This does not occur in other machine learning paradigms. In unsupervised learning it is similar, we receive some data and we have to classify it somehow. However, we are not guided by rewards/punishments but by their structure.

import sys

if __name__ == '__main__':
	print "\n".join(sys.argv)

The above example displays the arguments one below the other. The conditional expression if __name__ == '__main__': checks whether the script was invoked in the console. If the script was imported in another file, then the code will not execute.

docopt - we put docstrings to work​

sys.argv doesn't allow much though. You have to take care of the correct handling of the arguments yourself, and if we want to have help (when calling the script with --help) for our script, we must remember to ensure its compatibility with the rest of the code. For simple things, however, sys.argv is enough. However, if we want to build a tool that will be intensively used in the place that programmers like best (i.e. in the console), we have to use something else. A quite popular solution is argparse. However, I used the docopt tool, which seems quite interesting to me because the arguments are built using a docstring. Cool thing.

We start by installing the tool:

pip install docopt

Let's write something simple, let's say it's a script that adds numbers. For now, however, we are not adding up, we will see what we are dealing with.

"""Sum integer values.

Usage:
  sum.py <numbers>...
  sum.py (-h | --help)
  sum.py --version

Options:
  -h --help     Show this screen.
  --version     Show version.

"""
from docopt import docopt


if __name__ == '__main__':
    arguments = docopt(__doc__, version='Sum 1.0')

    numbers = arguments['<numbers>']

    try:
        numbers = list(map(int, numbers))
        print(sum(numbers))
    except ValueError:
        print('Cannot cast value(s) to integer.')

From the content of the docstring, it is easy to guess what to call and what arguments we are dealing with. Brackets indicate arguments, dashes (two or one) indicate options, and three dots indicate that arguments may be repeated. Brackets combined with a vertical line indicate mutually exclusive options.

Time for testing, let's call --version and --help:

Sprawdźmy co się stanie jak wywołamy z argumentami 1 2 3 4:

We see that the content of the arguments variable is a dictionary containing three fields --help, --version and <numbers>.

Okay, let's modify the code a bit, this time let's add up these numbers.

if __name__ == '__main__':
    arguments = docopt(__doc__, version='Sum 1.0')

    numbers = arguments['<numbers>']

    # we need to cast it to an int because we are dealing with a list in the form ['1', '2', '3', '4']
    numbers = list(map(int, numbers))

    print(sum(numbers))

It will display:

As you can see, the docopt library is very easy to use. You don't have to mess around with documentation like in the case of argparse and similar libraries.

More details about docopt can be found at http://docopt.org/.

Since the last post about the project, I've been working on the bot for quite some time, trying to write something that performs some simple strategy. After installing BWAPI, the bot starts and plays with the standard bot included in the game. As I had known the default bot and its weaknesses for years, I decided that my bot would perform a simple strategy called 5 pool with some modifications.

It looks like this:

1. If there are 50 minerals, then create a drone.
2. Take the created drone and send it to a potential enemy base. The maps on which bots play SSCAIT have from two to a maximum of four possible base locations, so a drone can be sent to one, two or three locations depending on the map.
3. If there are 200 minerals and five drones, create a spawning pool.
4. If the drone encounters an enemy base, let it attack the nearest building and then run back to its base to collect minerals.
5. If the spawning pool is already built, create zerglings (and if necessary, overlords if you can't create zerglings because the unit limit has been reached) and let them attack the enemy's base.

As you can see, the strategy is very simple. In point four, I use the exceptionally stupid behavior of a standard StarCraft bot. If his base is attacked, he takes all the units and tries to destroy the attacked one. If we start running away with this unit, the bot will chase it to our base. This behavior has a huge impact on the bot's economy, because it does not mine for quite a long time, which means lose of the game. This play is not necessary, because without it you can easily beat the bot with a 5-pool, but I like to bully the artificial intelligence :)

The code​

I placed the code of this bot in a separate repository. I don't want to litter the competition because I won't write in Java.

This time I will describe the most important fragments of the code:

@Override
public void onStart() {
	// ...

    isScouting = false;
    isScoutingIdle = false;
    isSpawningPool = false;
    scoutDrone = null;
    buildDrone = null;
    hatchery = null;
    playerStartLocation = null;
    possibleEnemyBaseLocations = null;
    baseToScout = null;
    enemyBase = null;
    enemyBuildings = new EnemyBuildings();

    // ...
}

The above fragment prevents various bugs. If we do not set the class fields at the start of each game, the bot may behave incorrectly in the next game, because the bot instance is not created from scratch, but is used all the time, and the onStart function is called before the game starts. For example, I had problems with the fact that the drone selected for building could not build a spawning pool because it simply did not exist. buildDrone contained a reference to an object from the previous game.

The most important thing, however, is the onFrame function called every frame:

@Override
public void onFrame() {
    float supplyUsed = self.supplyUsed() / 2;
    float supplyTotal = self.supplyTotal() / 2;
    int dronesCount = getDronesCount();

    enemyBuildings.update(game);

    // ...

    for (Unit myUnit : self.getUnits()) {
        if (myUnit.getType() == UnitType.Zerg_Hatchery) {
            if (supplyTotal - supplyUsed <= 1) {
                if (self.minerals() >= 100) {
                    myUnit.train(UnitType.Zerg_Overlord);
                }
            } else {
                if (supplyUsed < 5) {
                    if (self.minerals() >= 50) {
                        myUnit.train(UnitType.Zerg_Drone);
                    }
                } else {
                    if (self.minerals() >= 50) {
                        myUnit.train(UnitType.Zerg_Zergling);
                    }
                }
            }
        }

        if ((myUnit.getType().isWorker() && myUnit.isIdle())) {
            gatherMinerals(myUnit);
        }

        if (myUnit.getType() == UnitType.Zerg_Zergling && myUnit.isIdle()) {
            attack(myUnit);
        }
    }

    scouting(dronesCount);

    if (dronesCount >= 5 && !isSpawningPool && self.minerals() >= 200) {
        buildSpawningPool();
    }
}

First, three auxiliary variables are initialized: supplyUsed, supplyTotal and dronesCount. The first two are actually unnecessary. I created them because BWAPI returns all supply values ​​occupied by units multiplied by two due to the fact that one zergling takes 0.5 supply. I'm used to game values, so it was easier for me to work with these variables. Whereas dronesCount is the number of drones. It is used to determine when to build a spawning pool.

The entire onFrame function includes checking and updating the set (HashSet) of enemy buildings, the order of building and producing units, and orders for units. It's all terribly sloppy, but I'll just write that it looked worse, so be glad you don't have to read the original mess I created :)

What else should I describe here so that the post doesn't get too long? Maybe what the attack looks like:

private void attack(Unit myUnit) {
    HashSet<Position> enemyBuildingPositions = enemyBuildings.getBuildings();

    if (!enemyBuildingPositions.isEmpty()) {
        Position enemyBuildingPosition = enemyBuildingPositions.iterator().next();
        myUnit.attack(enemyBuildingPosition);
    } else {
        if (enemyBase != null) {
            myUnit.attack(enemyBase.getPosition());
        } else {
            ThreadLocalRandom random = ThreadLocalRandom.current();

            Position randomPosition = new Position(
                    random.nextInt(game.mapWidth() * 32),
                    random.nextInt(game.mapHeight() * 32)
            );

            if (myUnit.canAttack(randomPosition)) {
                myUnit.attack(randomPosition);
            }
        }
    }
}