6CCS3ML1 (Machine Learning)
Coursework 2
(Version 1.4)
1 Overview
For this coursework, you will have to implement Q-learning algorithm. Your code will again be
controlling Pacman, in the classic game, and the Q-learning algorithm will be helping Pacman
choose how to move. Your Q-learning algorithm should be able, once it has done its learning, to be
able to play a pretty good game, and its ability to play a good game will be part of what we assess.
No previous experience with Pacman (either in general or with the specific UC Berkeley AI implementation
that we use) is required, though you should have used it for Coursework 1 and Practical 8.
This coursework is worth 10% of the marks for the module.
Note: Failure to follow submission instructions will result in a deduction of 10% of the marks you
earn for this coursework.
2 Getting started
2.1 Start with Pacman
The Pacman code that we will be using for the 6CCS3ML1 coursework was developed at UC Berkeley
for their AI course. The person who developed this code then kindly made it available to everyone.
The homepage for the Berkeley AI Pacman projects is here:
http://ai.berkeley.edu/
Note that we will not be doing any of their projects. Note also that the code only supports Python
3, so that is what we will use1
.
You should:
1. Download:
pacman-cw2.zip
from KEATS.
2. Save that file to your account at KCL (or to your own computer).
3. Unzip the archive.
This will create a folder pacman-cw2
1
If you anything other than Python 3, you are on your own in terms of support, and if the code you submit
does not work (which is likely) you will lose marks.
1 cocarascu-yannakoudakis-6ccs3ml1-cw2
Figure 1: The smallGrid version of Pacman
4. From the command line (you will need to use the command line in order to use the various
options), switch to the folder pacman-cw2.
5. Now type:
python pacman.py -p RandomAgent -n 10 -l smallGrid
and watch it run.
This command illustrates a few aspects of the Pacman code that you will need to understand:
• -n 10 runs the game 10 times.
• -l smallGrid runs the very reduced game you see in Figure 1.
This is not a very interesting game for a human to play, but it is moderately challenging for a
reinforcement learning program to learn to play.
• -p RandomAgent tells the pacman.py code to let Pacman be controlled by an object that is
an instance of a class called RandomAgent.
The program then searches through all files with a name that ends in:
Agents.py
looking for this class. If the class isn’t in an appropriately named file, you will get the error:
Traceback (most recent call last):
File "pacman.py", line 679, in
args = readCommand( sys.argv[1:] ) # Get game components based on input
File "pacman.py", line 541, in readCommand
pacmanType = loadAgent(options.pacman, noKeyboard)
File "pacman.py", line 608, in loadAgent
raise Exception(‘The agent ’ + pacman + ‘ is not specified in any *Agents.py.’)
In this case the class is found in sampleAgents.py
RandomAgent, as its name suggests, just picks actions at random. When there is no ghost, it will
win a game eventually by eating all the food2
, but when a ghost is present, RandomAgent dies pretty
quickly.
2
If you want to see this happen, run python pacman.py -p RandomAgent -k 0 -l smallGrid. The flag -k
sets the number of ghosts.
2 cocarascu-yannakoudakis-6ccs3ml1-cw2
2.2 Towards an RL Pacman
Now, your job is to write code to learn how to play this small version of Pacman. To get you started,
we have provided a skeleton of code that will do this, found in:
mlLearningAgents.py
This file contains a class QLearnAgent, and that class includes several methods. (You should look
at the code in order to fully understand this description.)
init() This is the constructor for QLearnAgent. It is called by the game when the game starts up
(because the game starts up the learner).
The version of init() in QLearnAgent allows you to pass parameters from the command
line. Some of these you know from the lectures:
• alpha, the learning rate
• gamma, the discount rate
• epsilon, the exploration rate
and you will use them in the implementation of your reinforcement learning algorithm. The
other:
• numTraining
allows you to run some games as training episodes and some as real games.
All the constructor does is take these values from the command line, if you pass them, and
write them into sensibly named variables. If you don’t pass values from the command line,
they take the default values you see in the code. These values work perfectly well, but if you
want to play with different values, then you can do that like this:
python pacman.py -p QLearnAgent -l smallGrid -a numTraining=2 -a alpha=0.2
Note that you need to have no space between parameter and value alpha=0.2, and you need
a separate -a for each one.
getAction() This function is called by the game every time that it wants Pacman to make a move
(which is every step of the game).
This is the core of code that controls Pacman. It has access to information about the position
of Pacman, the position of ghosts, the location of food, and the score. The code shows how
to access all of these, and if you run the code (as above) all these values are printed out each
time getAction() is called.
The only bit that maybe needs some explanation is the food. Food is represented by a grid of
letters, one for each square in the game grid. If the letter is F, there is no food in that square.
If the letter is T, then there is food in that square. (Walls are not represented).
The main job of getAction() is to decide what move Pacman should make, so it has to
return an action. The current code shows how to do that but just makes the choice randomly.
final() This function is called by the game when a Pacman has been killed by a ghost, or when
Pacman eats the last food (and so wins).
Right now all this function does is to keep track of the number of episodes (more on what
they are below) and sets epsilon and alpha to zero when a certain number have happened
(more on that in a minute also).
3 cocarascu-yannakoudakis-6ccs3ml1-cw2
3 What you have to do (and what you aren’t allowed to do)
3.1 Write some code
Now you should fill out QLearnAgent with code that performs Q-learning. You should implement
your code in the mlLearningAgents.py file. You will be assessed on your implementation of Qlearning
by the functionality of several methods defined in the file. These are clearly indicated
in the file.
Because of the way that Pacman works, your learner needs to include two separate pieces:
1. Something that learns, adjusting utilities based on how well the learner plays the game.
2. Something that chooses how to act. This decision can be based on the utilities, but also has
to make sure that the learner does enough exploring.
You need to do that because using Pacman forces you to do online learning. The only way that
Pacman learns is by actually playing in a game. Initially they will play badly, but over time they
should get better and better. However, they will learn slowly. Indeed, they will need to play around
2000 games to get to be any good.
See Section 4 for a description of what to submit. Note that we will evaluate your code by running
the command:
python pacman.py -p QLearnAgent -x 2000 -n 2010 -l smallGrid
so we will train the learner for 2000 episodes and then run it for 10 non-training episodes (see below).
We will use the score that your learner gets in those 10 episodes as a way to decide how good your
solution is. To get good marks, your QLearnAgent needs to win around 8 of the games. That is
very achievable.
3.2 Things to know
Here are some things to think about when writing your learning code:
• Your code must implement Q-learning. Code that does not contain such an algorithm will
lose marks.
• The code has some support for learning. First, you can run several games one after another
by using the -n parameter. This:
python pacman.py -p QLearnAgent -l smallGrid -n 5
will run the same game 5 times. Each of these runs is one of the episodes that were mentioned
in the text about final(). The really useful bit is that the same instance of QLearnAgent
gets run all five times, so that it can learn over all five episodes.
Even more helpful, you can do some of the runs without the graphical interface, so that they
run quickly. If you run:
python pacman.py -p QLearnAgent -l smallGrid -x 5 -n 10
it will do the first 5 runs without the interface. The remainder of the runs will happen with
the interface. During the training runs, the system does not record the scores, so the average
score (and results) printed at the end do not include the results of the training episodes.
4 cocarascu-yannakoudakis-6ccs3ml1-cw2
The command-line switch -numTraining is just a synonym for -x, so the value you specify
with -x is the value that ends up in the variable numTraining in the code. This is why
final() counts the number of episodes and compares it to numTraining — that is how the
agent knows when training is over and it is showtime. We set epsilon and alpha to zero at
that point because (as you know from the lecture) these are parameters that control learning.
An -greedy learner chooses not to do the best action that it knows with a probability , so if
we set to zero the learner will always do what it thinks is best (and it won’t get killed because
it does something random)3
. Similarly, if you set α to zero in a Q-learning/SARSA/temporal
difference learner, then the update doesn’t change the Q-values/utilities.
• Though, as the lecture says, in theory you need to adjust the learning rate over time, I did not
find that was necessary.4
• There is no way to access the reward for winning (or the cost of losing) outside final().
(getAction() is not called once the episode is over.) You will need those rewards to learn
how to avoid the ghost and how to win the game.5
3.3 Limitations
There are some limitations on what you can submit.
1. Your code should be in Python 3.
Code written in a language other than Python will not be marked.
2. Your code will be tested in the same environment as we have been using in the lab. That is
the standard Anaconda Python 3 distribution. Code using libraries that are not in the standard
Python 3 distribution may not run when we test it. If you choose to use such libraries and
your code does not run when we test it, you will lose marks.
The reason for this is that we do not have the resources to deal with setting up arbitrarily
complex environments (with the possibility of libraries with arcane interactions) for every
submission.
3. You are not allowed to modify any of the files in pacman-cw2.zip.
In fact, the only thing you can do as part of the coursework is to write code in mlLearningAgents.py.
The idea is that everyone solves the same problem — you can’t change the problem by
modifying the base code that runs the Pacman environment.
4. You are not allowed to copy, without credit, code that you might get from other students
or find lying around on the Internet. (This includes the use of code that was distributed as
part of the module — if you use code from files other than mlLearningAgents.py without
attribution, we will consider that to be plagiarism.) We will be checking.
This is the usual plagiarism statement. When you submit work to be marked, you should only
seek to get credit for work you have done yourself.
3
It might still make a bad move, but it won’t know it is a bad move.
4
It is quite possible that the utilities did not converge as a result, or that they did not converge on the optimal
values, but the learner could play a good game, and that is what matters.
5Without those rewards, my learner will choose to run towards the ghost to end the game quickly and minimise
its losses — very rational when you don’t get the big positive reward for winning.
5 cocarascu-yannakoudakis-6ccs3ml1-cw2
When the work you are submitting is code, you can use code that other people wrote, but you
have to say clearly that the other person wrote it — you do that by putting in a comment
that says who wrote it. That way we can adjust your mark to take account of the work that
you didn’t do.
5. You may be tempted to use code from the files qlearningAgents.py or learningAgents.py
that are part of the Berkeley AI Pacman project. These files provide a big part of the solution
to the problem of using reinforcement learning to play Pacman, so if you use them, you are
avoiding doing an important part of the work of the coursework.
If you use them and acknowledge it, we will deduct marks because you have avoided doing
some of the work we want you to do. If you use them and do not acknowledge it, that is
plagiarism, and you will have to answer to the relevant Misconduct committee.
6. Following on the last point, the code you submit must be in a class called QLearnAgent, and
as in mlLearningAgents.py, this class must be a direct subclass of Agent:
class QLearnAgent(Agent):
This, again, is intended to make sure that everyone solves the same problem, starting from
the basic Pacman Agent.
4 What you have to hand in
Your submission should consist of a single ZIP file. (KEATS will be configured to only accept a
single file.) This ZIP file must include a single Python file (your code).
The ZIP file must be named:
cw2--.zip
so my ZIP file would be named cw2-cocarascu-oana.zip. Submissions that are not in zip format
will lose marks.
Remember that we are going to evaluate your code by running your code using variations on
python pacman.py -p QLearnAgent -x 2000 -n 2010 -l smallGrid
and we will do this in a vanilla copy of the pacman-cw2 folder, so the base class for your agent must
be called QLearnAgent.
To streamline the marking of the coursework, you must put all your code in one file, and this file
must be called mlLearningAgents.py. If your file is named otherwise, you will lose marks.
Do not just include the whole pacman-cw2 folder. You should only include the one file that includes
the code you have written.
Submissions that do not follow these instructions will lose marks.
5 How your work will be marked
There will be six components of the mark for your work, four concerned with functionality and two
with form.
6 cocarascu-yannakoudakis-6ccs3ml1-cw2
1. Functionality
We will test your code by running your .py file against a clean copy of pacman-cw2.
(a) As discussed above, for full marks for functionality, your code is required to run when we
invoke the command:
python pacman.py -p QLearnAgent -x 2000 -n 2010 -l smallGrid
If your code does not run, you will lose marks.
(b) We will also look at your code for evidence of the use of Q-learning. Code that does not
use Q-learning will lose marks.
(c) When we run your code using:
python pacman.py -p QLearnAgent -x 2000 -n 2010 -l smallGrid
it is required to win 8 of 10 games. (If it fails to do this on the first run, we will run it
again and return the maximum number of wins from the two runs). If your code wins
less than 8 games, you will get a mark that reflects how many games your code wins —
more wins equals more marks.
Note that, since we have a lot of coursework to mark, we will limit how long your code
has to demonstrate that it can win 8 games out of 10. We will terminate the run after
five minutes. If your code has won less than 8 games within these five minutes, you will
get a mark that reflects how many games your code has won — more wins equals more
marks.
(d) Finally, to test generalisability, we will run your code on another grid. Code that fails to
run on another grid will lose marks. We will not test the performance of your code, that
is how many games it wins, we will only test if it runs. (There is no point in asking us
what grid we will test your code on because we won’t tell you. For one thing, the point
is that the code should not be making assumptions about which grid it is running on
and so neither should you. For another, we know that if you know which grid, some of
you will write code that exploits particular features of the grid to improve performance,
which is exactly what makes for bad solutions.)
2. Form
(a) There are no particular requirements on the way that your code is structured, but it should
follow standard good practice in software development and will be marked accordingly.
HOWEVER, we do ask that by default you observe the following requirements in order
to be able to reach the full set of marks for style (i.e., you will lose marks if you do not):
Variable naming style should be set to PascalCase for class names; UPPER CASE for
constants; camelCase for others. Also, ensure you make good use of whitespaces (e.g.,
you typically add a whitespace before and after an operator such as ‘<’).
(b) All good code is well documented, and your work will be partly assessed by the comments
you provide in your code. If we cannot understand from the comments what your code
does, then you will lose marks.
A copy of the marksheet, which shows the distribution of marks across the different elements of the
coursework, is available on KEATS, together with FAQs.
7 cocarascu-yannakoudakis-6ccs3ml1-cw2
Version list
• Version 1.0, March 17th 2019
• Version 1.1, March 27th 2019.
• Version 1.2, March 13th 2020.
• Version 1.3, February 22nd 2021.
• Version 1.4, March 8th 2022.
8 cocarascu-yannakoudakis-6ccs3ml1-cw2