COMP9444 Neural Networks and Deep Learning
Term 2, 2023
Project 1 - Characters and Hidden Unit
Dynamics
Due: Wednesday 5 July, 23:59 pm
Marks: 20% of final assessment
In this assignment, you will be implementing and training
various neural network models for three different tasks,
and analysing the results.
You are to submit two Python files kuzu.py and
encoder.py, as well as a written report hw1.pdf (in pdf
format).
Provided Files
Copy the archive hw1.zip into your own filespace and
unzip it. This should create a directory hw1, subdirectory
net, and ten Python files kuzu.py, encoder.py,
kuzu_main.py, encoder_main.py, encoder_model.py,
seq_train.py, seq_models.py, seq_plot.py, reber.py and
anbn.py.
Your task is to complete the skeleton files kuzu.py and
encoder.py and submit them, along with your report.
Part 1: Japanese Character Recognition
For Part 1 of the assignment you will be implementing
networks to recognize handwritten Hiragana symbols. The
dataset to be used is Kuzushiji-MNIST or KMNIST for
short. The paper describing the dataset is available here.
It is worth reading, but in short: significant changes
occurred to the language when Japan reformed their
education system in 1868, and the majority of Japanese
today cannot read texts published over 150 years ago.
This paper presents a dataset of handwritten, labeled
examples of this old-style script (Kuzushiji). Along with
this dataset, however, they also provide a much simpler
one, containing 10 Hiragana characters with 7000
samples per class. This is the dataset we will be using.
Text from 1772 (left) compared to 1900 showing the standardization of written
Japanese.
1. [1 mark] Implement a model NetLin which computes
a linear function of the pixels in the image, followed
by log softmax. Run the code by typing:
python3 kuzu_main.py --net lin
Copy the final accuracy and confusion matrix into
your report. The final accuracy should be around
70%. Note that the rows of the confusion matrix
indicate the target character, while the columns
indicate the one chosen by the network. (0="o",
1="ki", 2="su", 3="tsu", 4="na", 5="ha", 6="ma",
7="ya", 8="re", 9="wo"). More examples of each
character can be found here.
2. [1 mark] Implement a fully connected 2-layer network
NetFull (i.e. one hidden layer, plus the output layer),
using tanh at the hidden nodes and log softmax at
the output node. Run the code by typing:
python3 kuzu_main.py --net full
Try different values (multiples of 10) for the number
of hidden nodes and try to determine a value that
achieves high accuracy (at least 84%) on the test
set. Copy the final accuracy and confusion matrix
into your report, and include a calculation of the total
number of independent parameters in the network.
3. [2 marks] Implement a convolutional network called
NetConv, with two convolutional layers plus one fully
connected layer, all using relu activation function,
followed by the output layer, using log softmax. You
are free to choose for yourself the number and size of
the filters, metaparameter values (learning rate and
momentum), and whether to use max pooling or a
fully convolutional architecture. Run the code by
typing:
python3 kuzu_main.py --net conv
Your network should consistently achieve at least
93% accuracy on the test set after 10 training
epochs. Copy the final accuracy and confusion
matrix into your report, and include a calculation of
the total number of independent parameters in the
network.
4. [4 marks] Briefly discuss the following points:
a. the relative accuracy of the three models,
b. the number of independent parameters in each
of the three models,
c. the confusion matrix for each model: which
characters are most likely to be mistaken for
which other characters, and why?
Part 2: Encoder Networks
In Part 2 you will be editing the file encoder.py to create a
dataset which, when run in combination with
encoder_main.py, produces the following image (which is
intended to be a stylized map of mainland China).
You should first run the code by typing
python3 encoder_main.py --target star16
Note that target is determined by the tensor star16 in
encoder.py, which has 16 rows and 8 columns, indicating
that there are 16 inputs and 8 outputs. The inputs use a
one-hot encoding and are generated in the form of an
identity matrix using torch.eye()
1. [2 marks] Create by hand a dataset in the form of a
tensor called ch34 in the file encoder.py which, when
run with the following command, will produce an
image essentially the same as the one shown above
(but possibly rotated or reflected).
python3 encoder_main.py --target ch34
The pattern of dots and lines must be topologically
identical. But, it is fine for it to be rotated or reflected,
compared to the image above. Note in particular the
five "anchor points" in the corners and on the edge
of the figure.
Your tensor should have 34 rows and 23 columns.
Include the final image in your report, and include the
tensor ch34 in your file encoder.py
Part 3: Hidden Unit Dynamics for Recurrent
Networks
In Part 3 you will be investigating the hidden unit
dynamics of recurrent networks trained on language
prediction tasks, using the supplied code seq_train.py
and seq_plot.py.
1. [2 marks] Train a Simple Recurrent Network (SRN) on
the Reber Grammar prediction task by typing
python3 seq_train.py --lang reber
This SRN has 7 inputs, 2 hidden units and 7 outputs.
The trained networks are stored every 10000 epochs,
in the net subdirectory. After the training finishes,
plot the hidden unit activations at epoch 50000 by
typing
python3 seq_plot.py --lang reber --epoch 50
The dots should be arranged in discernable clusters
by color. If they are not, run the code again until the
training is successful. The hidden unit activations are
printed according to their "state", using the colormap
"jet":
Based on this colormap, annotate your figure (either
electronically, or with a pen on a printout) by drawing
a circle around the cluster of points corresponding to
each state in the state machine, and drawing arrows
between the states, with each arrow labeled with its
corresponding symbol. Include the annotated figure
in your report.
2. [1 mark] Train an SRN on the anbn language
prediction task by typing
python3 seq_train.py --lang anbn
The anbn language is a concatenation of a random
number of A's followed by an equal number of B's.
The SRN has 2 inputs, 2 hidden units and 2 outputs.
Look at the predicted probabilities of A and B as the
training progresses. The first B in each sequence and
all A's after the first A are not deterministic and can
only be predicted in a probabilistic sense. But, if the
training is successful, all other symbols should be
correctly predicted. In particular, the network should
predict the last B in each sequence as well as the
subsequent A. The error should be consistently
below 0.01. If the network appears to have learned
the task successfully, you can stop it at any time
using ?cntrl?-c. If it appears to be stuck in a local
minimum, you can stop it and run the code again until
it is successful.
After the training finishes, plot the hidden unit
activations by typing
python3 seq_plot.py --lang anbn --epoch 100
Include the resulting figure in your report. The states
are again printed according to the colormap "jet".
Note, however, that these "states" are not unique but
are instead used to count either the number of A's we
have seen or the number of B's we are still expecting
to see.
3. [1 mark] Briefly explain how the anbn prediction task
is achieved by the network, based on the figure you
generated in Question 2. Specifically, you should
describe how the hidden unit activations change as
the string is processed, and how it is able to correctly
predict the last B in each sequence as well as the
following A.
4. [1 mark] Train an SRN on the anbncn language
prediction task by typing
python3 seq_train.py --lang anbncn
The SRN now has 3 inputs, 3 hidden units and 3
outputs. Again, the "state" is used to count up the A's
and count down the B's and C's. Continue training
(re-starting, if necessary) for 200k epochs, or until
the network is able to reliably predict all the C's as
well as the subsequent A, and the error is
consistently in the range of 0.01 or 0.02.
After the training finishes, plot the hidden unit
activations by typing
Rotate the figure in 3 dimensions to get one or more
good view(s) of the points in hidden unit space.
5. [1 mark] Briefly explain how the anbncn prediction
task is achieved by the network, based on the figure
you generated in Question 4. Specifically, you should
describe how the hidden unit activations change as
the string is processed, and how it is able to correctly
predict the last B in each sequence as well as all of
the C's and the following A.
6. [4 marks] This question is intended to be more
challenging. Train an LSTM network to predict the
Embedded Reber Grammar, by typing
You can adjust the number of hidden nodes if you
wish. Once the training is successful, try to analyse
the behavior of the LSTM and explain how the task is
accomplished (this might involve modifying the code
so that it returns and prints out the context units as
well as the hidden units).
Submission
You should submit by typing
python3 seq_plot.py --lang anbncn --epoch 200
python3 seq_train.py --lang reber --embed True --model lstm --hid 4
give cs9444 hw1 kuzu.py encoder.py hw1.pdf
You can submit as many times as you like - later
submissions will overwrite earlier ones. You can check
that your submission has been received by using the
following command:
9444 classrun -check hw1
The submission deadline is Wednesday 5 July, 23:59pm.
In accordance with new UNSW-wide policies, 5% penalty
will be applied for every 24 hours late after the deadline,
up to a maximum of 5 days, after which submissions will
not be accepted.
Additional information may be found in the FAQ and will be
considered as part of the specification for the project. You
should check this page regularly.
Plagiarism Policy
Group submissions will not be allowed for this
assignment. Your code and report must be entirely your
own work. Plagiarism detection software will be used to
compare all submissions pairwise (including submissions
for similar assignments from previous offering, if
appropriate) and serious penalties will be applied,
particularly in the case of repeat offences.
DO NOT COPY FROM OTHERS; DO NOT ALLOW
ANYONE TO SEE YOUR CODE
Please refer to the UNSW Policy on Academic Integrity
and Plagiarism if you require further clarification on this
matter.
Good luck!