School of Computing and Information Systems
The University of Melbourne
COMP90073 Security Analytics,
Semester 2 2022
Project 2: Machine learning based cyberattack detection
Release: Tue 30 Aug 2022
Due: 1pm, Tue 11 Oct 2022
Marks: The Project will contribute 25% of your overall mark for the subject;
you will be assigned a mark out of 25, according to the criteria below.
Overview
There are three tasks in this project: Task I aims to develop your skills in applying
unsupervised machine learning techniques for anomaly detection. Task II helps you better
understand how to use gradient descent-based methods to generate adversarial examples
against supervised learning models beyond the computer vision domain. In Task III, you are
asked to read and review a recent paper on adversarial machine learning.
Specifically, (1) for Tasks I and II, two network traffic (NetFlow) datasets are provided, one
for each task. Both datasets contain botnet traffic and normal traffic. You need to identify
botnet IP addresses from both two datasets. (2) For Task II you also need to choose a
botnet IP address, and explain how to manipulate the corresponding raw network traffic
records in order to bypass detection. (3) Each student has been assigned a paper for Task
III, which will be sent individually via email.
Deliverables
1. Task I – Source code (Python) and SPL queries used to do the following:
a. Generate/Select features from the packet capture files (training and test datasets)
using Splunk. You can use apps such as Splunk Machine Learning Toolkit, but all
features have to be generated/selected within Splunk.
b. Use two alternative feature generation/selection methods (filter-based, wrapperbased, etc.) to select features from packet capture files (training and test datasets).
c. Use Python/Splunk to build six models: apply two different anomaly detection
techniques on each of the three set of features generated/extracted from 1.a. and
1.b.
d. Score the test data such that cyberattacks are assigned the highest (or lowest)1
scores.
e. Return the IP addresses of attackers and the timestamps of their first and last
attempt for attacking the network service (per attack scenario).
1 Optionally anomalies may have lowest scores given the applied technique. Some anomaly detection techniques
assign high scores (e.g., distance measure) to anomalies and some of them assign low scores (e.g., probability)
to anomalies.
f. Compare and discuss the results from different feature extraction and different
anomaly detection techniques.
g. Prepare a TXT file including all stream ID which your program classifies as attack
traffic, separated by newlines (i.e., one stream ID in each line).
2. Task II
a. Source code in Python, including:
i. Building, training and testing the supervised learning model.
ii. Generating adversarial examples for a chosen botnet IP address, i.e., how to
modify its feature values.
b. Explain how to change the raw traffic sent from/to the chosen botnet IP address, in
order to reflect the modified feature values. For example, the following six features
are extracted for each IP address: (1) mean outbound packet size, (2) variance of
outbound packet size, (3) mean packet count per second, (4) max packet count per
second, (5) mean of packet jitter, (6) variance of packet jitter. A supervised model
is trained on these features to decide whether an IP address is malicious. You find
that by manipulating the values of the third and fourth features, a botnet IP address
is labelled as “normal” by the model. Then how do you change the raw traffic
records so that they are consistent with the modified feature values? For instance,
if 1000 raw traffic records were related to the bot, do you change all 1000 records,
or only a subset, e.g., 100/200 of them? How do you change each of the selected
traffic records?
* Note that for Task II, (1) the model is trained to classify each IP address, NOT each
traffic record, as demonstrated in the above example. (2) The focus is not to train an
accurate detection model (i.e., do not spend too much effort on improving model
performance), but to understand the difference of generating adversarial examples in
domains other than computer vision: in the vision domain, raw pixels are often taken as
input, and attackers can directly manipulate them. However, in other domains such as cyber
security, raw data cannot be fed into a model directly, and instead features need to be
extracted first. Therefore, although it would not be difficult to know how to manipulate the
features to bypass detection, there will be different ways to change the raw traffic records, in
order to be consistent with the modified feature values and without affecting the botnet
functionality.
3. Task III. In this task, you will learn how to write a review for an academic paper.
Typically, a review should include the following parts:
a. Summary. Your review starts with a brief summary of the main ideas of the paper.
It helps meta-reviewers, program chairs and the authors to determine whether
there are any misunderstandings.
b. Merits. List the main contributions of the paper in this section. Contributions can be
theoretical, methodological, algorithmic, empirical, etc.
c. Main review. Provide a thorough review of the paper, including:
i. Originality: Are the tasks or methods new? Is the work a novel combination of
well-known techniques? Is it clear how this work differs from previous
contributions? Is related work adequately cited?
ii. Quality: Is the submission technically sound? Are claims well supported (e.g., by
theoretical analysis or experimental results)? Are the methods used appropriate?
Is this a complete piece of work or work in progress? Are the authors careful and
honest about evaluating both the strengths and weaknesses of their work?
iii. Clarity: Is the submission clearly written? Is it well organized? If not, please make
constructive suggestions for improving its clarity.
iv. Significance: Are the results important? Are others (researchers or practitioners)
likely to use the ideas or build on them? Does the submission address a difficult
task in a better way than previous work? Does it advance the state of the art in a
demonstrable way? Does it provide unique data, unique conclusions about
existing data, or a unique theoretical or experimental approach?
*Note that (1) the questions listed in c.i -- c.iv are for explanation only. DO NOT write the
main review in the form of Q&A. Write it like an essay instead. (2) Some papers include
appendix, which may include proofs, additional experimental settings and results. The
appendix helps you better understand the paper, but your review should focus on the main
part of the paper.
For Tasks I & II:
4. A README that briefly details how your program(s)/queries work(s). You may use
any external resources for your program(s) that you wish. You must indicate (cite)
what these external resources are and where you obtained them, in the README
file.
*Note: please submit a separate README file for Tasks I & II.
Technical Report
A technical report of around 2000 words comprising:
Task I:
1. An overview of the test dataset using Splunk and explaining feature
generation/selection using SPL queries and Splunk native functionalities.
2. Description of your methodology for generating features. Briefly explain your method
for the first project, and discuss your modifications and new findings in Project 2.
3. Review of at least two anomaly detection methods that you have used.
4. Description of the experimental setup and evaluation of the (two) methods in
detecting anomalies on the test datasets using features generated in Splunk and also
features generated using alternative methods. Description should also comprise IP
addresses of attacker(s) and victim(s), the attacked service(s), the timestamp, and
the type of the attack per attack scenario identified.
5. Description of your final CSV file, the scoring and thresholding technique you used
for detecting the reported anomalies2
.
6. Conclusion and discussion: describe anomaly detection method worked best given
the attack scenario.
2 For example, you may choose the best model as your final model or make an ensemble of models.
Task II:
7. Explanation of the generated features and your choice of supervised learning
model. Note that supervised learning is used here, and the mode is the target against
which adversarial examples will be generated.
8. Choosing one IP address classified as botnet by your model, and explaining:
a. How to perturb its features via gradient descent-based method to bypass the
detection of your model;
b. How to change the raw network traffic sent from/to it, in order to be consistent
with the modified feature values and without affecting the botnet functionality.
You should include a bibliography and citations to relevant research papers and external
resources and code you have used.
Review
A review of 400 – 500 words of the assignment paper.
1. Summary. This part should contain no more than three sentences. Please be brief,
but specific.
2. Merits. List the top three or more main contributions.
3. Main review. In order for your reviews to provide useful feedback to authors, write
this section in a top-down manner and start from the most important aspects. Your
arguments should be objective, specific, concise and polite.
Assessment Criteria
Code quality and README (2 marks)
Technical report (17 marks)
1. Methodology: (4 marks)
You will describe your methodology in a manner that would make your work
reproducible. You should describe in detail:
Tasks I and II
a. The features that were generated and/or selected.
Task I
a. The training data that was used to learn the anomaly detection models. You
should explain how the parameter settings for your methods were performed
(e.g., setting the 𝜈𝜈 parameter in OCSVM3
). You should not use the test
data for setting the parameters.
b. The scoring that was performed in each model to rank the data instances.
c. The thresholding on the scores that was performed in each model to label the
attacks.
3 For anomaly detection methods that require validation set for parameter tuning, you can combine a small
amount of anomalies (about 5%) from the of the attack day dataset to your training set.
2. Accuracy of Results: (4 marks)
Task I
Your machine learning based technique should generate a report of detected attacks
on the test datasets. This should be the output of your algorithm and you should not
change it based on your analysis. It should indicate the IP address of the attacker
and the victim, the attacked service, and the period (timestamps) for which the attack
was happening. You are marked out of 4 based on the percentage of successfully
detected attacks by your anomaly detection model.
Task II
As explained in Deliverable 2, the focus of Task II is not to obtain an accurate
detection model. Therefore, accuracy will not be marked separately, but together with
the critical analysis – you are required to perturb the feature(s) and raw network
traffic of an IP address classified as botnet by your model, but if that IP address in
fact belongs to a normal user, i.e., your model misclassifies it, you will not get full
mark for the critical analysis, even if your methods for perturbing the features and the
raw network traffic are correct.
3. Critical Analysis: (7 marks)
Task I
a. Use of Splunk for feature generation/selection from packet capture files
(training and test datasets).
b. Discuss the differences in processes, scalability, and results identified using
the Python code developed for anomaly detection.
Task II
a. Explain the steps for generating adversarial examples, including which
features are chosen, how perturbations are calculated.
b. Explain the steps for changing the raw network records.
4. Report Quality: (2 marks)
You will produce a formal report and express your methodology and findings
concisely and clearly. The quality and description of figures, tables, and the
README file should be acceptable.
Review (6 marks)
1 mark each for summary and merits. 4 marks for the main review.
Description of the Data
The two datasets for Project 2 (A2_1.zip & A2_2.zip) contain the NetFlow data for a network
under cyberattacks. Each line of the dataset includes the following 15 fields: (1) stream ID,
(2) timestamp, (3) duration, (4) protocol, (5) source IP address, (6) source port, (7) direction,
(8) destination IP address, (9) destination port, (10) state, (11) source type of service, (12)
destination type of service, (13) the number of total packets, (14) the number of bytes
transferred in both directions, (15) the number of bytes transferred from the source to the
destination.
Changes/Updates to the Project Specifications
If we require any changes or clarifications to the project specifications, they will be posted on
the LMS. Any addendums will supersede information included in this document.
Academic Misconduct
For most people, collaboration will form a natural part of the undertaking of this project.
However, it is still an individual task, and so reuse of ideas or excessive influence in
algorithm choice and development will be considered cheating. We will be checking
submissions for originality and will invoke the University’s Academic Misconduct policy
(http://academichonesty.unimelb.edu.au/policy.html) where inappropriate levels of collusion
or plagiarism are deemed to have taken place.
Late Submission Policy
You are strongly encouraged to submit by the time and date specified above, however, if
circumstances do not permit this, then the marks will be adjusted as follows. Each day (or
part thereof) that this project is submitted after the due date (and time) specified above, 10%
will be deducted from the marks available, up until 5 days has passed, after which regular
submissions will no longer be accepted.
Extensions
If you require an extension, please email Mark Jiang