COMP226 Assignment 2: Strategy Development
Continuous
Assessment
Number
2 (of 2)
Weighting 15%
Assignment
Circulated
Monday 17 April 2023 (week 9)
Deadline Thursday 11 May 2023 (week 12)
Submission
Mode
Submit up to two files to the CodeGrade assignment on Canvas: strategy.R
(required to get marks) and results.yaml (optional).
Learning
Outcomes
Assessed
This assignment addresses the following learning outcomes:
• Understand the spectrum of computer-based trading applications and
techniques, from profit-seeking trading strategies to execution algorithms.
• Be able to design trading strategies and evaluate critically their historical
performance and robustness.
• Understand the common pitfalls in developing trading strategies with historical
data.
Summary of
Assessment • The goal is to implement and optimize a well-defined trading strategy within the
backtester_2023 framework.
• Marks are available for the correct implementation of 10 functions in
strategy.R (70%).
• Further marks (that require a correct implementation in strategy.R) are
available for the results of a cross-validated optimisation that you can include in
results.yaml (30%).
• CodeGrade pre-deadline tests and offline example outputs are available to help
you check the correctness of your work.
Submission
necessary to
pass module
No
Late Submission
Penalty
Standard UoL policy; resubmissions after the deadline may not be considered.
Expected time
taken
Roughly 8-12 hours
Before you move on and read more about the assignment and start working on it, please make sure you
have worked through "backtester.pdf" (and the corresponding lectures if you want), which is an intro to the
backtester_2023 framework. Only return to this document when you already have the framework up and
running.
First, let's recall the contents of the backtester_2023.zip:
10 directories, 28 files
In the above listing, the following files/directories are specifically there for assignment 2:
• a2_main_template.R
• a2_periods.R
• a2_test_checks_and_getTMA.R
• strategies/a2_strategy_template.R
• a2_example_yamls
• DATA/A2
The relevance of these files and directories will be explained below. The rest of the document is split into
three parts:
• Part 1 describes the 10 functions that should be implemented to fully complete strategy.R; you
should start from a2_strategy_template.R;
• Part 2 describes how to create (the optional) results.yaml;
• Part 3 describes submission via CodeGrade and the available pre-deadline tests.
In addition to pre-deadline tests on CodeGrade, example outputs are provided (in this document and as
files) so that you can test whether you have implemented things correctly.
As for assignment 1, the pre-deadline tests will determine your mark for the first part, corresponding to 70%
of the overall marks that are available. Assuming that you have achieved full marks on the first part, the
pre-deadline tests will check that the form of results.yaml is correct, and that it uses the expected student
username (i.e., your one) and corresponding time periods; the pre-deadline tests do not check the
correctness of the other fields in results.yaml, which will be checked post deadline only if you pass the
pre-deadline test for results.yaml. For those other fields, you should use the examples provided (which
are in the subdirectory a2_example_yamls).
Part 1: strategy implementation (70%)
The trading strategy that you should implement is a triple moving average (TMA) momentum strategy, which
is described in slides 4.7. The specification of the strategy and the functions that it should comprise are given
in full detail, so the correctness of your code can and will be checked automatically.
Two template files are provided to get you started:
• strategies/a2_strategy_template.R, which should become the file strategy.R that you
eventually submit;
• a2_main_template.R, which uses DATA/A2 and strategies/a2_strategy_template.R.
If you source a2_main_template.R with no edits to these two files you will get an error:
Error in if (store$iter > params$lookbacks$long) { :
argument is of length zero
This is because the strategy requires a parameter called lookbacks that you will need to pass in from
a2_main_template.R. Read on to see what form this parameter should take, and, more generally, how
you should be editing these two files.
a2_strategy_template.R contains 10 incomplete functions that you need to complete. The first 6
functions (checkE01,..., checkE06) are error checks for the inputs to getTMA. These error checks are all
one-liners, worth 3% each. They are intentionally meant to be straightforward to implement. The next three
functions compute the moving averages (getTMA), use them to compute the position sign
(getPosSignFromTMA), and compute the position size (getPosSize). The final, tenth function,
getOrders combines the last three to implement that actual trading strategy. Recall that every strategy in
the backtester framework has a getOrders function.
The TMA momentum strategy that you should implement uses three moving averages with different
lookbacks (window lengths). The short lookback should be smaller than the medium one, which in turn
should be smaller than the long lookback. In every trading period, the strategy will compute the value of these
three moving averages (for the series that it trades on, which will be determined by params$series). You
will achieve this by completing the implementation of the function getTMA.
The following table indicates the position that the strategy will take depending on the relative values of the
three moving averages (MAs). You will compute this position (sign, but not size) by completing the function
getPosSignFromTMA. The system is out of the market (i.e., flat) when the relationship between the short
MA and the medium MA does not match the relationship between the medium MA and the long MA.
MA MA MA Position
short MA < medium MA < long MA short
short MA > medium MA > long MA long
The function getPosSignFromTMA takes the output of getTMA as input. The position size, i.e., the number
of units to be long or short, is determined by getPosSize. As for all strategies in the backtester framework,
the positions are given to the backtester by getOrders. Here are the detailed specification and marks
available for these 10 functions.
Function
name
Input parameters Expected behaviour Marks available for a correct
implementation
checkE01 ...
checkE06
prices;
lookbacks.
The behaviour of these checks
are specified as comments in
the template. Hints are given
below.
3% for each of the 6 checks;
18% in total.
getTMA prices;
lookbacks. The
specific form that
these arguments
should take is
specified in the
template code via the
6 checks that you
need to implement.
The function should return a list
with three named elements,
short, medium, and long. Each
element should be equal to the
value of a simple moving
average with the respective
window size as defined by
lookbacks. The windows
should all end in the same
period, the final row of prices.
12%
getPosSign
FromTMA
tma_list is a list
with three named
elements, short,
medium, and long.
These correspond to
the simple moving
averages as returned
by getTMA.
This function should return
either 0, 1, or -1. If the short
value of tma_list is less than
the medium value, and the
medium value is less than the
long value, it should return -1
(indicating short). If the short
value of tma_list is greater
than the medium value, and the
medium value is greater than
the long value, it should return 1
(indicating long). Otherwise, the
return value should be 0
(indicating flat).
10%
getPosSize current_close:
this is the current
close for one of the
series. constant:
this argument should
have a default value
of 5000.
The function should return
(constant divided by
current_close) rounded
down to the nearest integer.
5%
getOrders The arguments to this
function are always
the same for all
strategies used in the
backtester
framework.
This function should implement
the strategy outlined below in
"Strategy specification".
25%
All-or-nothing tests
Since the check functions and getPosSignFromTMA function will only return a small number of
possible correct values (2 for the check functions, and 3 for getPosSign), these are implemented as
"all-or-nothing" tests where you either get full marks for passing all tests or no marks if you fail at least
one test. As a very simple function, getPosSign is also marked with all-or-nothing tests, so from the
first 10 functions, partial marks are only available for getTMA and getOrders.
Strategy specification
The strategy should apply the following logic independently to only the series in params$series
(e.g., params$series could be c(1,3), which would mean trade only on series 1 and 3).
It does nothing until there have been params$lookbacks$long-many periods.
In the (params$lookbacks$long+1)-th period, and in every period after, the strategy computes
three simple moving averages with window lengths equal to:
• params$lookbacks$short
• params$lookbacks$medium
• params$lookbacks$long
The corresponding windows always end in the current period. The strategy should in this period send
market orders to assume a position (make sure you take into account positions from earlier)
according to getPosSignFromTMA and getPosSize. (Limit orders are not required at all, and can
be left as all zero.)
Hints
You can develop the first 9 functions without running the backtester.
For the checks you may find the following functions useful:
• The operator ! means not, and can be used to negate a boolean.
• sapply allows one to apply a function element-wise to a vector or list (e.g., to
c("short","medium","long")).
• all is a function that checks if all elements of a vector are true (for example, it can be used
on the result of sapply).
• %in% can be used to check if an element exists inside a vector.
To compute the moving average in getTMA you can use SMA from the TTR package.
For getPosSize, you can use the function floor.
For getOrders some instructions are given as comments in a2_strategy_template.R.
If an error occurs within a function and you would like to inspect the contents of a variable that is local
to the function, in addition to printing, you can also use global assignment (<<-) for debugging.
Example output for checkE01 ... checkE06 and getTMA
The file a2_test_checks_and_getTMA.R is provided to give you guidance on how you can test the six
functions, checkE01 ... checkE06. For each one, two tests are provided: for a correct implementation, one
test should produce TRUE and the other FALSE. (You don't need to use these tests, as you can also just rely
on the tests on CodeGrade.)
To use these tests, first source a2_test_checks_and_getTMA.R and also source the implementations
that you would like to test. The tests that should return TRUE are test_checkE01() ... test_checkE06();
for tests that should return FALSE, there is single function, test_pass_all_checks, which takes the
function to test as its only argument. Here's an example of both types of test for E01 (where a correct
implementation of checkE01 has been sourced):
> test_checkE01()
[1] TRUE
> test_pass_all_checks(checkE01)
[1] FALSE
The way these tests work is clear from the source code in a2_test_checks_and_getTMA.R:
###############################################################################
# Source the functions that you would like to test, e.g., with
# source('strategies/a2_strategy_template.R') or source('strategies/strategy.R')
###############################################################################
source('framework/data.R'); dataList <- getData(directory="A2")
prices <- dataList[[1]]
prices_19_rows <- dataList[[1]]$Close[1:19]
prices_20_rows <- dataList[[1]]$Close[1:20]
prices_20_rows_renamed <- prices_20_rows
colnames(prices_20_rows_renamed) <- 'Closed'
bad_prices <- c(1,2,3)
lookbacks_no_names <- list(5,10,25) # list elements not named
lookbacks_not_integer <- list(short=5,medium=as.integer(10),long=as.integer(20))
lookbacks_wrong_order <- list(short=as.integer(15),medium=as.integer(10),long=as.integer(20))
lookbacks <- list(short=as.integer(5),medium=as.integer(10),long=as.integer(20))
test_checkE01 <- function()
checkE01(prices,lookbacks_no_names)
test_checkE02 <- function()
checkE02(prices,lookbacks_not_integer)
test_checkE03 <- function()
checkE03(prices,lookbacks_wrong_order)
test_checkE04 <- function()
checkE04(bad_prices,lookbacks)
test_checkE05 <- function()
checkE05(prices_19_rows,lookbacks)
test_checkE06 <- function()
checkE06(prices_20_rows_renamed,lookbacks)
test_pass_all_checks <- function(check_func)
check_func(prices_20_rows,lookbacks)
test_getTMA <- function() # same inputs as test_pass_all_checks()
getTMA(prices_20_rows,lookbacks)
The final test function in this file is for getTMA, where you should get the following return values for a correct
implementation:
> test_getTMA()
$short
[1] 3081.5
$medium
[1] 3122.5
$long
[1] 3128.875
If you want to do further testing, you can use the pre-deadline tests on CodeGrade, which applies to all 10
functions, or you can extend a2_test_checks_and_getTMA.R by adding alternative examples yourself.
Example output for getPosSize
Here is one example input for each of the three possible outputs:
> getPosSignFromTMA(list(short=10,medium=20,long=30))
[1] -1
> getPosSignFromTMA(list(short=10,medium=30,long=20))
[1] 0
> getPosSignFromTMA(list(short=30,medium=20,long=10))
[1] 1
Example output for getPosSignFromTMA
Here are two examples of correct outputs:
> current_close <- 100.5
> getPosSize(current_close,constant=100.5)
[1] 1
getPosSize(current_close,constant=100.4)
[1] 0
Example output for getOrders
The following table gives the correct value of "profit" across 3 different time periods, using the "EXAMPLE"
data, and the following parameters:
params$lookbacks <- list(short=as.integer(5),
medium=as.integer(50),
long=as.integer(100))
params$series <- 1:4
start period end period profit
1 250 2086.184
1 1000 4103.204
500 1500 -2179.298
The examples of results.yaml (details below) can also be used to further establish the correctness of
getOrders, along with all the tests on CodeGrade.
Part 2: cross-validation (30%)
Warning
You can only access the final 30% of marks if you get 70% for the first part; otherwise CodeGrade will
not process results.yaml.
In this part of the assignment you are asked to do a cross-validated parameter optimization of profit, where
you will use an in-sample and out-of-of-sample time period. Every student has their own in-sample and
out-of-sample periods based on their MWS username (only the part before the @, e.g., for Vladimir Gusev,
this username is is gusev, rather than the full email form gusev@liverpool.ac.uk). By having different
time periods for different sutdents, there is not one single correct results.yaml.
To get your in-sample and out-of-sample periods, use a2_periods.R as follows. Source it and run the
function getPeriods with your MWS username as per the following example (where we use the fake
username "x1xxx"). Use startIn, endIn, startOut, and endOut as the start and end of the in-sample
and out-of-sample periods respectively.
> source('a2_periods.R')
> getPeriods('x1xxx')
$startIn
[1] 1
$endIn
[1] 884
$startOut
[1] 885
$endOut
[1] 2000
You will do two parameter sweeps. One on your in-sample period, and one on your out-of-sample period
(normally one doesn't do a sweep on the out-of-sample period in practice; we do it here to allow detailed
cross-period performance analysis). The sweep will be over the following parameters: the short, medium, and
long lookbacks, and the subset of series that are traded on.
Parameter Values
short lookback 5, 10
medium lookback 50, 100
long lookback 200, 300
series All subsets of 1:4 that have at least two elements
The correct resulting number of parameter combinations is 88.
Hint
You can use expand.grid to create the relevant parameter combinations; alternatively you could
use nested for loops.
The following information (full example below) is needed in results.yaml:
1. Your username and the corresponding periods. This information is used for a pre-deadline check to give
you confidence that you are using the right periods.
2. The parameter combination that gives the best profit on the in-sample period (where the series
parameter is encoded in binary, see below); the corresponding profit.
3. The parameter combination that gives the best profit on the out-of-sample period (where the series
parameter is encoded in binary, see below); the corresponding profit.
4. rank_on_out: The rank (a possibly fractional number between 1 and 88) that describes where the
parameter combination from 2. ranks on the out-of-sample period.
5. rank_on_in: The rank (a possibly fractional number between 1 and 88) that describes where the
parameter combination from 3. ranks on the in-sample period.
How to compute the rank
Use the rank (package:base) with the argument ties.method='average'.
Interpretation
An ideal scenario is for the best in-sample parameter combination to also be the best out-of-sample
parameter combination. In practice, this is often not the case, as we have seen in the slides. Here, as
we did in the slides, we are exploring the difference between parameter combination performance on
in-sample and out-of-sample periods, where "a good outcome" is for the rank_on_out and
rank_on_in to both be close to 1 (where 1 is ideal).
NOTE: you will not submit the code used to do the optimisation, which takes a long time to run; you will only
submit the results of the optimisation in results.yaml.
Example output for results.yaml
In the a2_yamls subdirectory, three examples of results.yaml are provided for the fake usernames
"x1xxx", "x1yyy", and "x1zzz", and using the "EXAMPLE" data. For "x1xxx", the yaml file contents are:
username: x1xxx
periods:
startIn: 1
endIn: 884
startOut: 885
endOut: 2000
ins:
short: 5.0
medium: 50.0
long: 300.0
series1: 1
series2: 1
series3: 1
series4: 1
profit: 5963.63
rank_on_out: 2.0
out:
short: 10.0
medium: 50.0
long: 300.0
series1: 1
series2: 1
series3: 1
series4: 0
profit: 3072.562
rank_on_in: 19.0
Note how the params$series parameter is represented in the yaml, as 4 binary variables (taking values 0
or 1): series1, series2, series3, and series4.
Once you have correctly completed part 1, and have also created the code to do the parameter sweep and
ranking you can use these three examples to test your output. These examples are done using the
"EXAMPLE" data so that they do no leak information about the correct answers on the "A2" data.
Marks breakdown for results.yaml
The marks for results.yaml are only available if you have achieved 70% on the first part. Moreover, the
yaml file must have the right format, and must show the correct username and periods -- there is a
pre-deadline test that checks all of this for you.
Here is an example blank results.yaml, shown with additional line numbers:
1 username:
2 periods:
3 startIn:
4 endIn:
5 startOut:
6 endOut:
7 ins:
8 short:
9 medium:
10 long:
11 series1:
12 series2:
13 series3:
14 series4:
15 profit:
16 rank_on_out:
17 out:
18 short:
19 medium:
20 long:
21 series1:
22 series2:
23 series3:
24 series4:
25 profit:
26 rank_on_in:
Note that the line numbers on the left are not part of the file; they are shown since they are used in the
following tables.
Requred for passing the pre-deadline check:
Field(s) Line numbers in example Marks
username 1 0
periods 3-6 0
Assuming that your submitted yaml passed the pre-deadline check, which checks the username and periods
fields and the format of the yaml file, the following marks are available for the remaining fields:
Field(s) Line numbers in example Marks
In-sample best params (unique) 8-14 5
In-sample best profit 15 5
rank_on_out 16 5
Out-of-sample best params 18-24 5
Out-of-sample best profit 25 5
rank_on_in 26 5
Part 3: submission and pre-deadline tests
To get marks the submission of strategy.R is required; the submission of results.yaml is optional:
There are pre-deadline tests for all 10 functions in strategy.R that are needed for the first 70% of marks.
Randomisation
To reduce the incentive for trying to hardcode answers, tests involve randomness in the inputs. This
does mean that there can be some (small) variance in the mark for wrong answers, but there is none
for correct answers. This is a reasonable price to pay for being able to see all the tests that were run
openly.
For the functions checkE01,..., checkE06, these are "all-or-nothing" tests (to prevent always returning TRUE
or always returning FALSE from getting marks). If you do not pass all tests, you will be shown only the tests
that you failed. For example, here's what happens for checkE01 if it just returns TRUE:
When the output says "expected [1] FALSE" that means that the input arguments should have passed this
check.
One only sees tests where FALSE was expected but TRUE was returned. Here is what happens for
checkE02 if it always returns FALSE (one sees only tests where TRUE was expected):
For getTMA, partial marks are possible. Here's an example of the tests for a getTMA implementation that
passes some but not all tests:
The way this submission was created was to break a corerct implementation for the short TMA for certain
lookback values. Note that the errors on CodeGrade show that the problem is only with the short TMA and
only for certain values of the lookback; this type of information may be useful in debugging. Note also that a
broken getTMA (or getPosSignFromTMA or getPosSize) should also break getOrders, because it
should use them. For example, here's the output for getOrders when the same submission as used for
getTMA is used:
As for getTMA, partial marks are possible for getOrders. The tests for getOrders use the resulting profit
for comparison.
Since getPosSignFromTMA should only return 0,1,-1, it is an "all-or-nothing" test. Here's the output when a
wrong implementation that always return 1 is submitted (only failed tests where an expected output of 0 or -1
are shown):
For getPosSize, since it is very simple and there should not be lots of "edge cases" we again implement it
as an "all-or-nothing" test. Here's the output when the flooring of the position size has been ommitted:
So for getTMA and getOrders partial marks are possible, for the other 8 functions it is all or nothing, and
then only failed tests are shown if the test is not passed.
For results.yaml, the pre-deadline test checks that the username and periods are correct and that the format
of the yaml is correct. Here is any example of using the wrong username:
Errors can also arise for the wrong periods or for a badly formatted yaml or one with the wrong fields.
When the submitted yaml passes all pre-deadline tests, you will see the following:
Only in this case will your results.yaml submission be marked post-deadline.
Warning
Your code will be put through the department's automatic plagiarism and collusion detection system.
Student's found to have plagiarized or colluded will likely receive a mark of zero. Do not show your
work to other students, and do not search for answers online.
Good luck with the assignment.
THE END