COMP2017 9017
Assignment 3
Due: 23:59 12 May 2024
Assignment 3 Tests 5%
You have now under-taken another role within the Byte-Tide Project. You will be in-charge of devel- oping testing methods and procedures for the new version fo ‘ByteTide‘. However you may or may not have access to stable version of the build, so you will need to follow on with the specification and do your best to construct an ample set of test cases.
You have been directed to focus on testing the following components of the ‘ByteTide‘ project:
• Merkle Tree
• .bpkg file format
For each component, re-read the Assignment Specification and outline parts of the document that you believe will cause ‘issues‘ and provide a clear rationale behind them.
1 Testing Report
You will need to deliver a test procedures document that you will use and also inform the developer team about. For each test-case entry, make sure your document contains:
• An identifier for the test case
• Provide a brief description of the issue (guide: 20 to 50 words)
• Input and resources required
• Test script/method (how to execute the tests)
• The expected result
You are allowed to use generative AI or external resources to complete this assessment. However you must provide attribution and citations for all resources you have used for your test case.
Creating test cases and integration tests of the resulting file system. Prompts and their corresponding test cases and testing software must be reported. Testing software can be generated by AI.
Include the version and source of the generative AI used. See the example markdown file in section 5.
All these fields must be present and accurate
1 # Author
2 # Date
3 # AI used (yes/no)
4 # AI link
5
6 # Test #1 Description
7 # Test #2 Description
8 ...
9 # Test #10 Description
10
11 # AI Prompt #1
12 # AI Generated #1
13 # AI Prompt #2
14 # AI Generated #2
15 ...
16 # AI Prompt #M
17 # AI Generated #M
18
19 # Instructions to run testing script.
• AI used - yes with version, otherwise no
• AI link - what is the URL to access this service
• AI Prompt - what you entered (asked)
• AI Generated - what was returned. Please summarise in one line and place any data/code in files with a reference to them.
• Instructions to run testing script - all steps required for a person to setup and run a single test, or all tests.
There may be any number of prompts used, multiple instances (conversations), please cite the URL.
2 Testing script
Write 10 testing scripts that will independently run and test an implementation of the Merkle tree and package loader functionality.
Your report must include a section on how to setup and run tests. You are to organise your tests such that an implementation of btide or package_main binaries can be dropped in and tested. For examples, see 6
The test environment is an Ed Discussion workspace. Linux 5.15.0-101-generic x86_ 64 with GNU bash, version 5.2.26. No additional packages can be downloaded by the tester. If programs/libraries are small enough and the total repository size is within the limit, they can be included within the submission.
Be careful regarding assumptions surrounding:
• absolute file paths
• permissions for read, write, execute
• architecture dependent binary builds
• architecture dependent binary data (non-portable read and write from file)
• .gitignore to exclude necessary testing binaries or related files
• forgetting to add the necessary testing binaries or related files
• an external dependency can be built from source on Ed (check it!)
• available super user privileges (not available on Ed).
Be sure that the grader can execute your tests within an Ed workspace. You can always test this yourself by following your own instructions in a new Ed workspace.
3 Getting started
There are a number of test cases available from the Assignment 3 already. You are encouraged to look at these for inspiration. Those test cases do not cover all functionality or scenarios.
Think about what kind of scenarios would likely confirm the correctness rather than a unit test.
You may use generative AI, online resources, or other persons to help explore scenarios with these problems. Remember, you must always cite the sources.
4 Marking
4.1 Deliverables
You will need to upload the following deliverables to the ‘Testing - Bytetide‘ lesson slide on Ed.
• 4 marks - test_report.md - Markdown Document with at least 10 test-case entries
• 1 mark - tests directory - Test scripts and resources required to execute them
Marks are awarded on the test description and their precision in identifying test cases. Consider what are you searching for, how can you describe the specific program state(s) that you want to test. From what is the initial state to the new state and your expectations.
4.2 Deductions
Deductions will apply if the report file format is not followed:
• any of the sections are missing
• the author is missing/invalid
• the date is missing/invalid. Use Unix date time $ date
• the description is missing (automatic zero)
• the link is missing, or does not work
• the prompt is missing
• the generated output is missing in test directory
Deductions will apply if:
• tests are replicated from the already given test cases
• repository size exceeds 5 MB
• failure to clean up system resources used by testing script (e.g. create but never remove named pipe)
5 Sample report
** Sample AI generated test case and code for Backgammon implemented in Swift **
# Author
Joe Bloggs
# Date
Mon 15 Jan 2024 11:03:41 AEDT
# AI used (yes/no)
yes. ChatGPT 3.5
# AI link
https://chat.openai.com
# Test 1 Description − Initial board state
I am testing if the board has been setup correctly.
The input to this state is the program starting and configuration is set to defaults. A user makes a
,
→ selection to play a new game and the board is printed on the screen.
I am going to check and count that the characters appear in the correct positions on the screen.
# Test 2 Description − Player move is correct
The player must be able to make a valid move. I want to see if the legal move is considered first legal (
,
→ because there are illegal moves) and then to see if the board state is updated to reflect the token
,
→ being moved.
The input state of the program for this test is not the initial state, rather it is looking at a halfway
,
→ completed game whereby there are illegal moves possible, but a legal move is made.
The expected outcome is that the move is permitted and the board state
# Test 3 Description − A player can bear off
There is a rule in Backgammon that describes this rule: (https://www.gnu.org/software/gnubg/manual/
,
→ html_node/Bearing−Off.html). I want to check when bearing off is available and if the
,
→ implementation presents that option to the player.
The input state of the program is when all 15 checkers have moved into their home board.
The expected outcome is that after a dice roll, the player is able to remove the checker (as an option).
# AI Prompt 1
https://chat.openai.com/share/5d65ddd7−3a97−48a1−b8c2−44d07162dfd4
Write three test cases for a backgammon program written in the Swift programming language. The test
,
→ cases are checking:
− if the board is setup correctly
− if a player move has been effected correctly
− if a player can bear off (https://www.gnu.org/software/gnubg/manual/html_node/Bearing−Off.html)
# AI Generated 1
Test Case: Check if the board is setup correctly
See data in file tests/board_state/data1.in
Test Case: Check if a player move has been effected correctly
See data in file tests/board_state/data2.in
Test Case: Check if a player can bear off
See data in file tests/board_state/data3.in
# AI Prompt 2
Write the testing code required to run these test cases
# AI Generated 2
See code in file tests/board_state/testcode_1.swift
See code in file tests/board_state/testcode_2.swift
See code in file tests/board_state/testcode_3.swift
# Instructions to run testing script.
Compile all testing code beforehand
```bash
testNumber=1 # a number 1−3
swiftc src/*.swift tests/board_state/testcode_${testNumber}.swift −o test_binary
```
Run this by entering the folder tests/board_state
```bash
./run_test.sh ${testNumber}
```
6 Sample test directory structures
The following is a suggested directory structure. Each folder is categorised and binary files are sym- linked to a top level bin directory.
| -- bin
| | -- btide
| `-- package_main
| -- test_report.md
`-- tests
| -- chunk
| | -- btide -> ../../bin/btide
| | -- chunk_test4.in
| | -- chunk_test4.out
| | -- chunk_test5.in
| | -- chunk_test6.out
| | -- package_main -> ../../bin/package_main
| `-- run_test.sh
...
| -- merkle_simple
| | -- btide -> ../../bin/btide
| | -- merk_test1.in
| | -- merk_test1.out
| | -- package_main -> ../../bin/package_main
| `-- run_test.sh
| -- merkle_tricky
| | -- btide -> ../../bin/btide
| | -- merk_test3.in
| | -- merk_test3.out
| | -- package_main -> ../../bin/package_main
| `-- run_test.sh
...
`-- package
| -- data8.bpkg
| -- outcome8.out
| -- data9.bpkg
| -- outcome9.out
| -- package_main -> ../../bin/package_main
`-- run_test.sh
Another suggested directory structure. Each test is entirely self contained within the folder. The binary is assumed to be in the directory.
| -- test_report.md
`-- tests
| -- test01
| | -- btide
| | -- merk_test1.in
| | -- merk_test.out
| `-- run_test.sh
...
| -- test03
| | -- btide
| | -- merk_test3.in
| | -- merk_test.out
| `-- run_test.sh
...
| -- test09
| | -- btide
| | -- data9.bpkg
| | -- outcome9.out
| | -- package_main
| `-- run_test.sh
`-- test10
| -- btide
| -- data10.in
| -- outcome10.out
| -- package_main
`-- run_test.sh