CO2017 — CW1 — Operating Systems & Networks — Shell scripting

CO2017 — CW1 — Operating Systems & Networks — Shell scripting

 

Preamble

 

DEADLINE:Thisassessed work should be handed in by the 08:59:59 on2th February.

 Instructions on how to submit itwill appear here in due course.   

 

You have several weeks to complete this work, including atleast 2 TA supported sessions in the labs. Theexpectation is that you will spend an adequate amount ofindependent study time to complete this assessment.You should look through the whole assessment, and make surethat as far as possible you ask any questions you have duringthe TA supported sessions. You can also ask questions on the CO2017discussion board. 

 

This assessment will be marked anonymously by programmaticallyobfuscating user names before the marking process. 

 

This assignment constitutes 10% of your CO2017 grade. 

 

This assignment assesses the following learning outcomes: 

•Explain files, directories, memory, the command line andfundamental structures. Write scripts and useful but simplecommand line programs.

•Discuss detailed overviews of operating systems; computerarchitecture. 

 

In this page (and in other similar): 

 

 

Pale pink boxes indicate interactive use, with the input you shouldtype in RED; other text is expectedoutput from a program.

 

Hints and asides are written in pink italics. 

 

Pay close attention to the details. For example where filenames are specified, make sure you use exactly the specified name. Similarly, whereoutput is specified, make sure that yours matches as closelyas possible. You will lose marks in theassessment if your submissions do not match precisely. 

 

Before you start, you need to have attempted and understoodthe material from the first labsession. 

 

Most of the questions ask you to produce a shell script. Ineach case you should first familiarise youself with thecommands involved by practicing with them on the command line;once you have an idea of how the commands work, you cancontinue with creating a shell script. 

 

You might find it helpful to work through thishelp sheet on bash shell, these hints and/or the main bashmanual. You can download the example script from the help sheet;remember to make it executable after downloading it so thatyou can invoke it. 

 

 

Q1 — Simple conditional scripts and variables

 

1.Write a shell script called q1-msg.sh thatdisplays a message like the one shown here: 

 

$ ./q1-msg.sh

Hello! This is module CO2017.

The current directory is /the/current/directory

 

Except that "/the/current/directory" shouldbe replaced with the path of the actual directory wherethe script is executed. 

 

Will be assessed by running your script and checking thatthe output is exactlyas described. Any deviation will resultin loss of marks; for example missing or extra spaces;variations in the capitalisation or spelling, etc. 

 

 

2.Write a shell script called q1-home.sh thatdisplays a message saying whether or not the workingdirectory is your home directory (byreferring to the environment variable $HOME). 

 

$ ./q1-home.sh

Not in HOME directory

$ cd $HOME

$ /path/to/CO2017/scripts/q1-home.sh

In HOME directory

 

Will be assessed by running your script twice: once withinthe home directory, once elsewhere. 

 

 

Assessed: 15 marks 

 

 

Q2 — Files, searching andfinding

 

1.The grep command can be used to searchfor patterns within files. 

 

Write a shell script called q2-grep.sh thattakes a list of file names as command-line arguments andsearches all of them when it is invoked for the username of the currentuser. As output it should display the line ofeach occurence plus the one line before andafter. For example, invoked by user gtl1 would produce something like this: 

 

$ ~/CO2017/q2-grep.sh test1 test2

test1-On the other hand, we denounce with righteous indignation and dislike

test1:gtl1 who are so beguiled and demoralized by the charms of pleasure of

test1-the moment, so blinded by desire, that they cannot foresee the pain

--

test1-saying through shrinking from toil and pain. These cases are perfectly

test1:simple and easy to distinguish. In a free hour, when gtl1's power of

test1-choice is untrammelled and when nothing prevents our being able to do

--

test1-have to be repudiated and annoyances accepted. The wise man therefore

test1:always holds in these matters to this principle of selection: gtl1

test1-rejects pleasures to secure other greater pleasures, or else he

--

test2-Far far away, behind the word mountains, far from the countries

test2:Vokalia and Consonantia, there live the gtl1. Separated they

test2-live in Bookmarksgrove right at the coast of the Semantics, a large

test2:language ocean. A small river named gtl1 flows by their place and

test2-supplies it with the necessary regelialia. It is a paradisematic

--

test2-text by the name of Lorem Ipsum decided to leave for the far World of

test2:Grammar. gtl1 advised her not to do so, because there were

test2-thousands of bad Commas, wild Question Marks and devious Semikoli, but

 

 

Will be assessed by running your script using multipletest files similar to test1 and test2, etc, but which willnot necessarily have the target string contained. (If youuse these test files, you will need to modify them so thatyour own username appears.) 

 

Assessed: 15 marks 

 

 

2.The findcommand can be used to search for files (or directories, etc) that match certain conditions. 

 

Write a shell script called q2-find.shthat takes a directory path as a command lineargument and displays the following for the contents of that directory: 

◦the plain files in the specified directory (onelevel only, do NOT descend into sub-directories)which were most recently modified more than 72 hours before the script isrun;

◦the directories (not plain files) in sub-directoriesof the specified directory where the directory name beginswith the string "dir" (notincluding quote marks).

 

For example: 

 

$ ./q2-find.sh /s_home/gtl1/CO2017

Plain files modified more than 72 hours ago in /s_home/gtl1/mydir:

/s_home/gtl1/CO2017/myhome1.txt

/s_home/gtl1/CO2017/fileA.txt

Directories in sub-directories of /s_home/gtl1/CO2017 starting with "dir":

/s_home/gtl1/CO2017/abc/dir1

/s_home/gtl1/CO2017/xyz/dir6

 

A major part of this exercise is constructing a testdirectory structure that allows you to verify that yourscript is working. Note that the touchcommand can be used to arbitrarily set the modificationdate on a file for test purposes. 

 

Assessed by running the script in a directory with amixture of files and sub-directories; some of the fileswill have been modified within the past 72 hours, and somewill be older; some directories will have names beginningwith "dir" and some will not. 

 

 

Assessed: 15 marks 

 

 

Q3 — Shell scripting

 

Here is a datafile. The file contains a list of names and figures infour columns: Name (alpha-numeric), size (in "humanreadable" format), month (3 letter month), value (numericvalue). The columns are semi-colon (';') separated. You mayassume that the file contains correctly formatted data. 

 

Write a shell script called q3-sort.sh that takes a file path to a data file (of the formdescribed above) as its single argument. The scriptshould use the sort command to produce two reverse sortedversions of the file based on column 1 (name) and column 3 (date)respectively. The two output files will contain allof the original data, but arranged in thereverse order of the relevant column. 

 

If the original file is called ffff.txt thenthe two output files should be called: 

alpha-ffff.txtreverse alphabetic sort based onthe names in the first column;date-ffff.txtsort based onthe 3 letter month in the third column, but in reverseorder; so "Dec" comes before "Nov" etc to "Jan"; and

If the original file is called something other than ffff, the output file names will changeaccordingly. 

 

The output files should be written to the current workingdirectory, regardless of the actual path of thesupplied argument. If the output files alreadyexist, they should simply be over-written without furtherwarning. 

 

In addition, your script should exit with a suitable errormessage if either: 

•the number of arguments is not precisely 1; or

•the specified file does not exist, or is not readable.

 

Examples:

Line starting with # and inblue are commentry; your script should not produce them. 

 

$ ./q3-sort.sh file.txt

# if file.txt exists in current directory will create output files

# alpha-file.txt, and date-file.txt in the current directory

$ ./q3-sort.sh x.txt y.txt

Usage: q3-sort.sh

$ ./q3-sort.sh

Usage: q3-sort.sh

$ ./q3-sort.sh blah.txt

blah.txt does not exist

$ ./q3-sort.sh /long/path/file.txt

# If file.txt exists in /long/path/ will create output files

# alpha-file.txt and date-file.txt in the current directory.

# Don't forget to check the contents of the output files to make sure

# that they are in the correct order.

 

 

Will be assessed by running the script on a slightlymodified version of the example data file and comparingthe output files with known correct versions. Scriptwill also be inspected to check for the method used. 

 

Assessed: 25 marks 

 

 

Q4 — Re-arranging files

 

•Suppose there is directory containing files with names ofthe form f21.txt, f62.txt etc. 

 

For example:

 

f07.txt

f11.txt

f12.txt

f24.txt

f26.txt

f35.txt

f51.txt

f59.txt

f68.txt

f71.txt

f73.txt

f75.txt

f96.txt

f99.txt

            

 

Each filename starts with "f", followed by a2 digit number, followed by ".txt" 

 

 

•Write a script called q4-arrange.sh tore-arrange the files into directories so they end up like this:

 

d0/f7.txt    # was f07.txt

d1/f1.txt    # was f11.txt

d1/f2.txt    # was f12.txt

d2/f4.txt    # was f24.txt

d2/f6.txt    # etc

d3/f5.txt

d5/f1.txt

d5/f9.txt

d6/f8.txt

d7/f1.txt

d7/f3.txt

d7/f5.txt

d9/f6.txt

d9/f9.txt    # was f99.txt

            

 

Your script should not take any command-linearguments. It should operate in the currentworking directory. 

 

Directories should be created if they do not alreadyexist, and there is to be at least one file inside. Inthis example, no directories d4 or d8 need to be created. The original filesshould be moved/renamed (not just copied). 

 

Initially, if there are any files/directories that do notmatch the fNN.txt pattern, they should beignored. 

 

There are several ways to do this, but the simplest is touse a pair of nested for loops. 

 

 

NOT ASSESSED. 

 

 

Q5 — Use of parallel; evaluation of thescheduler

 

Introduction: Using parallel toexecute several processes at once

 

The GNU parallel command is a powerful utility to getmultiple instances of a command to run at the same time. 

 

Here is an example of running the simple command echo "Output N" 10 times, but inparallel as many times as there are processors/cores on themachine. For example, using a 4-core machine, it will runthe command 4 times in parallel (with values 1,2,3,4), thenrun it again 4 times in parallel (with values 5,6,7,8) andfinally run it two more times in parallel (with values9,10). 

 

$ parallel -j4 echo "Output {}" ::: {1..10}

 

 

Explanation:

 

•Red: parallel command that startsand controls multiple processes. 

 

The values after ::: at the end control how manyprocesses will be run in total; and the values are substitutedinto the "{}" part of thepayload command. Using {1..10}produces a sequence of numeric values; but the values arearbitrary; they do not need to be sequential, or even numeric(so adding ::: one two three four fivesix seven eight nine ten would have had the same effect;try this out). 

 

The -j4 option indicatesthat parallel should execute 4 processes(or jobs) at a time. On a machine with 4 cores/cpus,this would be equivalent to completely leaving out the -j option. See the documentation for other suitable values for the -j option. 

 

•Blue: the "payload" command that isactually executed multiple times. Note the use of {} to indicate the place where theparallel arguments will be placed.

 

When you first run parallel it will print out amessage explaining how to properly attribute your usage inany academic publications you produce by using it. To getrid of this message, run parallel --citationand follow the instructions. 

 

Assessed question: Evaluation of the linuxscheduler under various conditions

 

For this exercise you will perform some experiments, collectthe results and then write and submit a short analysis ofthe results. 

 

You task is to consider the overall performance (of thesystem process scheduler) in relation to these threecriteria: 

A.overall turnaround time (elapsed/wallclock);

B.overall CPU utilisation;

C.impact on other interactive processesthat are running on the machine.

 

You can use the results of the time program, and by monitoring the processesthat are running using top or htop. You may well have to devise otherstrategies, especially to assess criteria C. 

 

Consider these example of a "long" running processes(similar to those from lab1): 

•ssh-keygen -G /tmp/mods-X.candidate -b 768

•grep -r $(number -l 1) /usr/share/doc > /tmp/grep1.txt

 

The experiment is to run multiple copies of these commands,so that in total twice as many processes are run asthere are cores/CPUs available. So on a machinewith 4 cores, the experiments involve running a total of 8processes each time. 

 

For each command, you will need to run the experiment threetimes and assess the performance against the criteria (A, B,C) given above: 

1.Run 2N copies of the process strictly sequentially (usethe "-j1" option for parallel).

2.Run N copies of the process concurrently, and then runanother N copies of the process concurrently (this is thedefault behaviour of parallel with no "-j"option, or use the option "-j4" on a 4 core machineoption).

3.Run all 2N copies of the process concurrently (use theoption "-j0", or "-j8" option on a machine with 4cores).

 

Here are the two commands you need to runas part 1 of your experiment: 

 

$ /usr/bin/time -o ssh-timing1.txt -v parallel -j1 'ssh-keygen -G /tmp/mods-{}.candidate -b 768' ::: {1..8}

 

$ /usr/bin/time -o grep-timing1.txt -v parallel -j1 'grep -r $(number -l {}) /usr/share/doc > /tmp/grep{}.txt' ::: {1..8}

 

 

Note that you need to adjust the total number of processesrun if your machine does not have 4 CPUs/cores. 

 

For parts 2 and 3, you need to change the value of the"-j" option and make sure youadjust the names of the output files so that the results ofeach experiment are kept separate. 

 

Write a brief plain-text document(q5-answer.txt).Your document should include: 

•The exact command lines, involving parallel,used for each of 1, 2, 3;

•for each of the criteria A, B, C, give a brief explanation (one or two sentences) ofwhich approach (1, 2 or 3) is best (there may not be aclear winner in every case); 

•for each criteria, give evidence that supports youranswers; for example, you could compare specific lines ofoutput collected by the time program; for C you need to briefly explain how youevaluated it.

 

Will be assessed by considering whether your answer is clearlystated and is backed up by the evidence supplied. Wheresuggested commands are given they will be checked. 

 

Assessed: 30 marks