Project 1 – Processing .sgm files

In this project, you will reading-in and processing multiple .sgm files. There are 22 of them in 

the folder and each is named reut2-0??.sgm, where ?? denote the file number. In each file, there 

are hundreds of articles that look like the one found at the last page. Note that the articles are 

formatted like an html file in that it uses tags. Some articles may have missing tags. DO NOT try 

to fix these files or do any other kinds of modifications. 

Parser

In this project, you will create an .sgm parser. Your program will read through each file, find 

each article, and be able to pull out the information between the tags. Below are the specific 

things I would like you to do in your parser:

1. Be able to pull out the words within the topics and places tags and keep count of the 

words. You can combine the counts of the words found in these tags. 

2. Be able to pull out the words within the body tag and keep count of the words. These 

counts should be separated from the counts in part 1. 

3. In addition, keep count of the following things:

a. The number of articles you were able to reads successfully (either you got the 

places/topics, body, or both)

b. The number of articles where you were able to pull out the words from topics tag. 

c. The number of articles where you were able to pull out the words from places tag.

d. The number of articles where you were able to pull out the words from the body

tag. 

Note that these numbers may not be the same since the tags may not be complete. If the 

tag is not complete, do not count the words. For example, if there is a but no 

, then do not count the words after .

4. Write your results onto text files, one for both the topics and places and another for the 

body. Write the counts from part 3 at the top of both files. Name these files appropriately. 

Suggestions

Less words to count: 

• You may want to look up some of the common words in the English language (you can 

use up to 30 of them) and ignore these words when you are reading the body of the 

articles. 

• Another way to lessen the output is to implement a simple stemmer in your parser. A 

stemmer is an algorithm that strips a word down to some base word. For example, you 

could implement a suffix-stripping stemmer, which removes the suffixes we place at the 

end of verbs. 

Keeping the word count:

• One data structure you use that can hold a word and a number is the class you created in 

problem 6 of Homework 6. As you read through the articles, you can create new objects 

of this class when you encounter a new word or update the count when you read in the 

same word. Note that these objects have to be stored in a data structure that can hold 

multiple items like arrays or array lists. 

You can also use a map data structure, where the word (key) is mapped to a 

number(value) (think of the functions you work with in your math classes). There are a 

few implementations of the map data structure like HashMap and TreeMap. Whichever 

you choose, define it as Map, since the words are the keys and the count 

is the value. Note that this data structure already functions like an array or arraylist, 

where you can add multiple keys and values. So, do not combine it with the first 

suggestion. 

See https://docs.oracle.com/javase/8/docs/api/java/util/Map.html for more 

information on maps. 

Processing the files in one go:

• Since the file names are all the same except for the file number, you may want to use a 

for loop to loop through all the files.

What to submit

Submit the following through Moodle:

1. Your commented .java file or .java file together with a report explaining your program. 

If you commented your program and explained each method and non-trivial statements, 

then you do not need to submit a report that explains how your program works. If you 

feel your comments are not enough to explain your program or your comments are too 

long and distracts from the code, then submit a report detailing how your program works. 

Please format the report as a PDF. 

2. Your two output files. If you are submitting a separate report, you can paste these at the end 

of the report and not have to submit them separately. 

Sample article in an .sgm file.