Department of Computing

COMP348 Document Processing and the Semantic Web

Assignment 1, Part 2:
Feedback

As with Part 1, this was generally done well. The marks on your hardcopies are for each of the four subparts (quality of results, quality of code, correctness of accuracy, quality of repoty), along with the total. I have some general comments:

Quality of Results

• As noted in the specs, marks in this section depended on your system accuracy. I ran your programs on new test data you hadn't seen before. I assigned marks here as follows:

  • the most accurate system got 4.5
  • otherwise, systems scoring > 75% got 4.25
  • otherwise, systems scoring > my benchmark got 4
  • otherwise, systems getting some result on all test data got 3.5
  • otherwise, a partial score depending on completeness
  The accuracy scores clustered in each category, so in fact there wasn't anyone whose system only almost got 75%, or only almost reached my benchmark.

• The top 5 systems on the unseen test data were (accuracy to 2 d.p.):

  • Yenchi CAO 0.95
  • Mark JASKOLSKI 0.81
  • Sun Thurein AUNG 0.81
  • James PISKORZ 0.79
  • Jordan SHARPE 0.79
  Your accuracy is written on your report.

Quality of Code

• Don't cut and paste code when you should use a function. This happened quite often where people duplicated code for young and old categories.

Calculation of Accuracy: Baselines

• Baselines were generally better identified than in Part 1. I'd intended the baseline to be the majority class one; for the given test data it was 50% (because 50% of the files were Young, and 50% Old). If you chose the accuracy rate of my benchmark algorithm as your baseline, that was OK too.

• If you used the accuracy rate of my benchmark algorithm as your baseline, strictly speaking you should do a z-test of two proportions. The numbers are close enough that it's OK if you just used the one proportion version.

Report

• System Descriptions: The aim of the system description is for you to tell me things like how you chose your features, and how you tokenised. It should be sufficiently detailed that I could (with a bit of effort, and perhaps not exactly) reproduce your approach. It's not to describe things like "And then I open a file using open() ...".

• MAKE SURE YOU SPELLCHECK YOUR REPORTS. It's also useful to have someone proofread what you've written.