CMPT 479/817 Usability Engineering

Assignment 2


The purpose of this assignment is to provide students with

Students will use the e-Commerce Web site that they used for Assignment 1 and evaluate it via user testing.

NOTE: This is a complex assignment. It is worth 20% of the year's mark and thus takes considerable work to complete. Be sure to fully understand both the steps in completing this assignment and the marking structure to ensure that you fully answer what is needed.

The purpose of this assignment IS NOT to test the principles that were used it Assignment 1. This assignment wants you have users test the Web site without their knowing anything about the principles. i.e. do not do anything that tries to force users to discover or use any of the principles.

NOTE 1: If you have not completed Assignment 1, you will be unable to complete step 5 d below.
NOTE 2: This is a time consuming assignment. Do not leave this too long and do not expect any extensions. If necessary it is better to submit an incomplete assignment than not to submit anything.
NOTE 3: If you have any specific questions about this assignment, be sure to ask them as soon as possible.

Steps in completing this assignment:

  1. Identify further information on user testing
    1. Chapter 3 introduced some ideas relating to conducting user testing and introduced some of the usability methods that can be used within a user test.
    2. Chapter 5 introduced the concept of scenarios (which will be used in your user testing).
    3. While the text introduced these concepts, you probably need to know more about user testing before you actually try to perform it.
    4. You should carefully consider the material in these text chapters and then you will need to obtain further information on user testing, before completing step 2 (be sure to look over all the requirements of this assignment before you start so that you can evaluate what types of information you need to get to help you successfully complete this assignment).
      1. students registered in CMPT 479 are required to consult at least 3 different sources of further information.
      2. students registered in CMPT 817 are required to consult at least 5 different sources of further information.
    5. You are looking for significant ideas that are distinct from each other and from the materials in the text that can help you plan and conduct your user testing for this assignment.
      1. students registered in CMPT 479 are required to discuss at least 5 distinct ideas
      2. students registered in CMPT 817 are required to consult at least 10 distinct ideas 
  1. Plan your user testing
    1. your test goals,
    2. how you will conduct the user tests
    3. how you will collect and evaluate the results, including                                                              
      1.   performance measures
      2. observing the use
      3. follow up questions

  1. Identify and develop descriptions of what you will be testing, this will involve
    1. a minimum of 3 different tasks that a user could perform on the web site

NOTE: It is ideal for you to evaluate as many different tasks as possible, but while it would be nice to include more scenarios, it is important that all scenarios tested can be completed by your test subjects in a maximum of between 30 and 45 minutes of user testing.

    1. a minimum of 5 distinctly different scenarios (which are what you actually will be testing)
    2. there must be at least two distinct scenarios for at least one of the tasks.

NOTE 1: The descriptions of the scenarios should focus on what needs to be done rather than how to do it, so that you can give these scenarios to test subjects and see how they try and accomplish them with the web site. The scenarios should not be based on the Principles, but should be based on the most important tasks/scenarios that a user of the Web site is likely to want to perform.

NOTE 2: For scenarios to be of use, you need to identify in advance each of the logical steps that must be accomplished in order to successfully accomplish the scenario, whether or not you choose to provide this information to your test subjects.

NOTE 3: Any scenario that involves attempting to purchase a product should not actually involve submitting a bogus or real order to the owner of the web site, but it has to involve sufficient interactions up to the point of submitting an order to allow sufficient testing to determine relevant usability issues.

  1. Recruit a minimum of 4 individuals to test the web site for you. You need to obtain informed consent from each of them to participate. This informed consent should include your explaining:
    1. The purpose of your user test and the procedure that you will be using
    2. Any potential benefits and risks that they might receive from their participation
    3. Their right to withdraw from the user test at any point during the test
    4. How you will protect their identity (e.g. confidentiality of any published results)

NOTE 1: You should select your participants to represent a wide range of possible user characteristics. You will be expected to report the characteristics that apply to each user, but not the names or other personally identifying information about them. You should also consider how these characteristics relate to results in your analysis of results. 

NOTE 2: You may not use students from this class as participants in your usability tests.

  1. Individually conduct the user tests according to your plans.

NOTE 1: Do not wait too long to complete this, there is still a lot to do after the tests are completed.

NOTE 2: If you do not get useful results from an individual test subject, you should recruit an additional test subject and conduct an additional test so that you have 4 sets of useful results.

Reporting Requirements

Develop a well-structured and well-written report of your user testing that takes into account the marking structure (below), with the following sections:

    1. The results of your identifying further information (from step 1 above)
      1. Develop a short (less than 1 page in length for cmpt 479 students and less than 2 pages in length for cmpt 817 students) report on what you learned about user testing from the sources that you consulted which provided you ideas that were beyond the information presented in the text. Be user to use proper academic referencing to indicate which source(s) each idea came from. (it is expected that you will get some unique information from each source as well as some information that might be found in multiple sources. 
      2. Make sure that each your discussion of each idea includes:
        1. a unique name for the idea
        2. a brief description of the idea
        3. a brief description of how it can be applied to this assignment
        4. reference(s) to the source of the idea
      3. Also include a suitable bibliography that properly cites the sources of the ideas.
    1. The planning and conducting of the testing, including:
      1. Your plan for the testing (from step 2 above)
NOTE: You should formally acknowledge, with suitable academic references, the sources of information that helped you with constructing this plan

      1. Descriptions of the tasks and scenarios (from step 2 above) and discussions of how you will obtain the measures you chose to observe / measure during the testing (including any that will be asked in a pre-testing and/or a follow-up interview/questionnaire).
NOTE: This should include the actual wording of the scenarios that you provided to your research participants and any testing or interview scripts / questionnaires used

      1. Brief descriptions of the characteristics of each of your research participants, without actually identifying them. (from step 3 above)
      2. A brief discussion of how the testing was conducted and any changes to your initial plans that you had to make.
      3. A brief discussion of how you analyzed / evaluated the results.
    1. The results of the testing. Results are objective statements of what measures you obtained from the testing. 
      1. Results should be presented on their own without any personal discussions from the person doing the testing. 
      2. Results should be organized and formatted so that it is easy to read and understand them 
    1. The analysis of the results. Analysis of results involves 
      1. identifying / combining / summarizing potentially important results
      1. the commenting on these important results in terms of both
        1. what you think the results mean and 
        2. what should be done about them.
    1. What you have learned about the differences between user testing and expert evaluations. 
      1. This should start with a comparison of your results from this user testing with the results from the expert evaluation you conducted in Assignment 1. 
      2. This needs to go beyond presenting this comparison of results from the two approaches to include an analysis of what this means to you., in terms of which method to use to get what types of information in what situations.
NOTE : This needs to focus on the types of results you obtained (i.e. the types of good and bad features identified by the two approaches) and not the approaches themselves. From this you should be able to differentiate what expert evaluation is good for and what user testing is good for and what either one might be equally good for.
Be especially careful that you meet the expectations identified in the marking structure (below)!


Submit your assignment electronically via e-mail to Prof. Carter by noon on Thursday, October 20, 2016

Marking Structure

Total marks available:
In order to be eligible to receive full marks:
 The following marking structure will be used:
NOTE 1:  It is important that each of the {3 or 5} sources you use provides at least one unique item of information. If it does not then the maximum possible marks will be reduced by 2 marks for each source that does not provide something unique.
NOTE 2: each of the items in this section of the report will get {2 or 1 or 0 marks}  for  (name, description, and source) and for (discussion of how it applies)
2 marks for very good to excellent quality work
1 mark for adequate quality work
0 marks for unacceptable quality work
NOTE 1: for each of these marks of {5, 4, 3, 2, 1, or 0} will be assigned depending on the quality and completeness (in terms of content) of what you have produced.
NOTE 2: the marks on this section of the report are not subject to being diminished based on the actual testing involving less than the required minimums.
NOTE: the marks on this section of the report are subject to being diminished based on the actual testing involving less than the required minimums.
NOTE 1: for each of these marks of {5, 4, 3, 2, 1, or 0} will be assigned depending on the quality and completeness (in terms of content) of what you have produced.
NOTE 2: the marks on this section of the report are subject to being diminished based on the actual testing involving less than the required minimums.
NOTE: each of the significant items in this section of the report will get {2 or 1 or 0 marks}
2 marks for very good to excellent quality work
1 mark for adequate quality work
0 marks for unacceptable quality work