Project Evaluation
Due Date: 12:00 noon, Mon, Nov. 25, 2019
The evaluation of each
project will depend on the project being evaluated and the level of the
course (480 or 840 in which the student is registered). Students are
advised to discuss their plans for evaluation with the instructor to
ensure that they are suitable.
NOTE:
there are many similarities between what is needed for a good
evaluation report and the various published research papers that you
should have referred to in your analysis
Project Evaluation
While it is possible to
set up user testing of products (such as ADEs), without having products
to test the most appropriate type of testing is often expert
evaluation. Ideally it would be nice to have your analysis and design
reports evaluated by a real expert who is independent from the class.
However, this might be too time consuming to obtain in time for you to
complete an evaluation report by the due date.
The role of
evaluation in this project, is to recognize that the intial analysis
and design can benefit from iterative improvements. Thus, in addiiton
to evaluating challenges and opportinutues, it is important to beab le
to take action on these challenges and opportunities to improve your
project.
Therefore I am suggesting a three part procedure for you to do to perform a suitable evaluation:
- Instructor
Evaluation - the instructor will evaluate your analysis and design and
suggest challenges and opportunities that need your attention. This
instructor-based evaluation will focus on identifying the challenges
and opportunities and not necessarily on suggesting what needs to be
done about them.
- Self Evaluation - you should critically
evaluate your analysis and design to identify challenges and
opportunities that need to be addressed. This could include identifying
further useful information that could be used to improve your project.
- Proof
by Example - in addition to parts 1 and 2, you should create examples
to illustrate that your design works as it is intended to work. In
creating examples you will either prove that it can work or that it
needs improvement. If it needs improvement, then you should also make
the improvements and create examples to show that the improvements make
it work.
Your evaluation report should contain:
- A report on what you have done, including:
- A
discussion of the instructor based evaluation and what you have done
about it to improve your project. This can be done using the same
format as for class critiques.
- A discussion of your own critical analysis of the project. This also can be done using the same format as for class critiques.
- A
discussion of how you went about creating and using examples and of
what you found from using them, including any challenges or
opportunities thta they identified.
- A revised version of your analysis and design report, based on what you ahve done.
- changes
should be easily identified by the instructor (preferably using Word's
track changes feature) and should include a brief comment (either in
italics and parenthesis or using Word's comment feature)
- the examples should be included in the design portion of the report in an easy to use manner.
Background (used other years)
ReDesign
In order to have something to evaluate you require a suitable design (and implementation - if applicable).
Many
students might need to revise their design from the analysis and design
to take into account any important concerns provided in the marking of
your analysis and design report. Those marking comments provide a first
level of evaluation and need to be taken into account before you have a
suitable design/implementation to use in this phase of your project.
The
first section of your evaluation report needs to identify the comments
received in the marking of your design and explain what was done to
address them. If there were serious problems with your design, you
should also provide a revised version of your design as part of this
section of your evaluation report.Implementation (required for CMPT 840)
CMPT 840 students are required to report on their implementation as part of their report for this stage of the class project.
See below for details.
The particular way of developing your implementation will depend on
your project.
- It is important that you clearly explain what you have done to
implement your design (e.g. building a system or performing an experiment).
- It is also important to explain why you did what
you did and to discuss what worked and what did not work.
- Remember to
focus on those aspects of your project that have the most to do with
the material from this class.
In addition to actually implementing the design, you need to provide a report on the implementation, that should include:
- a point form list of the major implementation activities / decisions that you made in order to implement your system
- any other material that will help to describe your implementation
- e.g. for implementations involving designing computer software: a
set of screen captures and accompanying discussion of how these screens
are used to work with your system. It is recommended that this be in
the form of a tutorial that could be given to a user so that the user
would understand clearly how to use the system.
Evaluation
It is important that you find a way of evaluating your design/implementation.
- In some cases (especially where this does not involve an implementation) this can be done by selecting suitable test cases
and explaining how your project would deal with them. Remember, that
suitable involves enough to establish the range over which your project
works, as well as where it is not robust enough to work. If all your
test cases work fine, you might not have enough test cases.
- In
other cases (especially where there is an ADE or other tool) you will want to do some actual user testing. Remember to
follow the ethical expectations related to the use of human test
subjects.
Your evaluation report should include:
- a
report on what you did to your design to satisfy the evaluation
provided in the marking of your analysis and design report (as
discussed above)
- [CMPT 840 students only: a report on your implementation]
- a point form list of the major evaluation activities you conducted
- a set of the data that you collected from these activities
- an analysis of this data and the conclusions that you arrived at from this analysis
- a set of recommendations for improvements that could be made to your implementation
Additional
information on different forms of evaluation including user
testing and expert evaluation is contained in Chapter 16 of the course
text.
Students who are unsure of what will be a suitable
evaluation method for their project should discuss their ideas with the
instructor as soon as possible.
Evaluation of an ADE should involve actual user testing.
- Remember that the intended audience of an ADE are average people who
are interested in learning about accessibiltiy and are not
accessibility experts or persons with specific disabilities.
- Testing should involve a diverse set of at least 5 test subjects
- Remember to
follow the ethical expectations related to the use of human test
subjects.
CMPT 840 Evaluation Report
Your evaluation report should include:
- Redesign: a
report on what you did to your design to satisfy the feedback
provided in the marking of your analysis and design report (as
discussed above)
- Implementation: a report on your implementation (if applicable)
- a point form list of the major implementation activities / decisions that you made in order to implement your system
- any other material that will help to describe your implementation
- e.g. for implementations involving designing computer software: a
set of screen captures and accompanying discussion of how these screens
are used to work with your system. It is recommended that this be in
the form of a tutorial that could be given to a user so that the user
would understand clearly how to use the system.
- It is important that you clearly explain what you have done to
implement your design (e.g. building a system or performing an experiment).
- It is also important to explain why you did what
you did and to discuss what worked and what did not work.
- Remember to
focus on those aspects of your project that have the most to do with
the material from this class.
- Evaluation Activities: a point form list of the major evaluation activities you conducted
- Evalaution Results: a set of the data that you collected from these activities
- Evaluation Analysis: an analysis of this data and the conclusions that you arrived at from this analysis
- Recommendations: a set of recommendations for improvements that could be made to your implementation
- NOTE: There are always improvements that could be made, whether or not there is the time and resources to make them
Additional
information on different forms of evaluation including user
testing and expert evaluation is contained in Chapter 16 of the course
text.
Copyright © 2010, 2016 - Jim A Carter Jr