Measuring Learning Results: Eight Recommendations for Assessment Designers

Posted by Joan Phaup

Is it possible to build the perfect assessment design? Not likely, given the intricacies of the learning process! But a white paper available on the Questionmark Web site helps test authors respond effectively to the inevitable tradeoffs in order to create better assessments.

Measuring Learning Results, by Dr. Will Thalheimer of Work-Learning Research, considers findings from fundamental learning research and how they relate to assessment. The paper explores ...

how to create assessments that measure how well learning interventions are preparing learners to retrieve information in future situations—which as Will states it is the ultimate goal of training and education.

The eight bits of wisdom that conclude the paper give plenty of food for thought for test designers. You can download the paper to find out how Will arrived at them.

1. Figure out what learning outcomes you really care about. Measure them. Prioritize the importance of the learning outcomes you are targeting. Use more of your assessment time on high-priority information.

2. Figure out what retrieval situations you are preparing your learners for. Create assessment items that mirror or simulate those retrieval situations.

3. Consider using delayed assessments a week or month (or more) after the original learning ends—in addition to end-of-learning assessments.

4. Consider using delayed assessments instead of end-of-learning assessments, but be aware that there are significant tradeoffs in using this approach.

5. Utilize authentic questions, decisions, or demonstrations of skill that require learners to retrieve information from memory in a way that is similar to how they’ll have to retrieve it in the retrieval situations for which you are preparing them. Simulation-like questions that provide realistic decisions set in real-world contexts are ideal.

6. Cover a significant portion of the most important learning points you want your learners to understand or be able to utilize. This will require you to create a list of the objectives that will be targeted by the instruction.

7. Avoid factors that will bias your assessments. Or, if you can’t avoid them, make sure you understand them, mitigate them as much as possible, and report their influence. Beware of the biasing effects of end-of-learning assessments, pretests, assessments given in the learning context, and assessment items that are focused on low-level information.

8. Follow all the general rules about how to create assessment items. For example, write clearly, use only plausible alternatives (for multiple-choice questions), pilot-test your assessment items to improve them, and utilize psychometric techniques where applicable.

Related posts:Feedback in Questionmark LivePodcast: Dr. Will Thalheimer on the Use of FeedbackPsychometrics 101: How do I know if an assessment is reliable? (Part 2)

Hide