Cognitive Interview protocol

Cognitive interviews with students allowed us to determine if they interpreted the items as intended and if students had to use their knowledge of content to answer the items correctly. The interview protocol is shown below.

Prologue:

We are developing a test for middle school students, and we need your help to get the questions on the test just right.  I don’t expect you to answer all of the questions correctly.  If you get a few wrong, it will help me know whether we have written the items well.  Remember, the point is to help us write a good test, not to test what you do or don’t know.  Do you have any questions before we get started?  Remember that all of your answers are confidential.

Procedure

  • Ask students to read aloud and ”think aloud“ as they read the questions and answer choices, if they are comfortable doing so.
  • For each multiple-choice item, ask:
  1. Why did you choose that answer?  (probe for words or diagrams they keyed in on, as well as their thinking behind the response)
  2. Were there other answer choices that you almost chose? (why?)
  3. Were there any answer choices that you did not even consider? (why?)
  4. Was there an answer choice you were expecting to see, but did not?  What was it?
  5. Were there any words or diagrams you did not really understand, or situations that made the question confusing?
  6. Do you have any other comments on the item?