Cognitive Interview Protocol

Cognitive interviews with teachers allowed us to determine if teachers interpreted the items as intended and if teachers had to use their knowledge of content to answer the items correctly. The interview protocol is shown below.

Prologue:

We are developing a test for middle school teachers, and we need your help to improve the questions on the test.  I don’t expect you to answer all of the questions correctly.  If you get a few wrong, it will help me know whether we have written the items well.  Remember, the point is to help us write a good test, not to test what you do or don’t know.  Do you have any questions before we get started?  Remember that all of your answers are confidential.

Procedure

  • Ask teachers to read aloud and ”think aloud“ as they read the questions and answer choices, if they are comfortable doing so.
  • For each multiple-choice item, ask:
  1. Why did you choose that answer?  (probe for words or diagrams they keyed in on, as well as their thinking behind the response)
  2. Were there other answer choices that you almost chose? (why?)
  3. Were there any answer choices that you did not even consider? (why?)
  4. Was there an answer choice you were expecting to see, but did not?  What was it?
  5. Were there any words or diagrams you did not really understand, or situations that made the question confusing?
  6. Does this scenario seem realistic/plausible?
  7. Do you have any other comments on the item?