Instrument development included two administrations to samples of students nationwide. Pilot testing provided information about the difficulty of items and resulted in deleting some items and revising others. An additional round of cognitive interviews and revisions produced items used in the field tests. Field testing was done primarily to create the final scales. For both the pilot test and the field test, HRI recruited middle school science teachers to administer the items to at least one of their classes. These processes were varied to meet the needs for the content areas.
Force and Motion
Pilot Test
A pool of 34 force and motion student items were included in the initial pilot. Slightly fewer than 1,600 middle school students nationally responded. Analysis of the data indicated that the items were overall quite difficult. Therefore, approximately 20 new and intentionally easier items were written as a result. These new and revised items were developed through the same process as the original items.
Field Test
Using results from the pilot test and the newly-developed items, 50 items were selected for the second field test with just over 5,000 students nationally. As the pool of nearly 50 items was too large to administer in a single sitting, we created two forms with 32 items each; 16 items were common to each form.
Plate Tectonics and Flow of Matter & Energy in Living Systems
Pilot Test
A pool of approximately 50 items was included in the initial pilot for each content area. The items were spread across two forms with linking items. About 4,000 middle school students across the nation responded to each pilot form. We conducted classical and IRT analyses on both sets of pilot data.
Field Test
Using results from the pilot test, approximately 35 items were selected for the field test, based on psychometric properties and content coverage. The selected items were administered to approximately 4,600 (Flow of matter & Energy) and 4,000 (Plate Tectonics) middle school students nationally.