The ATLAST student assessments have strong evidence of reliability and validity.
Reliability
Using data from the final field test, the internal reliability of each assessment was calculated using both a classical approach (Cronbach’s alpha) and Item Response Theory (IRT). The IRT internal reliability for each student assessment is given below:
- Flow of Matter and Energy: .78
- Force & Motion: .75
- Plate Tectonics: .86
The Cronbach alphas were similarly high.
Validity
Three lines of evidence support the argument that the assessments are valid measures of students’ knowledge of force and motion ideas. First, cognitive interviews with students (see Writing Items and Cognitive Interviews) established that students interpret the items as intended and that students must use their knowledge of content to answer the items correctly. Second, a panel of three content experts (e.g. individuals with a Ph.D. in physics) reviewed the assessment items (see Expert Review) at three stages (see Student Assessment Development) to ensure content accuracy. They also reviewed the final assessment and judged it to be an adequate measure of the content domain.
Finally, dimensionality analyses (including both factor analysis and cluster analysis) indicate that all items on the Plate Tectonics and Flow of Matter and Energy assessments measure a single dominant trait. HRI termed this trait “content knowledge.” These analyses indicate that both a both a 1-factor and a 2-factor solution were supported for the Force and Motion assessment. We chose the 1-factor solution, with all items on the assessment measuring “content knowledge” as a single dominant trait.