Item Consistency Index: An Item-Fit Index for Cognitive Diagnostic Assessment

Hollis Lai, Mark J. Gierl, Ying Cui, Oksana Babenko

Abstract


An item-fit index is a measure of how accurately a set of item responses can be predicted using the test design model. In a diagnostic assessment where items are used to evaluate student mastery on a set of cognitive skills, this index helps determine the alignment between the item responses and skills that each item is designed to measure. In this study, we introduce the Item Consistency Index (ICI), a modification of an existing person-model fit index, for diagnostic assessments. The ICI can be used to evaluate item-model fit on assessments designed with a Q-matrix.  Results from both a simulation and real data study are presented. In the simulation study, the ICI identified poor-fitting items under three manipulated conditions: sample size, test length, and proportion of poor-fitting items. In the real-data study, the ICI detected three poor-fitting items for an operational diagnostic assessment in Grade 3 mathematics. Practical implications and future research directions for the ICI are also discussed.


Keywords


Item Consistency Index (ICI); cognitive diagnostic assessment (CDA); test development

Full Text:

PDF

References


References

Bock, R. (1972). Estimating item parameters and latent ability when responses are scored in two or more nominal categories. Psychometrika, 37, 29-51.

Cui, Y., & Leighton, J. (2009). The hierarchy consistency index: Evaluating person fit for cognitive diagnostic assessment. Journal of Educational Measurement, 46(4), 429-449.

Cui, Y, & Li, J. C.-H. (2014). Evaluating person fit for cognitive diagnostic assessment. Applied Psychological Measurement, 39, 223-238.

Cui, Y, & Mousavi, A. (2015). Explore the usefulness of person-fit analysis on large scale assessment. International Journal of Testing, 15, 23-49.

Gierl, M., Leighton, J., & Hunka, S. (2007). Using the attribute hierarchy method to make diagnostic inferences about examinees’ cognitive skills. In J. Leighton & M. Gierl (Eds.), Cognitive diagnostic assessment for education: Theory and applications (pp. 242-274). Cambridge, MA: Cambridge University Press.

Gierl, M., Cui, Y., & Zhou, J. (2009). Reliability and attribute-based scoring in cognitive diagnostic assessment. Journal of Educational Measurement, 46(3), 293-313.

Gierl, M., Alves, C., & Taylor-Majeau, R. (2010). Using the Attribute Hierarchy Method to Make Diagnostic Inferences about Examinees’ Knowledge and Skills in Mathematics: An Operational Implementation of Cognitive Diagnostic Assessment. International Journal of Testing, 10(4), 318-341.

Jang, E. (2005). A validity narrative: Effects of reading skills diagnosis on teaching and learning in the context of NG TOEFL (Doctoral dissertation). University of Illinois at Urbana-Champaign, IL, USA.

Orlando, M., & Thissen, D. (2003). Further investigation of the performance of S-X2: An item fit index for use with dichotomous item response theory models. Applied Psychological Measurement, 27(4), 289-298.

R Development Core Team (2011). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. .

Reise, S. (1990). A Comparison of item- and person-fit methods of assessing model-data fit in IRT. Applied Psychological Measurement, 14(2), 127-137.

Rost, J., & von Davier, M. (1994). A conditional item-fit index for Rasch models. Applied Psychological Measurement, 18(2), 171-182.

Sinharay, S., Puhan, G., & Haberman, S. (2009, April). Reporting diagnostic scores: Temptations, pitfalls, and some solutions. Paper presented at the National Council on Measurement in Education, San Diego, CA, USA.

Sinharay, S., & Almond, R. (2007). Assessing fit of cognitive diagnostic models a case study. Educational and Psychological Measurement. 67(2), 239-257.

Wang, C., Shu, Z., Shagn, Z., & Xu, G. (2015). Assessing Item-Level Fit for the DINA Model. Applied Psychological Measurement, 1-14.

Yen, W. (1981). Using simulation results to choose a latent trait model. Applied Psychological Measurement, 5, 245-262.


Refbacks

  • There are currently no refbacks.


e-ISSN: 1694-2116

p-ISSN: 1694-2493