PTAM 2014-3

ABSTRACTS

Evaluation of the chi²-statistic and different fit-indices under misspecified number of factors in confirmatory factor analysis
Michael Themessl-Huber

Abstract
Model evaluation is one of the most important parts in confirmatory factor analysis. There are different criteria to evaluate the fit of a model.
Using data simulation the type-I-risk and the type-II-risk of the chi²-statistic were investigated. Therefore, a correct specified model and two models with misspecified number of factors were tested under different simulation conditions. The behavior of the standardized root-mean-square residual (SRMR), the root-mean-square error of approximation (RMSEA) and the comparative fit index (CFI) were also investigated. Cut-off values provided by Hu and Bentler (1999) were used for these fit-indices. To compare different models the chi²-difference-test or the F-statistic (Kubinger, Litzenberger, & Mrakotsky, 2006) can be used. The behavior of these methods was investigated too.
It was shown, that the chi²-test did not hold the type-I-risk of 5 %. For the SRMR and the RMSEA different cut-off values should be used under present misspecification. The cut-off values for the CFI seem to be adequate. The F-test is an alternative to the chi²-difference-test. It has the advantage, that it can be also used even if models are non-nested.

Keywords: confirmatory factor analysis, fit-indices, chi-square statistics, model comparison, F-test

Michael Themessl-Huber, PhD
Faculty of Psychology
University of Vienna, Austria
michael.themessl-huber@univie.ac.at

Measured attention in prolonged over-learned response tasks and its correlation to high level scientific reasoning and school achievement
R. Hotulainen, H. Thuneberg, J. Hautamäki, M.-P. Vainikainen

Abstract
The relationship between attention and academic performance has been of interest starting with early studies on academic success and failure. In this study we examine how attention measured in simple and prolonged over-learned response tasks correlates with and contributes to scientific reasoning and school achievement (GPA). To study attention, the Attention Concentration Test (ACT) was used and to study scientific reasoning, a modified version of Science Reasoning Tasks, tapping control-of-variable schemata, was used. Of special interest was the highest performing attention group (+ 1 SD) formed from the ACT results. We gathered our data from Finnish ninth graders (n = 358; including 166 girls) from the city located in the eastern part of Finland. Statistically analysed results showed that attention contributed to scientific reasoning, which in turn explained the largest share of the GPA variance. The highest attention group differed from the lowest two attention groups in GPA and from all attention groups in scientific reasoning. For educational practitioners the ACT seems to be a useful tool in assessing exceptional academic learning potential in students.

Keywords: attention, prolonged over-learned working task, scientific reasoning, school achievement, high ability

Risto Hotulainen, PhD
Educational Assessment
University of Helsinki
P.O. Box 9 (Siltavuorenpenger 3A)
FI-00014 Helsinki, Finland
Risto.Hotulainen@Helsinki.fi

Optimal item pool design for computerized adaptive tests with polytomous items using GPCM
Xuechun Zhou  & Mark D. Reckase

Abstract
Computerized adaptive testing (CAT) is a testing procedure with advantages in improving measurement precision and increasing test efficiency. An item pool with optimal characteristics is the foundation for a CAT program to achieve those desirable psychometric features. This study proposed a method to design an optimal item pool for tests with polytomous items using the generalized partial credit model (G-PCM). It extended a method for approximating optimality with polytomous items being described succinctly for the purpose of pool design. Optimal item pools were generated using CAT simulations with and without practical constraints of content balancing and item exposure control. The performances of the item pools were evaluated against an operational item pool. The results indicated that the item pools designed with stratification based on discrimination parameters performed well with an efficient use of the less discriminative items within the target accuracy levels. The implications for developing item pools are also discussed.

Keywords: computerized adaptive testing, item pool design, G-PCM, p-optimality method

Xuechun Zhou, PhD
19500 Bulverde Road
San Antonio, TX 78259, USA
Xuechun.Zhou@pearson.com

Detection of Differential Item Functioning in the Cornell Critical Thinking Test across Korean and North American students
Brian F. French, Brian Hand, Jeonghee Nam, Hsiao-Ju Yen & Juan Antonio Valdivia Vazquez

Abstract
Critical thinking can be considered the conscious process a person does when he or she explores a situation or a problem from different perspectives. Accurate measurement of critical thinking skills, especially across cultures, in part, depends on the instrument’s measurement properties being invariant or similar across those groups. The assessment of item level invariance is a critical component of building a validity argument to ensure scores on the Cornell Critical Thinking Test (CCTT) have similar meaning across cultures. Item Response Theory methods were used to examine differential item functioning by culture in the CCTT-Form X. Results suggest that the items do function similarly across Korean and United States students with only 14% of items displaying DIF. Additionally, the majority of these DIF items appeared on one of the subscales. Practitioners and researchers can have confidence that mean differences observed at the total score level or on 3 of 4 subscales are not a function of a lack of measurement invariance. Findings support the validity of the inferences drawn when comparing students from these two countries.

Keywords: Differential Item Functioning, Critical Thinking, Invariance, Culture, Korea

Brian F. French, Ph.D.
Cleveland Hall, Rm 362
College of Education
Washington State University,
Pullman, Washington 99163, USA
frenchb@wsu.edu

Analysing latent constructs via a derived metric paired comparison approach: An application to students’ emotions in mathematics
Alexandra Grand

Abstract
In this article we suggest an approach for the analysis of sets of items using the method of paired comparisons. We applied the proposed approach to a students’ survey of emotions typically experienced while learning mathematics by focusing on the relative dominance of these emotions. The emotions of interest were: enjoyment, pride, anger, anxiety, boredom and shame which were each measured by a set of items and for which we want to obtain an ordering on a continuum. In a first step we evaluated the quality of items by using a method of non-parametric Rasch model tests. The item sets of the emotions enjoyment, anxiety and boredom met the properties of a Rasch model. As a result of fitting Rasch models, we obtained person “emotion” parameter estimates. We then derived for each individual metric paired comparison responses from the obtained person parameter estimates and directly modelled these derived relative responses by fitting a beta regression model. This model is similar to generalized linear models (GLMs). The proposed model accounts for bounded metric paired comparison data in (0,1) where subject covariates and object-specific covariates can also be incorporated. We found that there is a tendency, the higher the positive discrepancy between the self concept of maths ability and the averaged perceived maths ability of students the more enjoyment and the less anxiety is typically experienced while learning mathematics.

Keywords: beta regression model, latent constructs, logistic Bradley-Terry model, metric paired comparisons, Rasch model

Mag. Alexandra Grand
Institute for Statistics and Mathematics
Vienna University of Economics and Business (WU)
Welthandelsplatz 1
1020 Vienna, Austria
alexandra.grand@gmx.at

The Self Group Distinction Scale: A new approach to measure individualism and collectivism in adolescents
Takuya Yanagida, Dagmar Strohmeier, Yuichi Toda & Christiane Spiel

Abstract
Individualism/Collectivism (I/C) was defined as a group orientation characterized by the degree of the convergence of an individual’s opinion with an anchor group opinion. The Self Group Distinction (SGD) Scale as a new measurement using difference scores was developed. In sum, 532 Japanese adolescents with a mean age of 12.3 years (SD = 1.78 years) and 277 Austrian with a mean age of 11.96 years (SD = 1.81 years) were asked to indicate their own and the perceived class opinion with respect to seven items covering different aspects of I/C. Confirmatory factor analyses of difference scores demonstrated scalar measurement invariance between cultural groups. Validity was demonstrated by a smaller self-group distinction in Japanese compared with Austrian adolescents.

Keywords: individualism, collectivism, cultural differences, cross-cultural, measurement

Mag. Takuya Yanagida
School of Applied Health and Social Sciences
University of Applied Sciences Upper Austria
Garnisonstr. 21
4020 Linz, Austria
takuya.yanagida@fh-linz.at