Abstract

Before items can be implemented in a test, the item characteristics need to be calibrated through pretesting. To achieve high-quality tests, it's crucial to maximize the precision of estimates obtained during item calibration. Higher precision can be attained if calibration items are allocated to examinees based on their individual abilities. Methods from optimal experimental design can be used to derive an optimal ability-matched calibration design. However, such an optimal design assumes known abilities of the examinees. In practice, the abilities are unknown and estimated based on a limited number of operational items. We develop the theory for handling the uncertainty in abilities in a proper way and show how the optimal calibration design can be derived when taking account of this uncertainty. We demonstrate that the derived designs are more robust when the uncertainty in abilities is acknowledged. Additionally, the method has been implemented in the R-package optical.

Abstract

Integer programming (IP) is an extension of linear programming (LP) whereby the goal is to determine values for a set of decision variables (some or all of which have integer restrictions) so as to maximize or minimize a linear objective function of the variables subject to a set of linear constraints involving the variables. Although the psychological literature is replete with applications of multivariate statistics, implementations of mathematical modelling methods such as IP are comparatively far fewer. Nevertheless, over the decades, there have been a variety of important applications and the vast majority of these fall within the IP rather than the LP category. In this paper, we offer a brief overview of the history of IP methodology. We subsequently review some domains where IP has been gainfully applied in psychology, such as test assembly, cluster analysis and classification and seriation and unidimensional scaling. An illustrative example of using IP to cluster respondents measured on items pertaining to substance abuse disorder is provided. Finally, we identify areas where IP might be applied in emerging areas of psychology, such as in the domain of network psychometrics.

Abstract

Understanding students' learning trajectories is crucial for educators to effectively monitor and enhance progress. With the rise of computer-based testing, researchers now have access to rich datasets that provide deeper insights into student performance. This study introduces a general dynamic learning model framework that integrates response accuracy and response times to capture different test-taking behaviors and estimate learning trajectories related to polytomous attributes over time. A Bayesian estimation method is proposed to estimate model parameters. Rigorous validation through simulation studies confirms the effectiveness of the MCMC algorithm in parameter recovery and highlights the model's utility in understanding learning trajectories and detecting different test-taking behaviors in a learning environment. Applied to real data, the model demonstrates practical value in educational settings. Overall, this comprehensive and validated model offers educators and researchers nuanced insights into student learning progress and behavioral dynamics.

Abstract

Nowadays, multidimensional data are often available from educational testing. One natural issue is to identify whether more dimensional data are useful in fitting the item response data. To address this important issue, we develop a new decomposition of Widely Applicable Information Criterion (WAIC) via the posterior predictive ordinate (PPO) under the joint model for the response, response time and two additional educational testing scores. Based on this decomposition, a new model assessment criterion is then proposed, which allows us to determine which of the response time and two additional scores are most useful in fitting the response data and whether other dimensional data are further needed given that one of these dimensional data is already included in the joint model with the response data. In addition, an efficient Monte Carlo method is developed to compute PPO. An extensive simulation study is conducted to examine the empirical performance of the proposed joint model and the model assessment criterion in the psychological setting. The proposed methodology is further applied to an analysis of a real dataset from a computerized educational assessment program.

British Journal of Mathematical and Statistical Psychology, EarlyView.

Abstract

Item response theory models are commonly adopted in educational assessment and psychological measurement. Such models need to be modified to accommodate practical situations when statistical sampling assumptions are violated. Omission is a common phenomenon in educational testing. In modern computer-based testing, we have not only examinees' responses but also their response times. This paper utilizes response time and develops a joint model of responses and response times. The new approach is analogous to those developed in survival analysis for dealing with right-censored data. In particular, a key ingredient is the introduction of the omission time (OT), which corresponds to the censoring time in survival analysis. By competing risk formulation, the proposed method provides an alternative narrative to how an item becomes answered versus omitted, depending on the competing relationship of response time and OT, so that the likelihood function can be constructed properly. The maximum likelihood estimator can be computed via the expectation-maximization algorithm. Simulation studies were conducted to evaluate the performance of the proposed method and its robustness against various mis-specifications. The method was applied to a dataset from the PISA 2015 Science Test.

Abstract

Recent technological advancements have enabled the collection of intensive longitudinal data (ILD), consisting of repeated measurements from the same individual. The threshold autoregressive (TAR) model is often used to capture the dynamic outcome process in ILD, with autoregressive parameters varying based on outcome variable levels. For ILD from multiple individuals, multilevel TAR (ML-TAR) models have been proposed, with Bayesian approaches typically used for parameter estimation. However, fitting ML-TAR models can be computationally challenging. This study introduces a mean-field variational Bayes (MFVB) algorithm as an alternative to traditional Bayesian inference. By optimizing to approximate posterior densities, variational Bayes aims to find the best approximation within a defined set of distributions. Simulation results demonstrate that our MFVB algorithm is significantly faster than the standard Markov chain Monte Carlo (MCMC) approach. Moreover, increasing the number of individuals or time points enhances the accuracy of the parameter estimates using MFVB, suggesting that sufficient data are crucial for accurate estimation in complex models like ML-TAR models. When applied to real-world data, the MFVB algorithm was significantly more efficient than MCMC and maintained similar accuracy. Thus, the MFVB algorithm is a faster and more consistent alternative to MCMC for large-scale inference in ILD models.

Abstract

In various areas of science, researchers try to gain insight into important processes by jointly analysing different datasets containing information regarding common aspects of these processes. For example, to explain individual differences in personality, researchers collect, for the same set of persons, data regarding behavioural signatures (i.e., the reaction profile of a person across different situations), on the one hand, and traits or dispositions, on the other hand. To uncover the processes underlying such coupled data, to all N-way N$$ N $$-mode data blocks simultaneously a global model is fitted, in which each data block is represented by an N$$ N $$-way N$$ N $$-mode decomposition model (e.g., principal component analysis [PCA], Parafac, Tucker3) and the parameters underlying the common mode are required to be the same for all data blocks this mode belongs to. To estimate the parameters underlying the common mode, a simultaneous strategy is used that pools the information present in all data blocks (i.e., data fusion). In this paper, we propose the T3–PCA model, which represents three- and two-way data with Tucker3 and PCA respectively. This model is less restrictive than the already proposed LMPCA model in which the three-way data block is decomposed according to a Parafac model. To estimate the T3–PCA model parameters, an alternating least-squares algorithm is proposed. The superior performance of the simultaneous T3–PCA strategy over a sequential strategy (i.e., estimating common parameters using information from the three-way data block only) is demonstrated in an extensive simulation study and an application to empirical coupled anxiety data.

Abstract

This paper provides a literature review of assessment of fit of item response theory models. Various types of fit procedures for item response theory models are reviewed, with a focus on their advantages and disadvantages. Real data examples are used to demonstrate some of the fit procedures. Recommendations are provided for researchers and practitioners who are interested in assessing the fit of item response theory models.

British Journal of Mathematical and Statistical Psychology, Volume 78, Issue 1, Page i-iv, February 2025.