Psychological Methods, Mar 10, 2025, No Pagination Specified; doi:10.1037/met0000746

While online survey data collection has become popular in the social sciences, there is a risk of data contamination by computer-generated random responses (i.e., bots). Bot prevalence poses a significant threat to data quality. If deterrence efforts fail or were not set up in advance, researchers can still attempt to detect bots already present in the data. In this research, we study a recently developed algorithm to detect survey bots. The algorithm requires neither a measurement model nor a sample of known humans and bots; thus, it is model agnostic and unsupervised. It involves a permutation test under the assumption that Likert-type items are exchangeable for bots, but not humans. While the algorithm maintains a desired sensitivity for detecting bots (e.g., 95%), its classification accuracy may depend on other inventory-specific or demographic factors. Generating hypothetical human responses from a well-known item response theory model, we use simulations to understand how classification accuracy is affected by item properties, the number of items, the number of latent factors, and factor correlations. In an additional study, we simulate bots to contaminate real human data from 35 publicly available data sets to understand the algorithm’s classification accuracy under a variety of real measurement instruments. Through this work, we identify conditions under which classification accuracy is around 95% or above, but also conditions under which accuracy is quite low. In brief, performance is better with more items, more categories per item, and a variety in the difficulty or means of the survey items. (PsycInfo Database Record (c) 2025 APA, all rights reserved)

Psychological Methods, Mar 06, 2025, No Pagination Specified; doi:10.1037/met0000748

Psychological and behavioral scientists develop interventions toward addressing pressing societal challenges. But such endeavors are complicated by treatments that change over time as individuals’ needs and responses evolve. For instance, students initially in a multiyear mentoring program to improve future academic outcomes may not continue with the program after interim school engagement improves. Conventional interventions bound by rigid treatment assignments cannot adapt to such time-dependent heterogeneity, thus undermining the interventions’ practical relevance and leading to inefficient implementations. Dynamic treatment regimes (DTRs) are a class of interventions that are more tailored, relevant, and efficient than conventional interventions. DTRs, an established approach in the causal inference and personalized medicine literature, are designed to address the causal query: how can individual treatment assignments in successive time points be adapted, based on time-evolving responses, to optimize the intervention’s effectiveness? This tutorial offers an accessible introduction to DTRs using a simple example from the psychology literature. We describe how, using observational data from a single naturally occurring longitudinal study, to estimate the outcomes had different DTRs been counterfactually implemented. To improve accessibility, we implement the estimation procedure in lavaan, a freely available statistical software popular in psychology and social science research. We hope this tutorial guides researchers on framing, interpreting, and testing DTRs in their investigations. (PsycInfo Database Record (c) 2025 APA, all rights reserved)

Psychological Methods, Mar 03, 2025, No Pagination Specified; doi:10.1037/met0000747

When multivariate intensive longitudinal data are collected from a sample of individuals, the model-based clustering (e.g., vector autoregressive [VAR] based) approach can be used to cluster the individuals based on the (dis)similarity of their person-specific dynamics of the studied processes. To implement such clustering procedures, one needs to set the temporal order to be identical for all individuals; however, between-individual differences on temporal order have been evident for psychological and behavioral processes. One existing method is to apply the most complex structure or the highest order (HO) for all processes, while the other is to use the most parsimonious structure or the lowest order (LO). Up to date, the impact of these methods has not been well studied. In our simulation study, we examined the performance of HO and LO in conjunction with Gaussian mixture model (GMM) and k-means algorithms when a two-step VAR-based clustering procedure is implemented across various data conditions. We found that (a) the LO outperformed the HO in cluster identification, (b) the HO was more favorable than the LO in estimation of cluster-specific dynamics, (c) the GMM generally outperformed the k-means, and (d) the LO in conjunction with the GMM produced the best cluster identification outcome. We demonstrated the uses of the VAR-based clustering technique using the data collected from the “How Nuts are the Dutch” project. We then discussed the results from all our analyses, limitations of our study, and direction for future research, and meanwhile offered our recommendations on the empirical uses of the model-based clustering techniques. (PsycInfo Database Record (c) 2025 APA, all rights reserved)

Psychological Methods, Mar 03, 2025, No Pagination Specified; doi:10.1037/met0000741

Intensive longitudinal designs allow researchers to study the dynamics of psychological processes in daily life. Yet, because these methods are usually observational, they do not allow strong causal inferences. A promising solution is to incorporate (micro-)randomized interventions within intensive longitudinal designs to uncover within-person (Wp) causal effects. However, it remains unclear whether (or how) the resulting Wp causal effects translate into between-person (Bp) differences in outcomes. In this work, we show analytically and using simulated data that Wp causal effects translate into Bp differences if there are no counteracting forces that modulate this cross-level translation. Three possible counteracting forces that we consider here are (a) contextual effects, (b) correlated random effects, and (c) cross-level interactions. We illustrate these principles using empirical data from a 10-day microrandomized mindfulness intervention study (n = 91), in which participants were randomized to complete a treatment or control task at each occasion. We conclude by providing recommendations regarding the design of microrandomized experiments in intensive longitudinal designs, as well as the statistical analyses of data resulting from these designs. (PsycInfo Database Record (c) 2025 APA, all rights reserved)

Psychological Methods, Mar 03, 2025, No Pagination Specified; doi:10.1037/met0000729

How to evaluate how well a psychological scale measures just one construct is a recurring problem in assessment. We introduce an index, u, of the unidimensionality and homogeneity of a scale. u is just the product of two other indices: τ (a measure of τ equivalence) and ρc (a measure of congeneric fit). By combining these two indices into one, we provide a simple index of the unidimensionality and homogeneity of a scale. We evaluate u through simulations and with real data sets. Simulations of u across one-factor scales ranging from three to 24 items with various levels of factor homogeneity show that τ and, therefore, u are sensitive to the degree of factor homogeneity. Additional tests with multifactorial scales representing 9, 18, 27, and 36 items with a hierarchical factor structure varying in a general factor loading show that ρc and, therefore, u are sensitive to the general factor saturation of a test. We also demonstrate the performance of u on 45 different publicly available personality and ability measures. Comparisons with traditional measures (i.e., ωh, α, ωt, comparative fit index, and explained common variance) show that u has greater sensitivity to unidimensional structure and less sensitivity to the number of items in a scale. u is easily calculated with open source statistical packages and is relatively robust to sample sizes ranging from 100 to 5,000. (PsycInfo Database Record (c) 2025 APA, all rights reserved)

Psychological Methods, Mar 03, 2025, No Pagination Specified; doi:10.1037/met0000724

Bots (i.e., automated software programs that perform various tasks) and fraudulent responders pose a growing and costly threat to psychological research as well as affect data integrity. However, few studies have been published on this topic. (a) Describe our experience with bots and fraudulent responders using a case study, (b) present various bot and fraud detection tactics (BFDTs) and identify the number of suspected bot and fraudulent respondents removed, (c) propose a consensus confidence system for eliminating bots and fraudulent responders to determine the number of BFDTs researchers should use, and (d) examine the initial effectiveness of dynamic versus static BFDT protocols. This study is part of a larger 14-day experience sampling method study with trauma-exposed sexual minority cisgender women and transgender and/or nonbinary people. Faced with several bot and fraudulent responder infiltrations during data collection, we developed an evolving BFDT protocol to eliminate bots and fraudulent responders. Throughout this study, we received 24,053 responses on our baseline survey. After applying our BFDT protocols, we eliminated 99.75% of respondents that were likely bots or fraudulent responders. Some BFDTs seemed to be more effective and afford higher confidence than others, dynamic protocols seemed to be more effective than static protocols, and bots and fraudulent responders introduced significant bias in the results. This study advances online psychological research by curating one of the largest samples of bot and fraudulent respondents and pilot testing the largest number of BFDTs to date. Recommendations for future research are provided. (PsycInfo Database Record (c) 2025 APA, all rights reserved)

Psychological Methods, Mar 03, 2025, No Pagination Specified; doi:10.1037/met0000745

Social network analysis can answer research questions such as why or how individuals interact or form relationships and how those relationships impact other outcomes. Despite the breadth of methods available to address psychological research questions, social network analysis is not yet a standard practice. To promote the use of social network analysis in psychological research, we present an overview of network methods, situating each method within the context of research studies and questions in psychology. (PsycInfo Database Record (c) 2025 APA, all rights reserved)

Psychological Methods, Mar 03, 2025, No Pagination Specified; doi:10.1037/met0000744

Psychological network models often feature communities: subsets of nodes that are more densely connected to themselves than to other nodes. The Spinglass algorithm is a popular method of detecting communities within a network, but it is a nondeterministic algorithm, meaning that the results can vary from one iteration to the next. There is no established method for determining the optimal solution or for evaluating instability across iterations in the emerging discipline of network psychometrics. We addressed this need by introducing and evaluating iterated community detection: Spinglass (IComDetSpin), a method for aggregating across multiple Spinglass iterations to identify the most frequent solution and quantify and visualize the instability of the solution across iterations. In two simulation studies, we evaluated (a) the performance of IComDetSpin in identifying the true community structure and (b) information about the fuzziness of community boundaries; information that is not available with a single iteration of Spinglass. In Study 1, IComDetSpin outperformed single-iteration Spinglass in identifying the true number of communities and performed comparably to Walktrap. In Study 2, we extended our evaluation to networks estimated from simulated data and found that both IComDetSpin and Exploratory Graph Analysis (a well-established community detection method in network psychometrics) performed well and that IComDetSpin outperformed Exploratory Graph Analysis when correlations between communities were high and number of nodes per community was lower (5 vs. 10). Overall, IComDetSpin improved the performance of Spinglass and provided unique information about the stability of community detection results and fuzziness in community structure. (PsycInfo Database Record (c) 2025 APA, all rights reserved)

Psychological Methods, Feb 27, 2025, No Pagination Specified; doi:10.1037/met0000740

Reliability generalization (RG) studies frequently interpret meta-analytic heterogeneity in score reliability as evidence of differences in an instrument’s measurement quality across administrations. However, such interpretations ignore the fact that, under classical test theory, score reliability depends on two parameters: true score variance and error score variance. True score variance refers to the actual variation in the trait we aim to measure, while error score variance refers to nonsystematic variation arising in the observed, manifest variable. If the error score variance remains constant, variations in true score variance can result in heterogeneity in reliability coefficients. While this argument is not new, we argue that current approaches to addressing this issue in the RG literature are insufficient. Instead, we propose enriching an RG study with Boot-Err: Explicitly modeling the error score variance using bootstrapping and meta-analytic techniques. Through a comprehensive simulation scheme, we demonstrate that score reliability can vary while the measuring quality remains unaffected. The simulation also illustrates how explicitly modeling error score variances may improve inferences concerning random measurement error and under which conditions such enhancements occur. Furthermore, using openly available direct replication data, we show how explicitly modeling error score variance allows for an assessment to what extent measurement quality can be described as identical across administration sites. (PsycInfo Database Record (c) 2025 APA, all rights reserved)

Psychological Methods, Feb 24, 2025, No Pagination Specified; doi:10.1037/met0000743

Many psychological theories imply that the degree of congruence between two variables (e.g., self-rated and objectively measured intelligence) is related to some psychological outcome (e.g., life satisfaction). Such congruence hypotheses can be tested with response surface analysis (RSA), in which a second-order polynomial regression model is estimated and suitably interpreted. Whereas several strategies exist for this interpretation, they all contain rationales that diminish the probability of drawing correct conclusions. For example, a frequently applied strategy involves calculating six auxiliary parameters from the estimated regression weights and accepting the congruence hypothesis if they satisfy certain conditions. In testing the conditions, a nonsignificant null-hypothesis test of some parameters is taken as evidence that the parameter is zero. This interpretation is formally inadmissible and adversely affects the probability of making correct decisions about the congruence hypothesis. We address this limitation of the six-parameter strategy and other RSA strategies by proposing that statistical equivalence testing (SET) be integrated into RSA. We compare the existing and new RSA strategies with a simulation study and find that the SET strategies are sensible alternatives to the existing strategies. We provide code templates for implementing the SET strategies. (PsycInfo Database Record (c) 2025 APA, all rights reserved)

Psychological Methods, Feb 20, 2025, No Pagination Specified; doi:10.1037/met0000734

With the advancement of artificial intelligence (AI), many AI-derived techniques have been adapted into psychological and behavioral science research, including measure development, which is a key task for psychometricians and methodologists. Ant colony optimization (ACO) is an AI-derived metaheuristic algorithm that has been integrated into the structural equation modeling framework to search for optimal (or near optimal) solutions. ACO-driven measurement modeling is an emerging method for constructing scales, but psychological researchers generally lack the necessary understanding of ACO-optimized models and outcome solutions. This article aims to investigate whether ACO solutions are indeed optimal and whether the optimized measurement models of ACO are always more psychologically useful compared to conventional ones built by human psychometricians. To work toward these goals, we highlight five essential methodological considerations for using ACO in measure development: (a) pursuing a local or global optimum, (b) avoiding a subjective optimum, (c) optimizing content validity, (d) bridging the gap between theory and model, and (e) recognizing limitations of unidirectionality. A joint data set containing item-level data from German (n = 297) and the United States (n = 334) samples was employed, and seven illustrative ACO analyses with various configurations were conducted to illustrate or facilitate the discussions of these considerations. We conclude that measurement solutions from the current ACO have not yet become optimal or close to optimal, and the optimized measurement models of ACO may be becoming more useful. (PsycInfo Database Record (c) 2025 APA, all rights reserved)

Psychological Methods, Feb 20, 2025, No Pagination Specified; doi:10.1037/met0000730

Confirmatory bifactor models have become very popular in psychological applications, but they are increasingly criticized for statistical pitfalls such as tendency to overfit, tendency to produce anomalous results, instability of solutions, and underidentification problems. In part to combat this state of affairs, many different reliability and dimensionality measures have been proposed to help researchers evaluate the quality of the obtained bifactor solution. However, in empirical practice, the evaluation of bifactor models is largely based on structural equation model fit indices. Other critical indicators of solution quality, such as patterns of general and group factor loadings, whether all estimates are interpretable, and values of reliability coefficients, are often not taken into account. In addition, in the methodological literature, some confusion exists about the appropriate interpretation and application of some bifactor reliability coefficients. In this article, we accomplish several goals. First, we review reliability coefficients for bifactor models and their correct interpretations, and we provide expectations for their values. Second, to help steer researchers away from structural equation model fit indices and to improve current practice, we provide a checklist for evaluating the statistical fit of bifactor models. Third, we evaluate the state of current practice by examining 96 empirical articles employing confirmatory bifactor models across different areas of psychology. (PsycInfo Database Record (c) 2025 APA, all rights reserved)

Psychological Methods, Feb 17, 2025, No Pagination Specified; doi:10.1037/met0000737

Meta-regression is an essential meta-analytic tool for investigating sources of heterogeneity and assessing the impact of moderators. However, existing methods for meta-regression have limitations, such as inadequate consideration of model uncertainty and poor performance under publication bias. To overcome these limitations, we extend robust Bayesian meta-analysis (RoBMA) to meta-regression (RoBMA-regression). RoBMA-regression allows for moderator analyses while simultaneously taking into account the uncertainty about the presence and impact of other factors (i.e., the main effect, heterogeneity, publication bias, and other potential moderators). The methodology presents a coherent way of assessing the evidence for and against the presence of both continuous and categorical moderators. We further employ a Savage–Dickey density ratio test to quantify the evidence for and against the presence of the effect at different levels of categorical moderators. We illustrate RoBMA-regression in an empirical example and demonstrate its performance in a simulation study. We implemented the methodology in the RoBMA R package. Overall, RoBMA-regression presents researchers with a powerful and flexible tool for conducting robust and informative meta-regression analyses. (PsycInfo Database Record (c) 2025 APA, all rights reserved)

Psychological Methods, Feb 17, 2025, No Pagination Specified; doi:10.1037/met0000713

Questionnaire validation is indispensable in psychology and medicine and is essential for understanding differences across diverse populations in the measured construct. While traditional latent factor models have long dominated psychometric validation, recent advancements have introduced alternative methodologies, such as the “network framework.” This study presents a pioneering approach integrating information theory, machine learning (ML), and Bayesian networks (BNs) into questionnaire validation. Our proposed framework considers psychological constructs as complex, causally interacting systems, bridging theories, and empirical hypotheses. We emphasize the crucial link between questionnaire items and theoretical frameworks, validated through the known-groups method for effective differentiation of clinical and nonclinical groups. Information theory measures such as Jensen–Shannon divergence distance and ML for item selection enhance discriminative power while contextually reducing respondent burden. BNs are employed to uncover conditional dependences between items, illuminating the intricate systems underlying psychological constructs. Through this integrated framework encompassing item selection, theory formulation, and construct validation stages, we empirically validate our method on two simulated data sets—one with dichotomous and the other with Likert-scale data—and a real data set. Our approach demonstrates effectiveness in standard questionnaire research and validation practices, providing insights into criterion validity, content validity, and construct validity of the instrument. (PsycInfo Database Record (c) 2025 APA, all rights reserved)

Psychological Methods, Feb 17, 2025, No Pagination Specified; doi:10.1037/met0000719

Preregistration is gaining ground in psychology, and a consequence of this is that preregistered studies are more often included in meta-analyses. Preregistered studies likely mitigate the effect of publication bias in a meta-analysis, because preregistered studies can be located in the registries they were registered in even if they do not get published. However, current meta-analysis methods do not take into account that preregistered studies are less susceptible to publication bias. Traditional methods treat all studies as equivalent while meta-analytic conclusions can be improved by taking advantage of preregistered studies. The goal of this article is to introduce the hybrid extended meta-analysis (HYEMA) method that takes into account whether a study is preregistered or not and corrects for publication bias in only the nonpreregistered studies. The proposed method is applied to two meta-analyses on prominent effects in the psychological literature: the red-romance hypothesis and money priming. Applying HYEMA to these meta-analyses shows that the average effect size estimate is substantially closer to zero than the estimate of the random-effects meta-analysis model. Two simulation studies tailored to the two applications are also presented to illustrate the method’s superior performance compared to the random-effects meta-analysis model and precision-effect test and precision-effect estimate with standard error when publication bias is present. Hence, I recommend to apply HYEMA as a sensitivity analysis if a mix of both preregistered and nonpreregistered studies are present in a meta-analysis. R code as well as a web application (https://rcmvanaert.shinyapps.io/HYEMA) have been developed and are described in the article to facilitate application of the method. (PsycInfo Database Record (c) 2025 APA, all rights reserved)

Psychological Methods, Feb 13, 2025, No Pagination Specified; doi:10.1037/met0000727

For cluster randomized trials and individually randomized group treatment trials that compare two treatments on a continuous outcome, designs are presented that minimize the number of subjects or the amount of research budget, when aiming for a desired power level. These designs optimize the treatment-to-control allocation ratio of study participants but also optimize the choice between the number of clusters/groups versus the number of persons per cluster/group. Given that optimal designs require prior knowledge of parameters from the analysis model, which are often unknown during the design stage—especially outcome variances—maximin designs are introduced. These designs ensure a prespecified power level for plausible ranges of the unknown parameters and maximize power for the worst-case values of these parameters. The present study not only reviews but also extends the existing literature by deriving optimal and maximin designs when the number of clusters/groups are fixed because of practical constraints. How to calculate sample sizes in such practical designs and how much budget may be saved are illustrated for an empirical example. To facilitate sample size calculation for each of the variants of the maximin designs considered, an easy-to-use interactive R Shiny app has been developed and made available. (PsycInfo Database Record (c) 2025 APA, all rights reserved)

Psychological Methods, Feb 13, 2025, No Pagination Specified; doi:10.1037/met0000669

Evidence accumulation models (EAMs) are an important class of cognitive models used to analyze both response time and response choice data recorded from decision-making tasks. Developments in estimation procedures have helped EAMs become important both in basic scientific applications and solution-focused applied work. Hierarchical Bayesian estimation frameworks for the linear ballistic accumulator (LBA) model and the diffusion decision model (DDM) have been widely used, but still suffer from some key limitations, particularly for large sample sizes, for models with many parameters, and when linking decision-relevant covariates to model parameters. We extend upon previous work with methods for estimating the LBA and DDM in hierarchical Bayesian frameworks that include random effects that are correlated between people and include regression-model links between decision-relevant covariates and model parameters. Our methods work equally well in cases where the covariates are measured once per person (e.g., personality traits or psychological tests) or once per decision (e.g., neural or physiological data). We provide methods for exact Bayesian inference, using particle-based Markov chain Monte-Carlo, and also approximate methods based on variational Bayesian (VB) inference. The VB methods are sufficiently fast and efficient that they can address large-scale estimation problems, such as with very large data sets. We evaluate the performance of these methods in applications to data from three existing experiments. Detailed algorithmic implementations and code are freely available for all methods. (PsycInfo Database Record (c) 2025 APA, all rights reserved)

Psychological Methods, Feb 13, 2025, No Pagination Specified; doi:10.1037/met0000733

Determining the number of factors is a decisive, yet very difficult decision a researcher faces when conducting an exploratory factor analysis (EFA). Over the last decades, numerous so-called factor retention criteria have been developed to infer the latent dimensionality from empirical data. While some tutorials and review articles on EFA exist which give recommendations on how to determine the number of latent factors, there is no comprehensive overview that categorizes the existing approaches and integrates the results of existing simulation studies evaluating the various methods in different data conditions. With this article, we want to provide such an overview enabling (applied) researchers to make an informed decision when choosing a factor retention criterion. Summarizing the most important results from recent simulation studies, we provide guidance when to rely on which method and call for a more thoughtful handling of overly simple heuristics. (PsycInfo Database Record (c) 2025 APA, all rights reserved)

Psychological Methods, Feb 13, 2025, No Pagination Specified; doi:10.1037/met0000718

Classic confirmatory factor analysis (CFA) models are theoretically superior to exploratory factor analysis (EFA) models because they specify that each indicator only measures one factor. In contrast, in EFA, all loadings are permitted to be nonzero. In this article, we show that when fit to EFA structures and other models with many cross-loadings, classic CFA models often produce excellent fit. A key requirement for breaking this pattern is to have highly variable ratios of main loadings to corresponding cross-loadings in the true data-generating structure—and strongest misfit results when cross-loadings are of mixed sign. We show mathematically that EFA structures that are rotatable to a CFA representation are those where the main loadings and the cross-loadings are proportional for each group of indicators. With the help of a ShinyApp, we show that unless these proportionality constraints are violated severely in the true data structure, CFA models will fit well to most true models containing many cross-loadings by commonly accepted fit index cutoffs. We also show that fit indices are nonmonotone functions of the number of positive cross-loadings, and the relationship becomes monotone only when cross-loadings are of mixed sign. Overall, our findings indicate that good fit of a CFA model rules out that the true model is an EFA model with highly variable ratios of main and cross-loadings, but does not rule out most other plausible EFA structures. We discuss the implications of these findings. (PsycInfo Database Record (c) 2025 APA, all rights reserved)

Psychological Methods, Feb 13, 2025, No Pagination Specified; doi:10.1037/met0000725

Mediation analysis is widely used in psychological research to identify the relationship between independent and dependent variables through mediators. Assessing the relative importance of mediators in parallel mediator models can help researchers better understand mediation effects and guide interventions. The traditional coefficient-based measures of indirect effect merely focus on the partial effect of each mediator, which may reach undesirable results of importance assessment. This study develops a new method of measuring the importance of multiple mediators. Three R² measures of indirect effect proposed by MacKinnon (2008) are extended to parallel mediator models. Dominance analysis, a popular method of evaluating relative importance, is applied to decompose the R² indirect effect and attribute it to each mediator. This offers new measures of indirect effect in terms of relative importance. Both frequentist and Bayesian methods are used to make statistical inference for the dominance measures. Simulation studies investigate the performance of the dominance measures and their inference. A real data example illustrates how the relative importance can be assessed in multiple mediator models. (PsycInfo Database Record (c) 2025 APA, all rights reserved)