Statistical Models in Psychology (1): development and validation

By operationalizing competing theories as statistical models for behavioral outcomes (e.g., choice probabilities or response times), model-selection techniques such as the Bayes factor allow researchers to test competing strategies on the individual level. For each participant, this method provides posterior model probabilities defined as the probabilities that each of the competing models generated the behavioral outcomes. Recently, the theoretical and statistical sophistication of model-selection methods and models has steadily increased. However, given model-selection results on the individual level, it is not clear how to draw inferences on the group level (e.g., "do all or does a majority of participants adhere to a specific theory?"). As a remedy, we extend a recent model-based test for probabilistic classification by Cavagnaro and Davis-Stober (2019) to estimate the relative group sizes of the different theories. This method accounts for classification uncertainty and allows to test informative hypotheses on the group level (e.g., "most participants are best described by theory X or Y"). The method requires only posterior model probabilities (or information-theoretic weights) per individual as input and can easily be applied using the R package "multinomineq" for multinomial models with inequality constraints. Besides a reanalysis of empirical data, we present Monte Carlo simulations to assess the effect of ignoring classification uncertainty when drawing inferences on the group level.

In psychology, decision making is often – and successfully – modelled with the diffusion model, which is based on the assumption that evidence accumulation follows a Wiener diffusion process, that is, evidence accumulates over time with a constant drift and normal noise. Here, I will present a model suggesting that noise in evidence accumulation is not Gaussian but is better described by heavy-tailed distributions. Thus, the evidence accumulation process is mapped no longer by a diffusion process but by a so-called Lévy-flight. An important characteristic of Lévy-flights is the incorporation of jumps in the process. In decision making, such jumps indicate sudden changes in the subjective believes about the current situation. In the present talk I will (a) discuss possibilities to estimate parameters of the Lévy-Flight model, (b) compare the fit of the standard diffusion model and the Lévy-Flight model to empirical data, and (c) present first evidence of both individual-related and task-related predictors of the “heavy-tailed-ness” of the noise distribution.

In several contexts, it is useful to know whether there is consensus among a group of people. Such consensus may concern eyewitnesses reporting about the details of a crime, voters expressing their political views, or experts suggesting best practices for their field. Specifically, if each member of a group answers a set of questions, consensus analysis offers estimates of whether there is a group consensus, what this consensus is, and to what degree each respondent answered in line with the consensus. In previous consensus-analysis models, each model focused on a specific response format such as dichotomous yes-no responses or numerical continuous responses. Consequently, researchers had to adapt all questions to a specific response format—with mixed results. Dissatisfied with the constraints this placed on our research, we developed an extension of consensus analysis that allows the flexible mixture of response formats within a question set. This enables researchers to choose which response format best fits the content of each question rather than forcing the content into the constraints of the model. We present the conceptual foundations of the model and the results of model-validation studies. We conclude with a brief discussion about the implementation of varying response formats in measurement models and substantive models in Psychology.

Response-time extended multinomial processing tree (RT-MPT; Klauer and Kellen, 2018) models constitute a relatively new model class to model response latencies alongside the response frequencies of traditional multinomial processing tree (MPT) models. It enables the estimation of process-completion times and encoding plus motor-execution times (a.k.a. non-decision time), next to the typical process probabilities of traditional MPT models. Process-completion time refers to the time a process needs to finish with one of two outcomes (e.g. how long does it take to detect a word successfully, where detection is the process and the success its outcome). We developed an R-package, called “rtmpt” (Hartmann, Johannsen, & Klauer, in press), to fit models of the RT-MPT model class and show that the Bayesian algorithm underlying “rtmpt” is valid.

In a typical recognition task, participants study word lists and categorize single-words in a subsequent recognition phase as previously studied or not. Pairing two words randomly in a recognition phase (paired-words), combined judgements to each pair differ from two separate responses to the same words. Previously, two models have been used to describe the performance of paired-words: two-high threshold model (2HTM) as an example for discrete-state models and general recognition theory (GRT), a multidimensional signal detection theory, as an example for continuous models. Previous studies showed consistently that the discrete-state model could explain the paired-word recognition task best. However, both models have not been validated so far. Within this talk, we present a first attempt to validate those models using selective influence studies. We tested whether both models capture a base-rate manipulation in a meaningful way. Theoretically, it is expected that base-rate manipulations only affect parameters describing decisional processes. Inspecting those hypotheses within models allowing both, mnemonic and decisional parameters to differ between different base-rate manipulations, exactly those results can be observed. However, comparing model versions locating the effect on either the mnemonic or the decisional processes, former models are preferred. Within this talk, we will discuss the theoretical implications for those two models.