How pre-processing decisions affect the reliability and validity of the approach-avoidance task: Evidence from simulations and multiverse analyses with six datasets.
Approach-avoidance task (AAT)
Bias scores
Multiverse analysis
Outlier exclusion
Reliability
Simulation
Validity
Journal
Behavior research methods
ISSN: 1554-3528
Titre abrégé: Behav Res Methods
Pays: United States
ID NLM: 101244316
Informations de publication
Date de publication:
23 May 2023
23 May 2023
Historique:
accepted:
09
03
2023
medline:
24
5
2023
pubmed:
24
5
2023
entrez:
23
5
2023
Statut:
aheadofprint
Résumé
Reaction time (RT) data are often pre-processed before analysis by rejecting outliers and errors and aggregating the data. In stimulus-response compatibility paradigms such as the approach-avoidance task (AAT), researchers often decide how to pre-process the data without an empirical basis, leading to the use of methods that may harm data quality. To provide this empirical basis, we investigated how different pre-processing methods affect the reliability and validity of the AAT. Our literature review revealed 108 unique pre-processing pipelines among 163 examined studies. Using empirical datasets, we found that validity and reliability were negatively affected by retaining error trials, by replacing error RTs with the mean RT plus a penalty, and by retaining outliers. In the relevant-feature AAT, bias scores were more reliable and valid if computed with D-scores; medians were less reliable and more unpredictable, while means were also less valid. Simulations revealed bias scores were likely to be less accurate if computed by contrasting a single aggregate of all compatible conditions with that of all incompatible conditions, rather than by contrasting separate averages per condition. We also found that multilevel model random effects were less reliable, valid, and stable, arguing against their use as bias scores. We call upon the field to drop these suboptimal practices to improve the psychometric properties of the AAT. We also call for similar investigations in related RT-based bias measures such as the implicit association task, as their commonly accepted pre-processing practices involve many of the aforementioned discouraged methods. HIGHLIGHTS: • Rejecting RTs deviating more than 2 or 3 SD from the mean gives more reliable and valid results than other outlier rejection methods in empirical data • Removing error trials gives more reliable and valid results than retaining them or replacing them with the block mean and an added penalty • Double-difference scores are more reliable than compatibility scores under most circumstances • More reliable and valid results are obtained both in simulated and real data by using double-difference D-scores, which are obtained by dividing a participant's double mean difference score by the SD of their RTs.
Identifiants
pubmed: 37221345
doi: 10.3758/s13428-023-02109-1
pii: 10.3758/s13428-023-02109-1
doi:
Types de publication
Journal Article
Langues
eng
Sous-ensembles de citation
IM
Informations de copyright
© 2023. The Author(s).
Références
Barton, T., Constable, M. D., Sparks, S., & Kritikos, A. (2021). Self-bias effect: movement initiation to self-owned property is speeded for both approach and avoidance actions. Psychological Research, 85(4), 1391–1406. https://doi.org/10.1007/s00426-020-01325-0
doi: 10.1007/s00426-020-01325-0
pubmed: 32232562
Berger, A., & Kiefer, M. (2021). Comparison of different response time outlier exclusion methods: A simulation study. Frontiers in Psychology, 12, 675558. https://doi.org/10.3389/fpsyg.2021.675558
doi: 10.3389/fpsyg.2021.675558
pubmed: 34194371
pmcid: 8238084
Cousijn, J., Luijten, M., & Wiers, R. W. (2014). Mechanisms underlying alcohol-approach action tendencies: The role of emotional primes and drinking motives. Frontiers in Psychiatry, 5, 44. https://doi.org/10.3389/fpsyt.2014.00044
doi: 10.3389/fpsyt.2014.00044
pubmed: 24834057
pmcid: 4018525
De Houwer, J. (2003). The extrinsic affective Simon task. Experimental Psychology, 50(2), 77–85. https://doi.org/10.1026/1618-3169.50.2.77
doi: 10.1026/1618-3169.50.2.77
pubmed: 12693192
Dixon, W. J. (1953). Processing data for outliers. Biometrics, 9(1), 74–89. https://doi.org/10.2307/3001634
doi: 10.2307/3001634
Ernst, L. H., Ehlis, A.-C., Dresler, T., Tupak, S. V., Weidner, A., & Fallgatter, A. J. (2013). N1 and N2 ERPs reflect the regulation of automatic approach tendencies to positive stimuli. Neuroscience Research, 75(3), 239–249. https://doi.org/10.1016/j.neures.2012.12.005
doi: 10.1016/j.neures.2012.12.005
pubmed: 23298530
Fabre-Thorpe, M. (2011). The characteristics and limits of rapid visual categorization. Frontiers in Psychology, 2, 243. https://doi.org/10.3389/fpsyg.2011.00243
doi: 10.3389/fpsyg.2011.00243
pubmed: 22007180
pmcid: 3184650
Ferentzi, H., Scheibner, H., Wiers, R. W., Becker, E. S., Lindenmeyer, J., Beisel, S., & Rinck, M. (2018). Retraining of automatic action tendencies in individuals with obesity: A randomized controlled trial. Appetite, 126, 66–72. https://doi.org/10.1016/j.appet.2018.03.016
doi: 10.1016/j.appet.2018.03.016
pubmed: 29596869
Fokkema, M., Smits, N., Zeileis, A., Hothorn, T., & Kelderman, H. (2018). Detecting treatment-subgroup interactions in clustered data with generalized linear mixed-effects model trees. Behavior Research Methods, 50(5), 2016–2034. https://doi.org/10.3758/s13428-017-0971-x
doi: 10.3758/s13428-017-0971-x
pubmed: 29071652
Gelman, A., & Loken, E. (2013). The garden of forking paths: Why multiple comparisons can be a problem, even when there is no “fishing expedition” or “p-hacking” and the research hypothesis was posited ahead of time. Retrieved on 10 August, 2021, from http://www.stat.columbia.edu/~gelman/research/unpublished/p_hacking.pdf
Glashouwer, K. A., Timmerman, J., & de Jong, P. J. (2020). A personalized approach-avoidance modification intervention to reduce negative body image. A placebo-controlled pilot study. Journal of Behavior Therapy and Experimental Psychiatry, 68, 101544. https://doi.org/10.1016/j.jbtep.2019.101544
doi: 10.1016/j.jbtep.2019.101544
pubmed: 32086006
Gračanin, A., Krahmer, E., Rinck, M., & Vingerhoets, A. J. J. M. (2018). The effects of tears on approach–avoidance tendencies in observers. Evolutionary Psychology, 16(3), 1474704918791058. https://doi.org/10.1177/1474704918791058
doi: 10.1177/1474704918791058
pubmed: 30071754
Greenwald, A. G., McGhee, D. E., & Schwartz, J. L. (1998). Measuring individual differences in implicit cognition: the implicit association test. Journal of Personality and Social Psychology, 74(6), 1464–1480. https://doi.org/10.1037//0022-3514.74.6.1464
doi: 10.1037//0022-3514.74.6.1464
pubmed: 9654756
Greenwald, A. G., Nosek, B. A., & Banaji, M. R. (2003). Understanding and using the implicit association test: I. An improved scoring algorithm. Journal of Personality and Social Psychology, 85, 197–216. https://doi.org/10.1037/0022-3514.85.2.197
doi: 10.1037/0022-3514.85.2.197
pubmed: 12916565
Grubbs, F. E. (1950). Sample criteria for testing outlying observations. Annals of Mathematical Statistics, 21, 27–58. https://doi.org/10.1214/aoms/1177729885
doi: 10.1214/aoms/1177729885
Hampel, F. R. (1985). The breakdown points of the mean combined with some rejection rules. Technometrics, 27(2), 95–107. https://doi.org/10.2307/1268758
doi: 10.2307/1268758
Heuer, K., Rinck, M., & Becker, E. S. (2007). Avoidance of emotional facial expressions in social anxiety: The approach–avoidance task. Behaviour Research and Therapy, 45(12), 2990–3001. https://doi.org/10.1016/j.brat.2007.08.010
doi: 10.1016/j.brat.2007.08.010
pubmed: 17889827
Hofmann, W., Friese, M., & Gschwendner, T. (2009). Men on the “pull”: Automatic approach-avoidance tendencies and sexual interest behavior. Social Psychology, 40(2), 73–78. https://doi.org/10.1027/1864-9335.40.2.73
doi: 10.1027/1864-9335.40.2.73
Kahveci, S. (2020). AATtools: Reliability and scoring routines for the approach-avoidance task. R package version 0.0.1. Retrieved on 12 December, 2022, from https://cran.r-project.org/package=AATtools
Kahveci, S., Meule, A., Lender, A., & Blechert, J. (2020). Food approach bias is moderated by desire to eat specific foods. Appetite, 154, 104758. https://doi.org/10.1016/j.appet.2020.104758
doi: 10.1016/j.appet.2020.104758
pubmed: 32535212
Kahveci, S., Van Bockstaele, B., Blechert, J., & Wiers, R. W. (2020). Pulling for pleasure? Erotic approach-bias associated with porn use, not problems. Learning and Motivation, 72, 101656. https://doi.org/10.1016/j.lmot.2020.101656
doi: 10.1016/j.lmot.2020.101656
Kahveci, S., Van Alebeek, H., Berking, M., & Blechert, J. (2021). Touchscreen-based assessment of food approach biases: Investigating reliability and item-specific preferences. Appetite, 163, 105190. https://doi.org/10.1016/j.appet.2021.105190
doi: 10.1016/j.appet.2021.105190
pubmed: 33711352
Krieglmeyer, R., & Deutsch, R. (2010). Comparing measures of approach-avoidance behaviour: The manikin task vs. two versions of the joystick task. Cognition & Emotion, 24(5), 810–828. https://doi.org/10.1080/02699930903047298
doi: 10.1080/02699930903047298
Leins, J., Waldorf, M., Kollei, I., Rinck, M., & Steins-Loeber, S. (2018). Approach and avoidance: Relations with the thin body ideal in women with disordered eating behavior. Psychiatry Research, 269, 286–292. https://doi.org/10.1016/j.psychres.2018.08.029
doi: 10.1016/j.psychres.2018.08.029
pubmed: 30172185
Lender, A., Meule, A., Rinck, M., Brockmeyer, T., & Blechert, J. (2018). Measurement of food-related approach–avoidance biases: Larger biases when food stimuli are task relevant. Appetite, 125, 42–47. https://doi.org/10.1016/j.appet.2018.01.032
doi: 10.1016/j.appet.2018.01.032
pubmed: 29407526
Lindgren, K. P., Wiers, R. W., Teachman, B. A., Gasser, M. L., Westgate, E. C., Cousijn, J., ... Neighbors, C. (2015). Attempted training of alcohol approach and drinking identity associations in US undergraduate drinkers: Null results from two studies. PLOS ONE, 10(8), e0134642. https://doi.org/10.1371/journal.pone.0134642
Lobbestael, J., Cousijn, J., Brugman, S., & Wiers, R. W. (2016). Approach and avoidance towards aggressive stimuli and its relation to reactive and proactive aggression. Psychiatry Research, 240, 196–201. https://doi.org/10.1016/j.psychres.2016.04.038
doi: 10.1016/j.psychres.2016.04.038
pubmed: 27111213
Loijen, A., Vrijsen, J. N., Egger, J. I. M., Becker, E. S., & Rinck, M. (2020). Biased approach-avoidance tendencies in psychopathology: A systematic review of their assessment and modification. Clinical Psychology Review, 77, 101825. https://doi.org/10.1016/j.cpr.2020.101825
doi: 10.1016/j.cpr.2020.101825
pubmed: 32143108
Machulska, A., Kleinke, K., & Klucken, T. (2022). Same same, but different: A psychometric examination of three frequently used experimental tasks for cognitive bias assessment in a sample of healthy young adults. Behavior Research Methods. https://doi.org/10.3758/s13428-022-01804-9
Neimeijer, R. A., Roefs, A., Glashouwer, K. A., Jonker, N. C., & de Jong, P. J. (2019). Reduced automatic approach tendencies towards task-relevant and task-irrelevant food pictures in Anorexia Nervosa. Journal of Behavior Therapy and Experimental Psychiatry, 65, 101496. https://doi.org/10.1016/j.jbtep.2019.101496
doi: 10.1016/j.jbtep.2019.101496
pubmed: 31302385
Nosek, B. A., Bar-Anan, Y., Sriram, N., Axt, J., & Greenwald, A. G. (2014). Understanding and using the brief implicit association test: Recommended scoring procedures. PLoS One, 9(12), e110938. https://doi.org/10.1371/journal.pone.0110938
doi: 10.1371/journal.pone.0110938
pubmed: 25485938
pmcid: 4259300
Parsons, S. (2022). Exploring reliability heterogeneity with multiverse analyses: Data processing decisions unpredictably influence measurement reliability. Meta-Psychology, 6. https://doi.org/10.15626/MP.2020.2577
Payne, B. K. (2001). Prejudice and perception: the role of automatic and controlled processes in misperceiving a weapon. Journal of Personality and Social Psychology, 81(2), 181–192. https://doi.org/10.1037//0022-3514.81.2.181
doi: 10.1037//0022-3514.81.2.181
pubmed: 11519925
R Core Team. (2020). R: A language and environment for statistical computing. R Foundation for Statistical Computing.
Radke, S., Volman, I., Kokal, I., Roelofs, K., de Bruijn, E. R. A., & Toni, I. (2017). Oxytocin reduces amygdala responses during threat approach. Psychoneuroendocrinology, 79, 160–166. https://doi.org/10.1016/j.psyneuen.2017.02.028
doi: 10.1016/j.psyneuen.2017.02.028
pubmed: 28285187
Ratcliff, R. (1993). Methods for dealing with reaction time outliers. Psychological Bulletin, 114(3), 510–532. https://doi.org/10.1037/0033-2909.114.3.510
doi: 10.1037/0033-2909.114.3.510
pubmed: 8272468
Reinecke, A., Becker, E. S., & Rinck, M. (2010). Three indirect tasks assessing implicit threat associations and behavioral response tendencies: Test-retest reliability and validity. Zeitschrift für Psychologie/Journal of Psychology, 218(1), 4–11. https://doi.org/10.1027/0044-3409/a000002
doi: 10.1027/0044-3409/a000002
Rinck, M., Bundschuh, S., Engler, S., Müller, A., Wissmann, J., Ellwart, T., & Becker, E. S. (2002). Reliabilität und Validität dreier Instrumente zur Messung von Angst vor Spinnen. [Reliability and validity of German versions of three instruments measuring fear of spiders.]. Diagnostica, 48(3), 141–149. https://doi.org/10.1026/0012-1924.48.3.141
doi: 10.1026/0012-1924.48.3.141
Rinck, M., Dapprich, A., Lender, A., Kahveci, S., & Blechert, J. (2021). Grab it or not? Measuring avoidance of spiders with touchscreen-based hand movements. Journal of Behavior Therapy and Experimental Psychiatry, 73, 101670. https://doi.org/10.1016/j.jbtep.2021.101670
doi: 10.1016/j.jbtep.2021.101670
pubmed: 34157656
Rotteveel, M., & Phaf, R. H. (2004). Automatic affective evaluation does not automatically predispose for arm flexion and extension. Emotion, 4(2), 156–172. https://doi.org/10.1037/1528-3542.4.2.156
doi: 10.1037/1528-3542.4.2.156
pubmed: 15222853
Saraiva, A. C., Schüür, F., & Bestmann, S. (2013). Emotional valence and contextual affordances flexibly shape approach-avoidance movements. Frontiers in Psychology, 4, 933. https://doi.org/10.3389/fpsyg.2013.00933
doi: 10.3389/fpsyg.2013.00933
pubmed: 24379794
pmcid: 3861787
Solarz, A. K. (1960). Latency of instrumental responses as a function of compatibility with the meaning of eliciting verbal signs. Journal of Experimental Psychology, 59(4), 239. https://doi.org/10.1037/h0047274
doi: 10.1037/h0047274
pubmed: 13832584
Spearman, C. (1904). The proof and measurement of association between two things. The American Journal of Psychology, 15(1), 72–101. https://doi.org/10.2307/1412159
doi: 10.2307/1412159
Steegen, S., Tuerlinckx, F., Gelman, A., & Vanpaemel, W. (2016). Increasing transparency through a multiverse analysis. Perspectives on Psychological Science, 11(5), 702–712. https://doi.org/10.1177/1745691616658637
doi: 10.1177/1745691616658637
pubmed: 27694465
Tzavella, L., Lawrence, N. S., Button, K. S., Hart, E. A., Holmes, N. M., Houghton, K., ... Adams, R. C. (2021). Effects of go/no-go training on food-related action tendencies, liking and choice. Royal Society Open Science, 8(8), 210666. https://doi.org/10.1098/rsos.210666
Van Alebeek, H., Kahveci, S., & Blechert, J. (2021). Improving the touchscreen-based food approach-avoidance task: remediated block-order effects and initial findings regarding validity [version 3; peer review: 2 approved with reservations]. Open Research Europe, 1, 15. https://doi.org/10.12688/openreseurope.13241.3
doi: 10.12688/openreseurope.13241.3
Van Alebeek, H., Kahveci, S., Rinck, M., & Blechert, J. (2023). Touchscreen-based approach-avoidance responses to appetitive and threatening stimuli. Journal of Behavior Therapy and Experimental Psychiatry, 78, 101806. https://doi.org/10.1016/j.jbtep.2022.101806
doi: 10.1016/j.jbtep.2022.101806
pubmed: 36435548
van Peer, J. M., Roelofs, K., Rotteveel, M., van Dijk, J. G., Spinhoven, P., & Ridderinkhof, K. R. (2007). The effects of cortisol administration on approach–avoidance behavior: An event-related potential study. Biological Psychology, 76(3), 135–146. https://doi.org/10.1016/j.biopsycho.2007.07.003
doi: 10.1016/j.biopsycho.2007.07.003
pubmed: 17728047
van Strien, T., Frijters, J. E. R., Bergers, G. P. A., & Defares, P. B. (1986). The Dutch Eating Behavior Questionnaire (DEBQ) for assessment of restrained, emotional, and external eating behavior. International Journal of Eating Disorders, 5(2), 295–315. https://doi.org/10.1002/1098-108X(198602)5:2<295::AID-EAT2260050209>3.0.CO;2-T
doi: 10.1002/1098-108X(198602)5:2<295::AID-EAT2260050209>3.0.CO;2-T
von Borries, A. K. L., Volman, I., de Bruijn, E. R. A., Bulten, B. H., Verkes, R. J., & Roelofs, K. (2012). Psychopaths lack the automatic avoidance of social threat: Relation to instrumental aggression. Psychiatry Research, 200(2), 761–766. https://doi.org/10.1016/j.psychres.2012.06.026
doi: 10.1016/j.psychres.2012.06.026
Wagenmakers, E.-J., & Brown, S. (2007). On the linear relation between the mean and the standard deviation of a response time distribution. Psychological Review, 114(3), 830–841. https://doi.org/10.1037/0033-295X.114.3.830
doi: 10.1037/0033-295X.114.3.830
pubmed: 17638508
Wiers, R. W., Eberl, C., Rinck, M., Becker, E. S., & Lindenmeyer, J. (2011). Retraining automatic action tendencies changes alcoholic patients' approach bias for alcohol and improves treatment outcome. Psychological Science, 22, 490–497. https://doi.org/10.1177/0956797611400615
doi: 10.1177/0956797611400615
pubmed: 21389338
Wittekind, C. E., Reibert, E., Takano, K., Ehring, T., Pogarell, O., & Ruther, T. (2019). Approach-avoidance modification as an add-on in smoking cessation: A randomized-controlled study. Behaviour Research and Therapy, 114, 35–43. https://doi.org/10.1016/j.brat.2018.12.004
doi: 10.1016/j.brat.2018.12.004
pubmed: 30716613
Wittekind, C. E., Blechert, J., Schiebel, T., Lender, A., Kahveci, S., & Kühn, S. (2021). Comparison of different response devices to assess behavioral tendencies towards chocolate in the approach-avoidance task. Appetite, 165, 105294. https://doi.org/10.1016/j.appet.2021.105294
doi: 10.1016/j.appet.2021.105294
pubmed: 33991643
Zech, H. G., Rotteveel, M., van Dijk, W. W., & van Dillen, L. F. (2020). A mobile approach-avoidance task. Behavior Research Methods, 52(5), 2085–2097. https://doi.org/10.3758/s13428-020-01379-3
doi: 10.3758/s13428-020-01379-3
pubmed: 32180179
pmcid: 7575493
Zech, H. G., Gable, P., van Dijk, W. W., & van Dillen, L. F. (2022). Test-retest reliability of a smartphone-based approach-avoidance task: Effects of retest period, stimulus type, and demographics. Behavior Research Methods. https://doi.org/10.3758/s13428-022-01920-6