Institut für Psychologie der Humboldt-Universität zu Berlin

D I S S E R T A T I O N

Is there an interaction between facial expression and facial familiarity?
An investigation using performance data and event-related potentials.

zur Erlangung des akademischen Grades
doctor rerum naturalium (Dr. rer. nat.)
im Fach Psychologie

eingereicht an der Mathematisch-Naturwissenschaftlichen Fakultät II
der Humboldt-Universität zu Berlin

Dipl.-Psych. Nele Wild-Wall ,
geboren am 19.12.1973 in Hennigsdorf

Dekan:
Dekan der Mathematisch-Naturwissenschaftlichen Fakultät II Prof. Dr. Elmar Kulke

Gutachter:
1. Prof. Dr. Werner Sommer
2. Prof. Dr. Stefan Schweinberger
3. Prof. Dr. Norbert Kathmann

Datum der Promotion:28.05.2004

Abstract english

Contrasting traditional face recognition models previous research has revealed that the recognition of facial expressions and familiarity may not be independent. This dissertation attempts to localize this interaction within the information processing system by means of performance data and event-related potentials. Part I elucidated upon the question of whether there is an interaction between facial familiarity and the discrimination of facial expression. Participants had to discriminate two expressions which were displayed on familiar and unfamiliar faces. The discrimination was faster and less error prone for personally familiar faces displaying happiness. Results revealed a shorter peak latency for the P300 component (trend), reflecting stimulus categorization time, and for the onset of the lateralized readiness potential (S-LRP), reflecting the duration of pre-motor processes. A facilitation of perceptual stimulus categotization for personally familiar faces displaying happiness is suggested. The discrimination of expressions was not facilitated in further experiments using famous or experimentally familiarized, and unfamiliar faces. Part II raises the question of whether there is an interaction between facial expression and the discrimination of facial familiarity. In this task a facilitation was only observable for personally familiar faces displaying a neutral or happy expression, but not for experimentally familiarized, or unfamiliar faces. Event-related potentials reveal a shorter S-LRP interval for personally familiar faces, hence, suggesting a facilitated response selection stage. In summary, the results suggest that an interaction of facial familiarity and facial expression might be possible under some circumstances. Finally, the results are discussed in the context of possible interpretations, previous results, and face recognition models.

Keywords: face recognition , event-related potentials, expression, cognitive neuroscience

Abstract german

Entgegen traditioneller Gesichtererkennungsmodelle konnte in einigen Studien gezeigt werden, dass die Erkennung des Emotionsausdrucks und der Bekanntheit interagieren. In dieser Dissertation wurde mit Hilfe von ereigniskorrelierten Potentialen untersucht, welche funktionalen Prozesse bei einer Interaktion moduliert werden. Teil I untersuchte, ob die Bekanntheit eines Gesichtes die Emotionsdiskrimination erleichtert. In mehreren Experimenten diskriminierten Versuchspersonen zwei Emotionen, die von bekannten und unbekannten Gesichtern praesentiert wurden . Dabei war die Entscheidung fuer persoenlich bekannte Gesichter mit froehlichem Ausdruck schneller und fehlerfreier. Dies zeigt sich in einer kuerzeren Latenz der P300 Komponente (Trend), welche die Dauer der Reizklassifikation auswies, sowie in einem verkuerzten Intervall zwischen Stimulus und Beginn des Lateralisierten Bereitschaftspotentials (S-LRP), welches die handspezifische Reaktionsauswahl anzeigt. Diese Befunde sprechen fuer eine Erleichterung der Emotionsdiskrimination auf spaeten perzeptuellen Verarbeitungsstufen bei persoenlich bekannten Gesichtern. In weiteren Experimenten mit oeffentlich bekannten, gelernten und unbekannten Gesichtern zeigte sich keine Erleichterung der Emotionsdiskrimination für bekannte Gesichter. Teil II untersuchte, ob es einen Einfluss des Emotionsausdrucks auf die Bekanntheitsentscheidung gibt. Eine Erleichterung zeigte sich fuer neutrale oder froehliche Emotionen nur bei persoenlich bekannten Gesichtern, nicht aber bei gelernten oder unbekannten Gesichtern. Sie spiegelt sich in einer Verkuerzung des S-LRP fuer persoenlich bekannte Gesichter wider, was eine Erleichterung der Reaktionsauswahl nahelegt. Zusammenfassend konnte gezeigt werden, dass eine Interaktion der Bekanntheit mit der Emotionserkennung unter bestimmten Bedingungen auftritt. In einer abschließenden Diskussion werden die experimentellen Ergebnisse in Beziehung gesetzt und in Hinblick auf bisherige Befunde diskutiert.

Eigene Schlagworte: Gesichtererkennung, Ereigniskorrelierte Potentiale, Emotion, Kognitive Neurowissenschaft

Table of contents

• Zusammenfassung
• Summary
• 1.  Introduction
  • 1.1. Topic and overview
  • 1.2. Empirical overview and paradigm
    • 1.2.1. Face recognition and models of face recognition
    • 1.2.2. Facial expression
    • 1.2.3. Approach to the topic
    • 1.2.4. Mental Chronometry and Cognitive Psychophysiology
  • 1.3. Experimental design, model and hypotheses
    • 1.3.1. Working Model
    • 1.3.2. Hypotheses
• 2.  Part I: Does facial familiarity affect the discrimination of facial expression?
  • 2.1.  Experiment 1
    • 2.1.1. Rationale
    • 2.1.2. Method
    • 2.1.3. Results
    • 2.1.4.  Discussion
  • 2.2. Experiment 2
    • 2.2.1. Rationale
    • 2.2.2. Method
    • 2.2.3. Results
    • 2.2.4. Discussion
  • 2.3. Experiment 3
    • 2.3.1. Rationale
    • 2.3.2. Method
    • 2.3.3.  Results
    • 2.3.4.  Discussion
  • 2.4. Experiment 4
    • 2.4.1. Rationale
    • 2.4.2. Rating
    • 2.4.3. Method
    • 2.4.4. Results and Discussion
  • 2.5. Experiment 5
    • 2.5.1. Rationale
    • 2.5.2. Method
    • 2.5.3. Results and Discussion
  • 2.6. Experiment 6
    • 2.6.1. Rationale
    • 2.6.2. Method
    • 2.6.3. Rating
    • 2.6.4. Results
    • 2.6.5. Discussion
  • 2.7. Summary and Discussion of Part I
• 3.  Part II: Does facial expression affect the discrimination of facial familiarity?
  • 3.1. Experiment 7
    • 3.1.1. Rationale
    • 3.1.2. Method
    • 3.1.3.  Results
    • 3.1.4. Discussion
  • 3.2. Experiment 8
    • 3.2.1. Rationale
    • 3.2.2. Method
    • 3.2.3. Results and Discussion
  • 3.3. Summary and Discussion of Part II
• 4.  General Discussion
  • 4.1. A short review of the presented data
  • 4.2.  In search of interpretations for the symmetrical interaction
  • 4.3. Relevance of the data
  • 4.4. Perspective
• 5.  References
• 6.  Appendix
  • 6.1. Amplitude distribution and vector scaled data of Experment 2
  • 6.2. Amplitude distribution and vector scaled data of Experment 3
  • 6.3.  Amplitude distribution and vector scaled data of Experment 6
  • 6.4.  Amplitude distribution and vector scaled data of Experment 7
• Danksagung
• List of Abbreviations

Tables

• Table 1: Mean expression ratings of 4 subsets of unfamiliar faces.
• Table 2. Mean familiarity ratings for the stimulus set consisting of famous faces of Experiment 6 separated for the ratings in Scotland and Germany.
• Table 3. Mean expression ratings for the stimulus set consisting of famous faces of Experiment 6 separated for the ratings in Scotland and Germany.

Images

• Figure 1. The functional model of face recognition by Bruce and Young (1986)
• Figure 2. A model of the distributed human neural system for face perception by Haxby, Hoffman, and Gobbini (2000).
• Figure 3. Proposal for a functional working model on which the following experiments are based, illustrated by an expression discrimination task.
• Figure 4. Examples of portraits displaying one person with the expressions disgust (top row) and happiness (bottom row), three different head positions, and two different expressive intensities.
• Figure 5. Trial scheme of Experiment 1 and all following experiments.
• Figure 6. Reaction time and error rates for the experimental and the control group of Experiment 1.
• Figure 7. Recording positions of the scalp electrodes for recording the EEG.
• Figure 8. Reaction time and error rates of the expression discrimination task of Experiment 2 separated for familiarity and facial expressions.
• Figure 9. The N170 component for the expression discrimination task of Experiment 2 at the electrode site P10 separated for familiarity and disgust.
• Figure 10. The P300 component for the expression discrimination task of Experiment 2 at the electrode site Pz separated for familiarity and expression.
• Figure 11. The stimulus-locked LRP for the expression discrimination task of Experiment 2 separated for familiarity and expression.
• Figure 12. The response-locked LRP for the expression discrimination task of Experiment 2 separated for familiarity and expression.
• Figure 13. Differences of the mean amplitude distribution between unfamiliar (UF) and familiar (F) faces (top row) and between happiness (H) and disgust (D; bottom row) for the expression discrimination task of Experiment 2 in all tested time intervalls; a grey shading equals a negative difference.
• Figure 14. Skin conductance response as the mean of all valid trials for personally familiar and unfamiliar faces during a familiarity discrimination task.
• Figure 15. Reaction time and error rates for the expression discrimination task of Experiment 3 separated for familiarity and expression.
• Figure 16. The N170 component for the expression discrimination task of Experiment 3 at the electrode site P10 separated for familiarity and expression.
• Figure 17. The P300 component for the expression discrimination task of Experiment 3 at the electrode site Pz separated for familiarity and expression.
• Figure 18. The stimulus-locked LRP and LhEOG for the expression discrimination task of Experiment 3 separated for familiarity and expression.
• Figure 19. The response-locked LRP and LhEOG for the expression discrimination task of Experiment 3 separated for familiarity and expression.
• Figure 20. Differences of the mean amplitude distribution between unfamiliar (UF) and familiar faces (F; top row) and between happiness (H) and disgust (D; bottom row) for the expression discrimination task of Experiment 2.
• Figure 21. Reaction time and error rates for the expression discrimination task of Experiment 4 separated for familiarity and expression.
• Figure 22. Reaction time and error rates for the expression discrimination task of Experiment 5 separated for familiarity and expression.
• Figure 23. Reaction time and error rate for the expression discrimination task of Experiment 6 separated for familiarity and expression.
• Figure 24. The N170 component for the expression discrimination task of Experiment 6 at the electrode site P10 separated for familiarity and expression.
• Figure 25. The P300 component for the expression discrimination task of Experiment 6 at the electrode site Pz separated for familiarity and expression.
• Figure 26. The stimulus-locked LRP for the expression discrimination task of Experiment 6 separated for familiarity and expression.
• Figure 27. The response-locked LRP for the expression discrimination task of Experiment 6 separated for familiarity and expression.
• Figure 28. Differences of the mean amplitude distribution between pairs of the British (B), German (G), and International celebrities (I; top row), as well as between the neutral expression (N) and happiness (H; bottom row) for the expression discrimination task of Experiment 6 in all tested time intervalls; a grey shading equals a negative difference.
• Figure 29. Proposed working model for the familiarity discrimination task of Part II (S = stimulus, SE = structural encoding, ED = expression discrimination, FD = familiarity discrimination, RS = response selection, MP = motor preparation, R = response).
• Figure 30. Reaction time and error rates for the familiarity discrimination task of Experiment 7.
• Figure 31. The N170 component for the familiarity discrimination task of Experiment 7 at the electrode site P10 separated for expression and for familiarity.
• Figure 32. The P300 component for the familiarity discrimination task of Experiment 7 at the electrode site Pz separated for expression and separate graphs for familiarity.
• Figure 33. The stimulus-locked LRP for the familiarity discrimination task of Experiment 7 separated for expression and separate graphs for familiarity.
• Figure 34. The response-locked LRP for the familiarity discrimination task of Experiment 7 separated for expression and for familiarity.
• Figure 35. Differences of the mean amplitude distribution between pairs of happiness (H), disgust (D), and the neutral expression (N) as well as between unfamiliar (UF) and familiar faces (F; bottom row) for the familiarity discrimination task of Experiment 7 in all tested time intervalls; a grey shading equals a negative difference.
• Figure 36. Reaction time and error rates for the familiarity discrimination task of Experiment 8 separated for expression and familiarity.
• Figure 37. Depending on the temporal properties of the task relevant processes, the functional locus of interaction between facial familiarity and facial expression can be expected on different processing stages; (S = stimulus, SE = structural encoding, ED = expression discrimination, FD = familiarity discrimination, RS = response selection, MP = motor preparation, R = response).
• Figure 38. A proposed modification based on the functional model of face recognition by Bruce and Young (1986), including the modification that was suggested by Breen et al. (2000).