In the last decades face perception has become an important field in cognitive science and the body of literature addressing the issue of face recognition has grown to reflect its importance. Yet, as always, there are still many unclear points concerning the involved functional architecture, as well as the neural correlates of the already mentioned cognitive, emotional, and also automatic processes that are triggered by the multitude of facial information. Although the various processes involved in face recognition work easily in everyday life, it is complicated to explain them on a cognitive and functional basis. Many attempts have been made to model the involved processes and different aspects of face recognition (Bruce & Young, 1986; Burton & Bruce, 1993; Haxby, Hoffman, & Gobbini, 2000). However, the framework of a dissertation is narrow and can only mention the models that are relevant to the broader question. Only a short introduction is given about the empirical literature concerning the topic of face recognition.

One of the most influencial models of face recognition was introduced by Bruce & Young (1986; Fig. 1). It is a funtional model and based on empirical results as well as on data derived from clinical observations of patients who suffer from the selective loss of different aspects of face recognition. The model assumes several specialized modules which subserve the functional processes. The hierarchically ordered modules are thought to work in parallel and independently. Seven distinct codes are proposed as output information of the functional modules which can be derived from faces. An expression-independent description (structural code) is extracted from the first view-centered pictorial description (pictorial code) in an initial structural encoding process. This output information is matched with a face recognition unit (FRU) which is thought to exist uniquely for familiar faces. If the face is familiar, a following person identity node (PIN) contains semantic information about the recognized person (identity-specific semantic code). Finally, for familiar faces the name can be recalled (name code). The other processes of expression analysis (expression code), directed visual processing (visually derived semantic code), and facial speech analysis (speech code) are based on the earlier pictorial code as they can be performed on both unfamiliar and familiar faces.

Figure 1. The functional model of face recognition by Bruce and Young (1986)

The model makes also assumptions about the recognition of facial expression which is important for the topic of the present dissertation whereas other functional models do not (Burton, Bruce, & Johnston, 1990; Breen, Caine, & Coldheart, 2000). It is assumed that the recognition of facial familiarity and of facial expressions function in parallel and also independently. Both processes rely on different codes, the pictorial code and the view-independent structural code, respectively. Although the model of Bruce and Young (1986) can explain a wide range of empirical results (Young, 1998) there are also findings which contradict the assumptions of their model (e.g. Rossion, 2002; Baudouin et al., 2000; Schweinberger & Soukup, 1998; Endo et al., 1992). Especially the proposed independence and serial succession of the functional modules is questioned by these results and imply further research. The model has been complemented (Burton et al., 1990) in the last years and modifications have been suggested (Abdel Rahman, Sommer, & Schweinberger, 2002). Nonetheless, the original model by Bruce and Young (1986) and the refined version (Burton et al., 1990) has proven to be convenient to generate hypotheses as it can explain a wide range of results from studies that have been published since then (Le Gal & Bruce, 2002; Eimer, 2000; Bentin & Deouell, 2000; Ellis, 1989). However, it is a functional model which is all-embracing but only explains the gross processes. It is also important to understand the specific information which is processed by the proposed functional modules.

The recognition of faces can rely on different features or cues which supply information. Differentiations can be made between “first order” features such as eyes, nose, hair, mouth or shape information and “second order” features mainly including the configuration or spatial arrangement of the first order features. In addition, pigmentation and texture of the skin can also convey relevant information for facial recognition. Another differentiation can be made between external or „cardinal“ features (Ellis, 1986) like hair, hairline or face shape and internal facial features (eyes, nose or mouth as well as their spacial arrangement). Ellis, Shepherd, and Davies (1979) found facilitated recognition from internal, when compared to external, features only for famous faces. It is suggested that unfamiliar face recognition and face matching relies more on external facial features (Bruce, Henderson, Greenwood, et al., 1999). In contrast, internal features gain importance the more familiar a face becomes (Young, Hay, & Ellis, 1985) as they can vary less than external features (Bruce & Young, 1998). It is obvious that the recognition of facial expression relies mainly on internal features. One could suppose that overlapping information which is used for the recognition of facial familiarity and facial expression may cause an interaction between both processes. However, Calder, Young, Keane et al. (2000) examined this question by applying the composite effect to analyze the configural features that are used to discriminate facial expression or identity. The composite paradigm (see Bruce, 1988) shows that the recognition of configural features for facial identity or expression is disturbed when the top and bottom half of two different individuals or facial expressions are aligned (composite effect). For an expression discrimination task results of Calder et al. (2000) revealed composite effects in RT which were independent of the identities represented by the facial top and bottom half. For an identity discrimination task the composite effect was also independent of the expressions displayed by the two facial halfs. In another study Calder, Burton, Miller et al. (2001) applied a principle component analysis (PCA) of the pixel intensity information to the Ekman and Friesen (1976) expressive faces. The resulting factors from the PCA were analyzed further with a linear discrimination procedure in order to identify the factors that are most important to recognize facial expressions, facial identity or the gender of a face. The computational procedure revealed that the coding of facial expression relies largely on different components when compared to the coding of identity. Both studies of Calder et al. (2000; 2001) suggest that the configural information which is used to recognize either facial identity or facial expression is different and only overlapping partly.

Many studies have implied face recognition proceedes in a series of separable stages or functional processes (Campbell, Brooks, de Haan & Roberts, 1996; Nachson, Moscovitch, & Umilta, 1995). These processes can be selectively impaired as is evident from lesion studies. Just one example is prosopagnosia (see above), the inability to recognize previously familiar faces. In these patients the ability to recognize and match unfamiliar faces is still intact. Hence, different processes can be assumed which subserve the recognition of familiar and unfamiliar faces. Individuals who suffer from prosopagnosia do not recover from their impairment and are only able to compensate for it with nonfacial cues like the voice or clothing of a familiar person. This suggests that face recognition might rely on a specialized neuronal system in the brain (Kanwisher, McDermott, & Chun, 1997). In the last decade there has been an unsolved controversy concerning the topic of whether this system is independent of the general visual object recognition system (Gauthier & Tarr, 1997; Gauthier & Logothetis, 2000).

An important question, within face recognition research, attempts to isolate the underlying neuronal substrate which is involved. The loss of the ability to recognize previously familiar faces after damage of certain brain regions in patients with prosopagnosia may hint to the importance of these brain regions for familiar face recognition. This impairment recognisable mainly after bilateral damage of the inferior temporal and occipitotemporal cortex (Tranel, Damasio & Damasio, 1988; Damasio et al., 1986) although cases were observed after unilateral damage in the right hemisphere (Uttner, Bliem, Danek, 2002). These findings points to a right hemisphere advantage for face recognition which also has been suggested in other studies (Ellis, 1989; Schweinberger & Sommer, 1991; Rossion, Schiltz, & Crommelinck, 2003). Most of the studies, with heathy participants, addressing this question use functional magnetic resonance imaging (fMRI) or positron emission tomography (PET). Both methods have a high spatial resolution. Studies using ERP data,with the advantage of a high temporal resolution, give a nice complement in order to explicate the temporal interplay of the involved structures and processes. The model of the distributed human neural system for face perception (Figure 2) proposed by Haxby, Hoffman, and Gobbini (2000) brings together the most relevant results of fMRI, PET and also ERP studies from the last years. Hence, it is a good summary of the knowledge concering the neuronal substrate underlying face recognition. Haxby et al. (2000) identified a core sytem in the occipitotemporal visual extrastriate cortex. The inferior occipital gyri is important for the initial visual analysis of faces. Projections to the lateral fusiform gyrus and to the superior temporal sulcus subserve the analysis of invariant aspects (identity) as well as of changeable aspects of faces (facial expression, eye gaze, or lip movement). The core system is supported by an extended system including brain regions that are important for several aspects of face perception and processing but also for other cognitive tasks. It acts in concert with the core system and includes processes that facilitate spacially directed attention to faces, speech or expression perception as well as the processing of semantic mediated information. Although the model shares some elements of the afore mentioned model from Bruce and Young (1986) it is much more related to today’s knowledge about the neural system. Therefore, it may provide more plausible predictions for experiments that are concerned with questions of face perception and face processing.

Figure 2. A model of the distributed human neural system for face perception by Haxby, Hoffman, and Gobbini (2000).

Despite the model of Haxby et al. (2000) relying on a strong body of experimental results, there are still many unclear points and open questions. For example, the functional separation of the different regions, the temporal properties of the processes and interactions among the regions via back projections or links, or the role played by the lateral fusiform gyrus in expression perception because of possible characteristic expressions between individuals (Haxby et al., 2000). In addition, the distributed system may allow interactions of the functional processes that are ascribed to the different regions through the rich interlinking of the brain regions. Interactions between regions via neural linking and the temporal sequence of processing might be important prerequisites of an interaction between facial expressions and facial identity. The model allows this possibility, although it is unspecific about an interaction between both functional processes.

In summary, the research on face recognition has gained importance over last decades. The functional model of face recognition by Bruce and Young (1986) has itself proven to be influential and it is useful to explain a range of experimental results. Nonetheless, modifications based on conflicting results have been suggested. In general, the processing and recognition of faces relies on various featural, spatial, and configural information. An important question concernes the underlying neuronal system that subserves face recognition. The proposed model of Haxby et al. (2000) integrates the results of many studies examining this question.

Since the publication of the functional model of face recognition by Bruce and Young (1986) there exists a strong body of research which takes as its main question the recognition of facial identity. (There is a brief overview of this research above.) In contrast, fewer studies have been concerned with the recognition of facial expressions. Even though the research on facial expressions has a long tradition (e.g. Darwin, 1965). This imbalance only started to change in the last decade and still pertains (Calder, Lawrence, & Young, 2001a). In the paragraphs following, I will attempt to give a short introduction into the research of the recognition of facial expressions. The difference between the production of emotions and facial expressions within the sender and the detection or recognition of the facial expressions by the perceiver has to be pointed out. Although emotions can be perceived via different modalities (e.g. voice) or cues (facial cues, bodily gestures), the face seems to be the most important cue to perceive an emotional state. Thus, many studies focus exclusively on the perception of facial expressions.The production of emotions within the sender and also the expression of emotions via other non-facial gestures is not the topic of this dissertation. Hence, only a short overview will be given about research on the recognition and perception of facial expressions as well as the underlying functional and cognitive processes within the brain.

In his 1872 published book “The expression of the emotions in man and animal” Charles Darwin described in detail how humans and animals express facial emotions. Based on his thorough observations, he claimed that facial expressions are universal throughout all cultures and races, and they are not learned. They have their origions in the facial expressions of animals. After Darwins groundbraking work it almost took a century until his observations were confirmed by systematic studies of Ekman and Friesen (1968; 1971). They portrayed several expressions such as happiness, fear, surprise, and disgust and displayed them to participants from different cultures (e.g. USA, Brazil, Chile, Japan). Independent of the cultural background the participants were able to identify the facial expressions correctly. In 1971 Ekman and Friesen even presented the photographs to people in New Guinea who had no contact to western or eastern literate cultures. When describing antecedent situations for a certain expression these people picked the correct pictures in almost all cases. Until today it is widely accepted that expressions are innate and not learned. They are complex patterns of facial muscular, and neuronal actions controlled by the central nervous system and triggered by specific stimuli (Ekman, 1984). However, it has to be mentioned that there are also gestures that are learned and culture specific. In addition, this might also hold true for the situational conceptualization of facial expressions or the learned suppression of expressed emotions in several situations.

The universality of facial expressions has led to the proposal of so-called basic emotions. Although the number varies between studies, the most popular categorization is reflected in the set of emotional expressive faces by Ekman and Friesen (1976). Displayed by several male and female individuals it containes facial expressions of anger, disgust, fear, happiness, sadness, and surprise. Previously it has been an issue whether the perception of facial expressions is categorical or dimensional. The proposed basic emotions imply a categorical perception of emotions. Observations of categorization errors of facial expressions led Woodworth (1938) to the assumption that emotional expressions are conceptualized along the continuums pleasantness-unpleasantness, and attention-rejection. One of the most influencial dimensional so-called circumplex models comes from Russell (1980). He introduced two bipolar dimensions of pleasure-displeasure and degree of arousal. Nonetheless, recent results using morphed faces and expressions support the notion that facial expression perception is categorical (de Gelder, Teunisse, & Benson, 1997; Calder, Young, Benson, & Perret, 1996).

It is also still up for debate, as whether facial expressions are perceived as parts or as a whole. Configural information of the whole face may play an important role for expression recognition because face inversion makes it harder to recognize facial expressions (de Gelder et al., 1997). It is already known from identity recognition that face inversion disturbes the configural perception and therefore impairs face recognition (Bentin, Allision, Puce et al., 1996; Rossion, Delvenne, Debatisse et al., 1999; Eimer, 2000a). Results of Puce, Allison, Asgari et al. (1996) suggest that the eye region plays an important role in facial expression perception. Possibly, the relative importance of single features depends on the kind of expression because of different patterns of facial and muscular activation. It is conceivable that the eyes are more important for the perception of fear and anger. In contrast, the distinct feature for identifying happiness might be the mouth.

Recently, many studies have examined the neuronal system which subserves the recognition of facial expressions. As for face identity, recognition impairments of neuropsychological patients hint to specific brain regions which may be involved. It is strongly suggested that the amygdala plays a prominant role in the perception and recognition of fear (Adolps, Tranel, Damasio, & Damasio, 1994; Phillips, Young, Scott et al., 1998; Calder et al., 1996). In line with these clinical observations are also studies using fMRI or PET (Morris, Frith, Perret et al., 1996; Vuilleumiere, Armony, Clarke et al., 2002). Although fearful faces have often been used as expressive stimuli, recent studies also examine other facial expressions. Seemingly, a widely distributed system of brain structures subserves the recognition of facial expressions (Adolphs, 2002), and partly overlaps with structures which subserve face recognition in general (see Haxby et al., 2000). Many studies suggest different involvement of brain regions for particular expressions (Kesler/West, Andersen, Smith et al., 2001; Blair, Morris, Frith et al., 1999; Sprengelmeyer, Rausch, Eysel, & Przuntek, 1998; Whalen, Rauch, Etcoff et al., 1998). Structures like the limbic system, the occipitotemporal neocortex (Adolphs, 2002), or the sulcus temporalis superior (Haxby et al., 2000) are reported as important. In addition, the insula and basal ganglia proved to be relevant for the recognition of disgust (Calder et al., 2001a). This is underlined by selective impairments of disgust recognition in patients with Morbus Parkinson (Sprengelmeyer, Young, Mahn et al., 2003), a disease which is caused by the loss of dopaminergic neurons in the basal ganglia.

It was outlined in paragraph 1.2.2., that most researchers agree on a set of basic emotions. Accounts were given for the universality of these emotions because they are to a strong extent independent of cultural background or learning. Basic emotions seem to be categorically perceived. They seem to be recognized on part based facial information although configural information may also play a role. Neurophysiological results suggest a distributed system of brain structures which subserve the recognition of facial expressions. These structures are also important for face recognition in general. The involved brain regions might differ partly between several expressions.

According to the functional model of face recognition by Bruce & Young (1986), the processes in question, namely the recognition of facial expressions and of identity, are assumed to be independent. It is apparent, that we do not need the information about ones facial expression in order to identify somebody. On the other hand, we can perceive the expression easily from familiar and unfamiliar people. The claim of independence was tested in a study by Young et al. (1986). Participants had to match simultaneously presented faces that were familiar or unfamiliar and had to react to identity or to facial expressions. Results revealed faster RTs for familiar than for unfamiliar faces in the identity matching task, but not in the expression matching task. According to the hypothesis of the latter task, familiarity is assessed by face recognition units that do not affect the structural encoding nor the expression analysis stage. Hence no difference in RT between unfamiliar and familiar faces is expected. Similar RT results were obtained from Bobes et al. (2000) in an identity and expression matching task. Simultaneously recorded ERPs revealed different topographical distributions of scalp potentials for both tasks and therefore provide evidence for the idea of distinct neural subsystems subserving the recognition of facial identity and facial expression recognition. Results from studies using fMRI point to the same conclusion of distinct neural correlates of facial recognition memory and the perception of facial expressions (Phillips, Bullmore, Howard et al., 1998). In addition, evidence supporting independence of the systems comes from the double dissociation of both processes in patients suffering from brain injury. Tranel et al. (1988) report three patients with Prosopagnosia, an inability to detect facial identity, whose ability to recognize facial expressions was preserved. In another patient study from Young et al. (1993) the authors found a selective deficit in the processing of facial expressions which was completely unrelated to the recognition of familiar and unfamiliar faces. The same conclusion is derived from results of Alzheimer’s Disease patients who were impaired in discriminating facial identities and in naming and pointing to different expressions while the discrimination of facial expressions was preserved (Roudier, Marcie, Grancher et al., 1998). Another line of evidene for the independence of facial familiarity and facial expressions comes from the N170 component (Bentin et al., 1996), an ERP component which has been linked to the stuctural encoding of faces (Eimer, 2000a). It has been shown that this component is insensitive to facial familiarity and facial expressions (Eimer & Holmes, 2002; Herrmann, Aranda, Ellgring et al. 2002). Hence, an interaction between facial expression processing and facial familiarity can be denied at least on early strucural encoding stages.

However, it might be of benefit, from an evolutionary perspective, to perceive the facial expression especially from familiar fellows in order to get reward or to prevent punishment. Furthermore, certain facial expressions like happiness or even sadness are more likely to be expressed to familiar people. Therefore, finding an interaction between the perception of facial expressions and facial familiarity might be possible.

In addition, the computer analogy that the brain is organized in independent modules, which work serially and independently is not presentable anymore (Grossberg, 2000). With its rich interlinking the brain is easily capable of parallel and unifying information processing. The different neuroanatomical areas that are involved in the various aspects of face recognition are interconnected through many efferent and afferent links as can be drawn from the model of Haxby et al. (2000). Thus, interactions of the processes might be possible depending on the temporal properties and availability of different aspects of processed information. There is evidence that the processing of facial expressions starts as early as 80 ms (Pizzagalli, Regard, & Lehmann, 1999) to 120 ms after stimulus onset (Eimer & Holmes, 2002; ) in the human brain. Therefore, it stands to reason that the information extracted from expressive faces may modulate early structural face encoding processes (Pizzagalli, Lehmann, Hendrick et al., 2002; Sato, Kochiyama, Yoshikawa, & Matsumura, 2001).

Relevant to the topic of the present dissertation, the lateral fusiform gyrus, which is involved in the processing of invariant aspects of faces and identity (Haxby et al., 2000), is interlinked with the sulcus temporalis superior and also with the amygdala. Both areas are crucial for the processing of facial expressions. If information of the expressed emotions of a face is availlable early on, it may be used to boost attention or arousal. In return, following perceptual processes might work more efficiently. Krolak-Salmon, Fischer, Vighetto, & Mauguiére (2001) reported differential ERP activity between 250 and 750 ms in occipital and occipito-temporal areas that was related to emotional expression in a gender or expression counting task. They took this as support for top-down modulations of limbic (including amygdala) and frontal areas influencing extra-striate visual areas. It is also well proven that emotional stimuli, including expressive faces, can be processed more easily outside the focus of attention when compared to neutral stimuli (Fox, Russo, & Dutton 2002). Using fMRI, Viulleumier, Armony, Clarke et al. (2002) found increased activation in the amygdala for emotional expressive faces, on task irrelevant locations and independent of spatial attention. Emotional stimuli can guide focal attention to the relevant location because the amygdala is part of the attentional system (Eastwood, Smilek, & Merikle, 2001). Such mechanisms may have been important in the evolutionary development of many organisms to detect threats in the environment. Therefore, it is possible that under certain circumstances the recognition of facial expressions and identity may interact. Fast recognition of facial expressions, especially from conspecifics could have been relevant for survival in evolution. Furthermore, even if identity and expression analysis use different functional and neuroanatomical components (Bruce & Young, 1986; Haxby et al., 2000) they are linked through the cognitive system and an interaction is not necessarily excluded.

Recently, there have been studies that suggest an interaction between facial familiarity and the perception of facial expressions and vice versa. Schweinberger and Soukup (1998) used a selective attention paradigm by Garner (1976) to address the question of an asymetric relationship between facial identity and facial expressions. Four different stimuli varied along the two dimensions identity (person A vs. B) and expression (happy vs. sad). Participants had to perform a speeded discrimination task on either one dimension which is called as relevant. Three different experimental conditions were applied. In the control condition the relevant dimension is varied between stimuli, whereas the irrelevant dimension is kept constant (e.g. only person A displays a happy or sad expression in case of an expression discrimination task). In the orthogonal condition both dimensions varied orthogonally (person A and B displayed both facial expressions, respectively). For the correlated condition both stimulus dimensions are correlated (e.g. person A displayed only the happy expression whereas person B displayed only the sad expression). An increase in RT would be expected for the orthogonal condition when compared to the correlated one in case of an influence of the irrelevant dimension on the relevant one. Accordingly, Schweinberger and Soukup (1998) found increased RTs for the orthogonal condition of the expression discrimination task when compared to the correlated condition. This did not hold true for the identity discrimination task. The results point to an asymmetric interaction between facial identity and the discrimination of facial expressions but not vice versa. There is evidence for an interaction in both directions for famous versus unfamiliar faces. The familiarity of a face can facilitate the discrimination of expression. In a study by Baudouin et al. (2000) participants had to discriminate neutral from happy facial expressions. It was expected that an interaction between facial familiarity and the discrimination task would only emerge if expression discrimination is slowed down by a hard condition. Therefore, faces were displayed either with a shortened presentation time of 15 ms (vs. 400 ms) or with a concealed mouth (vs. the whole face). Results revealed a facilitation of the expression discrimination for famous faces when compared to unfamiliar faces only in the hard conditions. The authors concluded that facial familiarity increases the “perceptual fluency” and therewith the recognition of facial expressions under hard conditions.

Conversely, there is also evidence that facial expressions have an influence on the perception and recognition of familiarity (Baudouin et al., 2000a; Nagayama, 1999). In addition, differential effects were found for personally familiar and famous versus unfamiliar faces when performing a familiarity discrimination task (Endo et al., 1992). The recognition of personal familiarity was facilitated when faces displayed a neutral expression when compared to happy and angry expressions. In contrast, famous faces were recognized faster with a happy expression. It was argued by the authors that a neutral expression is more frequently seen on personally familiar faces whereas famous faces are more often seen with a happy expression.

Although all of these studies suggest an interaction of the perception of facial expressions and facial familiarity in one or the other direction some of them suffer from methodological insufficiencies. In the study of Schweinberger and Soukup (1998) a small stimulus set was used displaying only two different individuals. The paradigm of selective attention as introduced by Garner (1976) was originally designed to explore the perception of simple stimulus dimensions such as shape or colour. Therefore, it is not designed to handle the processing of such complex facial information with overlapping features. A detailed and critical review about the implementation of the Garner-paradigm on facial perception is given by Kaufmann (2002). There is a main problem that arises from using a small set of facial stimuli. Facial expressions and facial identity share at least some overlapping features (Calder et al., 2000), therefore it is not possible to increase the variability of the irrelevat dimension in the orthogonal condition without also affecting the variability of the relevant stimulus dimension. This latter increase of variability can lead to stimulus based differences between the orthogonal and the control condition that are not based on interactions between the two processes. Another important objection, especially when questioning the interaction of facial expressions and facial identity, are different picture based strategies that can be used within both tasks (Kaufmann, 2002). In the identity decision task pictorial strategies might be used to discriminate the individuals based on non-facial cues like overall contrast of the pictures. Such effective strategies may not be possible within the expression decision task because information about expressiveness relies only on internal facial features. Accordingly, this will lead to increased variability of the relevant stimulus dimension within the orthogonal condition when compared to the irrelevant dimension which should exclusively be increased in variability. Kaufmann (2002) was not able to replicate the results of Schweinberger and Soukup (1998) by trying to consider the problems of the Garner-paradigm when using faces as stimuli. However, when using a different paradigm an interaction may emerge.

It can be annotated, critically, to the study of Baudouin et al. (2000), that the perceptual variation (concealed mouth or short presentation time) in order to aggravate facial expression discrimination, may have affected the two particular expressions differently. Possibly, the mouth region is more important for the recognition of happiness when compared to the neutral expression. On the other hand, the perception of identity for familiar people relies more on internal facial features when compared to unfamiliar faces. This implies that the variability in the hard condition was not evenly distributed over the critical dimensions of facial familiarity and facial expression. Hence, an interaction of the hard/easy condition and familiarity in the expression discrimination task may arrise because of the differential effect of the perceptual manipulation on the variability of the familiarity dimension. Evidently, more research is needed to clearly speak for or against an interaction of facial familiarity and facial expressions.

To briefly summarize, experimental data were cited which favour the independence of facial expressions and facial familiarity. This is underlined by clinical observations from patients which suffer from selective impairments of one or the other process. Electrophysiological and functional imaging studies point to separable neuronal subsystems underlying both processes. However, an interaction of facial expressions and facial familiarity is reasonable from an evolutionary point of view and when considering the rich afferent and efferent linking within the brain. Temporal properties of the involved processes may also be important for a possible interaction. There is a strong body of evidence showing that facial expression information is processed early in the brain. Thus, other processes that are involved in face recognition might be affected by a top-down influence from this information. Most importantly, recent data suggest an interaction between the perception of facial expressions and facial familiarity. Thus, some studies suffer from methodological problems. Therefore, more research is needed to clarify the raised controversy.

The present dissertation is an attempt to elucidate this question with a 2-choice RT-paradigm. By means of ERPs a closer look is taken into the temporal properties of the involved processes and the functional locus of interaction. All principles and the basis of the experimental paradigm will be explained in the following section 1.2.4. In addition, an overview is given about the ERP components that are important for the hypotheses of the experimental parts.

The experimental logic of the present dissertation is mainly based on the overall paradigm of mental chronometry together with cognitive psychophysiology. Therefore, it is important for the reader to get an introduction to this methodology in order to comprehend the experimental design, the working model, and the derived hypotheses (in the experimental part). Due to the narrow framework of the dissertation this introductory overview is far from complete and will mention only the most important and relevant milestones in the history of mental chronometry and cognitive psychophysiology. Several important models can just be sketched roughly.

A major theme in the study of mental chronometry is the question of whether there are separable processing stages witin the cognitive information processing system and how these stages communicate with each other. The assumption is, that mental processes are time-consuming. The sum of all processing is reflected by behavioural data like the RT or error rate. The general experimental paradigm consists of a series of imperative stimuli (auditory, visual, or somatosensory) and a required response mostly as fast and correct as possible. Sometimes a warning stimulus is included ahead of the imperative stimulus either with or without provided information about the upcomming stimulus. Depending on the proposed model of cognitive processing, different hypotheses can be drawn concerning the dependent measures.

Mental chronometry (Posner, 1978) has a long history in the study of human information processing (Meyer, Osman, Irwin, & Yantis, 1988). Already the astronomers in the 19^th century searched for ways to measure the speed of mental processes because of individual differences in the subjective measurement of the movement of stars. Bessel (1823, cited after Meyer et al., 1988) for instance introduced a personal equation to measure the difference between these estimates for two different observers.

In 1850 Hermann von Helmholtz (cited after Meyer et al., 1988) introduced the simple RT procedure making him the most important forerunner of modern mental chronometry and cognitive psychophysiology. With his procedure he was able to estimate the rate of neutral conduction by considering the difference in RT between a simple reaction after a tactile lower limb stimulation compared to an upper limb stimulation. The mean RT of the latter task is somewhat shorter when compared to the mean RT of the affore mentioned. The RT difference being caused by the longer distance that the sensory nervous singal has to pass from the lower limbs.

Another major development was the subtraction method and the introduction of the choice RT procedure by Donders (1868, cited after Meyer et al., 1988). His technique used three types of RT-procedures in combination to calculate the duration of putative stimulus discrimination and response selection stages and the simple motor response. By subtracting the RT in the simple RT-task (which is supposed to consist just of the motor process) from the choice RT-task (including all three stages in question) and a go/nogo RT task, which does not include the response selection stage, the duration from either of the three stages can be estimated. Some assumptions are necessary to apply this method. First, the stimulus discrimination and the response selection stage are in strict succession and combine additively. Second, any processing stages may be inserted or deleted in a pure fashion.The procedure’s assumptions have major shortcomings which were pointed out by Külpe (1893, cited after Meyer et al., 1988) and caused a fall in favour for this method. It became quiet in the field of mental chronometry after the criticism by Külpe, although there were still some important develpoments like the discovery of the psychological refractory period by Telford (1931, cited after Meyer et al., 1988), the research on perceptual and response competition by Stroop (1935, cited after Meyer et al., 1988), or the calculation of speed-accuracy tradeoff curves from movement control tasks by Woodworth (1899, cited after Meyer et al., 1988). A speed-accuracy tradeoff curve is a function of error rate versus RT in a given task. In general it reveales a tradeoff between accuracy and movement speed, and shows that faster RTs lead to increased error rates and vice versa.

Stimulated by new developments in computer and communication science around the 1950^’s the chronometric paradigm regained importance (Meyer et al., 1988). This was mainly a result of the use of new tools to collect and process data as well as to test models of human information processing. Picking up the ideas of Donders (1868, cited after Meyer et al., 1988) scientists searched for alternative methods to study the durations of human information processing stages. In 1969 Sternberg developed the additive-factor method (AFM) to analyze RT data. The AFM overcomes the problematic assumption of pure insertion of processing stages. At least two experimental factors should be manipulated in order to draw conclusions from their RT effects on the involved stages. Additivity in RT is observed when both factors are manipulated simultaneously and the first factor affects the RT independently of the level of the second one. In contrast, an interaction is observed when one factor shows different effects on RT depending on the level of the second factor. If, for instance, two factors show additive effects on the RT they may influence two different stages in the information processing chain. On the other hand, if two factors show an interaction on RT they act on at least one processing stage in common. The AFM still has some problematic assumptions. It assumes serial stages without temporal overlap and a discrete output of the stages. Keeping in mind the rich interlinking of structures in the human brain these assumptions seem implausable. Anyhow, the AFM is a powerful tool to analyze RT data. Since its publication innummerable studies have relied on this method. Sanders (1998) gives a nice summary of the research about human performance including the AFM. At least six independent and physiologically plausible processing stages have been identified: preprocessing, feature extraction, feature identification, response selection, motor programming, and motor adjustment (Sanders, 1980).

Besides the AFM there are also other models concerning human information processing. In contrast to the assumption of serial processing their stages may overlap in time and work in parallel. The cascade model by McClelland (1979) assumes continuous information transmission between the distinct functional processing ‘levels’. The continuous information output from one processing level can be used by another processing level. This may enable processes to work in parallel. If a certain activation threshold is exceeded, a response is executed at the end. Until today several results point to the existance of parallel processing stages (Abdel Rahman et al., 2000; Miller, 1982, 1983; Eriksen, & Schultz, 1979). Another important issue is whether the information transmission between parallel processing stages consists of continuously increasing activation (McClelland, 1979) or of discrete chunks of information (Miller, 1982).

Although the methods for analyzing RT data described above are helpful to draw hypotheses and conclusions about the underlying processing stages, they are yet only based on the final output of many processing stages in common. In addition, the methods suffer from their more or less physiologically plausible presumptions. Psychophysiological measures were used within chronometric paradigms in the last quarter of the 20^th century in the hope of getting more direct measures of the underlaying processing stages. This so called ‘marriage between psychophysiology and cognitive psychology’ (Coles, 1989) brought benefits by using ERPs that are regarded as markers of physiological processes. ERPs are electrical potentials that are time locked to a specific event caused by the simultaneous activity of populations of neurons in the brain (Coles, Gratton, & Fabiani, 1989). If neurons have an optimal allignment they form an electrical dipole and the signal can be recorded at scalp electrodes. However, the recorded electroencephalogram (EEG) reflects all electrical activity within the brain that may not be related to a specific event. Usually, the event related signal is not visable within the noisy EEG. Therefore, ERPs are derived from the EEG by averaging samples of the EEG around this particular event. Hence, the ERP signal is discriminated from the randomly varying backround noise of the EEG. The result is a voltage by time function with positive and negative voltage peaks. Each peak or component can be described in numerous ways – according to polarity, latency, experimental / psychological precondition or according to topographical distribution on the head surface. In general, offline the ERP data are treated further by setting a baseline or applying a low-pass filter in order to increase the signal-to-noise ratio.

There are different possibilities of the quantification of components as they can be used as dependent measures in an experimental design. One possible latency measure is the onset latency of a component which represents the time in ms from the onset of the imperative stimulus to the onset of the component. The onset is defined in a certain way, e.g. the point in time when the signal exceeds a threshold. As another latency measure the peak latency is used, the time in ms from the onset of the imperative stimulus to the highest peak of a component. A third latency measure will be the time in ms from the onset of a particular component to the overt response. In addition to the latency measures, components can be described by using the baseline-to-peak amplitude (in µV) or an area measure by calculating the mean amplitude (in µV) in a defined time range.

In the present dissertation I will use three different ERP components. They are briefly outlined, in order to understand the experimental design and hypotheses of this dissertation.

The first ERP-component to be considered is the N170, most prominent at posterior-temporal and occipital sites. This component is selective for faces (Bentin et al., 1996; Eimer & McCarthy, 1999), although also emerging for single face parts like eyes (Eimer, 1998) and other face-like and well learned stimuli that are distinguishable on an item level (Carmel, & Bentin, 2002; Gauthier, & Tarr, 1997). In addition, it is delayed and enhanced in amplitude for inverted (Eimer, & Holmes, 2002; Rossion et al., 1999; Rossion, Gauthier, Tarr et al., 2000) and contrast reversed faces (Itier, & Taylor, 2002). The mentioned results indicate that the N170 is associated with the formation of a visual representation of a face like stimulus and may reflect the functional process of structural face encoding (Eimer, 2000; Bentin et al., 1996) as denoted by Bruce and Young (1986). Another frequent finding concerning the N170 is the insensitivity to facial familiarity and facial expressions (Eimer, & Holmes, 2002; Bentin & Deouell, 2000). Owing to these properties I propose the N170 to be the first functional marker in the working model outlined below which is supposed to be linked to the initial structural encoding process of a face (see Bruce & Young, 1986).

As another functional marker I use the P300 component, a positive centroparietal deflection peaking not earlier than 300 ms poststimulus. Its amplitude and latency are meaningful for the nature and timing of a participant’s cognitive response to a stimulus (Johnson, 1986). Usually it is elicted in an oddball task where the amplitude increases for infrequent stimuli of the auditory (Duncan-Jahnson, & Donchin, 1977) or visual modality. At this point it is noted that novel non-target stimuli sometimes elicit a more frontally distributed earlier positive deflection that has been referred to as the ‘P3a’ (Snyder, & Hillyard, 1976; Harmony, Bernal, Fernández et al., 2000). Distinction has been made to the ‘P3b’ or P300 as described here, which usually occures after target stimuli. Properties of the P300, like an increased amplitude depending on stimulus probability, on attention (Dujardin, Derambure,Bourriez et al., 1993), on stimulus complexity (Verbaten, 1983), or on subjective stimulus value (Begleiter, Porjesz, Chou, & Aunon, 1983) met in different functional assumptions and models. Frequently adopted is the view that the P300 reflects the online-updating of the working memory (Donchin, & Coles, 1988; Sommer, & Matt, 1990). It is undisputable that the P300 also reflects attentional function (Holdstock, & Rugg, 1995; Dujardin et al., 1993). Most importantly, it is elicited after the categorization of task-relevant stimuli. Therefore, its peak latency is highly correlated to the task relevant dimension. Findings that the P300 latency is affected by stimulus discriminability and degradation (Kutas, McCarthy, and Donchin, 1977; McCarthy & Donchin, 1983) indicate its contingency to perceptual stimulus evaluation. These last two mentioned properties of the P300 component will be most crucial for the working model proposed below.

The third event-related component, the LRP (Coles, 1989), is used to indicate the preparation of the response hand (De Jong, Wierda, Mulder, & Mulder, 1988). It is derived from the readiness potential (Kornhuber, & Deecke, 1965) which is recorded at the electrode sites C3’/C4’. They are placed over the left and right primary motor cortices (M1). In case of preparation of one or the other response hand the negativity at the contralateral recording site is increased because M1 controls the contralateral body side respectively. This asymmetry in the recorded scalp potential at the electrode sites C3’ and C4’ is isolated by a specific calculation procedure (as described below; or see Coles, 1989). The result is a negative going LRP for correct movements as well as a positive deflection for incorrectly prepared movements. The generation of the LRP is at least partly ascribable to M1 (Eimer, 1998). This is supported by the observation of a positive LRP after correctly prepared foot movements (Brunia, & Vingerhoets, 1980). Activated neurons in the M1 controlling the lower extremities are embedded in the central sulcus. They generate an electrical dipole which emanates to the contralateral hemisphere. This results in increased negativity over the ipsilateral movement side. Hence, according to the calculation procedure a positive LRP is derived.

Osman, Moore, and Ulrich (1995) proposed to separate the information processing from stimulus to overt response into two intervals with the LRP. The first interval from stimulus presentation until the beginning of central response activation – that is, the LRP – is calculated synchronized to the stimulus (S-LRP). It is informative about the time demand of processes running before or during the response activation (e.g. Leuthold, Sommer, & Ulrich, 1996). The second interval from the onset of the LRP until the overt response is averaged synchronized to the response (LRP-R). It indicates the time demand of motor processes beyond central response activation. Depending on the processes that are affected by an experimental manipulation latency differences can be expected in one or the other interval. If it acts on processes before or during central response preparation, the S-LRP interval will be affected. On the other hand, effects can be seen within the LRP-R interval if motor processes beyond hand activation are affected by the manipulation.

As outlined in section 1.2.3. there is reason to believe that the processing of facial expressions and facial identity may be interdependent under some circumstances. There are still many unsolved aspects concerning such interdependency as well as the prerequisits and conditions under which an interaction might be observed. In the present dissertation I elucidate upon the question of whether the recognition of facial expressions and of facial identity can interact under certain conditions. Most importantly, the functional locus of this interdependency comes into question. Performance data and ERP components can help to pinpoint the functional locus within the cognitive system. Because of the used paradigm and tasks participants had to perform, it is possible to take a closer look at the functional chronometry of the perception of facial expressions and identity independent of each other.

In order to clearly outline the hypotheses and draw predictions for all following experiments a simple working model will be introduced. It is mainly based on the functional model of face recognition by Bruce and Young (1986) which has already been described above. The working model tries to integrate the two different experimental tasks which will be used in the conducted experiments with dependent measures. In the experimental parts, participants had to perform a two-choice RT task either to discriminate between two different equiprobable facial expressions on separately presented portraits or to discriminate facial familiarity. In the task formerly mentioned, facial familiarity was varied indepenently of expression – that is, half of the presented portraits belonged either to familiar or unfamiliar faces. In the latter task the other dimension – facial expression – was varied independently. Each person was presented with either two or three different facial expressions. Within the working model different functional processes are assumed to possibly be affected (Figure 3).

Figure 3. Proposal for a functional working model on which the following experiments are based, illustrated by an expression discrimination task.

Firstly, the presented face has to be perceived and categorized as a face in the so called structural encoding stage. As outlined above, the peak latency of the N170 component reflects the temporal properties for this processing stage. Then the relevant stimulus information has to be extracted in order to perform the discrimination task. Here, the P300 component serves as a functional marker since it is related to the perceptual evaluation of task relevant information. It is worth noting that according to the model by Bruce and Young (1986) the expression analysis and the recognition of facial familiarity are thought to be independent processes functioning in parallel. Thus, depending on the experimental task the other irrelevant dimension (e.g. recognition of familiarity within an expression discrimination task) may still be recognized automatically and in parallel. In addition, it has to be mentioned that the recognition of facial expressions is in most cases somewhat faster than the recognition of facial identity. If the familiarity may affect the expression discrimination in an expression discrimination task this may only be possible when this task is slowed down (cf. Baudouin et al., 2000). In a third assumed stage the response hand has to be selected. The LRP reflects the preparation of a specific response side. After motor preparation and adjustment the overt response is executed.

The main hypothesis of this dissertation is to find a facilitative interaction between the recognition of facial expressions and of facial familiarity. This would contrast the assumptions of the functional model of face recognition (Bruce & Young, 1986). The interaction should manifest itself in the behavioural data. Admittedly, in Bruce and Young’s model both processes are interlinked through the cognitive system and an interaction is not excluded per se. Thus, the temporal properties of the two involved processes are an important prerequisite for an interaction. The present dissertation also questions, under which circumstances the predicted facilitative interaction can emerge. In addition, it is attempted to localize the functional processing stage which is connected to the expected facilitative interaction.

To understand the chronometric approach of the experimental parts this logic will be outlined here in more detail. Several possibilities are left by the attempt to detect the functional processing stage on which a facilitative interaction may occur. If faster RTs are present within a specific condition this facilitation should be reflected in the ERPs by shortened peak or onset latencies. An earlier peak of the N170 component would point to the facilitation of early perceptual processing – the structural encoding stage. If task relevant late perceptual processing stages depict the locus of confluence, an earlier P300 peak latency should be present in the facilitated condition. In addition, the peak latencies of all earlier components – in this case the N170 - have to be the same in this condition. Otherwise the facilitative effect of the earlier processing stage may just have propagated from this stage to the next one. Possibly, the facilitative effect due to an interaction between one process and the other may act on the response selection stage. In this case, shorter onset latencies of the S-LRP should be present within the facilitated condition. Again, this interpretation is only valid if all earlier components do not differ between the facilitated and non facilitated condition. The last functional locus of interaction to be proposed is motor preparation beyond hand selection. If this process is facilitated the interval between LRP onset and overt response (LRP-R) should be shortened.

Since the empirical backround, the main hypothesis, and the principle logic of the experimental paradigm are now outlined, a short overview is given about the following experimental parts. The experimental Part I will elucidate the question whether there is an interaction between facial familiarity and the discrimination of facial expressions. Experimental Part II reserves the question and asks whether there is an interaction between facial expressions and the discrimination of facial familiarity. After the experimental parts all results will be discussed thoroughly in a final general discussion and a perspective for further research will be given.

In Experiment 2 the same expression discrimination task was used as in Experiment 1. Participants had to discriminate between the two facial expressions happiness and disgust. The same stimulus set as approved in Experiment 1 was used. Again, the presented portraits displayed either a personally familiar or unfamiliar face with a weak or strong facial expression.

The main purpose of the present experiment was to replicate the facilitating effect of personal familiarity on the discrimination of facial expressions. This effect should be manifested in RT and error rates. In addition, the functional architecture of the underlying processes is assessed by means of ERPs. In order to pinpoint the functional locus of an interaction between facial familiarity and facial expressions different ERP components will be used as time and latency markers for the processes that are involved. Amplitude and topographical distributions of the potentials can give a hint to the availlability of extracted and processed stimulus information.

Concerning RT und error rates, the same facilitating effect of personal familiarity on the discrimination of facial expressions is expected as in Experiment 1. This may not necessarily be the case for personally familiar faces with weak expressive intensity. Based on the results of the previous experiment, a facilitative effect of personal familiarity regarding the discrimination of facial expressions is expected, especially for faces displaying happiness.

Different possibilities arise for the functional locus of the interaction between personal familiarity and the discrimination of facial expressions. If personal familiarity facilitates the structural encoding of a face, the temporal properties for early perceptual processes - as indexed by the N170 peak latency (Rossion et al., 1999a) - should be reduced for familiar faces. Although worth mentioning, this possibility is rather unlikely because of a strong body of results showing that the peak latency of the N170 is not influenced by facial familiarity nor facial expressions (Eimer, 2000; Rossion et al., 1999a). If late perceptual processes and accordingly the perception of facial expressions are influenced, an effect should be found on the peak latency of the P300 component peaking earlier for personally familiar faces. In addition, no effect of familiarity should be present for the peak latency of the N170 component. In the case of a later functional locus, namely the response selection stage before hand selection, the S-LRP interval is expected to be shorter for familiar faces when compared to unfamiliar ones. In this case, the N170 and P300 latency should not be affected by familiarity. Otherwise, a functional locus on late perceptual processing stages cannot be excluded because of propagation of the advantage for personally familiar faces to the following stages. Effects might also be found on the LRP-R interval. If familiarity shortens the time needs for motor preparation beyond hand selection, a reduced LRP-R interval for personally familiar faces should be present in the data.

Participants. 16 participants (11 women and 5 men; mean age = 24,6 years; aged between 20 and 32) took part in the experiment. They were personally familiar with half of the portrayed persons presented in the experiment. The participants fullfilled either course requirement or received a payment of 15 €. The mean handedness score (Oldfiled, 1971) was 78 (ranging from –82 to +100).

Design and Procedure. As in Experiment 1, participants had to discriminate the two facial expressions happiness and disgust. The same stimulus set was used as in the previous experiment. It was repeated three times with the exception that portraits of only 28 people were presented, half of them being personally familiar to the participant. For each participant 14 most familiar persons out of 16 potentially familiar ones were selected, respectively. The trial sequence was the same as in Experiment 1. The response side-to-expression key assignment was changed three times. This was done to calculate an LRP for the whole experiment and for every repetition of the stimulus set, respectively. Participants performed 40 practice trials by pressing the correct button according to the term refering to the crucial expression (“Freude”, happiness or “Ekel”, disgust) presented on the screen at the beginning of the experiment and before changing the key assignment.

Electrophysiological recording. The EEG was recorded from 31 electrode sites (Figure 7) including IO1, IO2, LO1, LO2, Fp1, Fp2, Fz, F3, F4, F7, F8, FT9, FT10, Cz, C’3 and C’4 (4 cm to the left and right of Cz, respectively), T7, T8, Pz, P3, P4, P7, P8, P9, P10, PO9, PO10, O1, O2, Iz and the right mastoid (M2) according to the modified 10-20 International System (Pivik, Broughton, Coppola et al., 1993). Tin electodes were used with ECI Electro-Gel^TM electrolyt paste and placed with an electrode cap (Electro-Cap International Inc. ). Electrode impedance was kept below 5 kΩ. The electrodes above and below the left eye (Fp1 and IO1) and next to the outer canthus of each eye (LO1 and LO2) served for controling EOG artefacts. All electrodes were referenced to the left mastoid (M1). A low pass filter was set at 30 Hz. The electrophysiological signal was digitized with 250 Hz and recorded continously together with triggers that marked stimulus onset and reaction.

Figure 7. Recording positions of the scalp electrodes for recording the EEG.

Offline the recording was segmented into epochs of 2300 ms for response synchronized onsets starting 1800 ms before the response. The EEG-data were bandpass filtered with high and low cutoff frequencies set to 0.01 and 8 Hz, respectively. All trials with incorrect responses, with signal drifts of more than 120 µV within the recording epoch or with other EEG artefacts were discarded. Blink trials were corrected by the method described by Elbert, Lutzenberger, Rockstroh, and Birbaumer (1985); if this was not possible, the trial was discarded. Stimulus synchronized epochs of 1200 ms length were generated by averaging the response synchronized epochs around a variable point representing the stimulus onset. It was calculated by subtracting the single trial RT from the response trigger for each epoch, respectively. An average reference was calculated disregarding the electrodes IO1, IO2, LO1 and, LO2. In order to calculate the ERPs, epochs were averaged according to the experimental conditions.

The LRP was derived by calculating the difference between the potentials contra- and ipsilateral to the responding hand at electrode sites C’3 and C’4 and averaged across hands (Coles, 1989). To assess possible influences of horizontal eye movements on the LRP, the lateralized horizontal EOG (LhEOG) was calculated for the electrode sites LO1 and LO2 in the same way as the LRP. This calculation was applied on response synchronized epochs as well as on stimulus synchronized ones.

Data analysis. Statistical analysis of RT and error rates were performed by means of Huyhn-Feldt corrected repeated measures ANOVAs, including the within-subject variables familiarity (personally familiar vs. unfamiliar), expressive intensity (weak vs. strong), and facial expression (happiness vs. disgust). As in the previous experiment Bonferroni-corrected significance levels were applied in the case of post-hoc comparisons by means of multiple t-Tests. In Experiment 1 it was hypothesized that personal familiarity can only act facilitatively on the discrimination of facial expressions when this process is slowed down, e.g. by weak expressive intensity. Surprisingly, a facilitation for personally familiar faces had been found within the strong expressive intensity condition. Therefore, an additional analysis of the RT data was applied based upon a median split. All trials were divided into trials with fast and slow RT according to the median of each condition and participant. A Huyhn-Feldt corrected repeated measures ANOVA was calculated including the factors RT- bin (fast RT trials vs. slow RT trials), familiarity, intensity, and facial expression.

For the event-related potentials a jackknifing based method (Miller, Patterson, & Ulrich, 1996) was applied to measure the onset-difference for any two conditions of the LRP and of the P300 peak latency. For the latter component the condition values at the electrode site Pz were determined. Because of the low signal-to-noise ratio of the LRP and sometimes indistinguishable P300-peaks within a single subject, signal quality was improved by averaging n times n-1 participants. The values of each new jackknifing-participant served as an estimate of variance for the participant which was left out. The t-values of the subsequent one-tailed t-Tests were adjusted by the value described by Miller et al. (1996). In order to assess possible influences of lateralized eye movements on the LRP, mean amplitudes over an interval of 100 ms around the point where the S-LRP and LRP-R onset occurred were derived from the LhEOG. Mean amplitudes were analyzed by means of Huyhn-Feldt corrected repeated measures ANOVA including the experimental factors familiarity and expression. For analysing the peak amplitude of the N170 component, measures were derived from averaged event related potentials at the electrode site P10. An Huyhn-Feldt corrected repeated measures ANOVA was performed with the factors familiarity, expressive intensity , and expression.

Analysis of mean amplitudes was conducted with average referenced data on a subset of 28 electrodes (omitting electrodes LO1, LO2, IO1 and, IO2). ANOVAs were performed within intervals of 50 ms starting from 200 ms until 600 ms after stimulus onset. The within-subject factors electrode site, familiarity , and facial expression were used. In addition, Huyhn-Feldt corrected repeated measures ANOVAs for all intervalls were performed on vector-scaled data (McCarthy & Wood, 1985) in order to assure that found differences are ascribable to differences in the topographical distribution.

Reaction time and error rate. Mean RTs and error rates for the different conditions of the expression discrimination task are displayed in Figure 8. Like in the previous experiment there was a strong effect of expressive intensity (F(1,15) = 159.8, p < .01) yielding faster RTs for portraits with strong when compared to weak intensity. However, there was no interaction of intensity with other factors. Participants showed also faster RTs on faces displaying disgust when compared to happiness (F(1,15) = 4.7, p < .05). Most important, RTs to familiar faces were faster than to unfamiliar faces (687 ms vs. 698 ms; F(1,15) = 9.9, p < .01). Again, this effect of familiarity is modulated by facial expression (F(1,15) = 5.2, p < .05) and only present in portraits with a happy expression (694 ms vs. 713 ms; t(15) = - 3.77, p < 0.01) but not for disgust.

An additional analysis was performed by splitting RTs according to the median per condition and participant. The 2-level factor RT- bin was included. The main effects of the factors familiarity, intensity, and expression as well as the interaction of familiarity and expression are not mentioned again, as they correspond to the afore presented analysis. In case of significant interactions with the factor RT- bin two post hoc ANOVAs were calculated for the separate bins, respectively. Most important, the speed of RT (factor RT- bin) significantly influenced the effect of familiarity in the expression discrimination task (F(1,15) = 12.5, p = .003). A facilitation for personally familiar over unfamiliar faces was only present for trials with slow RT (801 ms vs. 817 ms; F(1,15) = 11.8, p = .004), but not for trials with fast RT (773 ms vs. 777 ms; F(1,15) = 2.2, p = .16). The advantage of disgust over happy expressions was affected by response speed (F(1,15) = 5.2, p = .038) and is only observable for slow RTs (797 ms vs. 821 ms; F(1,15) = 5.8, p = .03), but not for fast ones (p = .16).

Figure 8. Reaction time and error rates of the expression discrimination task of Experiment 2 separated for familiarity and facial expressions.

Participants made less errors on portraits showing personally familiar faces when compared to unfamiliar faces (9,4 % vs. 11,6 %; F(1,15) = 16.1, p < .01) as well as on faces with strong expressive intensity when compared to weak intensity (5,8 % vs. 15,3 %; F(1,15) = 210.0, p < .01). This effect depends strongly on facial expression (F(1,15) = 19.7, p < .01) and in addition on intensity also (F(1,15) = 15.5, p < .01). Whereas for happy faces there are more correct reactions for familiar portraits independent of intensity (7,4 % vs. 13,9 %; ts(15) = -4.1, ps < .01); only with faces displaying weak disgust did participants make slightly less errors for unfamiliar faces when compared to familiar ones (t(15) = 3.4, p < .05).

Event related potentials. Figure 9 displays the N170 component at the electrode site P10 for the different conditions peaking exactly at 170 ms after stimulus onset. It is obvious that there are no peak differences and only minor amplitude differences between the different conditions. For this reason I abandoned any statistical calculation of the peak latency of the N170 component. Concering the peak amplitude an ANOVA confirmed the supposition of the lack of any differences between conditions (F < 1).

Figure 9. The N170 component for the expression discrimination task of Experiment 2 at the electrode site P10 separated for familiarity and disgust.

Figure 10. The P300 component for the expression discrimination task of Experiment 2 at the electrode site Pz separated for familiarity and expression.

The P300 is most pronounced at the electrode site Pz (Figure 10). Based on the significant interaction of the factors familiarity and expression in RTs, I expected an earlier peak of the P300 component for familiar portraits when compared to unfamiliar ones only for the happy expression. The jackknifing based comparison of the conditions familiar happiness versus unfamiliar happiness at the electrode site Pz revealed a trend (tJ(15) = 1.6, p < .10) with a slightly earlier peak for happy familiar faces.

The onset of the S-LRP (Figure 11) within the expression discrimination task started earlier for personally familiar portraits when compared to unfamiliar ones (tJ(15) = 1.98, p < .05). This difference was most prominent for portraits with a happy expression (tJ(15) = 1.78, p < .05) whereas it was absent for faces showing disgust (tJ _{< 1).}

No difference between conditions was found for the response-locked LRP (Figure 12).

An influence of the LhEOG on the S-LRP, and the LRP-R could be denied (ps > .10).

Figure 11. The stimulus-locked LRP for the expression discrimination task of Experiment 2 separated for familiarity and expression.

Figure 12. The response-locked LRP for the expression discrimination task of Experiment 2 separated for familiarity and expression.

Analysis of the mean amplitude distribution revealed significant interactions of electrode position by expression in all time intervals starting from 200 ms until 600 ms after stimulus onset (Fs(27,405) > 4.0, ps < .005; for all results see Appendix 6.1.). As can be seen in Figure 13, faces displaying disgut yielded higher amplitudes at parietal sites as well as lower amplitudes at temporal and occipital sites. In addition, in the same time intervals (200 ms until 600 ms post-stimulus), the interactions of electrode position by familiarity was also significant (Fs(27,405) > 5.2, p < .001). Higher amplitudes were observed for personally familiar faces at centroparietal sites as well as lower amplitudes at parietal and occipital sites.

Vector scaled data revealed topographical differences between faces displaying happiness and disgust in the intervals from 200 ms to 350 ms after stimulus onset (Fs(27,405) > 3.7, ps < .008). In addition, topographical differences between personally familiar and unfamiliar faces were evident in the time intervals starting from 200 ms until 450 ms after stimulus onset and again in the time interval from 550 ms until 600 ms after stimulus onset (Fs(27,405) > 4.3, ps< .003).

Figure 13. Differences of the mean amplitude distribution between unfamiliar (UF) and familiar (F) faces (top row) and between happiness (H) and disgust (D; bottom row) for the expression discrimination task of Experiment 2 in all tested time intervalls; a grey shading equals a negative difference.

Although numerically smaller than in Experiment 1, a facilitation was observed for familiar faces when participants discriminated facial expressions. It was especially pronounced when the face displayed a happy expression. A speed-accuracy tradeoff cannot explain the pattern of results because error rate was decreased for familiar faces and especially for happy familiar faces. As was already evident in Experiment 1 and replicated here, a facilitation of familiarity was only found for happy faces. Hence, happiness might have a special role when recognizing familiar facial expressions. Next to the neutral expression it is probably the most frequently encountered facial expression on familiar faces. This could be one reason why there is an advantage recognizing especially this expression on personally familiar faces. Results from Endo et al. (1992) point in a similar direction, although their results are not exactly compatible with these results. In an identity discrimination task participants were faster in classifying personally familiar faces, as familiar, with a neutral expression and famous faces, as familiar, with a happy expression. The expression with which a person encounters the most might influence the representation of this persons face. In return, the access to the stored representation might be faster, the more similar the currently encountered face is to this representation. Hence, if personal familiarity was accessed faster for happy faces it could act facilitatively in the expression discrimination task. This might not hold true for faces displaying disgust.

Compared to the previous experiment the faciliative effect of personal familiarity on the discrimination of facial expressions was diminished. Possibly, the whole context of an electrophysiological experiment increased the vigilance and attention of the participants. Results of Baudouin et al. (2002) suggest, that selective attention is important for the independence of both processes. Increased selective attention might have diminished an interaction between facial familiarity and the discrimination of facial expression. Admittedly, in this experiment mean RT decreased by over 150 ms when compared to Experiment 1. An important pre-requisite of the main hypothesis was defused because of overall faster RTs. Hence, the impact of familiarity on the discrimination of facial expressions decreased. The additional analysis of RT by splitting all trails according to the median shed some light on the diminished effect. It was shown that the facilitative effect of personal familiarity on the discrimination of facial expression was most pronounced for trials with slow response speed. This was predicted by the initial hypothesis. Within the slower RT bin the expression discrimination lasted long enough that the processed information about personally familiar faces could act act as facilitative on the expression discrimination task.

An unexpected result was the decreased error percentage for unfamiliar faces when displaying disgust. Possibly, positive expressions (like a happy face) and negative ones (with a disgusted expression) act differently on the neurocognitive system. The detection of negative expressions, in order to decide quickly upon the approach and avoidance of a given situation, might be important for the survival of an organism. Indeed, the perception of disgust in the present experiment was faster when compared to happiness. Perceiving disgust involves structures that are also implicated in the evaluation of offensive stimuli (Phillips, Young, Senior et al., 1997). In this sense, unfamiliar faces may be perceived as more offensive than familiar faces and hence the recognition of disgust is more accurate for unfamiliar face.

Most importantly, but only based on a small effect, the facilitated perception of expression for personally familiar happy faces is reflected in the electrophysiological data. There was no effect of familiarity, nor an interaction with expression, on the peak latency of the N170 component leaving an early perceptual locus of the found interaction apart from consideration. The same holds true for the LRP-R interval and concurrently for the preparation of the motor system beyond hand selection. In contrast, a shorter S-LRP interval was found for personally familiar faces when compared to unfamiliar ones. Corroborated by RT results, a significant difference within the S-LRP data was also found when comparing only personally familiar and unfamiliar faces displaying happiness. In addition, a trend was present for the P300 peak latency which was decreased for happy familiar faces when compared to happy unfamiliar faces. This observed trend for happy faces might have been propagated to the following response selection stage, because there a significant difference between happy familiar and unfamiliar faces, as reflected by the S-LRP, was found. Therefore, an effect of familiarity on the response selection stage is not safe to conclude anymore. Concerning the trend of the P300 peak latency, the relatively high RT variability may have broadened the peak of the P300 component making the peak detection less reliable. Thus, the effect of familiarity within happy faces was blurred to only a trend.

The analysis of the topographical distribution revealed both an early significant difference for facial familiarity and facial expression. There was no interaction found between those factors. The discriminative information on facial familiarity as well as on facial expressions seems to be available as early as 200 ms after stimulus onset. It may be possible that the information about facial familiarity can influence later processing stages that are relevant for the participant’s task, and this may cause the facilitative interaction between facial familiarity and the discrimination of happy facial expressions which was found for personally familiar faces in the present data. Had an interaction of facial familiarity and facial expression been present for the topographical distribution it could be expected that the facilitative effect for personally familiar happy faces might be subserved by slightly different brain regions as when faces display disgust. However, no such interaction was present. The validity of this supposition remains unclear because brain regions might be involved which do not affect the topographical distribution.

Overall, the results point towards an interaction between the perception of facial familiarity and the discrimination of facial expressions for personally familiar faces. As intended, the event related potentials elucidated the functional locus of interaction within the information processing stream more clearly. They point to late perceptual or response selection stages as the possible loci of interaction between personal familiarity and the discrimination of facial expressions. Based on the presented ERP data no safe decision can be made favouring one or the other functional locus of interaction. Thus, further data are necessary in order to clarify this unclear functional locus.

It was the intention of the present experiment to replicate the results of Experiment 2 with a slightly changed design in order to increase the rather small facilitative effect of familiarity on the discrimination of facial expressions. As discussed before, the numerically small effect of familiarity and the relatively high variability in RT in the previous experiment may be one reason for the somewhat unclear effect in the P300 peak latency. In addition, the present experiment intends to get better control of the degree of personal familiarity by recording the skin conductance response (SCR) and applying a questionaire about people being personally familiar.

Prior to the experimental blocks the SCR was recorded for all personally familiar and unfamiliar faces. Within face recognition the SCR is highly linked to emotional arousal processes, orientating response, and attention. Hence, it can give a clue to the individual significance and emotional value of a face (Tranel et al., 1985) or the familiarity (Breen et al., 2000). In healthy participants but also in a clinical context, it is often found that the SCR is more enhanced for famous faces when compared to unfamiliar ones. Accordingly, the expectation is to find this pattern for personally familiar and unfamiliar faces in the present experiment.

The previous experiments gave the impression that the personal contact to some members of the teaching staff who served as personally familiar faces was either too little or too long ago for some participants. Therefore, after recording the SCR but before the start of the experimental blocks, an additional block was added presenting all personally familiar people with a neutral facial expression and some personal background information about each (e.g. age, children, biographical data, chair, lectures or seminars). It was intended to refresh the memory for personally familiar people. Subsequently after this block the experimental blocks were started.

Again, it is expected that personal familiarity facilitates the discrimination of facial expressions expecially for happy portraits. Because no interaction had been found with the factor expressive intensity in the previous experiment this factor was omitted and only portraits with strong expressive intensity were used. Thus, the hypotheses do not consider expressive intensity anymore. For ERP data more clear cut results are expected than in the previous experiment because of the omission of portraits with weak expressiveness and thus decreased variability in RT. As already implied by the trend concerning the P300 peak latency in the previous experiment it is expected that this latency should be reduced for personally familiar happy faces when compared to unfamiliar happy faces. Due to propagation, this facilitation may also be present in the onset latency of the S-LRP. No effects should be present for the N170 component as well as for the LRP-R. Hence, the expectation for the present experiment is to pinpoint late task relevant perceptual processing stages as indexed by the P300 as the functional locus of interaction between personal familiarity and the discrimination of facial expressions.

Participants. In the third experiment 12 participants (10 women and 2 men; mean age = 25,8 years; aged between 21 and 54) took part. As in the previous experiments, they were personally familiar with half of the portrayed persons presented in the experiment. For each participant 14 of the most familiar people out of 16 potentially familiar ones were selected. Participants fullfilled either course requirement or received a payment of 15 €. The mean handedness score (Oldfiled, 1971) was 74,8 (ranging from –40 to +100).

Design and Procedure. The dedign and procedure differs from the previous experiments in that two blocks were added prior to the experimental blocks. At the beginning of the session, the SCR was recorded within one block. In order to keep the participants attention as well as to enhance the SCR, a familiarity discrimination task was required from the participants. They were instructed to view all presented portraits and discriminate personally familiar from unfamiliar faces by pressing one of two response keys on a computer keyboard with their index or middle finger of the dominant hand. Participants were told not to move the non-dominant hand where the SCR electrodes were affixed and which rested comfortably on a soft pad. After 3 warm-up trials at the beginning of the block showing unfamiliar people all 14 personally familiar faces as well as all 14 unfamiliar matches were presented successively and in random order with a neutral expression. The trial started with a fixation cross in the middle of the screen presented for a random duration between 11 and 12 seconds. In the last 5 seconds of the interval the fixation cross turned bold in order to capture the attention of the participant. It was replaced by a portrait of a familiar or unfamiliar person. All portraits remained on the screen for 2000 ms. Participants were supposed to make their familiarity decision within this time window. No feedback was provided.

After the SCR recording, the electrodes affixed to the hand were removed. Subsequently, participants had to perform a block to refresh their memory on the personally familiar people. All 14 familiar portraits were presented with a neutral expression together with information concerning age, children, position at institute, given lectures and seminars, hobbies, and biographical information (e.g. ‘was a competitive ice skater in his youth’). Participants were told that after the experiment there would be a short questionaire in order to test the information. Participants could look at the information at leisure.

Following the block to refresh memory the experimental blocks for the EEG recording started. As in the previous experiments, participants had to discriminate the two facial expressions happiness and disgust. The same stimulus set was used as in Experiment 2 with the exception that all portraits with weak expressive intensity were omitted because of the previous lack of interaction of this factor with any other factor. The trial sequence was the same as in the previous experiments. The whole stimulus set was repeated twice. The response side-to-expression key assignment was changed twice in order to calculate an LRP. At the beginning of the experiment and before changing the key assignment, participants performed 40 practice trials by pressing the correct button according to the expression (“Freude”, happiness or “Ekel”, disgust) written on the screen.

Electrophysiological recording. The SCR was recorded with Ag/AgCl electrodes (1 cm in diameter) and isotonic electrolyte gel (K-Y Jelly, Johnson & Johnson^TM). They were affixed to the thenar and hypothenar of the non-dominant hand. The ground electrode was placed on the forearm of the respective hand. A Coulbourn^TM Isolated Skin Conductance Coupler (Model V71-23) was used. The sampling rate was 200 Hz. Offline the signal was cut in 8200 ms epoch starting from 200 ms before until 8000 ms after the stimulus and converted in micro-Siemens (µS).

The EEG was recorded from the same 31 electrode sites as in Experiment 2 (see section 2.2.2.). For recording, the same settings and materials were used as in the previous experiment. Offline the recording was segmented into epochs of 2300 ms for response synchronized onsets starting 1800 ms before the response. The EEG-data were bandpass filtered with high and low cutoff frequencies set to 0.01 and 8 Hz, respectively. The criteria and procedures for artefact removal and blink trial correction was kept the same as in the previous experiment. Stimulus synchronized epochs of 1200 ms length were generated by averaging the response synchronized epochs around a variable point as in Experiment 2. An average reference was calculated disregarding the electrodes IO1, IO2, LO1 and, LO2.

The LRP at electrode sites C’3 and C’4 and the LhEOG at electrode sites LO1 and LO2 were calculated with the same procedure as described above (cf. Coles, 1989).

Data analysis. For calculation of the SCR data only artefact free trials with correct familiarity decisions were included. The prestimulus baseline was set at 200 ms. A valid SCR trial was counted if the maximum peak 1 to 7 sec after stimulus onset exceeded the criterion of 0.01 µS. If this was not the case the trial was counted as invalid and the value of the peak amplitude was set to zero. Different measures were calculated for familiar and unfamiliar faces. The magnitude was defined as the mean peak amplitude of all valid and invalid trials. For the peak amplitude and the peak latency only valid trials were included in the condition means. The frequency defined the percentage of valid responses. For each measure an ANOVA was performed with the two-level factor familiarity.

Statistical analyses of RT and error rate were performed by means of Huyhn-Feldt corrected repeated measures ANOVAs including the within-subject variables familiarity (personally familiar vs. unfamiliar) and facial expression (happiness vs. disgust). If post hoc comparisons were necessary, t-Test were calculated with Bonferroni corrected significance levels. Like in Experiment 2 all trials were split in two bins acording to the median for each condition and participant. Again, an ANOVA was performed with the additional two-level factor RT- bin (fast RT bin vs. slow RT bin).

As in the previous experiment a jackknifing based method (Miller et al., 1996) was used to measure the onset-difference for any two conditions of the LRP and of the P300 peak latency measured at the Pz electrode. As before, mean amplitudes were derived from the LhEOG for a 100 ms intervall in order to assess a possible influence on the LRP. For analysising the peak amplitudes of the N170 component measures were derived from averaged event related potentials at the electrode site P10. An ANOVA was performed with the factors familiarity and facial expression. Analysis of the mean amplitude distribution and the topography was performed the same way as in Experiment 2.

Figure 14. Skin conductance response as the mean of all valid trials for personally familiar and unfamiliar faces during a familiarity discrimination task.

In the previous experiments it has been shown that personal familiarity facilitates the discrimination of facial expressions especially for faces displaying happiness. ERP results suggested late perceptual processing stages – as indexed by the P300 peak latency – as the functional locus of interaction between both processes. However, due to high variability in RT, results have been unclear. The used stimulus set consisting of personally familiar and unfamiliar faces may have been one source of variance. It is the purpose of the present experiment to improve experimental control over the stimulus set by using only unfamiliar faces, whereas half of the stimulus set is experimentally familiarized within a learning block.

Although the degree of familiarity of experimentally familiarized faces is low when compared to personally familiar faces, an interaction could still emerge. This is conceivable when we consider that ERP results from Paller and coworkers (Paller, Bozic, Ranganath et al., 1999; Paller, Gonsalves, Grabowecky et al., 2000) found parietal ERP differences for visually familiarized faces when compared to newly presented faces. The authors ascribe this difference to the retrieval of stored visual face information. Striking evidence comes from Rossion et al. (2003) who found lower PET activation for unfamiliar versus learned familiarized faces within the right lateral fusiform gyrus and the right inferior occipital gyrus. Interestingly, the differences are found in visual extrastriate brain regions that subserve both, the categorization of faces on an object level as well as the discrimination based on previous encounters. An overlapping set of brain regions seems to be involved in face detection, individual discrimination, and presemantic recognition (Rossion et al., 2003). The involvement may just occur over different time scales, which ERP data suggests (e.g. Bentin et al., 1996; Eimer, 2000; Schweinberger, Pickering, Burton, & Kaufmann 2002a). Considering the mentioned results, neuronal differences between familiarized and unfamiliar faces are evident and reflected in ERP data. In addition, it is possible that the access to familiar face representations is faster when triggered by typical familiar stimuli such as an often encountered facial expression on a familiar face. Evidence comes from findings of Jemel, Pisani, Calabria et al. (in press), and Schweinberger, Pickering, Jentzsch et al. (2002) who found increased priming effects for the same facial stimuli when compared to different primes of the same face. Therefore, the expression with which a person is familiarized within a learning block is varied in the present experiment.

Only behavioural measures are recorded in Experiment 4 since it is the first time that I use this stimulus material. The expectation of the present experiment is to find an interaction between facial familiarity and the discrimination of facial expressions for experimentally familiarized faces. Thus, one half of the unfamiliar people will be familiarized in an initial learning block. As mentioned before, the visual experience with different facial expressions of the persons to be familiarized is varied. Within the learning block one half of the persons will be presented with a neutral and a happy expression. The other half will be displayed with a neutral and an angry expression. For the expression discrimination task this means that each familiarized person was seen before with only one expression, but not the other. The condition with the expression that was not encountered for a familiarized person is of particular interest. Previous results of Baudouin et al. (2000; Experiment 3) suggest that the facial expression of a person can be discriminated more accurately when it was familiarized with a happy when compared to a neutral expression. In contrast, it might also be possible that the previous encounter of a specific emotional expression can facilitate the discrimination of facial expressions. In this case it is hypothesized that, depending on the expression that was presented during the learning block, the discrimination is facilitated for a familiarized person displaying this already encountered facial expression. It could also be, that general visual familiarity of a face may facilitate the discrimination of facial expressions. In this case a facilitation should be present for all familiarized persons independent of the learned emotional expression.

In advance of the experiment a rating was conducted in order to select a stimulus set of 40 male and 40 female people out of 104 different unfamiliar people. For each person 3 pictures were available with either a neutral facial expression, happiness, or anger. Five participants (all female, mean age = 27.2 years, aged between 24 and 33) rated every portrait according to the six categories strong or weak happiness (“starke Freude”; “leichte Freude”), neutral (“Neutral”), strong or weak anger (“starker Ärger”; “leichter Ärger”), or none at all (“Keine von den angegebenen Kategorien”). Participants were instructed to use the last option mentioned sparingly. It did not belong to the rating scale but conveyed important information about the stimuli. Participants had as much time as they needed for rating the faces. Afterwards, the ratings were recoded on a scale ranging from -3 to + 3 and the percentage of correctly classified ratings was analyzed. People, whose portraits were less often correctly categorized were excluded, and 80 people (40 male) remained. They were divided into 4 subsets with 10 men and 10 women, respectively.

anger

happiness

neutral

Subset 1

- 2.34

2.71

1.05

Subset 2

- 2.35

2.73

1.11

Subset 3

- 2.35

2.69

1.11

Subset 4

- 2.34

2.75

1.13

Table 1: Mean expression ratings of 4 subsets of unfamiliar faces.

The decisive value was the mean rating of anger, because the variation was larger when compared to happy and neutral expressions. The latter two expressions were correctly categorized most often. The mean ratings for the three facial expressions happiness, anger, and neutral were comparable in each group (Table 1). That is, each group contained about an equal amount of people with high and low mean ratings of happiness or anger.

Participants. Twelve participants (7 women and 5 men, mean age = 25.5 years, aged between 19 and 36) took part in the present experiment. All participants had normal or corrected to normal vision. They received either course credit or 10,00 € for participation. Handedness was not determined because electrophysiological data were not recorded.

Stimuli and Apparatus. All portraits of the stimulus set consisting of 80 persons were edited in Adobe Photoshop to 256-colour pictures with a bluish grey background. For the initial learning phase, the size of the pictures was edited to 269 by 350 pixels. For the following experimental blocks they were resized to 190 by 247 pixels. This was done to present the pictures in about the same size as the portraits in Experiments 1 to 3. All stimuli were presented on a 17-inch screen with a size of 10.8 x 14.0 cm for the learning phase which equals a visual angle of 6.1^o horizontal and 8.0^o vertical at a viewing distance of 100 cm. For the experimental blocks the size of the stimuli (5.9 cm by 7.7 cm) yielded a visual angle of 3.4^o by 4.4^o at the same viewing distance. ERTS^® served as experimental software for stimulus presentation and response recording.

Learning-procedure. All participants had to undergo a 1 hour training session in order to become familiar with 40 people (two subsets of 20 people). One half of the presented faces displayed male faces. The assignment for the participants, to respond to two out of four subsets of faces, was counterbalanced. That is, half of the participants were familiarized with set one and two, whereas the other half was familiarized with sets three and four. For each participant, and counterbalanced between the participants, one set of people were presented only with a neutral and happy expression, the other set with a neutral and angry expression. In an initial learning block the participants viewed 80 portraits of 40 people with a neutral as well as a happy expression for the first subset or an angry expression for the second subset. All portraits were presented for 5000 ms. After a blank screen of 500 ms, the next portrait appeared in randomized order. Participants were instructed to view all presented portraits thoroughly in order to recognize them in the following blocks. No response was required. After a short break, 3 testblocks with a matching-to-sample task were added. On each trial, participants viewed 2 faces presented next to each other for 2000 ms. Within this time they had to decide, which of the two faces belonged to the previously learned ones by pressing the corresponsing right or left button on a computer keyboard. Subsequently, only the correct face remained in its position for an additional 1500 ms either with a green frame, when the previous decision was correct, or with a red frame in case of an error or a missed response. Reaction times over 2000 ms were also counted as errors. Forty different people served as distractor faces and were used only in the learning phase. In each block every portrait was presented twice with a distractor face of the same and of different gender. The two faces always had the same facial expression. At the end of each block, participants received feedback about mean RT, error count, and error percentage. After three test blocks, an error criterion of 5 % or lower had to be reached. If error percentage was above this criterion participants had to go on practising one, but at most three additional test blocks in order to meet this criterion. Overall, each person was viewed at least 14 times (52 s), whereas each single picture was viewed 7 times (26 s).

Design and data analysis. Following the learning phase, and a short break of about 10 minutes, the participants continued with the experimental blocks, that is, an expression discrimination task. The presented portraits comprised the 40 previously learned people as well as 40 completely unfamiliar people. Each person was represented with two portraits displaying either a happy or an angry expression. A fixation cross appeared in the middle of the screen for 500 ms. It was displaced by a target face displaying either a happy or angry expression. Participants had to discriminate the two expressions as fast and correct as possible by pressing the right or left key according to the key assignment. After pressing the button the screen went blank. After 1000 ms the next trial continued, while the order was randomized. If participants failed to respond within the maximum RT of 2000 ms, visual feedback for late response was provided (“Zu langsam!”, too slow). No feedback for wrong responses was provided because the stimulus set was presented twice and stimulus-to-reaction learning should be avoided. After one presentation of the whole set of portraits (divided into two blocks of trials) the key assignment changed. In order to prevent mistakes, it was practiced in a separate block at the beginning and before changing the key assignment. Participants responded to the stimulus words “Freude” (happiness) or “Ärger” (anger) by pressing the corresponding key. After each block, mean RT, error count and error percentage were provided as feedback.

Statistical analyses of RT and error rate were performed by means of Huyhn-Feldt corrected repeated measures ANOVA. The within-factors familiarity-type (familiar person with learned expression; familiar person with unlearned expression; unfamiliar person) and facial expression (happiness vs. anger) were used. If necessary, t-tests were calculated for post-hoc comparisons of conditions. If more than one comparision was necessary Bonferroni-corrected significance levels were applied.

Familiarization. Error rates for the three consecutive test-blocks of the learning procedure decreased from 6.6% over 3.4% for the second block to 2.7% for the last block. The decrease from the first to the second block was significant (t(11) = 5.5, p < .001).

Reaction time and error rates. Figure 21 summarizes RT and error rates for the expression discrimination task of Experiment 4. Faster RTs were expected for learned faces when compared to unfamiliar ones, especially when they were familiarized with a happy expression (Baudouin et al., 2000). As is obvious, portraits displaying happiness (506 ms) yielded faster RT (F(1,11)=11.9; p < 0.01) when compared to angry expressions (552 ms). Although not important for the main question of this dissertation, it has been found in previous studies that participants show faster RT to positive expressions of happiness when compared to negative expressions (Leppänen, Tenhunen, & Hietanen, in press; Crew, & Harrison, 1994; Kirita, & Endo, 1995; Hugdahl, Iversen, & Johnsen, 1993). It might be easier to recognize happiness, because less visual features may be necessary. Indeed, in the present stimulus set many happy faces display a salient smiling mouth. In contrast, the discrimination of angry faces may be more difficult. Already the ratings of the stimulus material conveyed more incorrect classifications for angry faces when compared to happy faces (t(4) = -5.6, p < .05). The main effect of facial expression in Experiment 2 with faster RTs for disgust when compared to happy faces does not stand against this interpretation. The closed mouth in the former stimulus set may have diminished the advantage for happy faces.

Figure 21. Reaction time and error rates for the expression discrimination task of Experiment 4 separated for familiarity and expression.

Unfortunately, the hypothesized effect of familiarity-type was not evident in a main effect nor an interaction (F < 1) with facial expression. For error percentage the data revealed only a trend for facial expression (F(1,11)=3.4; p=0.09) with slightly less errors for the angry (4,6%) when compared to happy expressions (6,6%).

As can be summarized from the present results no interaction was found between familiarity and the discrimination of facial expressions when using learned familiarized and unfamiliar faces. This independency is even more underlined by the fact that participants were not faster in discriminating the expression when they had seen the very same expressive portrait of a familiarized person during the learning block. Perceptual familiarity per se does not affect the perception or discrimination of facial expressions at all. Hence, other processes might have been involved which subserved the interaction between facial familiarity and facial expression discrimination for personally familiar faces in the previous experiments. An obvious difference is the missing personal importance and semantic information of the present stimulus set when compared to personally familiar faces. The latter mentioned faces may be more implemented in the neural system when compared to experimentally familiarized faces. Thus, in the present experiment the chance of finding an interaction between facial familiarity and the discrimination of facial expressions may have been smaller.

In general the mean RT of the present experiment was fairly fast when compared to the experiments which used personally familiar faces (Experiments 1-3). This may be due to the stimulus material used where the happy expression is clearly shown with an open mouth on all happy portraits. Faster RT for the happy facial expression when compared to anger underlines this supposition. Hence, because of the easy and fast expression discrimination task a necessary prerequisite of a possible interaction between facial familiarity and the discrimination of facial expressions is missing (c.f. Baudouin et al., 2000). On the other hand, the salient feature of an open mouth for happy portraits may have led the participants to a strategy where very little information is extracted from the internal facial features that are necessary to perform the task. There is evidence that the discrimination of facial expressions and identity share little featural information (Calder et al., 2001). Still, there is also overlapping feature information which is relevant for the discrimination of both processes. When participants have to use all featural information which is available and in part also relevant for identity recognition during an expression discrimination task, it can be assumed that an interaction might emerge. Possibly, the already mentioned lower degree of familiarity, a less distributed neural representation of familiarized faces, and also the lack of semantic knowledge and emotional importance may be more relevant for the unconfirmed hypothesis. To test whether the fast RT and the relatively easy discrimination of this experiment may have crossed a possible emerging of the expected interaction, a further experiment is conducted with a harder task in order to increase the mean RT for the expression discrimination.

As mentioned before, the mean RT of the previous experiment was very fast, presumably due to the stimulus material, which showed an open mouth as the most salient feature on portraits displaying happiness. In return, this may have led the participants to a strategy where most of the featural information of the internal face is ignored since it was irrelevant for the easy expression discrimination. In order to apply a harder condition and to increase the overall RT level different facial expressions were used within the expression discrimination tast of Experiment 5. Participants were asked to discriminate between neutral and angry facial expressions. If the lack of an interaction between familiarity type and discrimination of facial expressions is ascribable to the easy condition of the previous experiment, a facilitative effect of familiarity type for the discrimination of angry and neutral faces is expected in the present experiment.

Participants. Twelve participants (9 women and 3 men, mean age = 24,8 years, aged between 17 and 37) took part in this experiment. They all had normal or corrected to normal vision. Participants received either course credit or 10,00 € for participation. As in the previous experiment handedness was not assessed, because electrophysiological data were not recorded.

Stimulus and apparatus. For Experiment 5 the same stimulus set was used as in the previous experiment. The size of the stimuli was kept for the initial learning phase and for the following experimental blocks. Again, all stimuli were presented on a 17-inch screen with the size of stimuli corresponding to the same visual angles as in Experiment 4. The viewing distance was kept constant at 100 cm. ERTS® served as experimental software for stimulus presentation and response recording.

Learning-procedure. Like in Experiment 4, participants had to undergo a 1 hour training session in order to become familiar with two sets of 20 people (half of them being male). In an initial learning block the participants viewed 80 portraits of 40 people with a happy expression and with a neutral or angry expression, respectively. All portraits were presented for 5000 ms in randomized order. The learning procedure including the test blocks was the same as in the previous experiment with the exception that all people were familiarized with a happy expression. In addition, one set was familarized with a neutral facial expression. The other set was familiarized with anger. As before, the distractor faces in the matching-to-sample task displayed always the same facial expression.

Design and data analysis. Participants continued with the experimental blocks after a short break of 10 minutes. In the subsequent experimental blocks it was asked for an expression discrimination where neutral and angry faces were now displayed. Except for the expression of happiness which was replaced by a neutral expression, the experimental procedure and design were the same as in Experiment 4.

Statistical analyses of RT and error rate were calculated with the same constraints as in the previous experiment.

Familiarization. Error rates for the consecutive test blocks within the learning procedure decreased from 7.8% over 2.9% (t(11) = 3.9, p = .002) to 1.5% (t(11) = 4.2, p = .002).

Reaction time and error rates. The present experiment was conducted in order to exclude the possibility that the fairly fast RT, due to the easy discrimination in the previous experiment, might have been the reason for the lack of an interaction between facial familiarity and the discrimination of facial expression. As expected, mean RT was considerably slower in the present experiment when compared to the previous one (692 ms vs. 531 ms; t(11)=-4.3; p < 0.01). In Figure 22 it is evident, that portraits displaying an angry expression (639 ms) yielded faster RT (F(1,11)=34.4; p < 0.01) when compared to the neutral expression (729 ms). This result is in line with previous results showing that RT to expressive faces is faster than to neutral faces (Eastwood et al., 2001, Phillips et al., 1998). From an evolutionary perspective it is conceivable that it is important to recognize and react especially upon expressive faces since they may contain relevant information about an upcomming threat or a dangerous situation. There is a lot of evidence showing a more distributed neural network which is involved in the processing of expressive faces (Kesler/West et al., 2001; Sato et al., 2001). In addition, ERP results revealed that expressive faces are processed more rapidly within the brain (Eimer & Holmes, 2002).

Figure 22. Reaction time and error rates for the expression discrimination task of Experiment 5 separated for familiarity and expression.

Unfortunately, the expected effect of familiarity-type was not evident as a main effect nor an interaction (F < 1). No effect at all was present within error rates. Although the task of the present experiment was a lot harder when compared to the previous one, and response speed was fairly slow, no interaction emerged between the discrimination of facial expressions and facial familiarity. It can be supposed, that during the learning block, an extensive perceptual familiaritzation took place, which in return could have subserved an interaction. In contrast, the result of independency between the two processes are more underlined, because it is likely that participants had to use more featural information derived from the internal features in the fairly hard expression discrimination task of the present experiment. This information might even overlap with featural information which is relevant for perceiving identity.

From the present experiment it can be concluded that facial familiarity and the discrimination of facial expressions are independent processes for experimentally familiarized and unfamiliar faces. This also holds true in case of a harder task, which caused increased RT and possibly the increased use of information derived from internal facial features when compared to the easier task in the previous experiment. If an interaction between facial familiarity and the discrimination of facial expressions is not affected through perceptual familiarity, other features might be relevant for this effect. Possibly, semantic knowledge about a person may be sufficint to enhance the speed of expression discrimination. This can not be decided from these and previous results. Although a facilitative effect of familiarity was evident in Experiments 1 to 3, personally familiar faces comprise both semantic knowledge and personal importance. Both stimulus properties may be sufficient to subserve an interaction. Hence, the following experiment tries to solve this question by using famous faces, since they convey a lot of semantic information but are not necessarily of personal importance.

It has been shown in experiments 1 to 3 that facial familiarity can interact with the discrimination of facial expressions for personally familiar and unfamiliar faces. Personal familiarity facilitated the discrimination of facial expressions, especially for happy faces. The previous Experiments 4 and 5 contrasted these reuslts by showing no facilitation of expression discrimination for familiarized faces. It has been argued that the degree of familiarity, including the lack of semantic knowledge and personal importance, may have been the most obvious reason for the hypothesis to fail. Therefore, the present experiment will use famous faces since they already convey a lot of semantic knowledge. If pure perceptual familiarization is not sufficient to evoke an interaction of facial familiarity, and the discrimination of facial expressions, the question is raised whether this also holds for famous faces. Because of their higher degree of familiarity, the representation within the brain might be more distributed. Hence, they are processed differently to some degree. The basis for this assumption come from results by Paller et al. (1999; 2000). Participants had to view 40 faces. Half of them were presented with a name, and semantic information was given by a spoken voice. Completely new faces were added in a subsequent recognition test. ERP data revealed posterior differences between unnamed and new faces, which was interpreted as the retrieval of visual information. Most important, named faces elicted an additional anterior positivity when compared to new faces, which could be linked to the retrieval of semantic information. Thus, a small amount of semantic knowledge can alter the processing of previously learned faces when compared to completely unfamiliar ones.

Although the general neural processing of personally familiar and famous faces might be the same, it is conceivable that there are differences between them, because of a lack of direct encounter as well as the associated emotional importance which is not given for famous faces. Intracranial neurophysiological recordings show differences between these two facial types in the mesial and lateral temporal lobe and in the right amygdala (Seek, Mainwaring, Ives et al., 1993). These areas are linked to memory function as well as to emotional labeling (LeDoux, 1992), and might account for the differential processing of personally familiar faces on the one hand and famous faces on the other hand. Thus, similarities as well as differences in the processing of these face types are found. Therefore, the question is justifiable whether the observed interaction between facial familiarity and facial expressions for personally familiar faces also holds true when unfamiliar faces are contrasted with famous faces.

In addition, there are also differences in the processing of famous and unfamiliar faces. Already visual extrastriate areas are differently involved in the processing of both types of faces. Higher PET activation was found for unfamiliar versus familiar faces in the right fusiform gyrus and the right occipital inferior gyrus (George, Dolan, Fink et al., 1999; Gobbini, Leibenluft, Santiago et al., 2000). Evidence about differential processing of famous and unfamiliar faces within the brain also comes from ERP studies (Bentin, & Deouell, 2000; Eimer, 2000). Priming studies with parallel recorded ERPs suggest that these differences are already evident between 180 to 290 ms poststimulus (Begleiter, Porjesz, & Wang, 1995; Schweinberger, Pfütze, & Sommer, 1995). Schweinberger et al. (1995) proposed a face specific event-related component around 250 ms which emerges for repeated faces within a priming paradigm. For immediate repetitions this potential is higher in amplitude for famous faces when compared to unfamiliar ones. For unfamiliar faces the N250r component even disappeared with two to four intermediate items (Pfütze, Sommer, & Schweinberger, 2002). Converging results suggest that this potential “reflects the stimulus triggered access to stores facial representations in inferior temporal cortex as influenced by very recent visual experience” (Schweinberger et al., 2002, p 407). There are manifest processing differences between famous and unfamiliar faces which presumably lie in the availlability of a stored visual representation and of semantic knowledge. Referring to the purpose of this experiment, it is expected that these differences might also evoke a facilitation of the discrimination of facial expressions for famous faces.

For the present experiment a special stimulus set was created with german, international, and british celebrities. This experiment was planned in cooperation with J.Kaufmann at the University of Glasgow since the same experiment was to be conducted in Germany and Great Britain. The group of british celebrities was completely unfamiliar to the german participants. In a future experiment, to be conducted in Glasgow the german celebrities are thought to be completely unfamiliar to british participants. Hence, if an interaction between facial familiarity and the discrimination of facial expressions is found, this effect is expected to be reversed for the german and british celebrities in their respective country’s. For the present dissertation only the german experiment, conducted by myself, is described here.

Because of practical constraints and the accessibility of pictures, neutral and happy expressions were used. In contrast to personally familiar faces famous faces are emotionally less important, but the degree of percepual familiarity is high and semantic knowledge is available. The results from Experiments 1 to 3 point towards late perceptual processing stages which might show a facilitation for personally familiar faces. Possibly, not emotional value or attachment, but semantic knowledge only and therewith a more widespread neural distribution within the brain is the necessary pre-requisit for a facilitative effect. If so, it seems reasonable to expect the same facilitative effect for famous faces. Like in Experiment 1 to 3 this facilitative interaction should be evident in behavioural as well as in ERP data. As suggested by Experiment 2, a possible facilitation of the expression discrimination for famous faces should be reflected in a shorter peak latency of the P300 for familiar german and international celebrities when compared to unfamiliar british ones. Neutral and happy portraits may have a different effect when compared to the observed interaction between facial familiarity and facial expressions for personally familiar faces displaying happiness and disgust. Because both expressions are probably the expressions most encountered, the facilitative effect might generalize to both facial expressions. Hence, a pure main effect of familiarity within performance and ERP data is expected.

Participants. Sixteen participants (12 women and 4 men, mean age = 25,8 years, aged between 19 and 42 years) took part in the present experiment. They received either course credit or payment for participation. The mean handedness score (Oldfield, 1971) was +75,9 (ranging from +25 to +100). All participants had normal or corrected to normal vision.

Stimuli and Apparatus. Pictures of British, German and International celebrities with neutral and happy facial expressions were used. Since it was planned to conduct this study in Germany and Scotland as a cross national comparison of the familiarity effect, celebrities had to be famous either only in Germany, or Scottland, or in both countries. Frontal to three-quarter views were allowed. A set of 197 celebrities was collected, each of them was represented with two portraits showing a neutral and a happy expression. Pictures were edited in Adobe Photoshop to 8-bit black and white pictues with 170 by 216 pixels and a black backround. In the pre-rating, as well as in the experiment, they were presented on a 17 inch screen with a resolution of 800 x 600 pixels. At a viewing distance of 100 cm this corresponds to a size of 6,8 by 8,6 cm and a visual angle of 3,9^o horizontal and 4,9 ^o vertical. ERTS® served as experimental software for stimulus presentation and recording of behavioral responses.

Design and Procedure. In a two-choice reaction time task participants had to discriminate between neutral and happy facial expressions. In twelve consecutive blocks the whole stimulus set of 112 celebrities with 224 portraits, selected after a rating, was repeated three times. The possibility of participant determined breaks was given between blocks. Within each repetition of the stimulus set, the hand-to-key assignment was alternated once. At the beginning of the experiment and before every hand-to-key alternation participants performed a block of 24 practice trials. In each practice and experimental trial a fixation cross appeared in the middle of the screen for 1000 ms. Fixation was replaced by the stimulus which was presented for a maximum of 2000 ms or until the subject pressed the response key. For both, correct and incorrect responses, the next trial started after a fixed period of 1500 ms. In case of responses too early (under 100 ms) or too slow (above 2000 ms), feedback appeared for 800 ms after a blank screen of 500 ms. After the experimental blocks a familiarity rating was conducted. All 112 celebrities were presented to the participant with one picture at a time. The participants task was to rate the familiarity of each celebrity. No time limit was applied.

Electrophysiological recordings. The same experimental setup was used as in Experiments 2 and 3. The EEG was recorded from the same 31 electrode sites (see 2.2.2).

Offline the recording was cut into stimulus synchronized epochs of 2500 ms length (-500 ms prestimulus to 2000 ms poststimulus). Artefact treatment and ERP calculation were carried out with the same constraints as in Experiments 2 and 3. Response locked epochs were cut out of the stimulus locked epochs and averaged according to the point of time where the response occurred. An average reference was applied based on the same subset of 28 electrodes as in Experiments 2 and 3.

Data analysis. Statistical analysis of the behavioural data was performed by means of Huyhn-Feldt corrected repeated measures ANOVA including the within-variables familiarity (British, German, International) and expression (neutral vs. happy).

For peak latency measures of the P300 as well as onset latencies of the LRP, the jackknifing based method was used as outlined earlier. The F- and t-values of subsequent ANOVAs or two-tailed t-Tests were adjusted according to the method described by Miller et al. (1996), and by Ulrich and Miller (2001). Peak amplitude measures were derived from averaged event related potentials and ANOVAs were performed with the factors stimulus familiarity and expression. As in Experiments 2, and 3, mean amplitudes were derived from the LhEOG for a 100 ms interval around the S-LRP, and LRP-R onsets. Huyhn-Feldt corrected repeated measures ANOVAs including the factors familiarity , and expression were calculated in order to assess possible effects on LRPs.

Topographical analysis was performed the same way as in Experiments 2 and 3. ANOVAs were calculated for amplitude measures and vector scaled data (McCarthy & Wood, 1985) for every 50 ms segment starting from 200 ms until 600 ms after stimulus onset. The within factors electrode site, familiarity , and expression were included. In case of significant interactions of electrode site by familiarity post hoc ANOVAs were calculated including any two levels of this factor.

Before the Experiment a rating was conducted in order to choose a subset of the stimuli with the celebrities best representing the neutral and happy expression as well as being most familiar in either one or in both countries. In Germany as well as in Glasgow ten participants (11 women; mean age = 24,7 years, ranging from 19 to 32 years) took part in the rating. Participants had to rate each picture first on familiarity and on facial expression immediately afterwards.

Familiarity was rated on a 4-point scale (ranging from 0 to 3) with the categories: unfamiliar (0; “unbekannt”), slightly familiar (1; “kommt mir bekannt vor”), familiar but don’t know the name (2; “bekannt, weiß was, aber nicht den Namen”) and, know the name (3; “namentlich bekannt”). For the latter category it was supposed that when the participant knows the celebrity by name also other semantic knowledge is available (Burton, & Bruce, 1992; Hay, Young, & Ellis, 1991).

Facial expression was rated on a 3-category scale (ranging from 1 to 3) with the categories: neutral (1), happy (2; “eher fröhlich”) and, very happy (3; “sehr fröhlich”). An additional category “0” or none at all (“weder noch”) was added for pictures that did not represent any of the two facial expressions, neutral or happy. All of the 394 pictures were presented in randomized order on a computer screen. No time limit was applied for any of the ratings. On each trial a picture was presented at the top of the screen together with the familiarity rating scale. After the participant pressed the corresponding key for the rating choice with a left hand finger, the scale disappeared and the face remained on the screen for 500 ms. Afterwards, the facial expression rating scale was added to the picture. Then, the participant had to rate the same picture on facial expression with a right hand finger on four different keys on the computer keyboard. Visual presentation was terminated after the second key press corresponding to the expression rating. The next trial started after a blank screen of 1000 ms duration. The disconnection of the response hands was done to prevent participants from mixing up the successions of ratings which were accomplished on one picture in a single trial.

Mean ratings for each picture were calculated for both countries together (except for familiarity ratings) as well as for each country separately. To select the pictures a sequential strategy was used. The familiarity ratings served as a first criterion. The rating scale ranging from 0 to 3 provided the range limit of the mean value. British and german celebrities were selected if the mean value of the familiarity ratings for each picture was at least 1.6 by his/her own fellow countrymen and at the most 0.9 in the other country. Secondly, the expression ratings were considered. Here, the ratings of all participants from both countries were combined. The possible range of the mean values varied between 1 and 3. The lowest category (0; none at all) did not belong to the scale and was omitted for the calculation of the mean value for the expression ratings. Neutral pictures were accepted if a mean value of 1.4 was not exceeded. Happy pictures ought to have a minimum mean value of 1.5. The category ‘none at all’ (0) served as a final criterion on the expression rating scale. A maximum of 12 entries of a single picture was allowed. As the selection procedure was applied to a single picture a celebrity was only selected if both pictures fell within these criterions.

According to the procedure, a total of 112 celebrities (34 british, 37 german and 41 international) were selected, each represented by a neutral and a happy picture, respectively. The mean values of the familiarity and expression ratings are summarized in Table 2, and 3.

Ratings in Scotland

Ratings in Germany

Type of familiarity

Brit.

Ger.

Int.

Brit.

Ger.

Int.

Neutral pictures

2.48

0.23

2.72

0.23

2.53

2.38

Happy pictures

2.46

0.21

2.68

0.22

2.51

2.31

Combined

2.47

0.22

2.70

0.23

2.52

2.35

Table 2. Mean familiarity ratings for the stimulus set consisting of famous faces of Experiment 6 separated for the ratings in Scotland and Germany.

As can be seen in Table 2 the mean values of the familiarity ratings for british celebrities in Scotland, for german celebrities in Germany, and for international celebrities in both countries did not differ much. In addition, the mean values of the expression ratings (Table 3) for the three different groups of celebrities (British, German, and International) have only minor deviations as well.

Ratings in Scotland

Ratings in Germany

Ratings combined

Type of familiarity

Brit.

Ger.

Int.

Brit.

Ger.

Int.

Brit.

Ger.

Int.

Neutral pictures

1.06

1.05

1.05

1.11

1.15

1.13

1.09

1.10

1.09

Happy pictures

2.31

2.20

2.27

2.26

2.38

2.35

2.28

2.29

2.31

Table 3. Mean expression ratings for the stimulus set consisting of famous faces of Experiment 6 separated for the ratings in Scotland and Germany.

Reaction time and error rates. Figure 23 summarizes the RT and error rates for the expression discrimination task of Experiment 6. RT to happy faces was faster when compared to neutral faces (590 ms vs. 614 ms; F(1,15) = 4.5, p = .5). No other effect in RT was observed. Only the factor familiarity had an effect on error rates (F(2,30) = 4.7, p< .05). When compared to British and International celebrities, error rates in the expression discrimination task were lowest for German celebrities.

Event-related potentials. Figure 24 displays the N170 component at the electrode sites P10 for all conditions split up after familiarity and expression. As is evident from the figures the component peaks at exactly 170 ms after stimulus onset for all conditions. Statistical analysis of the peak amplitudes, and peak latencies revealed no difference between conditions.

Figure 23. Reaction time and error rate for the expression discrimination task of Experiment 6 separated for familiarity and expression.

Figure 24. The N170 component for the expression discrimination task of Experiment 6 at the electrode site P10 separated for familiarity and expression.

The P300 component averaged according to the familiarity by expression conditions is depicted in Figure 25. Calculation of the peak amplitude of the P300 component at the electrode site Pz revealed higher peak amplitudes for faces displaying happiness (M = 12.2 µV) when compared to the neutral expression (M = 10.9 µV; F(1,15) = 30.1, p < .001). There was no effect of facial familiarity (Fs < 1) nor an interaction of both factors (Fs < 1.6) for peak amplitude and peak latency of the P300 component.

Figure 25. The P300 component for the expression discrimination task of Experiment 6 at the electrode site Pz separated for familiarity and expression.

Figure 26 and 27 display the S-LRP and the LRP-R for all conditions, respectively. Both measures were not affected by horizontal eye movements, because there was no influence of experimental conditions on LhEOG (ps > .10). Despite small visible differences, jackknife based averages revealed no effect of any of the two factors expression nor familiarity. The same holds true for the response locked LRP.

Figure 26. The stimulus-locked LRP for the expression discrimination task of Experiment 6 separated for familiarity and expression.

Figure 27. The response-locked LRP for the expression discrimination task of Experiment 6 separated for familiarity and expression.

Analysis of the mean amplitude distribution over the scalp revealed a significant interaction of expression by electrode site starting from 200 ms until 550 ms (Fs(27,405) > 5.5, ps < .001) after stimulus onset (for all results see Appendix 6.3.). However, vector scaled data revealed topographical differences in the time intervals from 200 until 350 ms post-stimulus for the happy versus neutral expressions but not for later time intervals. Concerning the different amplitude distribution between the two expressions, difference maps (Figure 28) point to parieto-occipital and frontal electrode sites as containing the highest amplitude differences. Starting later from 350 ms after stimulus onset, the amplitude distribution depends also on familiarity (Fs(54,810) > 2.8, ps < .006). Different amplitude distributions are evident between british and german celebrities in the timerange from 450 until 550 ms after stimulus onset (Fs(27,405)>5.0, ps < .001) as well as between british and international celebrities from 350 to 400 and from 500 until 600 ms post-stimulus (Fs(27,405) > 3.9, ps < .009). No topographical differences for familiarity was found when comparing the vector scaled data of the three groups of celebrities.

Figure 28. Differences of the mean amplitude distribution between pairs of the British (B), German (G), and International celebrities (I; top row), as well as between the neutral expression (N) and happiness (H; bottom row) for the expression discrimination task of Experiment 6 in all tested time intervalls; a grey shading equals a negative difference.

No facilitative interaction was found for famous faces and the discrimination of facial expressions. This is in contrast to the RT results of Experiments 1 to 3, where a facilitation was found for personally familiar faces displaying happiness. It was an unexpected result, since a facilitative effect for famous faces during an expression discrimination task was suggested by the results of other studies (Boudouin et al., 2000). The already mentioned differences between personally familiar and famous faces might possibly account for the absence of an interaction between facial familiarity and the discrimination of facial expressions in the present experiment. The most obvious differences between these faces are the lack of direct encounter and of associated emotional importance. The latter difference becomes obvious in studies using the SCR. Here, lower arousal levels are evident for famous faces when compared to personally familiar ones (Herzmann, Schweinberger, Sommer et al., in press; Tranel, Damasio, & Damasio, 1995). In addition, differences in the activation of the right amygdala were found for personally familiar and unfamiliar faces but not for famous and unfamiliar faces in studies using intracranial recordings (Seeck et al., 2001). Sugiura, Kawashira, Nakamura et al. (2001) presented corroborating results by showing differential PET activation of the amygdala, the hypothalamus, and the medial frontal gyrus for personally familiar and unfamiliar faces. These regions are linked to the behavioural significance of a recognized stimuli. It stands to reason that the lower emotional importance (or lower behavioural significance) of famous faces might explain the lack of an interaction between familiarity and the discrimination of facial expressions.

Another clue concerning the still unconfirmed hypothesis might be given by the different results concerning the amplitude and topographical distribution of the present and the previous experiments. For personally familiar versus unfamiliar faces (Experiment 2), topographical differences derived from vector scaled data were evident as early as 200 ms poststimulus. In contrast, in the present experiment different amplitude distributions for famous and unfamiliar faces started as late as 350 ms poststimulus. Most importantly, topographies remained the same for famous and unfamiliar faces in all tested time intervalls, because no significant effect was found for vector scaled data. Possibly, the discriminative information about familiarity within the present stimulus set is available later as for personally familiar and unfamiliar faces.

The only evident effect within the RT results were faster RTs for happy facial expressions when compared to neutral ones. That expressive faces are recognized faster within several paradigms is a finding often seen (e.g. Holmes, Vuilleumier, & Eimer, 2003; Läppänen et al., 2003). Other brain regions might be involved when recognizing facial expressions with emotional valence when compared to neutral faces (Kessler/West et al., 2001; Dolan, Fletcher, Morris et al., 1996). The topographical differences between the neutral and happy portraits starting at 200 ms poststimulus in the present experiment corroborate these results. In addition, emotionally valenced faces may boost attention mostly through the involvement of e.g. the amygdala (Adolphs 2002; Allison, Puce, & McCarthy, 2000; Whalen, Rauch, Etcoff et al., 1998) and hence, faster RTs can be observed. Another hint of exceeded saliency or attention which is captured by the expressive happy faces when compared to the neutral expression, is the increased P300 peak amplitude for happy faces measured at the Pz electrode. This is in line with quite a few results showing increased P300 amplitudes for happy versus neutral faces (Carretie, & Iglesias, 1995; Marinkovic, & Halgren, 1998; Herrmann et al., 2002; Vanderplog, Brown, & March, 1987) or emotional arousing versus neutral scenes (Cuthbert, Schupp, Bradley et al., 2000). All these results suggest that happy faces are processed differently from neutral ones in the present expression discrimination task. As mentioned earlier, these differences might emerge because of higher saliency and increased attention to happy faces which might be based on differential involvement of brain areas that subserve these processes.

To summarize, the present experiment revealed no interaction between facial familiarity and the discrimination of facial expressions for famous and unfamiliar faces. This was an unexpected result and suggests that emotional importance may be a necessary prerequisite for an interaction of both processes. Since emotional importance cannot be premised for famous faces in this experiment, but surely for personally familiar faces in the previous section the just found independence of both processes might be explainable. In addition, differences in amplitude distribution between famous and unfamiliar faces started later than in Experiment 2 and 3. This might suggest that differential information about familiarity is available fairly late and therefore, familiarity cannot act as a facilitative for the discrimination of facial expressions as found for personally familiar faces.

In Experiment 7 the question is raised whether the observed interaction of personal familiarity onto the discrimination of facial expressions also holds true for an interaction in the opposite direction, that is, an effect of facial expression on the discrimination of facial familiarity for personally familiar and unfamiliar faces. Because the perception of facial expressions is a fast process and does not need to be slowed down by a hard condition in order to interfere in the functional processing streem, portraits with weak expressive intensity were omitted. Instead portraits with a neutral expression were added. Participants had to perform a speeded familiarity discrimination task in which all presented people displayed happiness, disgust, or a neutral expression in randomized order. It is expected that the frequently seen happy expression will facilitate the decision for familiarity (also see Endo et al. 1992). The neutral expression may also shed some light on the special role of the happy expression that was evident in all previous experiments. A neutral expression is probably the most frequently seen expression on personally familiar faces, but also on unfamiliar faces in general. If the pronounced facilitative effect of happiness is based on the frequency with which this expression is encountered on the face of a personally familiar person, comparable effects are expected for personally familiar portraits with happy and neutral expressions when compared to disgust. If the expressiveness resp. emotional arousal of a face is the crucial factor, the facilitation should only be evident for personally familiar faces displaying happiness, but not for a neutral expression. However, an expression of disgust may disturb the perception of familiarity, either because it is rarely encountered on a familiar face or because it may disturb the configuration of the internal features as a happy and certainly as a neutral expression more. In this case a facilitative effect is only expected for familiar happy faces.

Participants. Twenty participants (all women; mean age = 25,0 years; aged between 20 and 34) took part in Experiment 7. They were personally familiar with half of the presented persons displayed in the experiment. Participants received either course credit or a payment of 12 €. The mean handedness score was 75 (ranging from –83 to +100; Oldfiled, 1971).

Design and Procedure. In the present experiment the same stimulus set was used as in Experiments 1 to 3 with the exception that portraits with weak expressive intensity were omitted and portraits with a neutral expression were added. Hence 2 x 3 different conditions arose: familiar and unfamiliar happy faces displaying a neutral expression, happiness, and disgust. In a two-choice reaction time task participants had to discriminate between whether the presented face was personally familiar or not. All 28 people of the stimulus set were presented with happiness, disgust, and a neutral expression in three different perspectives. Trials were presented in randomized order with the same trial scheme as in the previous experiments. The stimulus set was repeated twice, whereas, after a single repetition, the hand-to-key assignment was changed in order to calculate an LRP for all conditions. The order of the key assignment was counterbalanced across participants. Before the experimental blocks participants viewed all 28 persons, who were included in the stimulus set, with a neutral expression and made a verbal response about personal familiarity. This was done to avoid errors, because people in the stimulus set were displayed repeatedly. In a pilot study one participant happened to classify an unfamiliar person as familiar, and error rates increased extremely due to the repetitions of the people.

Electrophysiological recordings. In the present experiment the same electrode setup was used as in all previous experiments, which recorded event-related potentials. Furthermore, electrophysiological data were treated the same way as in the previous experiment and averaged to the above mentioned six conditions.

Data analysis. For statistical analysis the same tests and procedures were used as in the previous experiments, with the exception that the within subject factor expressive intensity was omitted. The factor expression contained three levels (happy, disgust, and neutral). Analysis of the S-LRP, and LRP-R onsets, the LhEOG as well as of the peak amplitudes for the N170, and P300 components were also the same as in the previous experiments. Peak latency measures of the N170 and P300 components were analyzed with a repeated measure ANOVA including both within subject factors mentioned above. The latency values of the P300 component were based on jackknifing averages. The F-values were corrected according to the equation F_c = F/(n-1)² (Ulrich, & Miller, 2001).

Figure 30. Reaction time and error rates for the familiarity discrimination task of Experiment 7.

As expected, the categorization for a face as being personally familiar yielded faster RTs than the categorization as unfamiliar. This is a result often found for tasks involving a familiarity discrimination (Endo et al., 1992; Phillips et al., 1998). Most important, facial expression affected the recognition of personal familiarity in the respect that the frequently seen happy and neutral expressions facilitated the decision for a face as being personally familiar when compared to disgust. No such effect was found for unfamiliar faces. Complementing the results of Part I, which pointed to an interaction between personal familiarity and the discrimination of facial expressions, the present experiment revealed an interaction in the opposite direction or personally familiar faces. Happy or neutral facial expressions facilitated the discrimination of a face as being personally familiar. A note of caution should be added. As mentioned before, all persons were presented with a neutral expression in advance of the experimental blocks, in order to exclude systematic errors based on classifying a person as familiar or unfamiliar by mistake. Hence neutral pictures of all people might have been primed, and interpretations which rely on the results of these pictures have to be carried out with caution.

In line with the RT results an erlier S-LRP onset was found exclusively for happy familiar faces when compared to familiar faces displaying disgust. No differences were found between personally familiar faces displaying happines and disgust when considering the LRP-R onset as well as the N170 or P300 peak latency. Together, the results clearly point to the response selection stage as indexed by the S-LRP which is facilitated for happy personally familiar faces within a familiarity discrimination task.

The results are in line with other findings of e.g. Endo et al. (1992), who found faster RT in a familiarity discrimination task for familiar faces when they expressed happiness or a neutral expression. Admittedly, for personally familiar faces they found decreased RT only for the neutral expression. Due to the mouth being closed in all pictures, it is likely that the participants in the present experiment were also familiar with this rather weak happy expression of all personally familiar faces. Therefore, the findings of Endo et al. (1992) concerning personally familiar faces might not stand up against the present results.

The expressions might have disturbed the spatial arrangement of the internal features to a different degree. It is conceivable that this is not true for the neutral expression, and only partly true for happiness, but mainly true for disgust. Hence, familiar faces should be most easily recognized with a neutral or a happy expression, whereas disgust should disturb the recognition the most. The lack of a difference between the neutral and happy expression is also explainable within the present stimulus set since happiness was expressed with the mouth being closed. This makes it more comparable to a neutral expression.

The response selection stage as the locus of interaction stands in contrast to the late perceptual processing stage, which was considered as the possible functional locus of interaction for the expression discrimination task in Experiment 2. Other processes may subserve the interaction between facial expressions and facial familiarity within a familiarity discrimination task. Since a happy face is the most common gesture between aquainted people, it is probable that the facilitation is not due to personal familiarity. It may depend on a strong stimulus-response learning between a happy facial cue and the feeling of familiarity. Hence the response selection stage as the facilitated process is the most likely to be expected. However, the facilitation for personally familiar faces with a neutral expression did not fit into this reasoning. Possibly, the initial pre-exposure of the neutral faces had a much stronger effect than expected. Thus, to proof this reasoning, the same facilitative effect of happiness displayed on familiar faces should also emerge for experimentally familiarized faces. If, on the other hand, an unfamiliar face displays a happy expression, the decision for unfamiliarity should be more error prone.

An additional conclusion can be drawn from the data concerning the locus of facilitation for the familiarity decision. It is a result often found, that famous or experimentally familiarized faces are categorized faster in an identity discrimination task than unfamiliar ones (Phillips et al., 1998). To my knowledge it has been never addressed on which functional processing stage this advantage for familiar faces is located. As cited above, there is numerous evidence that the peak latency of the N170 component is not influenced by familiarity. Therefore, early visual perception may not be faster depending on facial familiarity. Studies which compared amplitudes and topographies from event-related potentials found differences between familiar and unfamiliar faces starting around 300 ms after stimulus onset (Paller et al., 2000; Endl, Walla, Lindinger et al., 1998). In the present experiment a clear effect in P300 peak latencies was found with an earlier peak for personally familiar faces. This difference is still evident in the later response selection stage (S-LRP), indicating a propagation of the effect which is already present in the P300 peak latency. No other processing stage was influenced by familiarity. Hence the results clearly point to late perceptual processing stages that are faster for familiar faces within a familiarity decision.

Taken together, the results of Experiment 7 point towards an interaction between facial expressions and the recognition of familiarity for personally familiar faces only. In addition, the effect of facial familiarity on S-LRP onset points to the response selection stage as the functional locus of the facilitation found for happy and neutral expressions. However, the results do not clearly elucidate on whether the found interaction between facial expressions and the discrimination of facial familiarity is due to personal importance of the familiar faces, or because of the mere perceptual familiarity which may be stronger for the often seen neutral and happy expression. A third explanation might be the special status a happy expression has as a social cue when aquainted people meet each other. It is necessary to have better control over the learning experience concerning facial expressions, when a face becomes familiarized. Therefore, a last experiment is carried out using unfamiliar faces, where half of the faces were familiarized in a learning block with different facial expressions.

In the present experiment the question was raised, whether the found interaction between facial expressions and the discrimination of personally familiar faces also holds true for experimentally familiarized faces. Assuming that the representations of familiar faces also relies on perceptual factors like facial expression when the face is familiarized, appearance of the same expression should facilitate the recognition of a familiar face. Whereas, if personal significance and arousal for familiar faces is crucial for an interaction of both processes, an interaction should not emerge between both processes.

The stimulus material in the present experiment consisted of initially unfamiliar faces. Half of the faces were familiarized in a learning block always with a neutral expression, and either a happy, or angry facial expression. For the following experimental block an equal amount of unfamiliar faces was added. Participants had to discriminate familiarized from unfamiliar faces. Faces were presented in randomized order either with a happy or angry facial expression. Importantly, for half of the familiarized faces the angry expression was never encountered, for the other half of faces the happy expression was not previously shown.

It is the main hypothesis that the facial expression which was encountered during familiarization in the learning block would facilitate the decision for a face as being familiar. This should be reflected in reduced RT and error rates for these conditions but not for the conditions with the facial expressions which were not encountered in the learning block.

Participants. Twelve participants (9 women and 3 men, mean age = 26,6 years, aged between 20 and 39) took part in the present experiment. All participants had normal or corrected to normal vision. They received either course credits or 10,00 € for participation. Handedness was not determined, because electrophysiological data were not recorded.

Stimulus and apparatus. For the present experiment the same stimulus set was used as in Experiment 4 and 5. The same size of the stimuli was used for the initial learning phase and for the following experimental blocks respectively. Again, all stimuli were presented on a 17-inch screen. The viewing distance was kept constant at 1m. ERTS® served as experimental software for stimulus presentation and response recording.

Learning-procedure. Participants had to undergo a 1 hour training session in order to become familiar with two sets of 20 people (half of them being male). The learning procedure was exactly the same as in Experiments 4. The presented facial expressions were the same. That is, one set of 20 people was familiarized with a neutral and happy expression, whereas the other set of 20 people was presented with neutral and angry expressions. The familiarized sets were counterbalanced across participants as described above.

Design and data analysis. Participants continued with the experimental blocks after a short break of about 10 minutes. They were then required to discriminate between whether the displayed portrait belonged to one of the 40 familiarized people or not by pressing a corresponding key on the right or left. All persons presented were displayed with the two facial expressions happiness, and anger. The whole stimulus set was presented in randomized order and repeated twice. The trial schema was the same as in all previous experiments. Participants had to perform 6 blocks of trials. In case of too slow (> 2000 ms) or incorrect responses feedback was provided after the trial by showing the words “Zu langsam!” (Too slow!) or “Falsch!” (Wrong!) for 500 ms on the screen. A summarized feedback for the block was provided during the breaks. At the beginning and in the middle of the experimental blocks the hand to key assignment was changed. In order to learn the key assignment, participants performed 40 practice trials by pressing the corresponding key according to the words “gelernt” (learned) or “ungelernt” (unlearned) presented on the screen.

Statistical analyses of RT and error rates was the same as in Experiments 4 and 5 with the exception that now the factor expression contained 3 levels (neutral, happy, and angry). In addition, the within-factor familiarity type was used. It contained the levels ‘familiar person with learned expression’, ‘familiar person with unlearned expression’, and ‘unfamiliar person’.

Familiarization. Error rates for the test-blocks of the learning procedure decreased from 11.0% over 4.7% (t(11) = 5.1, p < .001) to 3.5% for the third block.

Reaction time and error percent. Figure 36 displays the RT and error rate of Experiment 8. No effect at all emerged for the error rates in the present experiment. The RT strongly depends on the familiarity type (F(2,22)=15.3, p < .01). Post-hoc tests revealed faster RTs for faces that were familiarized in the previous learning block with (M = 608 ms) or without the displayed expression (M = 639 ms) when compared to unfamiliarized faces (M = 669 ms; ts(11) > -5.2, ps < .01). Although visible in Figure 36 and numerically evident, no statistical difference was found between the two types of familiarized faces (t(11) = -2.2, p = .15). Hence the main hypothesis of the present experiment was not confirmed. The discrimination of a face as being familiar was independent of the expression that was encountered during familiarization. However, the numerical difference between familiarized faces with learned expressions and familiarized faces with unlearned expressions lead towards a view that the perceptual experience with which a person is familiarized may play at least a minor role concerning the representations that are built up.

Figure 36. Reaction time and error rates for the familiarity discrimination task of Experiment 8 separated for expression and familiarity.