Overview
For more than a decade I have explored the functional neurophysiology of cognitive processes in behavioral sequence learning, gradually progressing from the oculomotor control, through sensoriotor sequence learning to language processing and visual scene analysis.  The common elements across this trajectory has been the organization of these behaviors in time, and the functional neurophysiology of cortex and basal ganglia that implement this organization.  A very brief overview is provided with pointers to appropriate publications in the following paragraphs:
 

The Saccade Model

Based on an extensive analysis of studies of the neurophysiology of the oculomotor saccade system Dominey & Arbib (1992) developed a model that described single and double step saccade behavior and electrophysiology of LIP and the frontal eye fields (FEF), caudate and substantia nigra of the basal ganglia, and the superior colliculus in these ocuomotor tasks.   We also used this model and its extenstion to investigate neural mechanisms for colliding saccades evoked by frontel eye field stimulation (Dominey et al. 1997), and  Nicolas Schweighoffer examined how the cerebellum could be added to the model to provide for adaptive gain control (Schweighofer et al. 1996a,b).
 


Sequence Learning Model

In order to account for ocolomotor and manual sequence learning behavior and electrophysiology in the prefrontal cortex as observed by Barone and Joseph (1989 Exp Brain Res), we augmented the saccade model with a prefrontal cortex, and dopaine modulated cortico-striatal plasticity (Dominey, Arbib and Joseph 1995 - DAJ95).  The model simulated sequence learning and performance behavior, as well as the neurophysiolgical task-related activity of PFC neurons.  We also demonstrated how a behavioral task that required context dependant responses to sequential stimuli could be learned by the model, with insights again into the underlying neurophysiology (Dominey & Boussaoud 1997).

It is worth noting that this (Dominey, Arbib and Joseph 1995) was the first neurocomputational model that proposed that reward-related dopamine would provide a mechanism for cortico-striatal plasticity, based on the work at the time of Schultz and colleagues, and Calabresi and colleagues.
 

Serial and Temporal Structure Learning

Part of the design goal of the DAJ95 model was to accomodate input and behavioral output that was organized in time in the same way that behavioral tasks are presented to behaving primates.  That is, we were interested in both the serial order of sequences, and their temporal structure, i.e. the durations of sequence elements themselves and the delays between them.  The model was thus demonstrated to perform quite well both in complex seqeunce learning (Dominey 1995) and in its sensitivity to the temporal organization of behavioral sequences (Dominey 1998a,b).

Dominey PF (1995) Complex Sensory-Motor Sequence Learning Based on Recurrent State-Representation and Reinforcement Learning, Biological Cybernetics, 73, 265-274
Dominey PF (1998) Influences of Temporal Organization on Transfer in Sequence Learning:  Comments on Stadler (1995) and Curran and Keele (1993)  J. Exp Psychology: Learning, Memory and Cognition, 24 (1) 234-248
Dominey PF (1998)  A shared system for learning serial and temporal structure of sensori-motor sequences? Evidence from simulation and human experiments. Cognitive Brain Research. 6, 163-174
 

Motor Imagery in Parkinson's Disease

At the same time, based on the functional organization of the model we predicted that dopaine depletion in Parkinson's disease should lead to correlated impairments in the imagery without execution, and in the execution of motor seqeucnes in patients.  This was behaviorally confirmed in Dominey et al. (1997), and led to a series of brain imagery studies that further revealed the underying neurrophysiology (Thobois et al. 2000; 2001 etc).
 


Abstract Structure Learning

During this period I began a series of sequence learning experiments that led to the development of the concept of sequences (e.g. ABC-BAC, and DEF-EDF) that share a common abstract structure that can be used to generate new isomorphic sequences.  We argued (Dominey et al. 1998) that the DAJ95 model (rebaptized the Temporal Recurrent Network or TRN) could not learn abstract structure, similar to the point argued by Marcus et al. in their controversial paper in Science (1999).  As part of the arguement for dissociable systems of learrning serial and abstract structure, we demonstrated relatively selectie failures for abstract processing in schizophrenics (Dominey and Georgieff 1997) and serial processing in Parkinsonian patients (Dominey et al. 1997).
 
 


Sequential Cognition and Language

The strategy in the language domain is to start with infant capabilities and then build up to those of the adult.

Temporal Structure of Language

Developmental studies indicate that between birth and 8 months of age, infants display sensitivity to the serial, temporal and abstract structure of language and language-like sound seqeunces.  We thus used this well documented behavior as the simulation targets, and indeed demonstrated that the model could simulate serial, temporal and abstract structure sensitivity as observed by Saffran et al (1996), Nazzi et al. (1998), and Marcus et al. (1999), respectively in Dominey and Ramus (2000).    This sensitivity to temporal structure likely contributes to the initial lexical categorization of open vs. closed class words (Shi et al. 2000), that will play a crucial role in subsequent phrasal sementics processing including thematic role assignment.  We thus demonstrated (Blanc, Dodane & Dominey 2003) that the TRN could detect differences in the temporal structure of F0 to perform lexical categorization in French and English.  Similarly, Blanc & Dominey (2003) demonstrated that the TRN can exploit the temporal structure of the fundamental frequency to discriminate between different prosodic attitudes in spoken language.
 


Thematic Role Assignment and Syntactic Comprehension:

With these developmental studies underway, and the introduction of abstract structure, the stage was set for addressing aspects of adult language processing.  A first naive attempt was made in Dominey (1997).  Then in Dominey (2000 chapter), thematic role assignment (i.e. determining "who did what to whom") was addressed in the context of abstract structure procesisng.  Different abstract structures corrresponded to different mappings from input sentences to their canonically ordered thematic roles, and these mappings were under the control of closed class words processed in the TRN.  This work was further developped and generated predictions that were tested (Dominey et al. 2003) in ERP (Hoen and Dominey 2000, Lelekov et al. 2000) and behavioral experients (Hoen et al. 2003).
 
 

Grammatical Constructions and Phrasal Semantics

The previous work demonstrated the concept that thematic role assignements could be associated with different patterns of closed class words characteristic of different sentence types.  This concept was further extended into the domain of grammatical constructions as sentence to meaning mappings, both in the extension of the model (Dominey 2000/2002), and in the development of scene analysis systems that can extract meaning from scenes based on sequcnes of physical primitives including contact and changes in position (Dominey 2003a, b).

The future work rests on two interesting and chalenging positions:  The first is theoretical and holds that it is the generative structure of semantics and conceputal structure - combined with communication requirments - that drives syntactic structure and not the other way arround.   The second position is technical and holds that the proof is in the pudding, and that to test and validate our ideas we must build systems based on these ideas.  The currrent state of developent in robotics, computer vision and human language technology holds great promise in this domain.

Dominey PF (2000) Conceptual Grounding in Simulation Studies of Language Acquisition, Evolution of Communication, 4 :2, 57-85.
Dominey PF (2003a) Learning Grammatical Constructions in a Miniature Language from Narrated Video Events, Proceedings of the 25th Annual Meeting of the Society for Cognitive Science, Boston.
Dominey PF (2003b) Learning Grammatical Constructions from Narrated Video Events for Human-Robot Interaction, Proceedings  of the IEEE Humanoid Robotics Conference, Karlsruhe, Germany.
Dominey PF, Inui, Toshio Miniature Language Learning via Mapping of Grammatical Structure to Visual Scene Structure in English and Japanese
 

Robot cognition working group
 
 

retour équipes ISC

Institut des Sciences Cognitives  UMR 5015  CNRS UCB Lyon 1
67, boulevard Pinel 69675 BRON cedex
33 (0)4 37 91 12 12   33 (0)4 37 91 12 10 web@isc.cnrs.fr

ACCUEIL ISC