Professional activities

2010-now               Managing Director Technical Sciences at TNO

TNO Technical Sciences includes 1500 researchers in the following fields of expertise : mechanics, physics, chemistry, constructive engineering and ICT. As a director I am responsible for the world class excellence of our research, the quality and motivation of employees, the innovation and maintenance of our facilities and infrastructure, and for symbiotic partnerships with other research organisations.

2001-now               Professor “Multimodal Interaction in Virtual Environments at Utrecht University

From 2001 to 2006 I was a full professor (part time) in “Multimedia Interaction” at the University of Amsterdam. Arnold Smeulders (UvA), Jan Biemond (TUD), Martin Kersten (CWI)  and I were the founders of the Euro 32M program MultimediaN.  Today I am a full professor (part time) in "Multimodal Interaction in Virtual Environments" at Utrecht University. Mark Overmars (UU) and I founded the Center for Advanced Gaming and Simulation (AGS) and acquired the Eur 20M GATE program (Game Research for Training and Entertainment).

My research focuses on:
- Multimodal game environments: how do multimodal game environments (stimulating the auditory, visual, tactile, and vestibular sensory modalities) improve object identification, location memory, situation awareness, navigation and affective appraisal?
- Brain computer interfaces: can we use ‘event related potentials” (specific EEG signals) in combination with tactile cues to enable thought controled navigation in virtual environments?
- Intelligent environments: how can we make information systems more intelligent to support people in complex decision making (e.g. serious gaming). More specifically, how can we improve multi-agent-actor technology to do that?

I supervise(d) the following PhD-students:
- Niels Bakker, University of Delft : Spatial orientation in Virtual Worlds (1997-2001)
- Summit Mehra, University Amsterdam : Hand-held Interactive Multimedia (2002-2004)
- Nienke Weder, TNO : Cross modal interfaces (2005-2007)
- Jan van Erp, TNO : The tactile information channel (1997-2007)
- Sander Jansen,
Utrecht University : Human Manoeuvring in virtual environments (2007-2012)
- Tom Philippi,
Utrecht University : Multimodal memory (2007-2012)
- Joske Houtkamp,
Utrecht University : Affective Appraisal (2006-2012)
- Marieke Thurlings,
Utrecht University : Brain computer interfaces for gaming (2008-2013)
- Ksander de Winkel,
Utrecht University : Multimodal cue combination (2008-2013)
- Benny van der Vijgh, Utrecht University: Controlled game-based stress manipulation (2010-2014)
- Anna van der Zalm, Utrecht University: Persuasive Technology for Personalized Sleep Coaching

2005-2010             Director Research TNO Defence, Security and Safety

In 2005 I became the Director Research of TNO Defence, Security and Safety. This core area of TNO encompasses 1040 researchers and works for the Defence market. I was responsible for the 60M€ explorative research portfolio. Challenges were to realise the transition from a budget driven to a demand driven research organisation.  The goal was excellent  knowledge development with a substantial societal impact. This required an increase in scale at technology level (focus, power), demand driven programming of the research portfolio (utilisation) and intensive international collaboration (brokerage). 

2003-2005            Director of Information Office TNO

January 2003 I was asked to lead the corporate TNO staff that manages the TNO ICT infrastructure and corporate applications. The challenge was to transform TNO from a company with 15 locally separated and heterogeneous ICT infrastructures into a single TNO wide  ICT environment. This involved the development of vision, infrastructure, applications, the redesign of company processes and above all communication and culture.  In 2 years time we set up an ICT governance model within TNO (operations, innovation and strategy boards), formulated and propagated a vision on goals, functionality and technology and implemented the TNO information system called Spider.

Spider is a fully thin client system for 6000 TNO employees offering a fully integrated portal environment for office work, project management, customer relationship management, virtual collaboration, news, IP Telephony, etc. The project involved a 10M€ budget over 2 years and 60 internal and external project members. Spider fully integrates SAP, Microsoft and Websphere Portal environments with military safety standards. Moreover numerous advisory and user boards were involved. To be successful the ICT department had to be transformed from a small focus computation center to a shared service center and double in size to 40 fte.  With Spider TNO can call itself a leading organization in the area of information and knowledge management. 

2001-2003            Director of TNO Multimedia & Telecommunications

In 2001 the board of TNO asked me to become the director of the business center TNO Multimedia and Telecommunications (TNO MET). This again broadened my scope substantially because TNO MET worked for three TNO institutes: TNO Strategy, Technology and Policy. TNO Human Factors and TNO Physics and Electronics Laboratory.  I became responsible for the development of public-private research programs and contract research in the area of  ICT.
Together with the Technical University of Delft and the University of Amsterdam we founded
MultimediaN with multimedia research and technology as its core business. MultimediaN is funded by government and industry with a research portfolio of 8M€ per year between 2004 and 2008.
As the director of TNO MET I was a member of the TNO team that investigated acquisition of the 350 fte research laboratory KPN Research of the Dutch telecommunications company KPN.  As of October 2001 we investigated the technology portfolio, business perspective, due diligence and IPR issues and developed a business plan. In January 2003 TNO took over KPN Research which became formally the TNO institute TNO Telecom. 

1999-2001            Head of the Department of Information Processing

At this time I became the head of the department of Information processing (25 fte, 3M€ turnover) and member of the management team of TNO Human Factors. My activities shifted towards project acquisition and  resource management. My contribution was the qualitative and quantitative growth of the department and a growth in the civil market by 50%. 

1998-1999           Steering & Control research at TNO

In 1998 I was appointed  coordinator of the research group Steering and Control Tasks at TNO Human Factors. Major projects were 3D audio in cockpits (AUDIS  for the Royal Dutch Airforce) and advanced interfaces for ship bridges (ATOMOS II for the European Union). Further, I boosted explorative research on tactile interfaces together with PhD-student Jan van Erp. 

1994-1996          Virtual Reality research  at TNO

My knowledge of human visual perception fitted nicely in TNO’s ambitions to develop VR techniques for ship design and prototyping. After moving to TNO Human Factors I developed software for PC and Evans and Sutherland machines to simulate ship bridges and command and control rooms. Further, I set up a VR-lab at TNO Soesterberg and validated  virtual mock-up and real versions with respect to spatial perception, navigation and manipulation (virtual hand control). I supervised PhD-student Niels Bakker and various students on navigation in VR.
As a project leader I managed various design projects for the Royal Netherlands Marine and Boskalis using VR-techniques with substantial media exposure.   At this time my scope broadened to related research field such as anthropometry (virtual humans in VR), video-communication (communication in VR) and man-machine interfaces (virtual hand control). I represented The Netherlands in the  RSG 28 NATO study group on virtual reality applications in military environments.
In 1996 I became the principal Scientist and coordinator of the research group Workplace Ergonomics. I initiated and acquired the funding for various large national research projects such as MCCW (Media Communication and Collaborative Work) and European projects such as
VIRTUE (Virtual Team User Environments) which aimed at advanced multi-user teleconferencing technology in collaboration with BT Research and Sony.  A major project was the design of a national demonstration centre for VR and internet applications in Utrecht. This multi-million project by order of KPN involved the business plan, facility design, content design with numerous interactions  with scientists, engineers, marketeers en architects.

1992-1994            Psychophysics at Utrecht University

In 1992 I returned to Utrecht, again with Jan Koenderink, and worked in the context of the European ESPRIT Insight 2 project. Here, previously developed models on human motion perception and psychophysical data converged to a predicting model describing human 3D affine structure-from-motion perception. Besides, I coached and collaborated with various PhD-students amongst whom Sjoerd de Vries and Hendrik-Jan van Veen.  

1990-1992          Psychophysics at  New York University

At New York University I worked with George Sperling and Charly Chubb on Grant AFOSR 91-0178 from the Air Force Office of  Scientific Research. The focus was the mathematical modelling of  more complex motion types such as second order motion and texture perception. In addition, I made a next step from linear motion to non-linear motion (visual acceleration), in collaboration with TNO in the Netherlands (Lex Toet en Herman Snippe).  I showed humans have great difficulty in discriminating visual acceleration. This was expected to have a serious impact on the accuracy with which humans can extract 3D structure.
During this period I applied as an astronaut for the European Space Agency. After many tests I ended in the “Top seven” of  over 700 selected applicants, but did not make it to the last five.  

1986-1990        Psychophysics at Utrecht University

As a PhD-student I worked with Jan Koenderink at the physics department of Utrecht University.  The topic was human visual motion perception which is the foundation for  figure-ground segmentation, attentional processes and 3D-structure-from-motion. My grand challenge was to understand how people process visual motion and to program computers  to see visual motion. I designed a new paradigm to study psychophysically  the strength of apparent motion and the effects of luminance and orientation of linearly moving elements. Further, I studied various perceptual properties of rotating patterns.  All this resulted in models for understanding how motion perception is implemented in our brain.  The most important finding was that motion perception is based on simple energy correlations rather than on matching of similar features in consecutive images.

In parallel we developed theory and computer algorithms for automatically extracting motion patterns in moving images. It turned out to be possible to extract local linear deformations of the image without any a priori knowledge about image features, just by locally monitoring the temporal and spatial luminance derivatives.