Sensing Curiosity in Play and Responding


Contemporary pedagogy shows that, in order to be effective, science education must not only teach students scientific facts and principles, but also enable them to create new facts and discover principles, enjoy engaging in scientific processes, and develop intrinsic interests and confidence in working scientifically (Von Stumm et al., 2011). And yet contemporary schools are often required to “teach to the test” in ways that are virtually antithetical to hands-on practice and knowledge creation. And today’s students are consequently losing interest and ability in the process of science. In this work we believe that fostering curiosity and exploration, seeking and creating, is essential to science learning and so we are developing technologies to scaffold these activities in elementary school students. The larger goal of our research is to leverage the cognitive and social benefits of learning in small groups. Here we focus on triggering students’ curiosity in order to achieve that goal.

While emerging digital technologies have demonstrated advantages as new forms of learning media, little research investigates the many challenges of raising and sustaining curiosity in science learning. We address these challenges by incorporating four aspects of an ideal curiosity-evoking environment:

  1. a role-playing game to encourage exploration of science concepts by allowing children to switch amongst the different roles in science exploration (Jessica Hammer, HCII and ETC)
  2. a virtual child that models and scaffolds exploratory and collaborative hands-on learning in the role-playing game in ways that trigger increased curiosity on the part of the real children in the group (Justine Cassell, LTI and HCII)
  3. a multimodal sensing system that automatically recognizes children’s learning states and conveys them to the virtual child (Louis-Philippe Morency, LTI)
  4. a set of curiosity-evoking augmented reality toys that allow real and virtual children to play together (Zhen Bai, HCII)

Findings from this work will deepen our understanding of students’ learning behaviors in science classes, provide design guidelines and frameworks for technology-enhanced learning environments that foster epistemic curiosity in science education, and inform teachers and educators how to incorporate new instructional approaches in both high- and low-tech learning contexts that lead to increased scientific interest in students while young, and then in their later life and work.




The labor market today increasingly demands higher order skills called socio-emotional (SEL) skills, which refer to a broad repertoire of competencies and character qualities that complement foundational literacy and numeracy, and enable individuals to approach complex challenges in their changing environment. Some examples include creativity, collaboration, persistence (grit), leadership, curiosity etc. Curiosity is a vital SEL skill in educational contexts, and curiosity and exploratory behaviors are believed by many to be tightly intertwined with one another. According to contemporary accounts curiosity is caused by: (1) the desire for information, which leads to exploratory behaviors in search of missing knowledge and is satisfied when this knowledge gap is filled appropriately (Loewenstein, 1994); and (2) pleasurable feelings aroused by the anticipation of acquiring new knowledge through exploratory behavior (Spielberger and Starr, 1994). In a virtuous cycle, positive feelings derived from both learning experiences make positive impacts on students’ beliefs about their competence in mastering scientific processes, in turn promoting greater breadth and depth of exploration of information related to problem solving (Wu & Miao, 2013). Consequently, curiosity has been identified as a critical character quality to prepare students for lifelong learning in the 21st century (World Economic Forum, 2015).

While on one hand, curiosity has traditionally been described as an inherently individual and stable disposition toward seeking novelty and approaching unfamiliarity, other research also suggests that curiosity can be mediated by and fostered through social interactions with parents, teachers and other children (Engel, 2011). Social factors in learning environments play a critical role in influencing student’s curiosity and exploration both in and outside of schools. This is because knowledge can be constructed socially through peer interaction and collaboration, and academic motivation and outcomes are also often influenced by peers (Parr and Townsend, 2002).

With the recent increase of computer technology in science education, a research agenda is emerging to understand the opportunities and challenges of designing and adopting technologies in school classrooms. Although the novelty of technologies often motivates students briefly to explore new technology-enhanced learning materials, it remains challenging to shift from pure perceptual curiosity to epistemic curiosity related to specific science topics, and maintain the latter curiosity at an optimal level. In addition to the learning materials, instructional approaches within the classroom such as teacher’s scaffolding and social dynamics in collaborative learning also contribute in nurturing students’ interests and skills, which are unfortunately often neglected in technology-driven investigations. In addition, while the importance of SEL factors in the context of collaborative learning has long been recognized (Kreijns et al., 2003), only a small subset of the current generation of learning technologies actually aim to foster or develop SEL skills in students. We believe that expanding the social capability or the ability of learning technologies to provide a social infrastructure is important to facilitate acquisition of SEL skills in students. By social capabilities, we not only refer to the nature of the inner technical system that makes the interaction social (underlying model of behavior understanding/intelligence) (Cassell et al., 1999), but also to generating sensory/motor capabilities that affect social behavior (ability to engage group members in an interesting, relevant conversation with appropriate speech, body behaviors etc) (Breazeal, 2004).

In this project, we are motivated to explore the state-of-the-art learning and instructional approaches in schools, develop a systematic framework to foster curiosity in science education, and apply this framework to the design of a playful learning environment integrating advanced technologies in game, virtual and augmented reality, and multimodal sensing.

Role-playing games provide a natural role-switching mechanism, which may help children experience different roles such as question-asker and question-answerer in group interaction. In addition, role-playing games enable children to play very different roles in games than in real life, which may enhance their competencies in problem solving (Hammer, 2007). Furthermore, the game itself can provide a safe environment for children to experiment and explore without worrying about social risks such as failure.

Virtual peer scaffolding is a way to provide the educational benefits of collaborative learning before children naturally know how to collaborate. Since the intelligent virtual peer has demonstrated special capacity in supporting both learning and collaborative skills for young children (Cassell, 2004), we are working on designing a virtual child that can provide (1) necessary guidance such as introducing games and illustrating ways to play, and (2) key scaffolding such as taking risks, playing and switching between different roles, and collaborating with other players in scientific reasoning.

As curiosity is highly transient, it is important that the virtual child recognizes and responds to real-time learning states of the students. For example, noticing that the child playing the leading investigator role is not engaged in the game, the virtual child might try to provoke her curiosity in order to get her to explore solutions. This learning state information will be provided by MultiSense, a multimodal sensing system based on detection of visual, verbal and vocal cues and inferences about underlying learning states drawn from those cues. Multisense will automatically analyze behaviors such as facial expression, eye gaze, head gesture and voice quality to infer children’s engagement, collaborative behavior, curiosity, enthusiasm and scientific risk-taking.

Ideally a virtual peer is expected to interact with human players in the way human children interact with one another, except with advanced knowledge in scaffolding learning activities. There is, however, often an interaction barrier between the virtual peer, who is confined to virtual space, and children, who are situated in physical space. As recent research has demonstrated the special capacity of augmented reality technologies in externalizing and sharing human thoughts in physical space (Bai et al., 2015), we intend to explore opportunities of mixed-reality play environments to support collaborative learning experiences and face-to-face interaction between the virtual child and human players. In addition, research has shown that the tangible features of digitally augmented objects accelerate conceptualization development through direct manipulation with knowledge components in three-dimensional space. Therefore it will also be interesting to investigate the design space of digitally augmented objects as novel curiosity-evoking stimuli in supporting observation, experimentation and communication about scientific knowledge.



The SCIPR project was launched in September 2015 and is applying a combination of design-based research, observational and interventional studies of children in natural play environments, game design and playtesting, and machine learning to investigate the whole range of learning science and developmental psychology approaches to increasing SEL skills such as curiosity in the service of science learning.

We have conducted observational research across different learning contexts to find ways in which children may naturally express curiosity and evoke curiosity in one another in small group learning. Using a combination of manual and automatic annotation procedures, we are analyzing sequential patterns of these observable signs of high and low curiosity across group members. Specifically, we are looking at three classes of behaviors: (i) verbal – question asking, argument, justification, making suggestion, idea verbalization, hypothesis generation etc, (ii) non-verbal – head pose orientation, emotional expressions of delight, surprise, confusion etc, (iii) turn taking related behaviors. Ground truth for curiosity has been obtained using thin-slice technique (Ambady & Rosenthal, 1992) on a crowdsourcing platform, where raters are presented 10-second video segments in randomized order to provide a rating for curiosity level.

Based on results from these investigations and existing research from psychology, learning sciences and group dynamics, we have articulated a computational model of curiosity in small group science learning, and how it can be fostered by role-playing games. Specifically, we make a tripartite distinction between observable behaviors, their individual and interpersonal functions and corresponding strategies to support those functions via a learning technology or human coach. We are currently looking into longitudinal latent variable modeling approaches to verify this proposed integrated psychological and social infrastructure of curiosity.

In parallel, we are also working on equipping the virtual peer to be able to engage in gameplay, by investigating the design space of digitally augmented toys as novel curiosity-evoking stimuli in supporting scientific inquiry, collaboration and social interaction between the virtual child and real children.

These will lead us to iterative evaluations of the effectiveness of the computational model of curiosity, which will be first conducted by involving students in the game and virtual peer design using a Wizard-of-Oz system. Finally, we will summarize design guidelines for technology-enhanced learning environments, and generalize the findings to inform teachers and educators of approaches to promote curiosity exploratory learning behaviors and self-efficacy in science classrooms.


  • Principal Investigator
    • Justine Cassell
  • ArticuLab
    • Zhen Bai
    • Bhargavi Paranjape
    • Tanmay Sinha (alumnus)
    • David Slebodnick
    • Luo Yi Tan
  • Oberlin Wintern 2016: Patrice DiChristina
  • Spring 2016 Intern: Tristan Marino, Aliya Blackwood
  • Spring 2016 Capstone students: Su Baykal, Benjamin Boesel, Ryan Donegan, Sam Gao, Ian Go
  • Summer 2016 Interns: Jiamin Ping, Lu Sun, Shrey Shah, Bo Kim, Hyunji Do, Julia Luo, Sam Li, Stephanie Bao, Sina Siddiqi
  • Fall 2016 Interns: Zoey Feng, Angela Liang, Michelle Deng, Hea Jin Cha, Jarret Lin, Changning Shou, Gaurav Lahiry, Olamitundun Oladipo
  • Spring 2017 interns: Lu Sun, Danny Choo, Zoey Feng, Angela Liang, Michelle Deng, Lauren Yan, Kaily Bruch, Prachi Mahableshwarkar, Kanisha Vaughn, Vivian He


Bai, Z., Blackwell, A. F., & Coulouris, G. (2015, April). Exploring Expressive Augmented Reality: The FingAR Puppet System for Social Pretend Play. In Proc. CHI’15, 1035-1044.

Baltrušaitis, T., Robinson, P., & Morency, L. P. (2014). Continuous conditional neural fields for structured regression. In Computer Vision–ECCV 2014 (pp. 593-608). Springer International Publishing.

Cassell, J. (2004). Towards a model of technology and literacy development: Story listening systems. Journal of Applied Developmental Psychology, 25(1), 75-105.

Hammer, J. (2007). Agency and authority in role-playing “texts”. Michele Knobel and Colin Lankshear, 67.

Loewenstein, G. (1994). The psychology of curiosity: A review and reinterpretation. Psychological bulletin, 116(1), 75.

Spielberger, C. D., & Starr, L. M. (1994). Curiosity and exploratory behavior. Motivation: Theory and research, 221-243.

New Vision for Education: Unlocking the Potential of Technology. (2015) Retrieved from

Ambady, N., & Rosenthal, R. (1992). Thin slices of expressive behavior as predictors of interpersonal consequences: A meta-analysis. Psychological bulletin, 111(2), 256.

Von Stumm, S., Hell, B., & Chamorro-Premuzic, T. (2011). The hungry mind intellectual curiosity is the third pillar of academic performance. Perspectives on Psychological Science, 6(6), 574-588.

Wu, Q., & Miao, C. (2013). Modeling curiosity-related emotions for virtual peer learners. IEEE Computational Intelligence Magazine, 8(2), 50-62.

Engel, S. (2011). Children’s need to know: Curiosity in schools. Harvard Educational Review, 81(4), 625-645.

Parr, J. M., & Townsend, M. A. (2002). Environments, processes, and mechanisms in peer learning. International journal of educational research, 37(5), 403-423.

Kreijns, K., Kirschner, P. A., & Jochems, W. (2003). Identifying the pitfalls for social interaction in computer supported collaborative learning environments: a review of the research. Computers in human behavior, 19(3), 335-353.

Breazeal, C. L. (2004). Designing sociable robots. MIT press.

Cassell, J., Bickmore, T., Billinghurst, M., Campbell, L., Chang, K., Vilhjálmsson, H., & Yan, H. (1999, May). Embodiment in conversational interfaces: Rea. In Proceedings of the SIGCHI conference on Human Factors in Computing Systems (pp. 520-527). ACM.

See Other Projects

Alex Projects
RAPT (Rapport-Aligning Peer Tutor)
SARA: the Socially Aware Robot Assistant
Yahoo!-CMU InMind
Yahoo!-CMU InMind