Combining Tangible and Gestural Interaction

Experimental setup. Here, tangibles are placed on the tabletop to match the reference model on the wall display, which can be rotated using gesture.

We explore the use of combined gestural and tangible interfaces in collaborative gameplay for youth in small groups. In our setup, gestural interaction manipulates reference content projected on a wall-mounted display, to support tangible interaction on a tabletop display. We designed two games to explore the impact of this interface on collaborative gameplay: Block Rock involves placing tangibles (blocks) in the target configuration shown on the wall display, and Furniture Finder requires toy furniture to be placed in a target configuration, but the specific furniture to place must first be “uncovered” on the wall display. We conducted a within-subjects’ user study with 92 children between the ages of 8 and 15, and considered the impact of several factors (group size, age range, game type and interaction style) on collaboration. Using structured coding we derived three basic interaction control strategies: one player does both gestural and tangible interaction, one player does gestural interaction while the others share tangible interaction, and all players share both gestural and tangible control. We find that game design impacted coordination: Furniture Finder encouraged sequential role-based coordination, as one player uncovered furniture on the 3D model while others placed that furniture on the table, while Block Rock permitted players to work independently, which could lead to resource contention.  Interaction style also impacted coordination: gestural control occurred more often in Block Rock when discrete gestures were used to rotate the 3D model in canonical orientations than when a grab gesture was available to control rotation continuously. In general, infringement occurred more often over tangible control than gestural control, but discrete gestures led to more contention over getting the “right” view on the 3D model in Block Rock, particularly with dyads. However, dyads exhibited significantly more turn taking behaviour overall than triads. 

Combining Tangible and Gestural Interaction