Mixed Reality, Mixed Presence

A basic premise of collaborative mixed reality systems is that they enable mixed presence (collocated and remote) collaboration by effectively blending physical and virtual spaces. In this project we consider what it means to ‘effectively blend’ real and virtual for collaboration. We take a multi-pronged approach, combining infrastructure development, physical-virtual co-design, iterative prototyping and controlled experimentation.

IMRCE

IMRCE is a lightweight, flexible, and robust Unity toolkit that allows designers and researchers to rapidly prototype mixed reality mixed presence (MR-MP) environments that connect physical spaces, virtual spaces, and devices. IMRCE helps collaborators maintain awareness of the shared collaborative environment by providing visual cues such as position indicators and virtual hands. At the same time IMRCE provides flexibility in how physical and virtual spaces are mapped, allowing work environments to be optimised for each collaborator while maintaining a sense of integration. IMRCE will be presented at SUI 2018.

Mapping Physical and Virtual Actions

In a controlled study we consider two scenarios involving hiding and sharing blended physical- virtual documents around a tabletop, under two vertical display conditions: a solo display showing the remote collaborator as an avatar, and circumambient displays showing the rest of the connected virtual environment. We explore four types of cues (interactive, communication- based, ambient, and infrastructure) and their impact on how collaborators hide and share blended physical-virtual documents. Many participants did not obviously develop a sense that the physical and virtual surroundings were fused; as a result, certain physical privacy behaviours (e.g., orienting one’s body to shield documents from the remote collaborator) were less apparent in the study. However, the circumambient displays generated curiosity about how the spaces were connected, and episodes where breaches were enacted or the spatial correlation was otherwise suggested led some participants to trust the environment less. On the table, the presence of fiducial markers and digital document “shadows” served to cue participants about the impact of hiding and sharing physical documents, however accidental breaches usually went unnoticed. We present these results in our ISS 2016 paper.

We developed SecSpace, a software toolkit for usable privacy and security research in mixed reality collaborative environments. SecSpace permits privacy-related actions in either physical or virtual space to generate effects simultaneously in both spaces. These effects will be the same in terms of their impact on privacy but they may be functionally tailored to suit the requirements of each space. We detail the architecture of SecSpace and present three prototypes that illustrate the flexibility and capabilities of our approach in our EICS 2014 paper.

 

TwinSpace

TwinSpace is an infrastructure for collaborative mixed reality built on top of a desktop virtual environment that supports document sharing and editing. It provides a robust platform for rapid prototyping of applications that span physical and virtual spaces, allowing for comparative evaluation of designs. The system is applicable to a wide range of physical settings, virtual environments, and collaborative activities. TwinSpace provides the following four key features:

  1. A communications layer that seamlessly links the event-notification and transactional mechanisms in the virtual space with those in the physical space.
  2. A common model for both physical and virtual spaces, promoting the interoperability of physical and virtual devices, services and representations.
  3. A mapping capability, managing how physical and virtual spaces are connected and synchronized.
  4. Specialized virtual world clients that participate fully in the larger ecology of collaborative mixed reality devices and services.

TwinSpace has been used in several research projects; most are outlined below.

 

 

inSpace Lab

Video

The inSpace Lab is an experimental environment modelled after the project room, a room that provides a physical home for a project (its members, activities, tools, and products) over its lifespan. The lab is designed to utilize the spatial layout and ‘collaboration affordances’ of the physical space in order to connect with a complementary virtual space.

The inSpace lab is composed of three main regions (see figure). Region A supports brainstorming and generating new content on a whiteboard, region B supports extended group discussions and sharing content on wall displays, and region C supports composing, modifying, and discussing content over an interactive tabletop display.

inSpace serves as our primary testing ground for mixed reality interaction techniques, physical-virtual co-design, policies for privacy, sharing mechanisms, and physical-virtual mappings.

Or de l’Acadie

Video

Or de l’Acadie is a prototype collaborative game built using TwinSpace. Set in an abandoned 18th Century French colonial fort, two pairs (French and British sides) compete for valuables distributed throughout the town, with the goal to collect the most valuables within a time limit.

The game emphasizes asymmetry in both game controls and team dynamics. Each British player controls a single soldier using keyboard controls. Players are not collocated, and can communicate with each other only when their soldiers are within earshot. Each soldier is equipped with a static top-down map of the fort, and can collect a limited number of riches before needing to return to a location at the edge of the fort to stash them.

The French side consists of a Mi’kmaq mystic who is able to communicate with spirits, and a deceased French Gendarme. The French players are physically collocated, and use a range of physical controls to compete for the valuables (see figure).

Virtual Supermarket

Video

This project implemented a mixed reality solution to the design contest held at the 3DUI conference in 2010. Items were selected from virtual supermarket shelves by moving a physical trolley around a carpet that was mapped to the supermarket floor. The touchscreen on the trolley presented the first-person perspective of the shopper.

Each selected item was dynamically associated with a physical token. When it was time to ‘check out’, the physical tokens were carried to a physical table/display. The display presented the surface of a corresponding virtual table. When tokens were placed on the physical table, the associated items appear on the virtual table at the same position and orientation.

2018

Salimian, Hossein; Brooks, Stephen; Reilly, Derek

IMRCE: A Unity Toolkit for Virtual Co-Presence Conference

Proceedings of the 6th ACM Symposium on Spatial User Interaction (SUI 2018), ACM, Berlin, Germany, 2018.

BibTeX

2016

Salimian, Hossein; Brooks, Stephen; Mackay, Bonnie; Reilly, Derek

Physical-Digital Privacy Interfaces for Mixed Reality Collaboration: An Exploratory Study Conference

Proceedings of the ACM Conference on Interactive Surfaces and Spaces (ISS 2016), ACM, Niagara Falls, Canada, 2016.

Links | BibTeX

2015

Reilly, D; Echenique, A; Wu, A; Tang, A; Edwards, W K

Mapping out work in a mixed reality project room Conference

Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI'15), ACM, Seoul, Republic of Korea, 2015, ISBN: 978-1-4503-3145-6.

Links | BibTeX

2014

Reilly, D; Mathiasen, N; Salimian, M; Edwards, W K; Franz, J; MacKay, B

SecSpace: Prototyping Usable Privacy and Security for Mixed Reality Collaborative Environments Conference

6th ACM SIGCHI Symposium on Engineering Interactive Computing Systems (EICS 2014), Rome, Italy, 2014.

BibTeX

2013

Reilly, D; Salimian, M; Brooks, S

Document-Centric Mixed Reality and Informal Communication in a Brazilian Neurological Institution Conference

CSCW 2013, 2013.

BibTeX

2011

Reilly, D; Tang, A; Wu, A; Echenique, A; Massey, J; Mathiasen, N; Mazalek, A; Edwards, W K

Organic UIs in Cross-Reality Spaces Conference

Second International Workshop on Organic User Interfaces, TEI 2011, 2011.

BibTeX

Wu, A; Reilly, D; Tang, A; Mazalek, A

Tangible Navigation and Object Manipulation in Virtual Environments Conference

Proceedings of the Fifth International Conference on Tangible, Embedded and Embodied Interaction (TEI '11), Funchal, Portugal, 2011.

BibTeX

Reilly, D; Tang, A; Wu, A; Mathiasen, N; Echenique, A; Massey, J; Rouzati, H

Toward a Framework for Prototyping Physical Interfaces in Multiplayer Gaming: TwinSpace Experiences Conference

Proceedings of International Conference on Entertainment Computing (ICEC 2011), 6972 , LNCS Springer, 2011.

BibTeX

2010

Reilly, Derek; Rouzati, Hafez; Wu, Andy; Brudvik, Jeremy; Hwang, Jee-Yeon; Edwards, Keith

TwinSpace: an infrastructure for cross-reality team spaces Conference

In Proc. ACM Symposium on User Interface Software and Technology (UIST '10), ACM, New York, USA, 2010.

Abstract | Links | BibTeX

Wu, A; Reilly, D; Hwang, J Y; Echenique, A; Santos, E; Edwards, W K

A Tangible, Cross-Reality Supermarket interface Conference

3DUI 2010, 2010.

BibTeX

Reilly, D; Tang, A; Wu, A; Echenique, A; Chamoli, S; Massey, J; Edwards, W K

Or de l'€™Acadie: a TwinSpace demo Conference

UIST 2010, 2010.

BibTeX

Reilly, D; Voida, S; McKeon, M; Dantec, Le C; Edwards, W K; Mynatt, E; Mazalek, A

Space Matters: Physical-Digital and Physical-Virtual Co-Design in the Inspace Project Journal Article

IEEE Pervasive Computing, 9 (3), pp. 54–63, 2010.

BibTeX

Mixed Reality, Mixed Presence