Sang-won Leigh, Philipp Schoessler, Felix Heibeck, Pattie Maes, Hiroshi Ishii

Abstract

In this paper, we present a novel interaction system that allows a collocated large display and small handheld devices to seamlessly work together. The smartphone acts both as a physical interface and as an additional graphics layer for near-surface interaction on a computer screen. Our system enables accurate position tracking of a smartphone placed on or over any screen by displaying a 2D color pattern that is captured using the smartphone's back-facing camera. The proposed technique can be implemented on existing devices without the need for additional hardware.