With The Diorama Project we are putting virtual objects in real world spaces. Our goal is to create a system that allows people to build an imaginative parallel universe superimposed on their everyday space. The research challenges are technical (precise location sensing), perceptual (creating a coherent hybrid space) and social (developing new interface metaphors for physically located virtual objects).