Virtual Garden: A Vision-based Multimedia Installation

Andrew Senior


Andrew Senior


This paper describes a project proposal for a vision-based multimedia installation, bringing together computer vision, computer graphics and other modalities to generate a virtual Japanese garden. The installation provides a platform for exploring human-computer interaction through physical objects.

Categories and Subject Descriptors

J.5 Computer Applications Arts and Humanities
H.5 Information Systems Information Interfaces and Presentation
H.4 Information Systems Information Systems Applications
K.4 Computing Milieux Computers and society


Computer vision, computer graphics, projection displays, multimedia installation, Japanese garden


Increasing computing power and decreasing hardware prices will soon bring a wide variety of digital multimedia into the lives of many more people. In many ways the general public are unaware of the power of these technologies and the impact that they will have on peoples' lives. Being commercially driven many of the technologies are for business applications, but are also being pushed into entertainment. Computer vision technologies are bringing about possibilities such as ubiquitous automated surveillance and device-free human-computer interaction that will similarly influence many peoples' lives.

This paper presents a multimedia installation which is enabled by current technologies in computer vision and audio-visual display, combining to create dynamically an aesthetically pleasing virtual Japanese garden. The creation is brought about by engaging with the user in an interactive process. In addition to involving the designer/user/viewer in the act of creation, the installation is designed to expand the users' awareness -- of the technologies involved and their capabilities, and of an aesthetic and culture of which they may have little awareness.

Much research effort is currently being devoted to the development of these multimedia and vision technologies, one of the aims being to develop future computer interfaces to allow human-computer interaction. Vision is being stressed in this work as it allows a natural interaction with people, being seemingly device-less, and very natural [1] -- it is a medium that humans are well adapted to using and understanding. A particularly important sector of this research deals with the ambiguous boundary between the physical and digital world and a number of research projects have begun to deal with how the digital world can be manipulated by the manipulation of physical objects (e.g. physical icons `phicons' [5]). This installation is a further exploration of how people can interact with the virtual world through physical objects interpreted as a user interface.

The Virtual Garden

The work is an interactive, part physical, part virtual, realization of a Japanese garden. The work allows members of the public to design and experience a Japanese-style stone garden in full size in the gallery space.


Figure 1: The virtual Japanese garden. (a) shows an elevation with the camera viewing the projection area on the floor; (b) shows the schematic of operations in the system;

The work is conceived as shown in figure 1(a): A number of light-weight synthetic rocks are placed within a rectangular floor area. The floor area would be covered with sand or gravel to generate the sensations of walking across a real garden. Any number of users may wander about the `garden' and rearrange the rocks. As the rocks are reconfigured, their positions are detected by a computer vision system, and a raking pattern is dynamically produced and projected into the garden to reproduce the appearance of a Japanese karesansui [3, p.13] garden. This being a garden for the 21st century, the raking pattern could be dynamic, evolving over time like ripples in a pond, or taking on more unusual configurations or appearances. Such patterns could be influenced by other modalities of input, including sketched or scanned graphics and acoustic input. Similar effects have been produced in PingPongPlus [4] which augments a game of table-tennis with computer projections.

A number of refinements are also proposed, including dynamic response (visual and aural) to users' presence. In particular, ripples could radiate from footfalls located with microphones embedded in the floor (using the techniques proven in [4]), or in response to utterances from the viewers. An acoustic background could be modulated in accordance with the locations of the rocks, and the actions of the users.

Additional garden features (moss, trees) could also be added under the control of supplementary algorithms. An aesthetic guide could also be programmed, to instruct in the methods of Japanese garden designers [2,3], giving insight into the principles that guide the design of traditional physical gardens. Naturally this exhibit lends itself well to being displayed, and even interacted with, over the Internet. Indeed the system could be linked to professional garden designers who could remotely realise or influence gardens with the system and display them in the gallery.

Supplementary information on the technical construction of the installation would also be displayed to increase awareness of the technologies used, ideally through an installation web page that would contain links to related projects and other applications of the technologies.

Technical details

An initial, small-scale demonstration version has been created, with computer vision acquisition of the rock locations, driving a PERL-generated VRML model of the garden and the rocks. The process is illustrated in figure 1(b).

The system is implemented on a Linux laptop computer. The vision input is from a Philips USB camera interfaced through Video For Linux. Images are acquired from the camera by a C program which carries out elementary image processing operations. Morphological operators (dilation and erosion) are applied to the thresholded image of the scene, to isolate the rock images. Connected components analysis is used to distinguish the rocks from one another and simple moments analysis finds the major and minor axes of the rock images, approximated as ellipses.

The C program outputs the rock locations, which are fed into a PERL program. This in turn generates a VRML model, consisting of virtual rocks (as ellipsoids) and a mathematically-generated raking pattern. Endless possibilities exist for the generation of such patterns, though the current implementation is a traditional pattern with each rock surrounded by ripples, and a general wave pattern filling the remaining space. Finally the VRML is displayed, with the FreeWRL browser-- either on screen or projected down onto the garden space with a projection display. Sample images of the garden, generated from the video input of figure 2(a) can be seen in figure 2(b-c)

Garden images

Figure 2: The vision and graphical systems. (a) The raw camera input; (b) The corresponding rendered garden from above; (c & d) oblique views.

Currently the system is not running as a closed loop because of the slowness of the VRML browser, but a final implementation entirely in C/OpenGL should achieve the speeds necessary.


When operating as an installation, the issue of calibrating the visual display with the camera needs to be tackled. Calibration would be initiated by projection of a known calibration image, imaged by the camera, to calculate the mapping between camera image and display, so that these can be aligned. Feedback must also be taken into account, since the vision system is imaging the same area projected onto by the display. To get a true picture of the physical world, independent of the projected artefacts, a thresholding technique may be sufficient, but one of the following could also be used:

Past and future work

This installation stems from a number of sources of inspiration, originally being conceived during a visit to Japan in 1998. Most of the author's previous work1 has been in welded steel sculpture and environmental art. The steel works are abstract constructions which play with a number of ideas to elicit a response from the viewer, aiming to evoke interest, surprise and wonder. Much of this work is strongly geometric and reflects the author's fascination with the modulation of space, and his sources of inspiration in the natural world, in the abstract world of mathematics and in language.

This project can be seen to be more closely related to the author's environmental art works which primarily consist of cairns constructed in the natural environment from found stones. These are works of natural materials, resonant with the genius loci. However, with their clearly man-made origin, they highlight and bring into question the presence and action of man in the landscape. Aesthetically they are simple forms, generally in the traditional arrangement of a way-marking cairn, but with transformations and rearrangements to suit the specific site.

More recent works have begun to combine steel with natural stones, and also to explore the relationship of these physical media with electronic art forms, in particular making electronic renderings and adaptations of physical works. Planned future works involve the incorporation of video works in steel and stone sculptures as video installations, progressing to the incorporating of computer vision and speech recognition techniques to make interactive installations that are aware of environmental changes including the presence of observers. The author's current research into face detection and recognition [7,8] provide many starting points for interactive art installations, as demonstrated in ``The Magic Morphin' Mirror'' [6].


[1] P. Maes, T. Darrell, B. Blumberg and A. Pentland. The ALIVE System: Wireless, Full-Body Interaction with Autonomous Agents, In: The ACM Special Issue on Multimedia and Multisensory Virtual Worlds, Spring 1996.

[2] David A. Slawson, Secret teachings in the art of Japanese gardens Kodansha, 1987.

[3] Mitsukuni Yoshida and Takeji Iwamiya, Japanese Gardens: Images, Concepts, Symbolism Hitachi, 1990.

[4] H. Ishii, C. Wisneski, J. Orbanes, B. Chun, and J. Paradiso, PingPongPlus: Design of an Athletic-Tangible Interface for Computer-Supported Cooperative Play, in Proceedings of Conference on Human Factors in Computing Systems (CHI '99), (Pittsburgh, Pennsylvania USA, May 15-20, 1999), ACM Press, pp. 394-401.

[5] B. Ullmer, and H. Ishii, mediaBlocks: Tangible Interfaces for Online Media (video), in Extended Abstracts of Conference on Human Factors in Computing Systems (CHI '99), (Pittsburgh, Pennsylvania USA, May 15-20, 1999), ACM Press, pp. 31-32.

[6] T. Darrell, G. Gordon, W. Woodfill, H. Baker, A Magic Morphin' Mirror SIGGRAPH '97 Visual Proceedings. ACM Press 1997.

[7] A. Senior. Recognizing faces in broadcast video in Proceedings of the IEEE International Workshop on Recognition, Analysis, and Tracking of Faces and Gestures in Real-Time Systems, September 1999.

[8] S. Basu, C. Neti, N. Rajput, A. Senior. and L. Subramaniam. and A. Verma. Audio-visual Large Vocabulary Continuous Speech Recognition in the Broadcast Domain in Proceedings of Multimedia Signal Processing 1999.

1 http://www.andrewsenior.com

Copyright © 2000 ACM.