What?

The Khronos Projector is an interactive-art installation allowing people to explore pre-recorded movie content in an entirely new way. A classic video-tape allows a simple control of the reproducing process (stop, backward, forward, and elementary control on the reproduction speed). Modern digital players add little more than the possibility to perform random temporal jumps between image frames.

The goal of the Khronos Projector is to go beyond these forms of exclusive temporal control, by giving the user an entirely new dimension to play with: by touching the projection screen, the user is able to send parts of the image forward or backwards in time. By actually touching a deformable projection screen, shaking it or curling it, separate "islands of time" as well as "temporal waves" are created within the visible frame. This is done by interactively reshaping a two-dimensional spatio-temporal surface that "cuts" the spatio-temporal volume of data generated by a movie.

There are many ways to visually explore the spatio-temporal volume generated by a movie; most of them result from cutting the spatio-temporal volume by a two-dimensional surface. The usual way of visualizing video content, consist on showing consecutively each image of the sequence. In other words, the intersecting surface is a plane that remains always perpendicular to the time arrow. Now, for instance, by intersecting the spatio-temporal volume by planes that are not perpendicular with the time axis, the resulting image will show a spatio-temporal gradient. Of course, there have been already many works and art-works that develop on such ideas.

Now, to the best of my knowledge, the Khronos Projector is the first Art-Installation enabling the interactive shaping of an arbitrarily complex spatio-temporal cutting surface thanks to a dedicated tangible and "sensual" human-machine interface, and thus giving the user a strong feeling of being actually sculpting the space-time "substance" with its own hands.

Why?

When contemplating a still image or a motionless sculpture, we are free to direct our sight wherever we want over the whole work - perhaps only secretly compelled by the compositional forces the author has succeeded instilling in it. This is barely possible in a movie - we are forced to adopt a point of view in space and time. Using the power of computers, we can free ourselves from this constraint.

The Khronos projector unties time and space in a pre-recorded movie sequence, opening the door for an infinite number of interactive visualizations. Using the Khronos projector, event's causality become relative to the spatial path we decide to walk on the image, allowing for a multiple interpretation of the recorded facts. In this sense, the Khronos projector can be seen as an exploratory interface that transforms a movie sequence into a spatio-temporal sculpture for people to explore at their own pace and will.

From the technical point of view, this work expands on the work I've been doing at the Ishikawa-Namiki Lab concerning human-machine interfaces using laser-based active tracking and dedicated, real-time image-processing vision circuits. The feedback I received during the presentation of a prototype laser-based tracking system at SIGGRAPH 2004 convinced me of the necessity to develop a tangible human-machine interface capable of interpreting touch - and if possible, capable of sensing even the delicacy of a caress - while at the same time able to react in a subtle and natural way, also through tactile feedback. The Khronos-Projector tissue-based deformable screen is a first step in that direction.

How?

The spatio-temporal fusion algorithm is the core of the Khronos program: it consist on blending hundreds of images from a movie sequence to produce a unique displayed image. The blending operation is controlled by a "spatio-temporal filter", or "spatio-temporal cutting surface", which is a two-dimensional surface lying inside the "spatio-temporal volume" of the movie. The fused image is the union of all the intersections of the spatio-temporal surface with each image in the sequence.

The program (extremely computationally hungry) was prototyped using Matlab, and finally coded in C++ using OpenGL. The present version can display the dynamic blended image both in 2D but also in 3D by mapping the image on a NURBS-generated surface representing the actual time/pressure map.

In the most basic embedding for the proposed installation, the spatio-temporal filter is shaped interactively using the mouse, a graphic tablet or a touch-screen LCD display. A physically-based model of a membrane is then used to compute in real time a realistic shape for the spatio-temporal filter.