MOVING PICTURES YOU MOVE

THE CONTROL OF TIME IN NEW MEDIA AESTHETICS

BY JIM GASPERINI

Suppose we could pose an anachronistic question to someone from the 19th century: "In the century to come, something called 'moving pictures' will come to dominate art and communication. What do you imagine these 'moving pictures' will be like?"
If our interlocutor happened to be visionary, she might imagine an experience similar to the theater. An audience enters a darkened hall to watch actors perform a dramatic production. Instead of live actors on a real stage, however, the story unfolds in a series of projected photographs. If she were visionary to the point of prescience, she might further imagine this ephemeral 'stage' miniaturized and fit into a box, viewable at will in the drawing room or parlor. The phrase 'motion picture,' however, might just as likely conjure up something more like the experience of wandering through the gallery of a museum, or leafing through the photographs in an album. Instead of fixed single images, the portrait or landscape would somehow move. And since of all forms the plastic arts are the ones in which the audience has the most complete control over the pace and order of the experience, it is not unreasonable to suppose she might guess that the picture would move when you move it .
Such a 19th-century respondent would have been dead wrong about the dominant media of the 20th century. She might be right on target, however, about the media that will dominate the 21st.


TIME IN TRADITIONAL FORMS

Of the many possible ways to classify different forms of artistic expression, the one most relevant to this discussion centers on the issue of who controls the passage of time . The different traditional forms range from ones in which the author keeps nearly complete control to ones in which temporal order and duration is left entirely to the audience.

  • In the temporal arts of film, radio, recorded music, and video, works are fixed in a way that leaves little or no role to the audience. Start here, the audience is told, and watch or listen until we (director, producer, conductor or composer) say it's finished. This extreme has only been reached in the twentieth century, with the ability to fix performances in the atemporal amber of recording technology.

  • With live music, opera, legitimate theater and other forms of performance, control over the passage of time is shared. The composer or playwright sets the order and structure of the experience, but the precise pace, duration and nuances of expression are strongly influenced by actors, directors, musicians and conductors. The paying audience participates to some extent as well, though more subtly, by influencing the performance of actors and sometimes interrupting or prolonging performances with applause, boos or demands for encores. Influenced perhaps by the motion picture, audiences have become more and more passive in this century. Certainly they are tamer than the days when hooting at poor strutting and fretting players and cries of "bravo" and "encore" enlivened performances at the Globe theater, vaudeville and the Grands Boulevards. The tradition of the uncompromisingly judgmental audience lives on at La Scala, however. New ways for participatory involvement in theatrical events continue to be invented, such as the 'mosh pit' in which today's rock fans hurl themselves from the edge of the stage onto swarms of tightly-packed writhing cohorts.

  • In works composed of text, control of pace shifts even further to the side of the audience. Though the author of a novel or book of poems usually sets a strict linear order to the experience, the precise pace with which someone reads, whether they run through the whole thing in one sitting or work through it in bits and snatches over days or years, how many times they reread a difficult line or stanza before moving on to the next, whether they skip through certain passages or read every word, whether they open the book of poems at the beginning or start at random somewhere in the middle--all these options are considered the normal purview of the audience.

  • With art forms based on space--painting, photography, architecture and sculpture--audience control over time becomes nearly complete. It is up to the gallery-goer how long they linger before a painting, from how many angles they view it, which details they notice in what sequence, whether they walk around a work of sculpture or pass on to the next distraction in another corner of the room. To fully experience a visual work does require what Umberto Eco calls "circumnavigational time," particularly in the case of architecture.* A work like Chartres cathedral or the Taj Mahal should be explored at length, despite the claims of bus tour guides that purport to "show you" such a work in a couple hours. Yet just how much of such work must be viewed, and for how long, remains entirely up to the audience.


    THE EFFECTS OF DIGITAL TECHNIQUES

    So at which point in all this range of creative expression does digital technology change the role of the audience? The answer is sweeping: in all of them. With varying degrees of success, artists and other storytellers are beginning to use digital media to create works that bear enough resemblance to earlier forms to be given such labels as "interactive movies," "interactive fiction" and "interactive music." These terms are misleading, since the best of these works enable such different experiences that they should not be considered sub-genres of earlier forms but new forms themselves, together creating a new order of aesthetic experience that occasionally parallels but at heart profoundly differs from the old.

    Digital technology can be used, of course, to create complete or partial works within old media. It has been used with particular success in electronic music and special effects films. Truly new works, however, are those that involve some form of what is known as "interactivity."

    Ask a dozen people involved with new media what they mean by "interactivity" and you'll get nearly as many definitions. For some, a work is only "truly interactive" when the machine meets some measure of sophistication in the way it emulates human behavior, or enables the user to create a customized experience with as few limitations as possible. Others are less interested in the human-machine relationship than in human-to-human interaction with the machine as mere intermediary. Still others point out that any work of art can be called "interactive," since significance is ultimately a judgment of the human mind and each individual "interacts" with the work to create a unique experience. So broad a definition, however, robs the word of any usefulness.

    The question of what is or is not "interactive" is less interesting than the question of just how deep an experience based on new technology can be. I will use "interactive" quite broadly: a work is "interactive" if the audience is given any kind of technologically-occasioned ability to intervene in the order or pace of the aesthetic work. Interactive works have been created that draw heavily on each of the traditional art forms. About the only thing they all have in common is a shift of control over time toward the audience. Sometimes this shift parallels the control traditionally exercised by performers. Sometimes it goes further, making the audience a kind of co-author.

    In the productions of Interactive Hollywood the audience is typically given very limited freedom of action. Usually you are placed in a role drawn from classic genre stories, challenged to use the available time wisely enough to achieve an optimal ending. Typically you play a key character in the story--often, but not always, the protagonist--and are given carefully circumscribed ways to influence the action. In some works, short bursts of filmed motion alternate with long periods in which you gather information; others approximate a 'real time' clock and offer short windows of opportunity for choice before moving on anyway if you take no action. By offering the audience choices these works take a few steps away from the total authorial control of the traditional forms, though they usually don't get very far.

    Works drawing inspiration largely from literature include hyperfiction, which offers readers the same contemplative pace as normal fiction but allows them to explore a kind of structural ambiguity by retracing their steps multiple times and experimenting with variable consequences. Networked environments like MUDs and graphical 'chat rooms' resemble works of improvisatory theater, in which the audience members can play author-devised roles, create roles for themselves, help to craft and extend the fictional environment, and interact both with other real-time performers and pre-authored simulations. The balance between pre-authored and participatory elements varies greatly from work to work, but the audience is almost always given a wide range of expressive freedom that moves beyond the control traditionally exercised by performers into the realm of the author.

    In some works the role of the author diminishes to the point where it can become hard to distinguish an interactive work from an instrument. Take a sophisticated audio synthesizer that comes loaded with pre-recorded sound samples and techniques for recombining them in various ways. Compare that with a work of interactive music, which offers sound structures originally authored by a composer but allows the user to electronically manipulate them in various ways. One is an instrument, designed by engineers; the other is a work of art, designed by a composer. As software and hardware evolve, it becomes harder and harder to maintain the distinction.

    The correspondence between interactive forms and analog counterparts is always inexact, and sometimes close to nonexistent. Many new works resemble several pre-digital forms at once. In graphical adventures like Myst, for example, the audience explores the circumnavigational time of a virtual architecture while performing a role in a story whose basic structure is literary. Judgments about aesthetic merit aside, it can be argued that various forms of computer game offer forms of popular entertainment and storytelling with no clear pre-digital counterparts. The most popular forms of game turn control of time into a challenge: control time or die. The game sets a fast, insistent pace which you must match, carefully picking or blasting your way through a minefield of obstacles (laser-toting aliens, poisonous mushrooms, musclebound martial arts experts) or the story is over.


    A NEW 'MOVING PICTURE'

    A particularly fruitful approach to interactive aesthetic experience draws on the traditions of the spatial arts such as painting and sculpture. Though the audience retains a great degree of control over the pace and order of these experiences, it shares that control with the author in ways that can be compared to a variety of traditional art forms.

    A good example is a short work by Alex Mayhew, sent out on floppy disks by Peter Gabriel's Real World organization as a Christmas card a couple years ago and available for download at a site maintained by the British Arts Council (http://www.sonnet.co.uk/hub). The screen shows a cloudy, semi-abstract environment in the midst of which sits a young girl, twitching occasionally in her sleep. Your cursor is a big fly, which buzzes when you move it around the stage. Click on the girl and she squints at you, yawns or asks you to let her sleep in peace. Gradually you discover that she is an angel, and has lost her halo. Once you find it for her she awakens and thanks you. A door opens to reveal a 'Happy Christmas' message, and the work is over.

    This simple work poses a plausible answer to the anachronistic question with which we began this discussion. A 19th century visionary might easily have imagined something like this if asked to conjure up a vision of the coming 'moving picture.' At first glance it resembles a painting, and with one exception the pace of the experience is under your control just as it would be if it were a painting. Yet it moves, when you make it move, and a gentle story tells itself in the course of your prodding.

    A more complex example is ScruTiny in the Great Round, my collaboration with Tennessee Rice Dixon published on CD-ROM by Calliope Media and Macmillan Interactive. Control of time drifts back and forth between player and author in "ScruTiny," and depending on how it is used it can resemble a variety of traditional art forms. Each screen of the work is a complex collage, and may be contemplated like any painting at whatever length the audience chooses. As you move through it, however, your choices set in motion a variety of audio-visual elements. Sounds and sometimes graphic animations are remixed in response to your cursor movements. Your actions can be compared to a performer of music, creating an individualized audio-visual concert for yourself and anyone who happens to be watching with you. Though we like to think of our audience as "the last collaborator," as authors we try to make things easy. Unlike mastering a traditional instrument, you are not asked to put in long hours of practice before you can master the interface.

    "ScruTiny" contains numerous animations, which can be launched one by one or in series. While the animations are playing it resembles a traditional film, though a film more in the painterly tradition than the dramatic tradition that has dominated the twentieth century. Painterly works have been part of the cinematic tradition since the earliest days, continuing through Un Chien Andalou to the intensely personal, richly layered work of such video artists as Daniel Reeves in our day. That such works have never been as popular as more traditional dramatic works is due in part to the fact that their audiences have not had sufficient control over time to be able to savor and understand their ambiguities.

    Contemplating a painting, the audience has the time to consider its various complexities, spending as much time as they might need to understand its multiple layers of meaning. When a series of complexly composed images are strung together into a film and rushed past an audience that cannot stop, rewind, or otherwise control the flow of imagery, the audience may become numb, or at least oblivious to much of the work's significance. Building a measure of interactivity into a painterly audiovisual work, allowing the audience to replay elements at will, should serve to validate that relatively neglected approach to cinematic experience.

    Perusing a work like "ScruTiny" can also be compared to reading a book of poems--the order in which you choose to read an audiovisual sequence, and how long you linger over it, is now under your control. Since "ScruTiny" is a multidimensional collage, with different versions of each scene (some 'lit' by the sun, some by the moon) wandering around it can also be compared to navigating through a conceptual space, like a work of virtual architecture. The sense of virtual space central to many 2-D and 3-D virtual worlds has sometimes been proposed as the defining characteristic of aesthetic digital media. Though spatial concepts are quite appropriate to works with architectural or sculptural dimensions , however, they become uselessly strained when applied to many musical and most literary works. Giving primacy to the temporal authority of the audience allows for the application of spatial metaphors only when appropriate.


    A NEW ORDER

    As we move into the next century, and the new forms outgrow their infancy, the comparison of new forms to older ones should gradually become unnecessary. Unfortunately, many theoretical discussions about new media still get lost in debates about the finer points of analogies to older media, missing the larger point of the comparison. These analogies are only analogies, and analogies are by definition inexact--otherwise they would be identities. Digital technology has enabled the creation not just of one new interactive medium, but an entire new order of interactive media full of rich expressive and emotive possibilities, which we have only begun to explore. The familiar boundaries between author, performer and audience have been broken forever. Shifting back and forth in this churning mix of aesthetic possibilities, but always central to a clear understanding of how they move us and we move them, is the control of time.


    * Umberto Eco, "Six Walks in the Fictional Woods," Harvard U. Press 1975.

    Jim Gasperini (jimg@well.com) is a multimedia artist, writer, programmer and partner in ScruTiny Associates, a new media developer based in New York (http://www.didi.com/scrutiny)


    back