New Media from Borges to html

Software Design and Modern Art: Parallel Projects

Ten years after the appearance of first cultural institutions solely focused on new media, the field has matured and solidified. But what exactly is new media? And what is new media art? Surprisingly, these questions remain to be not so easy to answer. The book you are now holding in your hands does provide very interesting answers to these questions; it also provides the most comprehensive foundation for new media field, in the process redefining it a very productive way. In short, this book is not just a map of the field as it already exists but a creative intervention into it.

Through the particular selections and their juxtaposition this book re-defines new media as parallel tendencies in modern art and computing technology after the World War II. Although the editors of the anthology may not agree with this move, I would like to argue that eventually this parallelism changes the relationship between art and technology. In the last few decades of the twentieth century, modern computing and network technology materialized certain key projects of modern art developed approximately at the same time. In the process of this materialization, the technologies overtake art. That is, not only new media technologies – computer programming, graphical human-computer interface, hypertext, computer multimedia, networking (both wired-based and wireless) – have actualized the ideas behind the projects by artists, but they extended them much further than the artists originally imagined. As a result these technologies themselves have become the greatest art works of today. The greatest hypertext text is the Web itself, because it is more complex, unpredictable and dynamic than any novel that could have been written by a single human writer, even James Joyce. The greatest interactive work is the interactive human-computer interface itself: the fact that the user can easily change everything which appears on her screen, in the process changing the internal state of a computer, or even commanding reality outside of it. The greatest avant-garde film is software such as Final Cut Pro or After Effects which contains the possibilities to combining together thousands of separate tracks into a single movie, as well as setting various relationships between all these different tracks – and it thus it develops the avant-garde idea a film as an abstract visual score to its logical end – and beyond. Which means that computer scientists who invented these technologies – J.C. Licklider, Douglas Engelbart, Ivan Sutherland, Ted Nelson, Seymor Papert, Tim Berners-Lee, and others – are the important artists of our time – maybe the only artists who are truly important and who will be remembered from this historical period.

To prove the existence of historical parallelism, New Media Reader systematically positions next to each the key texts by modern art that articulate certain ideas and the key texts by modern computer scientists which articulate similar ideas in relation to software and hardware design. Thus we find next to each the story by Jorge Borges (1941) and the article by Vannevar Bush (1945) which both contain the idea of a massive branching structure as a better way to organize data and to represent human experience.¹

The parallelism between texts by artists and by computer scientists involves not only the ideas in the texts but also the form of the texts. In the twentieth century artists typically presented their ideas either by writing manifestos or by creating actual art works. In the case of computer scientists, we either have theoretical articles that develop plans for particular software and/or hardware design or more descriptive articles about already created prototypes or the actual working systems. Structurally manifestos correspond to the theoretical programs of computer scientists, while completed artworks correspond to working prototypes or systems designed by scientists to see if their ideas do work, to demonstrate these ideas to colleagues, sponsors and clients. Therefore New Media Reader to a large extent consists from these two types of texts: either theoretical presentations of new ideas and speculations about projects or types of projects that would follow from them; or the descriptions of the projects actually realized.

Institutions of modern culture that are responsible for selecting what makes it into the canon of our cultural memory and what is left behind are always behind the times. It may take a few decades or even longer for a new field which is making an important contribution to modern culture to “make it” into museums, books and other official registers of cultural memory. In general, our official cultural histories tend to privilege art (understood in a romantic sense as individual products an individual artists) over mass industrial culture. For instance, while modern graphical and industrial designers do have some level of cultural visibility, their names, with the exception of a few contemporary celebrity designers such as Bruce Mau and [Philip Stark are generally not as known as the names of fine artists or fiction writers. Some examples of key contemporary fields that so far have not been given heir due are music videos, cinematography, set design, and industrial design. But no cultural field so far remained more unrecognized than computer science and, in particular, its specific branch of human-computer interaction, or HCI (also called human-computer interface design, or HCI).

1. 
   New media versus cyberculture.
   

What are these new cultural objects? Given that digital computing is now used in most areas of cultural production, from publishing and advertising to filmmaking and architecture, how can we single out the area of culture that specifically owes its existence to computing? In my The Language of New Media I begin the discussion of new media by invoking its definition which can be deduced from how the term is used in popular press: new media are the cultural objects which use digital computer technology for distribution and exhibition.³ Thus, Internet, Web sites, computer multimedia, computer games, CD-ROMs and DVD, Virtual Reality, and computer-generated special effects all fall under new media. Other cultural objects which use computing for production and storage but not for final distribution -- television programs, feature films, magazines, books and other paper-based publications, etc. – are not new media.

The problems with this definition are three-fold. Firstly, it has to be revised every few years, as yet another part of culture comes to rely on computing technology for distribution (for instance, the shift from analog to digital television; the shift from film-based to digital projection of feature films in movie theatres; e-books, and so on) Secondly, we may suspect that eventually most forms of culture will use computer distribution, and therefore the term “new media” defined in this way will lose any specificity. Thirdly, this definition does not tell us anything about the possible effects of computer-based distribution on the aesthetics of what is being distributed. In other words, do Web sites, computer multimedia, computer games, CD-ROMs and Virtual Reality all have something in common because they are delivered to the user via a computer? Only if the answer is at least partial yes, it makes sense to think about new media as a useful theoretical category.
3. New Media as Digital Data Controlled by Software.

The Language of New Media is based on the assumption that, in fact, all cultural objects that rely on digital representation and computer-based delivery do share a number of common qualities. In the book I articulate a number of principles of new media: numerical representation, modularity, automation, variability and transcoding. I do not assume that any computer-based cultural object will necessary be structured according to these principles today. Rather, these are tendencies of a culture undergoing computerization that gradually will manifest themselves more and more. For instance, the principle of variability states that a new media cultural object may exist in potentially infinite different states. Today the examples of variability are commercial Web sites programmed to customize Web pages for every user as she is accessing the site particular user, or DJs remixes of already existing recordings; tomorrow the principle of variability may also structure a digital film which will similarly exist in multiple versions.

I deduce these principles, or tendencies, from the basic fact of digital representation of media. New media is reduced to digital data that can be manipulated by software as any other data. This allows automating many media operations, to generate multiple versions of the same object, etc. For instance, once an image is represented as a matrix of numbers, it can be manipulated or even generated automatically by running various algorithms, such as sharpen, blue, colorize, change contrast, etc.

More generally, extending what I proposed in my book, I could say that two basic ways in which computers models reality – through data structures and algorithms – can also be applied to media once it is represented digitally. In other words, given that new media is digital data controlled by particular “cultural” software, it make sense to think of any new media object in terms of particular data structures and/or particular algorithms it embodies.⁴ Here are the examples of data structures: an image can be thought of as a two-dimensional array (x. y), while a movie can be thought of as a three-dimensional array (x, y, t). Thinking about digital media in terms of algorithms, we discover that many of these algorithms can be applied to any media (such as copy, cut, paste, compress, find, match) while some still retain media specificity. For instance, one can easily search for a particular text string in a text but not for a particular object in an image. Conversely, one can composite a number of still or moving images together but not different texts. These differences have to do with different semiotic logics of different media in our culture: for example, we are ready to read practically any image or a composite of images as being meaningful, while for a text string to be meaningful we require that it obeys the laws of grammar. On the other hand, language has a priori discrete structure (a sentence consists from words which consist from morphemes, and so on) that makes it very easily to automate various operations on it (such as search, match, replace, index), while digital representation of images does not by itself allow for automation of semantic operations.
4. New Media as the Mix Between Existing Cultural Conventions and the Conventions of Software.

As particular type of media is turned into digital data controlled by software, we may expect that eventually it will fully obey the principles of modularity, variability, and automation. However, in practice these processes may take a long time and they do not proceed in a linear fashion – rather, we witness “uneven development.” For instance, today some media are already totally automated while in other cases this automation hardly exists – even though technologically it can be easily implemented.

Let us take as the example contemporary Hollywood film production. Logically we could have expected something like the following scenario. An individual viewer receives a customized version of the film that takes into account her/his previous viewing preferences, current preferences, and marketing profile. The film is completely assembled on the fly by AI software using pre-defined script schemas. The software also generates, again on the fly characters, dialog and sets (this makes product placement particularly easy) that are taken from a massive “assets” database.

The reality today is quite different. Software is used in some areas of film production but not in others. While some visuals may be created using computer animation, cinema sill centers around the system of human stars whose salaries amount for a large percent of a film budget. Similarly, script writing (and countless re-writing) is also trusted to humans. In short, the computer is kept out of the key “creative” decisions, and is delegated to the position of a technician.

If we look at another type of contemporary media -- computer games – we will discover that they follow the principle of automation much more thoroughly. Game characters are modeled in 3D; they move and speak under software control. Software also decides what happens next in the game, generating new characters, spaces and scenarios in response to user’s behavior. It is not hard to understand why automation in computer games is much more advanced than in cinema. Computer games is one of the few cultural form “native “ to computers; they begun as singular computer programs (before turning into a complex multimedia productions which they are today) -- rather than being an already established medium (such as cinema) which is now slowly undergoing computerization.

Given that the principles of modularity, automation, variability and transcoding are tendencies that slow and unevenly manifest themselves, is there a more precise way to describe new media, as it exists today? The Language of New Media analyzes the language of contemporary new media (or, to put this differently, “early new media”) as the mix (we can also use software metaphors of “morph” or “composite”) between two different sets of cultural forces, or cultural conventions: on the one hand, the conventions of already mature cultural forms (such as a page, a rectangular frame, a mobile point of view) and, on the other hand, the conventions of computer software and, in particular, of HCI, as they developed until now.

Let me illustrate this idea with two examples. In modern visual culture a representational image was something one gazed at, rather than interacted with. An image was also one continuous representational field, i.e. a single scene. In the 1980s GUI redefined an image as a figure-ground opposition between a non-interactive, passive ground (typically a desktop pattern) and active icons and hyperlinks (such as the icons of documents and applications appearing on the desktop). The treatment of representational images in new media represents a mix between these two very different conventions. An image retains its representational function while at the same time is treated as a set of hot spots (“image-map”). This is the standard convention in interactive multimedia, computer games and Web pages. So while visually an image still appears as a single continuous field, in fact it is broken into a number of regions with hyperlinks connected to these regions, so clicking on a region opens a new page, or re-starts game narrative, etc.

This example illustrates how a HCI convention is “superimposed” (in this case, both metaphorically and literally, as a designer places hot spots over an existing image) over an older representational convention. Another way to think about this is to say that a technique normally used for control and data management is mixed with a technique of fictional representation and fictional narration. I will use another example to illustrate the opposite process: how a cultural convention normally used for fictional representation and narration is “superimposed” over software techniques of data management and presentation. The cultural convention in this example is the mobile camera model borrowed from cinema. In The Language of New Media I analyze how it became a generic interface used to access any type of data:

As a result of this mix, we get such strange hybrids as clickable “image-maps,” navigable landscapes of financial data, QuickTime (which was defined as the format to represent any time-based data but which in practice is used exclusively for digital video), animated icons – a kind of micro-movies of computer culture – and so on.

As can be seen, this particular approach to new media assumes the existence of historically particular aesthetics that characterizes new media, or “early new media,” today. (We may also call it the “aesthetics of early information culture.”) This aesthetics results from the convergence of historically particular cultural forces: already existing cultural conventions and the conventions of HCI. Therefore, it could not have existed in the past and it unlikely to stay without changes for a long time. But we can also define new media in the opposite way: as specific aesthetic features which keep re-appearing at an early stage of deployment of every new modern media and telecommunication technologies.
5. New Media as the Aesthetics that Accompanies the Early Stage of Every New Modern Media and Communication Technology.

Rather than reserving the term new media to refer to the cultural uses of current computer and computer-based network technologies, some authors have suggested that every modern media and telecommunication technology passes through its “new media stage.” In other words, at some point photography, telephone, cinema, television each were “new media.” This perspective redirects our research efforts: rather than trying to identity what is unique about digital computers functioning as media creation, media distribution and telecommunication devices, we may instead look for certain aesthetic techniques and ideological tropes which accompany every new modern media and telecommunication technology at the initial stage of its introduction and dissemination. Here are a few examples of such ideological tropes: new technology will allow for “better democracy; it will give us a better access to the “real” (by offering “more immediacy” and/or the possibility “to represent what before could not be represented”); it will contribute to “the erosion of moral values”; it will destroy the “natural relationship between humans and the world” by “eliminating the distance” between the observer and the observed.

And here are two examples of aesthetic strategies that seem to often accompany the appearance of a new media and telecommunication technology (not surprisingly, these aesthetic strategies are directly related to ideological tropes I just mentioned). In the mid 1990s a number of filmmakers started to use inexpensive digital cameras (DV) to create films characterized by a documentary style (for instance, Timecode, Celebration, Mifune). Rather than treating live action as a raw material to be later re-arranged in post-production, these filmmakers place premier importance on the authenticity of the actors’ performances. The smallness of DV equipment allows a filmmaker to literally be inside the action as it unfolds. In addition to adopting a more intimate filmic approach, a filmmaker can keep shooting for a whole duration of a 60 or 120 minute DV tape as opposed to the standard ten-minute film roll. This gives the filmmaker and the actors more freedom to improvise around a theme, rather than being shackled to the tightly scripted short shots of traditional filmmaking. (In fact the length of Time Code exactly corresponds to the length of a standard DV tape.)

These aesthetic strategies for representing real which at first may appear to be unique to digital revolution in cinema and in fact not unique. DV-style filmmaking has a predecessor in an international filmmaking movement that begun in the late 1950s and unfolded throughout the 1960s. Called “direct cinema,” “candid” cinema, “uncontrolled” cinema, “observational” cinema, or cinéma vérité (“cinema truth”), it also involved filmmakers using lighter and more mobile (in comparison to what was available before) equipment. Like today’s DV realists,” the 1960s “direct cinema” proponents avoided tight staging and scripting, preferring to let events unfold naturally. Both then and now, the filmmakers used new filmmaking technology to revolt against the existing cinema conventions that were perceived as being too artificial. Both then and now, the key word of this revolt was the same: “immediacy.”