"Virtual Reality: Directions of Growth"
Hi folks:
Over the next few days, I'll be posting the sections of a recent
article called "Virtual Reality: Directions of Growth". Several
current discussion topics on sci.virtual-worlds are addressed.
Here are the section headings, grouped into six postings:
I. Introduction
II. The Research Suite
III. The Essence is Inclusion
IV. Virtual World Projects at HITL
V. Other Research Areas
VI. Virtual World Tools
VII. Educational Approaches
VIII. Lessons Learned
IX. Coming Attractions
X. Risks
XI. Evolving Philosophies
William Bricken
Virtual Reality: Directions of Growth
Notes from the SIGGRAPH '90 Panel
Copyright (C) 1990 All Rights Reserved by William Bricken
William Bricken
Human Interface Technology Laboratory
University of Washington, FU-20
Seattle, WA 98125
9/10/90
william@hitl.vrnet.washington.edu
I. INTRODUCTION
Virtual reality (VR) systems were introduced to the general public by
VPL and by Autodesk on June 6, 1989, VR Day, at two trade shows. This
event was preceded by about four months of media coverage. Since
then, VR has captured the public's imagination. It is also in the
unique position of being commercially available before being
academically understood.
Any technology which has the audacity to call itself a variety of
reality must also propose a paradigm shift. In essence, a paradigm
shift expands the potential of an entire discipline. For me, VR has
expanded every aspect of Computer Science, and is providing a base for
a very satisfying philosophy as well.
And just what is the paradigm shift? Computers are not only symbol
processors, they are reality generators. Until recently, computers
have generated only one dimensional symbolic strings. Text and
numbers. Text is a code which, when read, generates images of reality
in our minds. During the 80s, we enhanced the expressability of
computation by adding space and time dimensions to the realities being
generated. Two dimensional windows, 2D animation, solid modeling,
simulation. Now, in the 1990s, computer systems can generate virtual
environments, entire multisensory worlds which include us as
interactive participants. Digital information can seem as-if-real,
changing our notions of computation, symbolism, meaning, metaphysics,
self, and culture. Virtual realities are more than real.
The potential for VR to contribute to societal infrastructures such as
manufacturing, marketing, telecommunications, science, entertainment,
art, education, medicine, and media, suggests an economic impact that
rivals the Gross National Product. We live in two superimposed
worlds, the one of mass and the one of information. The huge
accumulation of difficult to access words on paper indicates that the
world of mass is not particularly well-suited for dealing with
information. As our culture matures into an information society, we
are now discovering the virtual world, an ideal place for interacting
with information.
What follows is a wide ranging discussion of interesting growth areas
for VR. I'll define VR, point to some active areas of research, tell
you about virtual world tools, outline some things we have learned
from working in the field, and discuss some risks and philosophies
engendered by VR techniques.
II. THE RESEARCH SUITE
VR is the body of techniques that apply computation to the generation
of experientially valid realities. HITL is forming its research
agenda around a suite of three interrelated technologies:
Behavior Transducers
hardware interface devices
Inclusive Computation
software interaction techniques
Intentional Psychology
biological constraints and plasticity
Behavior transducers map natural behavior onto digital streams.
Natural behavior is what two year olds do: point, grab, issue single
word commands, look around, toddle around. Behavior transducing
interface devices include body trackers, voice recognizers, spatial
sensors, kinesthetic feedback devices, and subjective audio and video
displays. Transducers work in both directions, physical behavior to
digital information (the virtual body) and virtual display to
subjective experience (the physiological model).
Inclusive software provides tools for construction of, management of,
and interaction with digital environments which surround a
participant/user. The central design issue for VR is getting behavior
transducers and virtual environments to feel good to a participant.
The intentional psychology of VR will require a deep knowledge of how
we work, our physiology, our sensations, our cognition. We must
refocus the effort of interface from the needs of symbol processors to
the needs of people.
III. THE ESSENCE IS INCLUSION
We at HITL believe that the primary defining characteristic of VR is
inclusion, being surrounded by an environment. VR places the
participant inside information. Some of the changes in perspective:
picture --> place
observe --> experience
use --> participate
interface --> inhabit
When we extend our field of view onto a computational environment
beyond about 60 degrees, a remarkable phenomenon occurs. We shift
from a feeling of viewing a picture to a feeling of being in a place.
This shift is accompanied by an emotional response. It seems as
though the unification of our symbolic processes with our visual
processes creates a feeling of wholeness, of empowerment. We shift
from external users (exercising rights) to internal participants
(exercising responsibilities), from being observers to having
experiences, from interfacing with a display to inhabiting an
environment.
My colleague Meredith Bricken and I have collected videotaped behavior
and exit interviews from over 500 people that we have guided through
initial VR experiences. We have seen overwhelmingly positive
responses, eagerness to return to "that place", willing suspension of
disbelief.
IV. VIRTUAL WORLD PROJECTS AT HITL
Our knowledge about VR and about how people respond to the VR
experience is being extended at HITL through several active projects:
information database, sci.virtual-worlds
simulation laboratory
virtual environment operating shell
laser microscanner display techniques
design and construction of worlds
3D audio display
instrument display prototypes
multiple participant worlds
educational experiences and environments
virtual prostheses
The information database is a project for NASA to follow the
development of VR and to serve as a clearinghouse for references and
research in the field. Sci.virtual-worlds is a moderated USENET
newsgroup for the discussion of VR issues.
The simulation laboratory provides a research environment for
prototyping VR hardware and for testing and evaluating effects on
human sensory, perceptual and psychomotor behavior.
The Virtual Environment Operating Shell is a software suite currently
written in C that wraps around the UNIX operating system. VEOS
provides resource and communication management for coordination of the
modules which make a VR system:
i/o hardware, behavior transducing input and display devices
world construction kits, CAD packages
dynamic simulation kits, for interaction and animation
virtual world tools
computational and display processors
The laser microscanner is a hardware research project to design a high
performance, low cost virtual display. Rather than creating an aerial
image using cathode-ray tubes or matrix element devices, the laser
microscanner scans a color image directly onto the retina. We don't
think in terms of addressing pixels, we think in terms of addressing
rods and cones directly. The head-mounted unit will integrate 3D
visual and audio display, voice recognition, and head and eye
tracking.
We build worlds for presentation, evaluation, and experimentation.
Our interest is the design of comfortable, functional worlds.
For Boeing, we are exploring 3D audio display techniques, and building
prototypes for design and display of complex instrument panels and
machines, in essence simulating the design of aircraft cockpits.
We are working on the implementation of multiple participant worlds
for an application to telecommunications. You can think of VR as a
very sophisticated replacement for the telephone.
Education and industrial training are natural applications of VR
techniques. We are designing virtual environments conducive to
learning, we're studying the transfer of skills between virtual and
actual tasks, and we're exploring the implications of VR for
educational theory and practice.
And we have great interest in the application of VR to prostheses for
the handicapped, for providing virtual bodies which extend individual
capacities, for providing alternative control devices for interaction
in virtual worlds.
V. OTHER RESEARCH AREAS
VR has intersected other areas of research in some surprising ways:
audio modeling
teleoperation, telepresence
image integration, HDTV
interactive drama
military simulation
3D audio hardware is commercially available, we should expect to hear
of inclusive sound systems in the stores soon. Audio theorists are
interested in specification languages for 3D music, in audio lenses
and icons (earcons), and in modeling ambience, the analog of
ray-tracing for sound.
Telepresence, the development of remotely controlled robots, requires
the same interface techniques as VR. The primary difference between
these disciplines is that teleoperation looks at interaction with real
(usually inaccessable) images, VR looks at virtual images. Both want
inclusive, interactive environments. The possibility of inhabiting
real worlds shook me out of a self-imposed computer graphics
narrowness. We can apply VR interaction and hardware techniques to
explore anywhere we can place a probe. We can inhabit a remote
undersea vehicle, processing digitized images into worlds that mix the
actual with the virtual. We can swallow a miniaturized transmitter
and explore our own stomach. We can build artificial bees with fiber
optic visual links and micromotors for dancing and for rubbing
antennae. We can then put our virtual bee-selves into the physical
hive and interact with real bees in their home environment. I can
hardly wait.
The multimedia community is very interested in digital images. It
seems only natural that we should port these flatlander tools into VR.
We could tile polygons with TV. More importantly, automated
conversion of images to 3D objects (the image recognition problem)
would permit a seamless integration of video-real with
graphic-virtual.
Hypertext has raised the question of interactive fiction. The
theatrical community is working to install plot and character into
virtual worlds, creating interactive drama. What do a good story and
a good experience have in common? Can we construct participatory
plots, guided experiences, autonomous characters?
Actually, VR grew up in the military. The first substantive
application of VR was to help Air Force pilots improve their ability
to aim missiles. The most refined and widely distributed VR
environment today is SIMNET, a large scale, simulated tank combat
system. Recently, I saw a paper on training close combat fighters in
VR. Sort of reminds me of the video arcade.
VI. VIRTUAL WORLD TOOLS
To give you an idea of what work and play will be like in VR, I'll
describe some of the tools we're designing at HITL:
the wand
the virtual body
virtual home, virtual community
concurrent inconsistent worlds
autonomous entities
concrete mathematics, experiential programming
The Wand is an evolution of the Mouse. It is a simple physical device
with a wide diversity of uses, ideal characteristics for a tool.
Physically, the wand is a spatial position and orientation sensor on a
handheld stick. In software, the Wand emanates a ray which can be
used for pointing at virtual objects. Coupled with voice commands,
the Wand can be used to identify objects, to attach to and move
objects, to bring things closer or place them at a distance, to
indicate a direction for flying, to identify a location to teleport
to, to measure distance, as a pen for drawing, as a knife, as a
switch, as a spotlight. Lots of functionality from a little hardware.
People achieve presence in VR by inhabiting a virtual body. The
virtual body is a software toolkit for associating an arbitrary suite
of behavior transducers (such as wands, voice command systems,
headtracking, etc.) to a display of self in a virtual world. What we
do physically is sensed and converted to virtual behavior. Don't
think that the virtual body is necessarily in the shape of our
physical body; any object in VR can be inhabited. If you are
controlling a physical robot, you may prefer your virtual body to be
the shape of that robot. If you are navigating a data terrain, you
may prefer to have a virtual body shaped like a jeep or an airplane.
The virtual body can filter and map physical behavior onto superhuman
capacities. One of the first things we did to figure out how a
virtual body might be used was to search the old comic books for super
powers.
The virtual home is an environment designed for personalized comfort,
for work and for play. My virtual home will have a cozy chair, a
fireplace, some cats, and a cabinet full of virtual tools and toys,
essentially what I now have at (physical) home. Physical reality is a
great starting model for virtual reality. Take what we like and
delete what we don't.
Virtual homes will be customized, personalized environments. The
virtual home extends to a virtual community. People we work with are
not organized by some cryptic email address that is basically a
program to tell the network where to find them. They are organized in
close proximity in space. In a virtual community, friends have
virtual homes that are visible from our own virtual home. They are
our neighbors. We visit them by pointing to their home and saying
"jack me there". Less frequent acquaintances may be down the road or
over the hill. The idea is to organize virtual space to accommodate
to human culture.
One profound capability in VR is to maintain inconsistent views for
different participants, to intermix personal realities. In physical
reality, mass has a way of being unarguable. We quickly default to
assuming a consistent, objective reality that is communal to everyone.
Consistency is an assumption and is widely overgeneralized. Each
person in physical reality, for example, has a viewpoint, each
viewpoint is necessarily in a different physical place, each
perspective provides different information about the inclusive
environment. Every experience is unique. We agree to suppress our
differences for massive objects, but the line is always fuzzy. We
certainly tolerate differences within the domain of conversation. How
we talk is an excellent example of concurrent inconsistent worlds.
In VR, communality can be negotiated rather than assumed. In VR, the
color of my shirt can appear to be green to me, but blue to you. So
long as we do not talk about or interact with the color of the shirt,
how it is rendered to each of us is irrelevant. Carry this a bit
further: I can be sitting in my virtual home next to an empty chair.
You jack a duplicate of your virtual body into that chair. From my
perspective, you are visiting me. Now, from your perspective, you are
still sitting in your virtual home, in your customized environment.
You have an empty chair, and I jack a duplicate of my virtual body
into it. We are now sitting in two totally different environments
while sharing a mutual conversation. For me, you are in my home, for
you, I am in your home. So long as the inconsistencies in our
environments are not items of contention or confusion, the differences
will not interfer with communication. When they do interfer, the
explicit differences become subject to negotiated resolution.
But the pluralism of VR is much deeper. It is possible to maintain
inconsistencies directly, without resolution, using a mathematical
technique called the imaginary boolean value. We could choose to
represent the color of my shirt as ambiguous, as context dependent.
Both green and blue. We can then discuss the color of the shirt as
being inconsistent, as information about which we simply do not see
eye to eye. I bring up these ideas from an esoteric branch of
representation theory to illustrate a fundamental point. VR is not
bounded by the assumptions of physical reality. We can have whatever
we can formally specify.
The HITL architecture specifies that every object in VR, including
space itself, have processing and memory resources. Entities are
objects with the capabilities of operating systems. Every entity is a
system, every entity is a variant of the same system. This means that
we can use the same editing, debugging, and interaction tools for
modifying each entity. Entities are running a sense-process-act loop;
in artificial intelligence terms, each entity is an agent, an actor.
This means that VR is inhabited with artificial life. Every entity is
capable of independent action, in response to environmental changes,
in response to internal memory or process changes, or in response to
changes in the rules, the disposition, specifying that entity's
internal processes. Each entity is an expert system using
pattern-matching on its input to trigger disposition rules and
metarules which generate outputs to the context. The environment
itself is just another entity, one that includes other entities within
it. All cyberspace is Toontown.
We have been able to demonstrate that mathematics itself (in
particular logic, integers, and sets) can be expressed concretely,
using 3D arrangements of physical things, such as blocks on a table,
doors open or shut, rockwalls that respond to gravity, the things of
everyday life. String-based symbolic representations of mathematical
concepts are typographically convenient, but tokens are not at all
essential to mathematical expression. VR makes it convenient to
express abstract ideas using spatial configurations of familiar
objects. One benefit of this approach is that we can build visual
programs, set them on a virtual table, and watch them work. We can
experience programs as other entities rather than as dumps of text.
Bugs would manifest as structural anomalies, as visual irregularities.
Architectural design has a sensual, experiential semantics. It is but
a quirk of typography that we have ignored the experiential semantics
of computational languages. More fundamentally, experiential
computing unites our spatial and our symbolic cognitive skills,
permitting mathematical visualization, analytic gestalt, whole brain
processing.
VII. EDUCATIONAL APPROACHES
VR provides an exciting educational medium for exploring worlds and
for exploring ourselves. It provides a training environment that is
rich, replicable, and responsive. It permits direct evaluation of
educational theory. The central educational issue for VR is one of
transfer of experience. Do skills and habits learned in VR transfer
to the physical world? Here are some educational issues:
Constructivism
"Human knowledge is essentially active." Piaget
Natural Semantics
non-symbolic, preoperational interaction
Programmable Participation
conducive and responsive environments
Cognitive Presence
modifiable self-concepts, learning by becoming
Social Reality
unique concurrent worlds
Educational psychologists have long known that people actively
construct their experience of reality. In VR, students will construct
their knowledge, then dwell within it, exploring their understanding.
Natural semantics means that the computational environment hides
symbolism in favor of displaying information in an innately
recognizable form. The two-year-old criterion: if a kid recognizes
it, its natural. The three Rs, all symbolic, will become the three
ACTs: enact, interact, and abstract.
VR provides the potential for completely customized, individualized
learning. Educational environments will uniquely respond to the
participant-learner, in terms of both needs and preferences. A
student model will not be necessary, instead the teacher and student
will modify the environment in support of student behavior.
We also have a tool for affective education, for sharing perspectives,
for mapping perspectives into broader contexts, for changing
self-image, for remapping capabilities.
Education is inherently social. Explicitly shared worlds and multiple
concurrent agreements provide the opportunity for groupwork, social
consensus, and the construction of functional, multiparticipant
environments.
In general, everything we do to educate with words and pictures can be
provided as virtual experience.
VIII. LESSONS LEARNED
Tom Furness, the Director of HITL, has over twenty years experience in
VR. He pioneered most of the hardware interface devices we use today,
in the extremely demanding environment of military aircraft.
Personally, I have worked on VR related projects for six years,
beginning at Atari Research Labs in 1984. Meredith Bricken designed
Autodesk's worlds, built Virtual Seattle for CHI'90, and has pioneered
research into the design of comfortable virtual environments. Over
the years we have learned some lessons:
Psychology is the Physics of VR.
Our body is our interface.
Knowledge is in experience.
Data is in the environment.
Scale and time are explorable dimensions.
One experience is worth a trillion bits.
Realism is not necessary.
A major theme of VR research is that Psychology, in the broad sense of
behavior, perception, cognition and intention, provides the rules and
the constraints of virtual worlds. Psychology is the Physics of VR.
This may come as a shock, it is one of those truths that is obvious
after it is said, but elusive before it is stated explicitly. Our
body is our interface. Interface is not something that is out there,
in some machine. Interface is a boundary which both connects and
separates, interface takes place at the surface of our skin. From the
perspective of VR, interface is physiology, interaction is natural
behavior. We simply want to use the power of computers to make
computation invisible.
Knowledge is in experience, it is not in some abstract, symbolic
representation. Data is in the environment, it is not stored away in
some memory array. These observations serve to remind us that we are
not the computer. To understand computation, we should participate
within it, rather than writing programs to dominate it. Humans have a
great skill for projecting outward, for becoming the tool we are
handling. We need reminding that we are creatures who dwell inside an
environment.
VR is inherently multidimensional. As well as freedom of translation
and rotation, in VR we can travel in scale and in time. Think of
scale as simply another direction; when we traverse scale, size
instead of location changes. We can also travel through time using
any of the techniques of film editing, including slow-motion, fast
forward, and temporal discontinuity.
There is a tremendous compression ratio between digital information
and human experience. Very approximately, it takes a hundred million
polygons to simulate what we see in one scene. Add duration,
multisensory channels, and interaction, and you get a lot of digital
information being transacted with each moment of consciousness.
Computation will not come close to this bandwidth for a long time.
Fortunately, virtual world experience does not require the information
density of physical reality.
Because our minds provide such tremendous flexibility in interpreting
what is outside of us, realism in VR is simply not necessary. Our
cognitive plasticity permits even simple cartoon worlds of 500
polygons to be experientially satisfying. We must design worlds that
respect our physiological needs. For example, we conceptualize
perspective in physical space as having six degrees of freedom, three
in translation and three in rotation. But our bodies have roughly
four and one-twelth degrees of freedom. We move easily in all
directions on a plane, forward and to the side, but not up, off the
surface we stand on. We rotate freely around the vertical by turning,
but our natural rotation forward, around our waist by leaning, is at
best 270 degrees (3/4 of a full 360 degree rotation). And our ability
to bend side to side is only about 120 degrees, one-third of a full
rotation. This adds up to a little more than four degrees of
physiological freedom. Input devices which permit complete freedom of
translation and rotation usually get people lost in space. The
dimensionality of our abstract perspective does not match that of our
physical construction. We must also differentiate that which is
innate from that which is learned. Pilots, for example, have learned
to fly in all six degrees of freedom. Realism is both physiological
necessity and cognitive interpretation. In VR, world design that
conforms to physiological necessity frees our minds to furnish the
rest of our reality.
IX. COMING ATTRACTIONS
Here are the coming attractions, what I believe will be available by
the end of the decade:
public domain VR software
massive database access
fabric of space
negotiable group space
conversational programming
artificial life
crossvalidation of realities
HITL is electing to distribute its software in the public domain. We
hope to create a context for the growth of an industry and for the
understanding of alternative realities. We hope to encourage the
evolution of a shared software and hardware environment which will
permit researchers to share progress and results. The commercial
marketplace can then improve on public work, selling value-added
features like customer support, prebuilt worlds, faster hardware,
better algorithms, realer time.
VR requires a new approach to database management. We want to access
massive databases such as Landsat as a function of our perspective,
our location in the database. We expect to see interactive databases
which we can explore through movement. Already waiting is the entire
Earth to one meter resolution, the location of every aircraft and
ship, large hunks of the Moon, the human body down to the resolution
of a cell, the flow of the economy, the network of computation. We
have digital worlds to explore.
I have mentioned that space is an entity. Many interactions between
entities can be expressed as internal processes of the spaces which
include them. Gravity is a primary example; we can implement
simplistic local gravity by decrementing the Z component of the
velocity vector of each entity in a space at each time tick. Rules
that apply uniformly to every entity in a space instead can be
ascribed once to the space itself. The inclusive space enacts local
gravity by owning the locations of the entities it includes. We want
to be able to place fields in space, to have space maintain its local
version of continuity, gradient, and metric, to build space-filling
logics which branch as a function of location.
One advantage of customized environments is that we will have to be
explicit about what is shared. VR suggests an approach to cooperative
work in a computational environment: rather than assume communality
and specify differences, assume complete difference and specify what
is common. It may turn out to be fun to build communal, consensual
contexts, to negotiate the group space.
One consequence of autonomous entities is that they can respond to our
communications. With voice recognition, we will be able to speak to
virtual entities as a means of programming their structure and
behavior. "I want the green cube I'm looking at to double in size."
The cube has a sensor for voice. Its rulebased disposition matches
the vocal input to its own identity and to its size changing function.
If you have permission, it changes itself to your specification.
Another consequence of autonomous entities is that they may have their
own agenda. The coupling of the behaviors of several entities could
determine events. Rulebases that support emergent behavior are
tremendously difficult to construct. We hope that the programmable
environment of VR will provide autonomous entities with a context for
the growth of interesting virtual life.
Fundamentally, VR forms a new reality, at least to the extent that we
are willing to relax our minds. We will need to calibrate the effects
of transfer across worlds and across realities. VR is the first
empirical tool of metaphysics, it permits us to compare realities, to
ask which alternative reality is preferable for which tasks.
X. RISKS
Do virtual worlds pose significant risks? I have prepared a list of
what I believe are the issues and problems for VR:
descriptive confusion
lack of experience
cognitive remodeling
fluid self
sensory overload, sensory ecstasy
power and control
cultural adaptability
VR is seeking definition, it could be anything from email to a fully
surrounding, multi-sensory environment. We are struggling with
appropriate comparisons. VR is not a drug and is not physically
addictive. Drugs change our perspective from inside the body, VR
changes our external environment. VR may well be psychologically
addictive (that is, entertaining), just like all good media
experiences can be. And there is that constant tension between
physical responsibility and cognitive exploration. Is VR escapist?
Escapism means seeking diversion from physical reality. VR cannot
escape being escapism, VR is perfect escapism. Is VR theater, or
interactive drama, or is it more than art? Is it scientific
visualization, or physical simulation, or is it more than science? Is
it financial modeling, or the perfect sales tool, or is it more than
economics? It's a good idea to spend some time figuring out what VR
is.
To me, the greatest problem is that we have virtually no experience in
VR. There are perhaps around ten thousand VR non-virgins. But I
estimate that there are no more than fifty people who have spent
twenty hours in VR. All of this excitement is purely conceptually, we
have very little experience with what we are talking about. The first
item on the VR agenda must be to construct and distribute hundreds of
systems, so that many people can contribute to our understanding. We
should know at least something about the cognitive effects of VR
before it is a consumer item with the distribution of Nintendo. When
a representative of MCC asked the lab the best way to invest two
million research dollars in VR, the answer was clear: give away forty
$50,000 systems.
The most complex, and potentially dangerous, risk is what we are
calling cognitive remodeling. Those who spend a lot of time in VR
bring back to physical reality some strange habits, like navigating
across a room by pointing, like bumping into walls cause they aren't
just images, like dreaming in polygons. VR effects dreaming strongly,
it seems to provide tools for control of the dreamlife from within the
dream. VR changes mental models. Now, it is not dangerous that this
is happening, cause all intense work produces similar effects. Anyone
who has programmed all night will know that the programming slips into
dreams. The problem is not that these things happen, it is that we
don't have the faintest clue what is going on. We do not know the
borders between virtual and actual. We have not yet had the
opportunity to evaluate current theories of reality crossing.
And how will we react when we are able to redefine our bodies, swap
our perspectives, mix our senses. We will have the ability to map
arbitrarily across sensory input, self-image, and behavioral output.
What will a fluid self be like? We will need to understand the
cognitive and behavioral effects of transportable perspectives, of
programmable bodies, of synesthetic sensations, of exchangable body
parts, of inhabiting arbitrary objects, of masslessness, of negotiable
communality, of complete empowerment.
Are there limits to the degree of warpage our senses can tolerate?
This is, of course, an empirical question. What are the functional
constraints of sensory modification for enhanced productivity, for
enhanced enjoyment? Are there sensory pathways to insanity or to
ecstasy? Just which side of the monitor do you stand on?
We have been discussing a domain which emphasizes personal freedom.
VR could be used for horrible purposes, but that negative assumes that
we are strapped to a chair. So long as each individual has the
freedom to reach up and turn off the experience, VR itself is quite
benign. But how will authority respond to this frontier? VR is
interactive, but will I have the right to remove the virtual arches in
my prebuilt reality given away with each hamburger? Are
advertisements from the creator necessarily non-interactive? Where
are the edges of property and ownership in a world which is digital?
Will there be commodities? What are the rights of autonomous
computational entities? Will there be stability? Will there be a
Virtual Environmental Protection Agency? I don't know, but I
certainly look forward to negotiating the communal rules of personal
responsibility in cyberspace.
The biggest issue is how our culture will respond to this new reality.
We have amassed hundreds of years of favoritism for the objective, the
scientific. Our values, ethics, and aesthetics are predisposed toward
Objectivism. Is VR a better place for transacting information? How
will physical reality react to competition? What will socialization
without material consequence be like? What kind of intimacy will
arise from explicitly penetrating world views? What kind of cultures
will arise when the VR network is standardized? Are we like Columbus,
discovering a completely new land in an unexpected place? Is living
in VR necessarily pathological? These are indeed exciting times.
XI. EVOLVING PHILOSOPHIES
We have come very close to talking about philosophy, so here are some
comments on philosophical concepts:
situated semantics
pervasion
immaterial realism
constructivism
boundary mathematics
more than reality
Situated activity is a growing school of thought in AI. The idea is
simple: what we do depends on our environment as well as our internal
state. We react and respond, constantly bringing external context
into our interpretation of the moment. Currently, symbolic logic is
split in half, between syntax (representation) and semantics
(meaning). Syntax is strictly formal, it has no basis in experience.
Semantics attempts to connect syntactic symbols to the reality of the
world by mapping representation onto meaning. The problem is that it
does so without regard to context external to the formal symbols.
Since environments necessarily introduce external unknowns, standard
semantics is just too literal. Situated activity is an attempt to
build a theory of context.
VR, in comparison, is totally situated. By defining natural behavior
as the rules of interaction, by displaying recognizable spatial
structures as output, by providing context in toto, and by including
the participant, VR redefines the relation between syntax and
semantics. Semantics, what we consider to be anchored to reality, is
displayed directly as (virtual) reality. Syntax, the symbols that
guide computational activity, is hidden in the background, out of
sight.
Environments include their participants, they pervade their contents.
Pervasion is a non-dualistic concept, more familiar to Buddhism than
to Christianity. A pervasive space is one which is diffused
throughout every part of itself, including those parts occupied by
other spaces and objects. Objects themselves are those boundaries of
spaces that we can sense. When we look at the container of an
environment, from the outside, we see that it surrounds a portion of
space. The important point to understand is that environments focus
our attention on a particular portion of space, they do not separate
space into two opposite parts. The outside space still pervades the
inside space. For VR, the physical pervades the virtual. When we
enter a virtual world, we always bring our physical body. VR is not a
separate reality, in a dualistic sense, it is a pervaded reality. The
shift from duality to pervasion is from networks to maps, from
separation to unity, from confrontation to cooperation, from male to
female, from one to zero.
The Copernican revolution introduced a physics that differed
fundamentally from appearance. VR introduces a metaphysics that
differs fundamentally from the material. At the foundation of
Objectivism is an attempt to be realistic about the material world.
VR calls for immaterial realism, for being realistic about
information. The currency of VR is organization, not possession, not
accumulation, not territory. All laws are transmutable, we can
satisfy fantasy rather than fact. It is science itself that is
redefined. In VR, we can choose to be reductionalist, but at the
bottom of it all, there is not Mass or Nature, there is the Void. VR
is representational, but not a priori rational, empirical, or
verifiable. VR is illogical positivism: if you can specify it, it is
meaningful. All empirical hypotheses are true.
Another fundamental philosophical position engendered by VR is that of
constructivism, that our minds and our bodies coparticipate in
defining reality. Objectivism places an overemphasis on the input of
the physical body. Solipsism overemphasizes the mind. Constructivism
recognizes that reality is like light, it is both particle and wave,
both objective and cognitive, both observation and participation. In
VR, we cannot escape the realization that we are the architects of our
environment.
We are applying boundary mathematics to VR in three different ways: as
a foundational mathematics, as a technique for logical deduction and
maintenance of inconsistency, and as a spatial embodiment of abstract
concepts.
Boundary mathematics is a calculus of inclusion. The essence of VR is
inclusion, the relationship between an environment and a participant.
The primitives of boundary mathematics are also participant and
environment. Let ( ) represent an environment, and let i represent a
participant. A variation of Spencer Brown's Laws of Form provides the
axiomatic basis:
Observe: i ( ) = ( )
Participate: ( i ) =
The left-hand-side of each equation is descriptive (objective),
explicitly mentioning the participant. The right-hand-side is
experiential (participatory), implicitly using the participant's
perspective. We read the left-hand-side from our traditional
externalized, objective perspective. The right-hand-side refers to
our experience, from the subjective perspective. When we observe an
empty environment, we perceive its boundary. When we are included in
an otherwise empty environment, we perceive emptiness. That's all
there is at the foundations of experience.
The most important thing to realize about VR is that it is more than
reality, more than a simulation of reality. You add physical realism
to a virtual world by adding constraints that reduce the possibilities
in that world. Native VR lets you walk through walls, we add
collision detection to disallow this power. Native VR has no gravity,
we add gravitational equations to simulate a gravitational reality.
Reality simulation is a subset of potential VR experiences. The least
elaborated virtual world is the Void.
We describe innovations in terms of what they replace. Only after
decades do we come to understand the pervasive impact of new
technologies on our culture. The automobile was first the horseless
carriage. It replaced the carriage, looked like a carriage, and moved
at the speed of a horse. Decades later, the automobile has
transformed our landscapes, the pace of our travels, and our concepts
of time and space. The television replaced the radio. Television
programs were first radio programs with pictures. Decades later, the
television has transformed our evenings, the pace of our senses, and
our concepts of news and entertainment.
The computer is first a symbol processor. Although decades have
barely passed, it is transforming our concepts of information and
calculation. Computers are thought to replace typewriters and
desktops and filing cabinets. But the computer has yet to be
understood for what it is of itself, we still view it from the
impoverished model of what it replaces. McLuhan said that computers
extend our central nervous system. Our CNS is not a symbol processor,
it is a generator of personal realities. VR marks the end of the
infancy of computation, the essence of the computer revolution is yet
to come. Essentially computers are reality generators.
And reality is in the eye of the participant.
Over the next few days, I'll be posting the sections of a recent
article called "Virtual Reality: Directions of Growth". Several
current discussion topics on sci.virtual-worlds are addressed.
Here are the section headings, grouped into six postings:
I. Introduction
II. The Research Suite
III. The Essence is Inclusion
IV. Virtual World Projects at HITL
V. Other Research Areas
VI. Virtual World Tools
VII. Educational Approaches
VIII. Lessons Learned
IX. Coming Attractions
X. Risks
XI. Evolving Philosophies
William Bricken
Virtual Reality: Directions of Growth
Notes from the SIGGRAPH '90 Panel
Copyright (C) 1990 All Rights Reserved by William Bricken
William Bricken
Human Interface Technology Laboratory
University of Washington, FU-20
Seattle, WA 98125
9/10/90
william@hitl.vrnet.washington.edu
I. INTRODUCTION
Virtual reality (VR) systems were introduced to the general public by
VPL and by Autodesk on June 6, 1989, VR Day, at two trade shows. This
event was preceded by about four months of media coverage. Since
then, VR has captured the public's imagination. It is also in the
unique position of being commercially available before being
academically understood.
Any technology which has the audacity to call itself a variety of
reality must also propose a paradigm shift. In essence, a paradigm
shift expands the potential of an entire discipline. For me, VR has
expanded every aspect of Computer Science, and is providing a base for
a very satisfying philosophy as well.
And just what is the paradigm shift? Computers are not only symbol
processors, they are reality generators. Until recently, computers
have generated only one dimensional symbolic strings. Text and
numbers. Text is a code which, when read, generates images of reality
in our minds. During the 80s, we enhanced the expressability of
computation by adding space and time dimensions to the realities being
generated. Two dimensional windows, 2D animation, solid modeling,
simulation. Now, in the 1990s, computer systems can generate virtual
environments, entire multisensory worlds which include us as
interactive participants. Digital information can seem as-if-real,
changing our notions of computation, symbolism, meaning, metaphysics,
self, and culture. Virtual realities are more than real.
The potential for VR to contribute to societal infrastructures such as
manufacturing, marketing, telecommunications, science, entertainment,
art, education, medicine, and media, suggests an economic impact that
rivals the Gross National Product. We live in two superimposed
worlds, the one of mass and the one of information. The huge
accumulation of difficult to access words on paper indicates that the
world of mass is not particularly well-suited for dealing with
information. As our culture matures into an information society, we
are now discovering the virtual world, an ideal place for interacting
with information.
What follows is a wide ranging discussion of interesting growth areas
for VR. I'll define VR, point to some active areas of research, tell
you about virtual world tools, outline some things we have learned
from working in the field, and discuss some risks and philosophies
engendered by VR techniques.
II. THE RESEARCH SUITE
VR is the body of techniques that apply computation to the generation
of experientially valid realities. HITL is forming its research
agenda around a suite of three interrelated technologies:
Behavior Transducers
hardware interface devices
Inclusive Computation
software interaction techniques
Intentional Psychology
biological constraints and plasticity
Behavior transducers map natural behavior onto digital streams.
Natural behavior is what two year olds do: point, grab, issue single
word commands, look around, toddle around. Behavior transducing
interface devices include body trackers, voice recognizers, spatial
sensors, kinesthetic feedback devices, and subjective audio and video
displays. Transducers work in both directions, physical behavior to
digital information (the virtual body) and virtual display to
subjective experience (the physiological model).
Inclusive software provides tools for construction of, management of,
and interaction with digital environments which surround a
participant/user. The central design issue for VR is getting behavior
transducers and virtual environments to feel good to a participant.
The intentional psychology of VR will require a deep knowledge of how
we work, our physiology, our sensations, our cognition. We must
refocus the effort of interface from the needs of symbol processors to
the needs of people.
III. THE ESSENCE IS INCLUSION
We at HITL believe that the primary defining characteristic of VR is
inclusion, being surrounded by an environment. VR places the
participant inside information. Some of the changes in perspective:
picture --> place
observe --> experience
use --> participate
interface --> inhabit
When we extend our field of view onto a computational environment
beyond about 60 degrees, a remarkable phenomenon occurs. We shift
from a feeling of viewing a picture to a feeling of being in a place.
This shift is accompanied by an emotional response. It seems as
though the unification of our symbolic processes with our visual
processes creates a feeling of wholeness, of empowerment. We shift
from external users (exercising rights) to internal participants
(exercising responsibilities), from being observers to having
experiences, from interfacing with a display to inhabiting an
environment.
My colleague Meredith Bricken and I have collected videotaped behavior
and exit interviews from over 500 people that we have guided through
initial VR experiences. We have seen overwhelmingly positive
responses, eagerness to return to "that place", willing suspension of
disbelief.
IV. VIRTUAL WORLD PROJECTS AT HITL
Our knowledge about VR and about how people respond to the VR
experience is being extended at HITL through several active projects:
information database, sci.virtual-worlds
simulation laboratory
virtual environment operating shell
laser microscanner display techniques
design and construction of worlds
3D audio display
instrument display prototypes
multiple participant worlds
educational experiences and environments
virtual prostheses
The information database is a project for NASA to follow the
development of VR and to serve as a clearinghouse for references and
research in the field. Sci.virtual-worlds is a moderated USENET
newsgroup for the discussion of VR issues.
The simulation laboratory provides a research environment for
prototyping VR hardware and for testing and evaluating effects on
human sensory, perceptual and psychomotor behavior.
The Virtual Environment Operating Shell is a software suite currently
written in C that wraps around the UNIX operating system. VEOS
provides resource and communication management for coordination of the
modules which make a VR system:
i/o hardware, behavior transducing input and display devices
world construction kits, CAD packages
dynamic simulation kits, for interaction and animation
virtual world tools
computational and display processors
The laser microscanner is a hardware research project to design a high
performance, low cost virtual display. Rather than creating an aerial
image using cathode-ray tubes or matrix element devices, the laser
microscanner scans a color image directly onto the retina. We don't
think in terms of addressing pixels, we think in terms of addressing
rods and cones directly. The head-mounted unit will integrate 3D
visual and audio display, voice recognition, and head and eye
tracking.
We build worlds for presentation, evaluation, and experimentation.
Our interest is the design of comfortable, functional worlds.
For Boeing, we are exploring 3D audio display techniques, and building
prototypes for design and display of complex instrument panels and
machines, in essence simulating the design of aircraft cockpits.
We are working on the implementation of multiple participant worlds
for an application to telecommunications. You can think of VR as a
very sophisticated replacement for the telephone.
Education and industrial training are natural applications of VR
techniques. We are designing virtual environments conducive to
learning, we're studying the transfer of skills between virtual and
actual tasks, and we're exploring the implications of VR for
educational theory and practice.
And we have great interest in the application of VR to prostheses for
the handicapped, for providing virtual bodies which extend individual
capacities, for providing alternative control devices for interaction
in virtual worlds.
V. OTHER RESEARCH AREAS
VR has intersected other areas of research in some surprising ways:
audio modeling
teleoperation, telepresence
image integration, HDTV
interactive drama
military simulation
3D audio hardware is commercially available, we should expect to hear
of inclusive sound systems in the stores soon. Audio theorists are
interested in specification languages for 3D music, in audio lenses
and icons (earcons), and in modeling ambience, the analog of
ray-tracing for sound.
Telepresence, the development of remotely controlled robots, requires
the same interface techniques as VR. The primary difference between
these disciplines is that teleoperation looks at interaction with real
(usually inaccessable) images, VR looks at virtual images. Both want
inclusive, interactive environments. The possibility of inhabiting
real worlds shook me out of a self-imposed computer graphics
narrowness. We can apply VR interaction and hardware techniques to
explore anywhere we can place a probe. We can inhabit a remote
undersea vehicle, processing digitized images into worlds that mix the
actual with the virtual. We can swallow a miniaturized transmitter
and explore our own stomach. We can build artificial bees with fiber
optic visual links and micromotors for dancing and for rubbing
antennae. We can then put our virtual bee-selves into the physical
hive and interact with real bees in their home environment. I can
hardly wait.
The multimedia community is very interested in digital images. It
seems only natural that we should port these flatlander tools into VR.
We could tile polygons with TV. More importantly, automated
conversion of images to 3D objects (the image recognition problem)
would permit a seamless integration of video-real with
graphic-virtual.
Hypertext has raised the question of interactive fiction. The
theatrical community is working to install plot and character into
virtual worlds, creating interactive drama. What do a good story and
a good experience have in common? Can we construct participatory
plots, guided experiences, autonomous characters?
Actually, VR grew up in the military. The first substantive
application of VR was to help Air Force pilots improve their ability
to aim missiles. The most refined and widely distributed VR
environment today is SIMNET, a large scale, simulated tank combat
system. Recently, I saw a paper on training close combat fighters in
VR. Sort of reminds me of the video arcade.
VI. VIRTUAL WORLD TOOLS
To give you an idea of what work and play will be like in VR, I'll
describe some of the tools we're designing at HITL:
the wand
the virtual body
virtual home, virtual community
concurrent inconsistent worlds
autonomous entities
concrete mathematics, experiential programming
The Wand is an evolution of the Mouse. It is a simple physical device
with a wide diversity of uses, ideal characteristics for a tool.
Physically, the wand is a spatial position and orientation sensor on a
handheld stick. In software, the Wand emanates a ray which can be
used for pointing at virtual objects. Coupled with voice commands,
the Wand can be used to identify objects, to attach to and move
objects, to bring things closer or place them at a distance, to
indicate a direction for flying, to identify a location to teleport
to, to measure distance, as a pen for drawing, as a knife, as a
switch, as a spotlight. Lots of functionality from a little hardware.
People achieve presence in VR by inhabiting a virtual body. The
virtual body is a software toolkit for associating an arbitrary suite
of behavior transducers (such as wands, voice command systems,
headtracking, etc.) to a display of self in a virtual world. What we
do physically is sensed and converted to virtual behavior. Don't
think that the virtual body is necessarily in the shape of our
physical body; any object in VR can be inhabited. If you are
controlling a physical robot, you may prefer your virtual body to be
the shape of that robot. If you are navigating a data terrain, you
may prefer to have a virtual body shaped like a jeep or an airplane.
The virtual body can filter and map physical behavior onto superhuman
capacities. One of the first things we did to figure out how a
virtual body might be used was to search the old comic books for super
powers.
The virtual home is an environment designed for personalized comfort,
for work and for play. My virtual home will have a cozy chair, a
fireplace, some cats, and a cabinet full of virtual tools and toys,
essentially what I now have at (physical) home. Physical reality is a
great starting model for virtual reality. Take what we like and
delete what we don't.
Virtual homes will be customized, personalized environments. The
virtual home extends to a virtual community. People we work with are
not organized by some cryptic email address that is basically a
program to tell the network where to find them. They are organized in
close proximity in space. In a virtual community, friends have
virtual homes that are visible from our own virtual home. They are
our neighbors. We visit them by pointing to their home and saying
"jack me there". Less frequent acquaintances may be down the road or
over the hill. The idea is to organize virtual space to accommodate
to human culture.
One profound capability in VR is to maintain inconsistent views for
different participants, to intermix personal realities. In physical
reality, mass has a way of being unarguable. We quickly default to
assuming a consistent, objective reality that is communal to everyone.
Consistency is an assumption and is widely overgeneralized. Each
person in physical reality, for example, has a viewpoint, each
viewpoint is necessarily in a different physical place, each
perspective provides different information about the inclusive
environment. Every experience is unique. We agree to suppress our
differences for massive objects, but the line is always fuzzy. We
certainly tolerate differences within the domain of conversation. How
we talk is an excellent example of concurrent inconsistent worlds.
In VR, communality can be negotiated rather than assumed. In VR, the
color of my shirt can appear to be green to me, but blue to you. So
long as we do not talk about or interact with the color of the shirt,
how it is rendered to each of us is irrelevant. Carry this a bit
further: I can be sitting in my virtual home next to an empty chair.
You jack a duplicate of your virtual body into that chair. From my
perspective, you are visiting me. Now, from your perspective, you are
still sitting in your virtual home, in your customized environment.
You have an empty chair, and I jack a duplicate of my virtual body
into it. We are now sitting in two totally different environments
while sharing a mutual conversation. For me, you are in my home, for
you, I am in your home. So long as the inconsistencies in our
environments are not items of contention or confusion, the differences
will not interfer with communication. When they do interfer, the
explicit differences become subject to negotiated resolution.
But the pluralism of VR is much deeper. It is possible to maintain
inconsistencies directly, without resolution, using a mathematical
technique called the imaginary boolean value. We could choose to
represent the color of my shirt as ambiguous, as context dependent.
Both green and blue. We can then discuss the color of the shirt as
being inconsistent, as information about which we simply do not see
eye to eye. I bring up these ideas from an esoteric branch of
representation theory to illustrate a fundamental point. VR is not
bounded by the assumptions of physical reality. We can have whatever
we can formally specify.
The HITL architecture specifies that every object in VR, including
space itself, have processing and memory resources. Entities are
objects with the capabilities of operating systems. Every entity is a
system, every entity is a variant of the same system. This means that
we can use the same editing, debugging, and interaction tools for
modifying each entity. Entities are running a sense-process-act loop;
in artificial intelligence terms, each entity is an agent, an actor.
This means that VR is inhabited with artificial life. Every entity is
capable of independent action, in response to environmental changes,
in response to internal memory or process changes, or in response to
changes in the rules, the disposition, specifying that entity's
internal processes. Each entity is an expert system using
pattern-matching on its input to trigger disposition rules and
metarules which generate outputs to the context. The environment
itself is just another entity, one that includes other entities within
it. All cyberspace is Toontown.
We have been able to demonstrate that mathematics itself (in
particular logic, integers, and sets) can be expressed concretely,
using 3D arrangements of physical things, such as blocks on a table,
doors open or shut, rockwalls that respond to gravity, the things of
everyday life. String-based symbolic representations of mathematical
concepts are typographically convenient, but tokens are not at all
essential to mathematical expression. VR makes it convenient to
express abstract ideas using spatial configurations of familiar
objects. One benefit of this approach is that we can build visual
programs, set them on a virtual table, and watch them work. We can
experience programs as other entities rather than as dumps of text.
Bugs would manifest as structural anomalies, as visual irregularities.
Architectural design has a sensual, experiential semantics. It is but
a quirk of typography that we have ignored the experiential semantics
of computational languages. More fundamentally, experiential
computing unites our spatial and our symbolic cognitive skills,
permitting mathematical visualization, analytic gestalt, whole brain
processing.
VII. EDUCATIONAL APPROACHES
VR provides an exciting educational medium for exploring worlds and
for exploring ourselves. It provides a training environment that is
rich, replicable, and responsive. It permits direct evaluation of
educational theory. The central educational issue for VR is one of
transfer of experience. Do skills and habits learned in VR transfer
to the physical world? Here are some educational issues:
Constructivism
"Human knowledge is essentially active." Piaget
Natural Semantics
non-symbolic, preoperational interaction
Programmable Participation
conducive and responsive environments
Cognitive Presence
modifiable self-concepts, learning by becoming
Social Reality
unique concurrent worlds
Educational psychologists have long known that people actively
construct their experience of reality. In VR, students will construct
their knowledge, then dwell within it, exploring their understanding.
Natural semantics means that the computational environment hides
symbolism in favor of displaying information in an innately
recognizable form. The two-year-old criterion: if a kid recognizes
it, its natural. The three Rs, all symbolic, will become the three
ACTs: enact, interact, and abstract.
VR provides the potential for completely customized, individualized
learning. Educational environments will uniquely respond to the
participant-learner, in terms of both needs and preferences. A
student model will not be necessary, instead the teacher and student
will modify the environment in support of student behavior.
We also have a tool for affective education, for sharing perspectives,
for mapping perspectives into broader contexts, for changing
self-image, for remapping capabilities.
Education is inherently social. Explicitly shared worlds and multiple
concurrent agreements provide the opportunity for groupwork, social
consensus, and the construction of functional, multiparticipant
environments.
In general, everything we do to educate with words and pictures can be
provided as virtual experience.
VIII. LESSONS LEARNED
Tom Furness, the Director of HITL, has over twenty years experience in
VR. He pioneered most of the hardware interface devices we use today,
in the extremely demanding environment of military aircraft.
Personally, I have worked on VR related projects for six years,
beginning at Atari Research Labs in 1984. Meredith Bricken designed
Autodesk's worlds, built Virtual Seattle for CHI'90, and has pioneered
research into the design of comfortable virtual environments. Over
the years we have learned some lessons:
Psychology is the Physics of VR.
Our body is our interface.
Knowledge is in experience.
Data is in the environment.
Scale and time are explorable dimensions.
One experience is worth a trillion bits.
Realism is not necessary.
A major theme of VR research is that Psychology, in the broad sense of
behavior, perception, cognition and intention, provides the rules and
the constraints of virtual worlds. Psychology is the Physics of VR.
This may come as a shock, it is one of those truths that is obvious
after it is said, but elusive before it is stated explicitly. Our
body is our interface. Interface is not something that is out there,
in some machine. Interface is a boundary which both connects and
separates, interface takes place at the surface of our skin. From the
perspective of VR, interface is physiology, interaction is natural
behavior. We simply want to use the power of computers to make
computation invisible.
Knowledge is in experience, it is not in some abstract, symbolic
representation. Data is in the environment, it is not stored away in
some memory array. These observations serve to remind us that we are
not the computer. To understand computation, we should participate
within it, rather than writing programs to dominate it. Humans have a
great skill for projecting outward, for becoming the tool we are
handling. We need reminding that we are creatures who dwell inside an
environment.
VR is inherently multidimensional. As well as freedom of translation
and rotation, in VR we can travel in scale and in time. Think of
scale as simply another direction; when we traverse scale, size
instead of location changes. We can also travel through time using
any of the techniques of film editing, including slow-motion, fast
forward, and temporal discontinuity.
There is a tremendous compression ratio between digital information
and human experience. Very approximately, it takes a hundred million
polygons to simulate what we see in one scene. Add duration,
multisensory channels, and interaction, and you get a lot of digital
information being transacted with each moment of consciousness.
Computation will not come close to this bandwidth for a long time.
Fortunately, virtual world experience does not require the information
density of physical reality.
Because our minds provide such tremendous flexibility in interpreting
what is outside of us, realism in VR is simply not necessary. Our
cognitive plasticity permits even simple cartoon worlds of 500
polygons to be experientially satisfying. We must design worlds that
respect our physiological needs. For example, we conceptualize
perspective in physical space as having six degrees of freedom, three
in translation and three in rotation. But our bodies have roughly
four and one-twelth degrees of freedom. We move easily in all
directions on a plane, forward and to the side, but not up, off the
surface we stand on. We rotate freely around the vertical by turning,
but our natural rotation forward, around our waist by leaning, is at
best 270 degrees (3/4 of a full 360 degree rotation). And our ability
to bend side to side is only about 120 degrees, one-third of a full
rotation. This adds up to a little more than four degrees of
physiological freedom. Input devices which permit complete freedom of
translation and rotation usually get people lost in space. The
dimensionality of our abstract perspective does not match that of our
physical construction. We must also differentiate that which is
innate from that which is learned. Pilots, for example, have learned
to fly in all six degrees of freedom. Realism is both physiological
necessity and cognitive interpretation. In VR, world design that
conforms to physiological necessity frees our minds to furnish the
rest of our reality.
IX. COMING ATTRACTIONS
Here are the coming attractions, what I believe will be available by
the end of the decade:
public domain VR software
massive database access
fabric of space
negotiable group space
conversational programming
artificial life
crossvalidation of realities
HITL is electing to distribute its software in the public domain. We
hope to create a context for the growth of an industry and for the
understanding of alternative realities. We hope to encourage the
evolution of a shared software and hardware environment which will
permit researchers to share progress and results. The commercial
marketplace can then improve on public work, selling value-added
features like customer support, prebuilt worlds, faster hardware,
better algorithms, realer time.
VR requires a new approach to database management. We want to access
massive databases such as Landsat as a function of our perspective,
our location in the database. We expect to see interactive databases
which we can explore through movement. Already waiting is the entire
Earth to one meter resolution, the location of every aircraft and
ship, large hunks of the Moon, the human body down to the resolution
of a cell, the flow of the economy, the network of computation. We
have digital worlds to explore.
I have mentioned that space is an entity. Many interactions between
entities can be expressed as internal processes of the spaces which
include them. Gravity is a primary example; we can implement
simplistic local gravity by decrementing the Z component of the
velocity vector of each entity in a space at each time tick. Rules
that apply uniformly to every entity in a space instead can be
ascribed once to the space itself. The inclusive space enacts local
gravity by owning the locations of the entities it includes. We want
to be able to place fields in space, to have space maintain its local
version of continuity, gradient, and metric, to build space-filling
logics which branch as a function of location.
One advantage of customized environments is that we will have to be
explicit about what is shared. VR suggests an approach to cooperative
work in a computational environment: rather than assume communality
and specify differences, assume complete difference and specify what
is common. It may turn out to be fun to build communal, consensual
contexts, to negotiate the group space.
One consequence of autonomous entities is that they can respond to our
communications. With voice recognition, we will be able to speak to
virtual entities as a means of programming their structure and
behavior. "I want the green cube I'm looking at to double in size."
The cube has a sensor for voice. Its rulebased disposition matches
the vocal input to its own identity and to its size changing function.
If you have permission, it changes itself to your specification.
Another consequence of autonomous entities is that they may have their
own agenda. The coupling of the behaviors of several entities could
determine events. Rulebases that support emergent behavior are
tremendously difficult to construct. We hope that the programmable
environment of VR will provide autonomous entities with a context for
the growth of interesting virtual life.
Fundamentally, VR forms a new reality, at least to the extent that we
are willing to relax our minds. We will need to calibrate the effects
of transfer across worlds and across realities. VR is the first
empirical tool of metaphysics, it permits us to compare realities, to
ask which alternative reality is preferable for which tasks.
X. RISKS
Do virtual worlds pose significant risks? I have prepared a list of
what I believe are the issues and problems for VR:
descriptive confusion
lack of experience
cognitive remodeling
fluid self
sensory overload, sensory ecstasy
power and control
cultural adaptability
VR is seeking definition, it could be anything from email to a fully
surrounding, multi-sensory environment. We are struggling with
appropriate comparisons. VR is not a drug and is not physically
addictive. Drugs change our perspective from inside the body, VR
changes our external environment. VR may well be psychologically
addictive (that is, entertaining), just like all good media
experiences can be. And there is that constant tension between
physical responsibility and cognitive exploration. Is VR escapist?
Escapism means seeking diversion from physical reality. VR cannot
escape being escapism, VR is perfect escapism. Is VR theater, or
interactive drama, or is it more than art? Is it scientific
visualization, or physical simulation, or is it more than science? Is
it financial modeling, or the perfect sales tool, or is it more than
economics? It's a good idea to spend some time figuring out what VR
is.
To me, the greatest problem is that we have virtually no experience in
VR. There are perhaps around ten thousand VR non-virgins. But I
estimate that there are no more than fifty people who have spent
twenty hours in VR. All of this excitement is purely conceptually, we
have very little experience with what we are talking about. The first
item on the VR agenda must be to construct and distribute hundreds of
systems, so that many people can contribute to our understanding. We
should know at least something about the cognitive effects of VR
before it is a consumer item with the distribution of Nintendo. When
a representative of MCC asked the lab the best way to invest two
million research dollars in VR, the answer was clear: give away forty
$50,000 systems.
The most complex, and potentially dangerous, risk is what we are
calling cognitive remodeling. Those who spend a lot of time in VR
bring back to physical reality some strange habits, like navigating
across a room by pointing, like bumping into walls cause they aren't
just images, like dreaming in polygons. VR effects dreaming strongly,
it seems to provide tools for control of the dreamlife from within the
dream. VR changes mental models. Now, it is not dangerous that this
is happening, cause all intense work produces similar effects. Anyone
who has programmed all night will know that the programming slips into
dreams. The problem is not that these things happen, it is that we
don't have the faintest clue what is going on. We do not know the
borders between virtual and actual. We have not yet had the
opportunity to evaluate current theories of reality crossing.
And how will we react when we are able to redefine our bodies, swap
our perspectives, mix our senses. We will have the ability to map
arbitrarily across sensory input, self-image, and behavioral output.
What will a fluid self be like? We will need to understand the
cognitive and behavioral effects of transportable perspectives, of
programmable bodies, of synesthetic sensations, of exchangable body
parts, of inhabiting arbitrary objects, of masslessness, of negotiable
communality, of complete empowerment.
Are there limits to the degree of warpage our senses can tolerate?
This is, of course, an empirical question. What are the functional
constraints of sensory modification for enhanced productivity, for
enhanced enjoyment? Are there sensory pathways to insanity or to
ecstasy? Just which side of the monitor do you stand on?
We have been discussing a domain which emphasizes personal freedom.
VR could be used for horrible purposes, but that negative assumes that
we are strapped to a chair. So long as each individual has the
freedom to reach up and turn off the experience, VR itself is quite
benign. But how will authority respond to this frontier? VR is
interactive, but will I have the right to remove the virtual arches in
my prebuilt reality given away with each hamburger? Are
advertisements from the creator necessarily non-interactive? Where
are the edges of property and ownership in a world which is digital?
Will there be commodities? What are the rights of autonomous
computational entities? Will there be stability? Will there be a
Virtual Environmental Protection Agency? I don't know, but I
certainly look forward to negotiating the communal rules of personal
responsibility in cyberspace.
The biggest issue is how our culture will respond to this new reality.
We have amassed hundreds of years of favoritism for the objective, the
scientific. Our values, ethics, and aesthetics are predisposed toward
Objectivism. Is VR a better place for transacting information? How
will physical reality react to competition? What will socialization
without material consequence be like? What kind of intimacy will
arise from explicitly penetrating world views? What kind of cultures
will arise when the VR network is standardized? Are we like Columbus,
discovering a completely new land in an unexpected place? Is living
in VR necessarily pathological? These are indeed exciting times.
XI. EVOLVING PHILOSOPHIES
We have come very close to talking about philosophy, so here are some
comments on philosophical concepts:
situated semantics
pervasion
immaterial realism
constructivism
boundary mathematics
more than reality
Situated activity is a growing school of thought in AI. The idea is
simple: what we do depends on our environment as well as our internal
state. We react and respond, constantly bringing external context
into our interpretation of the moment. Currently, symbolic logic is
split in half, between syntax (representation) and semantics
(meaning). Syntax is strictly formal, it has no basis in experience.
Semantics attempts to connect syntactic symbols to the reality of the
world by mapping representation onto meaning. The problem is that it
does so without regard to context external to the formal symbols.
Since environments necessarily introduce external unknowns, standard
semantics is just too literal. Situated activity is an attempt to
build a theory of context.
VR, in comparison, is totally situated. By defining natural behavior
as the rules of interaction, by displaying recognizable spatial
structures as output, by providing context in toto, and by including
the participant, VR redefines the relation between syntax and
semantics. Semantics, what we consider to be anchored to reality, is
displayed directly as (virtual) reality. Syntax, the symbols that
guide computational activity, is hidden in the background, out of
sight.
Environments include their participants, they pervade their contents.
Pervasion is a non-dualistic concept, more familiar to Buddhism than
to Christianity. A pervasive space is one which is diffused
throughout every part of itself, including those parts occupied by
other spaces and objects. Objects themselves are those boundaries of
spaces that we can sense. When we look at the container of an
environment, from the outside, we see that it surrounds a portion of
space. The important point to understand is that environments focus
our attention on a particular portion of space, they do not separate
space into two opposite parts. The outside space still pervades the
inside space. For VR, the physical pervades the virtual. When we
enter a virtual world, we always bring our physical body. VR is not a
separate reality, in a dualistic sense, it is a pervaded reality. The
shift from duality to pervasion is from networks to maps, from
separation to unity, from confrontation to cooperation, from male to
female, from one to zero.
The Copernican revolution introduced a physics that differed
fundamentally from appearance. VR introduces a metaphysics that
differs fundamentally from the material. At the foundation of
Objectivism is an attempt to be realistic about the material world.
VR calls for immaterial realism, for being realistic about
information. The currency of VR is organization, not possession, not
accumulation, not territory. All laws are transmutable, we can
satisfy fantasy rather than fact. It is science itself that is
redefined. In VR, we can choose to be reductionalist, but at the
bottom of it all, there is not Mass or Nature, there is the Void. VR
is representational, but not a priori rational, empirical, or
verifiable. VR is illogical positivism: if you can specify it, it is
meaningful. All empirical hypotheses are true.
Another fundamental philosophical position engendered by VR is that of
constructivism, that our minds and our bodies coparticipate in
defining reality. Objectivism places an overemphasis on the input of
the physical body. Solipsism overemphasizes the mind. Constructivism
recognizes that reality is like light, it is both particle and wave,
both objective and cognitive, both observation and participation. In
VR, we cannot escape the realization that we are the architects of our
environment.
We are applying boundary mathematics to VR in three different ways: as
a foundational mathematics, as a technique for logical deduction and
maintenance of inconsistency, and as a spatial embodiment of abstract
concepts.
Boundary mathematics is a calculus of inclusion. The essence of VR is
inclusion, the relationship between an environment and a participant.
The primitives of boundary mathematics are also participant and
environment. Let ( ) represent an environment, and let i represent a
participant. A variation of Spencer Brown's Laws of Form provides the
axiomatic basis:
Observe: i ( ) = ( )
Participate: ( i ) =
The left-hand-side of each equation is descriptive (objective),
explicitly mentioning the participant. The right-hand-side is
experiential (participatory), implicitly using the participant's
perspective. We read the left-hand-side from our traditional
externalized, objective perspective. The right-hand-side refers to
our experience, from the subjective perspective. When we observe an
empty environment, we perceive its boundary. When we are included in
an otherwise empty environment, we perceive emptiness. That's all
there is at the foundations of experience.
The most important thing to realize about VR is that it is more than
reality, more than a simulation of reality. You add physical realism
to a virtual world by adding constraints that reduce the possibilities
in that world. Native VR lets you walk through walls, we add
collision detection to disallow this power. Native VR has no gravity,
we add gravitational equations to simulate a gravitational reality.
Reality simulation is a subset of potential VR experiences. The least
elaborated virtual world is the Void.
We describe innovations in terms of what they replace. Only after
decades do we come to understand the pervasive impact of new
technologies on our culture. The automobile was first the horseless
carriage. It replaced the carriage, looked like a carriage, and moved
at the speed of a horse. Decades later, the automobile has
transformed our landscapes, the pace of our travels, and our concepts
of time and space. The television replaced the radio. Television
programs were first radio programs with pictures. Decades later, the
television has transformed our evenings, the pace of our senses, and
our concepts of news and entertainment.
The computer is first a symbol processor. Although decades have
barely passed, it is transforming our concepts of information and
calculation. Computers are thought to replace typewriters and
desktops and filing cabinets. But the computer has yet to be
understood for what it is of itself, we still view it from the
impoverished model of what it replaces. McLuhan said that computers
extend our central nervous system. Our CNS is not a symbol processor,
it is a generator of personal realities. VR marks the end of the
infancy of computation, the essence of the computer revolution is yet
to come. Essentially computers are reality generators.
And reality is in the eye of the participant.
Comments
Post a Comment