VRML Behaviour - a proposal

This should no longer be considered as a "proposal for behavior in VRML". This process has gone a slightly different way nowdays, and I have not had time to update this. However, what is still interesting here, is my ideas about multiuser spaces, about "engines" and "brains", a duality that is necessary for making a shared virtual universe real.....

I believe this message to be truly interesting and to actually contain useful insights. I apologize in advance for it's size, but I couldn't make it much smaller....

This icon is used to mark new stuff added, or otherwize changed since last revision. Click on it to move to the next new item.

Introduction

• Actually, I did the thinking while completing our harvest of Oats. But that is beside the point. Working with farming machinery is a great thinking opportunity....

Typography

"Levels" of Behaviour

Level 0

Directly modifying an object "set position to 10,10,10"

Level 1

Simple(?) deterministic behaviour: "move in this direction 30 centimeter per second"

Level 2

More complex behaviours "Search for food", "Find girlfriend", "Do her" e.t.c.

Level 3

Decitionmaking. Decide which Level 2 behaviour to do. "Should I find a girl, or go eat?"

Therefore I suggest the following division: The Level 0/1 behaviours are the "engines" (to borrow a term from Open Inventor). Level 2/3 behaviours are the "brain". In this text I will mostly talk about the distinction between "engine" and "brain".

Engines, DIS and "Dead reckoning"

An engine script doesn't respond to outside events. (The brain does.) It only responds to time, and it's initial parameters. The brain provides these initial parameters, and may modify them, or replace the engine script completely.

The engine script is called each frame by the renderer, to query the position of the objects in the scene. The script should avoid storing information from one invokation to the next. It should not rely on variables, other than the initial parameters. An engine must be written in such a way that it can be passed a time T, and the enginge should know what to do at that time.

However, an engine can be of any complexity. It can actually borrow a lot from Bernies level 2 behaviours! We could create an engine for "Run to the tree and climb up". Actually, the more complex the engines are, the less network traffic will be generated!

Time stamps

To: Object (squirrel#7): 
   Execute function "WalkTo"
   Start time: 1995-10-01 14:31:09.55
   Duration:   5 seconds
   Parameters:
      InitialPos = 30.0, 25.3, 11.5
      Goal       = 50.0, 30.0, 11.5

The representation would naturally not be text, but much more compact. E.g. the function name could be an enumerator giving that functions ordinal in the squirrel engine script.

Since the packet contains the initial timestamp, a small latency in the net is hardly noticable. As soon as the packet arrives at any host running the simulation, the squirrel will be in sync, because the engine is a function of time, and can easily be "jumped into". And when the end-time arrives, it will stop.

But what if the squirrel "brain" detects a wolf after two of those five seconds? No problem. A new packet is sent:

To: Object (squirrel#7): 
   Execute function "Flee"
   Start time: 1995-10-01 14:31:12.04
   Duration:   
   Parameters:
      InitialPos       = 42.0, 27.5, 11.5
      DirectionVector  = -12.0, -5.0, 0.0

The "WalkTo" and "Flee" scripts are in this case in one file. They could alse be two completely different files. It could also be one script with a different parameter (WhatToDo="walk" or "flee"). The important thing is that the brain supplies the correct parameters for a particular script. The brain must know the engines interface.

The forseable future

Here's my little blockbuster.... :-)

The "brain" could easily forsee the future of a simple deterministic behaviour. A ball dropping to the floor will bounce. A vase will shatter. Therefore the brain is allowed to post FUTURE messages with a timestamp of the FUTURE.

Allowing looks into the forseable future, can inprove remote responsivness greatly. The vase will shatter and the ball will bounce at the same instant for all viewers.

Of course the bounce or the shatter could have been built into a complex fall-and-bounce or fall-and-shatter "engine" behaviours. As said before, these can be arbitrarily complex. But this is just an example.

In a complex case, the brain could forsee that object A and object B will collide in three seconds, and post a forsight message about the resulting bounce.

This means that for each virtual object, there is a sort of "message queue", which can store these "future time-stamped" events, and sequentially execute them as their respective start-time arrives.

Even in the squirrel case, these future messages can be used without sacrificing realism. Lets say the brain detects the wolf. So the brain posts a packet saying "Start fleeing in 2 seconds". This gives us the following "features":

• The squirrel gets a pretty natural "reaction time", just like a real squirrel.
• Even if the wolf walks out-of-range immediately, the squirrel will still run away, also, just like a real squirrel.
• And most importantly, all squirrels in all simulations are completely in sync. There is no lag whatsoever.

If it does this, it posts another future packet, that has an earlier timestamp than the "Flee" packet, plus an override flag. This flag essentially means "when a packet of this type arrives, flush any events with later times from the queue.

The Brain

   Def Squirrel {
       WWWInline geometry "http://www.xyz/~zap/squirrel.vrml"
       WWWScript engine "http://www.xyz/~zap/squirreng.vrbl#SitNWait"
       WWWScript brain "http://www.xyz/~zap/squirrel.vrbl"
   }

The brain is a scripted brain: squirrel.vrbl. In this case it is loaded on the same spot as the engine. This automatically gives the brain the same scope as the engine. The brain could have been loaded on a higher level.

For language choices, I would suggest ATLAST (a Forth-like language). It has the advantages in simplicity, and being in the Public Domain.

For a special brain, something of the following:

   Def MyAvatar {
       WWWInline geometry  "http://www.xyz/~zap/me.vrml"
       WWWScript engine    "http://www.xyz/~zap/me.vrbl"
       WWWScript brainlink "http://www.xyz:9876/~zap/"
   }

Allowed modifications

The VRML specification specifically states that a browser isn't required to keep the scene graph model in memory. However, the key issue when connecting the engine to the geometry, is that the scene graph must be retained, but only for the modifyable objects.

Here is a proposal for how it might look:

   DEF Squirrel DYNAMIC Separator {
      # Load squirrel behaviour script

      WWWScript engine "http://www.lysator.liu.se/~zap/squirrel.vrbl"
      WWWScript brain "http://www.lysator.liu.se/~zap/squirrel.vrbl"

      DYNAMIC Transform "pos" {
         translation 0 0 0  # Position the squirrel
      }
      # Squirrel body
      Cube {
         width 10
         height 20
         depth 10
      }

      DYNAMIC Separator "rgt_thigh" {
         DYNAMIC Transform "pos" {
            translation . . .
            rotation . . .
         }
         # Squirrel's RIGHT thigh
         Cube {
            width 2
            height 2
            depth 15
         }
         DYNAMIC Separator "ankle" {
            DYNAMIC Transform "pos" {
               translation . . .
               rotation . . .
            }
            # Squirrel's ankle
            Cube {
               width 2
               height 2
               depth 15
            }
         }
      }
      DYNAMIC Separator "left_thigh" {
         DYNAMIC Transform "pos" {
            translation . . .
            rotation . . .
         }
         # Squirrel's LEFT thigh
         Cube {
            width 2
            height 2
            depth 15
         }
         DYNAMIC Separator "ankle" {
            DYNAMIC Transform "pos" {
               translation . . .
               rotation . . .
            }
            # Squirrel's ankle
            Cube {
               width 2
               height 2
               depth 15
            }
         }
      }
      DYNAMIC Separator "head" {
         DYNAMIC Transform "pos" {
            translation . . .
         }         
         DYNAMIC Material "blush" {
            diffusecolor 0.6 0.4 0.2
         } 
         DYNAMIC Coordinate3 "mesh" {
            point [-2 0 -2,  -2 0 2,  2 0 2,  . . . ]
         }
         IndexedFaceSet {
            coordIndex [ 0, 1, 2, . . . ]
         }
      }
   }

• DYNAMIC is a keyword telling the browser that this node will be dynamically modified. Which means that it must not "globalize" this object. It must be kept as-is, with it's transformation. Essentially, the scene-graph must still exist, but only for objects marked as DYNAMIC. Any object that isn't inside a DYNAMIC separator does not need to retain it's scene-graph structure. Transformations that are not declared DYNAMIC does not need to be retained as a scene-graph structure element.

• The syntax of DYNAMIC is:
  

    DYNAMIC nodename "refferable name" { ....
  

  Some nodes can be defined as DYNAMIC. As you do that, you give them a refferable name. This is used by the behaviour engine to modify the position of the object.

• The behaviour engine has a scope who's root is the same as the node for which it is loaded. In our example, the engine would reffer to:
  • "pos" - The position of the entire squirrel.
  • "left_thigh.pos" - The position of the left thigh.
  • "left_thigh.ankle.pos" - The position of the left ankle
  • "head.pos" - The position of the head
  • "head.mesh" - The verticies that make up the head.
  • "head.blush" - The material of the head.
  This scoping also forbids the squirrel engine to modify anything outside the scope of the squirrel itself. (If the squirrel wants to pick up a peanut, it's brain must do it by sending a pick up message to the peanut.)

  The scoping works across WWWInlines too. As long as the WWWInline is the child of the squirrel, the squirrel behaviour engine can resolve references into it.

  Also, if a separate behaviour engine was loaded for the "left_thigh", it would only need to reference "pos" (which resolves to the left thighs position) and "ankle.pos" (which resolves to the left ankles position).

• The following nodes can be declarable as DYNAMIC:
  • Separator
    Defines a named container.
  • MatrixTransform
    The engine may replace the tranformation matrix itself.
  • Transform
    Transform is really just an alternate way to specify a transformation matrix (and hence, the result of a transform node is often stored internally as a transformation matrix). However, for a DYNAMIC Transform node, the engine should be able to change the fields in the same way as they are written in the VRML file, i.e. change the 'translation' or the 'rotation'. It is the courtesy of the browser to once again calculate that into a transformation matrix.
  • Rotation, Translation, Scale
    Similar to Transform, the engine modifies these in their respective way. It is the browsers task to convert that to the appropriate matrix.
  • Switch
    Allows an engine to turn an object on or off, or to cycle among several representations of an object by setting the whichChild attribute. (Useful for poor mans animation, custom LOD, e.t.c.)
  • Coordinate3
    The engine may modify any of the verticies.
  • Material
    The engine may change material properties.
    Note that material changes inside a DYNAMIC separator only apply inside that separator, even if they initially "leaked" out of the separator and modified subsequent nodes in the initial load of the VRML file.
  • Texture
    Change texture parameters
  • Directional/Point/SpotLight
    Change light parameters
  • Cameras
    Change camera parameters
  • (what else?)
    

Reusability of behaviour

If, for instance, we set attributes defining the leg-length and allowed turning-angles for joints in the squirrel, we could use a standard "walk" model for walking without having to write our own squirrel-walk. We could just pick one off the shelf, adopt is't naming scheme, set up the attributes to make the "walk" behaviour behave, and off it goes.

By this method, lots of people will start writing useful behaviours. With the correct division between attributes (defined in the model) and their use (in the engines), reusability is ensured.

What about ZOI's and such?

Today most worlds are small, because of the rendering power of the contemporary machines. A single .vrml file wouldn't fit much more than five people before the RENDERING is bogged down anyway, so I suggest:

For now, we igonre this issue. (Using the discussion below, doing so now is sort of "safe"). Why? Well, if the current shape of vrml worlds will set the standard, we will not be building "huge, seamless worlds". We are building little islands-with-links. Therefore, IMHO, the ZOI == the top level vrml world.

Of course I would love it if we could add WWWAnchors for user position and direction-of-motion. That way we could walk-thru-doors and flu-thru-windows instead of this unintuive "click on the door to go elswhere" of today.

Network Protocol

Minimizing info

Very Complex "Engines" reduce net traffic: If the "walking" of a person is accomplished by a spline interpolated table of samples of real walking people's legjoint rotation patterns, and the "walking" behaviour includes such complex "parameters" as "walk to this spot following a spline with these control points at these points in time", no net traffic needs to happen during the entire, highly realistic walk! Compare that to transferring the angle of each joint each frame.......

Each object created in the world would receive it's own instance identifier (e.g. IP adress of creator + index no). This is used througout to reference this object.

Do we need a server?

1. Art's computer sends a "what's happni'n dude?" message to Charlies computer.
2. Charlies machine notes that a new user has appeared. It adds Art's IP adress to a list (which up till now only contained himself)
3. Charlies machine answers by revealing:
   • The world name
   • Any objects loaded (that are not in the world description)
   • The currently running engines and their parameters for all objects
   • A list of the brains already running
   • A list of IP adresses where to send messages. (In this case, only the address of Charlies and Art's machines)
4. Art's machine loads all this, sets up all engines.
5. While this is happening, Charlie is moving some furniture. He sends messages to all IP's in the list except himself.
6. Art's machine receives the updates, changes the engines accordingly. When everything is loaded (maybe even while it is loading), Art can see the furniture moving.
7. Art's machine sends a "Create Object" package. The package contains:
   • The URL to the .vrml file of Art's avatar
   • The URL to the .vrbl file with Art's "engines"
   • The URL to art's "brain" (= his mouse/joystick/powerglove)
8. Charlie's machine receives this, and updates his picture accordingly
9. Art goes to the table who Charlie just moved. The brain for this object is now running in Charlies machine. Art grabs the table. By doing this, the brain transfers itself to Art's machine, to improve responsiveness. Art shakes the table violently. He grins to himself because of the nice rapid responses.
10. Charlie looks puzzled and ponders the waving table.
11. Then suddenly Jamie connects to Art's machine. He sends the "what's happni'n dude's" message.
12. Art's machine informs Jamie's about the world, the objects, their state. He also sends out a new list, now of three IP adresses, where all info should go.
13. Charlie has had enough, he logs off. That generates a delete-message for his avatar, and his IP adress is deleted from the list.
14. .... e.t.c.

With a bit of though, you can see that this approach scales nicely too. Assume there is a server. This is the same scenario again, now WITH server:

1. Charlie loads the VRML file (He is the first to do so). The file contains a line like:

     WWWServer {
        name = "vrtp://vr.wired.com:9876/this_world"
     } 

   This sever is contacted by the browser. The browser sends the "what's up dude?" message to the server. The server sends:
   • The world name
   • A list of IP adresses where to send messages. (In this case, only the address of the server and Charlie)
2. The world appears on Charlies screen. He grabs the table.
3. The table brain decides to move the Charlies machine, to improve his performance. The server gives the brain to Charlie, and charlie moves around the furniture a little.
4. Now Art connects to the server. Art gets:
   • The world name
   • Any objects loaded (Charlies avatar!)
   • The currently running engines and their parameters for all objects (some moving furniture)
   • A list of the brains already running (the table....)
   • A list of IP adresses where to send messages. (Charlie, Art, the server)
5. ...e.t.c.

But lets assume 2093 people log in. The server may decide to move to some broadcasting mode. It sends out a list of IP adresses to send to: Only the servers IP itself.

So when Art logs in, he is never informed about Charlies IP adress (or any other of the 2093 people there). He never directly sends packets to Art. He sends them to everybody-but-himself on the list (which is, only the server).

The server then starts to broadcast the stuff instead......

Similarly, the server could pass a multicast adress as the one to be used...

To summarize:

In my view, the server doesn't do much more than maintain the logins/outs. In a future, scalable scenario, the server would be responsible for ZOI testing, and seeing WHAT to send to WHOM, instead of sending EVERYTHING to EVERYBODY withing the same .vrml file. (But I also think, that the vrml file itself can BE the ZOI for the forseable future, due to rendering limitations.) The server also decides when and if to switch to multicasting.

The different interfaces

Here is an overview of the interfaces:

• ETGI - Engine To Geometry Interface
  • I call this Interface instead of protocol, because this never has to pass a network boundary.
  • This is the only interface the people advocating for the "API" approach wants to define. But the API approach only moves the problem, and does not solve it. (But it keeps VRML itself leaner, which is a valid argument.)
  • This is where we need to define what modifications an engine can do on geometry. My suggestions are listed above.
  • Engines can query the geometrys current matrix, e.t.c. per above
  • Engines can query the geometrys "attributes".

• BTEP - Brain To Engine Protocol
  • This is a Protocol because this has to pass a network boundary.
  • It is the universal way in which the brain informs the engine of what to do within a certain timespan.
  • The protocol is basically a function call, stored in a network readable format. There are many ways this can be done very simple. It needs to include:
    1. The timestamp, in some universal time format.
    2. The duration of the event, or zero for infinite.
    3. The function name (or an ordinal number, for compactness)
    4. An option to contain the full URL of a different engine script to replace the old one.
    5. The override flag.
    6. The number of parameters passed.
    7. The parameters for the function, in order. Any left out parameters assume default values. So the "most commonly used" parameters go first in a function call, then less and less common one follows.
  • These will probably be sent as UDP packets. There is no return value.
  Packets sent to the engine that runs on the same host as the brain are not sent over the network. They are directly put in the message queue of the local engine.

  The engine should also support a few standard return functions. These are called directly by the brain to the local copy of the engine. These should include things like:

  WhereAreYou

  Query the objects current position. (Returns the transformation matrix).

  GetBS

  Return the bounding sphere in world coordinates (very approximate)

  GetBB

  Return the bounding box in world coordinates (very approximate)

  Status

  Returns status info about the engine.

  These functions are standardized because they are available for any brain. E.g. when the Wolf wants to know where the squirrel is, the wolfs brain can query the local copy of the squirrel directly.

  Naturally it could ask the squirrels brain, which would give a more accurate position of the squirrel, but that information would be subject to lag. Asking the local squirrel might be incorrect (because of a pending update packet subject to lag) but doesn't have any lag at all. It is up to the application to decide which method to use for position testing. For simple proximity testing, asking the local copy of an object is encouraged, since it saves bandwidth.

  
  

• BTBP - Brain To Brain Protocol
  • This is also a Protocol because this has to pass a network boundary.
  • This is the way brains communicate, send and recieve messages, and soforth. If is essentially a trans-network function call.
  • The protocol is very similar to the BTEP. Both formats would be founded on a common base. The call must include:
    1. The function name
    2. Parameter count
    3. Parameter values
  • A return packet with the return value is expected. When a brain calls another brain, the calling brain's thread is held in wait for the response from the called brain. (We probably need a timeout here.)

• Other - Special interfaces of application specific nature
  • These are special application specific interfaces between a "special" brain and any input devices.

Summary of Document

• Two types of behaviour: "Engine" and "Brain". "Engine" runs everywhere, "Brain" runs at one spot.
• "Engine" and "Brain" loaded separatly for an object
• "Brain" posts timestamped invokations of "Engines"
• Invokations in the FUTURE is allowed
• Two types of "Brain"
• Scripted brains can move around at will.
• Engines modify object and children
• We can ignore ZOI's for now (it's a server issue anyway)
• Server only needs to handle logins/logouts