||Java Access ||Language ||Work Areas & Persistence ||Nested Databases ||Configuration Options ||Functions ||Scripts ||Local Variables ||Method Invocation ||Compiled Lazy and Variable Number of Arguments Functions ||Constructors ||Scoping ||Type Checking ||Lazy Evaluation ||Overloading Functions ||Dynamically Compiled Classes ||User Level Classes ||Casting ||Dynamic Class Location ||Loading Classes ||Array Initialization ||Subsettting ||Utilities ||Dynamic CORBA ||Compilation ||Signal Handlers ||Prompt Evaluation ||Multiple Evaluator Classes, Instances and Threads ||Others ||Lexers and Parsers ||Substitute ||Comments ||Language Changes ||Input Continuation Separator ||Internal Utility Functions ||
||Missing/Future Features ||

This short document outlines the features currently in this interactive/dynamic Java language.

Features

Firstly, this a dynamic Java in the sense that one can type commands at a prompt rather than iterating the write-compile-run cycle. Since exceptions are caught by the interpreter (and passed to the user), there are few fatal errors.

Java Access

The interpreter relies entirely on the Java reflectance mechanism. This allows an application to interrogate a Java Class object and determine its list of methods, fields and constructors. From this information, we can instantiate objects of any public class and then invoke methods and get and set values of fields in the instance or class. This interpreted (i.e. non-compiled) method invocation provides access from the command line to ``almost'' all the existing Java classes, be they those in the Java core classes, the Omega system classes, or arbitrary classes available on disk or the web. (See class loaders and Dynamic Class Location.)

Where fields or methods are not public but necessary, we have provided proxy methods in the evaluator which act as Internals. There is now support in the method calling mechanism to invoke public methods in protected classes. This allows us to access objects such as UNIXProcess which are not directly accessible in the earlier reflectance setup. This is the preferred way to provide access to these methods unless additionally functionality is to be provided.


Language

There is a extensive language that is very similar to Java and provides all the same features as Java and a few more. Looping, conditionals, method invocation, field access, etc. are all available. Definition of classes, methods. Notably, exceptions and exception handling are also available at the interactive/interpreted level as are functions and system-level methods. Subsetting, and general operator overloading is also available.

Object Databases & Persistence

Workspaces or databases are similar to the S/R model (and numerous others). One assigns objects to a database via a name. The simple assignment operator (=) invokes the assign() method of the current Evaluator and this communicates with its SearchPath to determine the appropriate database to which the object is assigned. At that point the characteristics of the selected Database apply. The other database operators in the evaluator include objects(), remove() (no rm) exists(). One can manipulate objects directly in a database rather than via the default assignment behavior of the evaluator using the same methods but on the target database. It does not have to be in the search path.

The searchPath is an ordered list/table of database objects maintained by an evaluator. Each can be shared across multiple evaluator instances. Similarly, elements of one search path can be elements of another search path.

There are several different types of databases, all of which can be used in the basic evaluation interchangeably. What differentiates them is the level of support they provide for 1) persistence, 2) program support.

Persistence is provided essentially by the Java serialization with a little bit of bookkeeping. There are two models for persistence: 1) the S-style of individual objects stored in separate files, and 2) collections of objects serialized as a single unit (more similar to the R style). These different models are implemented by related but different classes - PersistentObjectDatabase and ObjectDatabase. Assignments in the first model are immediately committed to disk. When the databases is first attached, its contents are filtered and "matching" filenames are taken as objects and the database (partially) populated.

 
In the second model, you must explicitly serialize the database. (This can be done on session completion.) The Evaluator's serialize() methods can be used for this.

To serialize an object, one can use the family of serialize() methods in the evaluator. Basically, specify the object and the destination (i.e. java.io.File or String filename) So, the following works

 ? serialize(1::10, "/tmp/foo")
 ? in = new ObjectInputStream(new FileInputStream("/tmp/foo"))
 ? in.readObject()
So for the default database, you can do
? serialize(defaultDatabase(), "/tmp/foo")
To retrieve it, there is a special convenience method
 ? attach(new File("/tmp/foo"))
or
 ? attach(new File("/tmp/foo"), 0)
to specify the position of the attachment.

The biggest difference between the two models is what is stored. In the first scenario, if the object being serialized contains references to other objects, their contents will be serialized in the serialized version. When it is restored, these associated objects will not be restored into the database. This means that dynamic links will not be conserved and more disk space will be used.
The second form of persistence does preserve links and takes minimal copies. This is provided for free by Java. For example,

 ? x = y = 1::10
 ? x[0] = 100
 ? y[0]
 100
 ? serialize(defaultDatabase(), "temp.ser")
 ? exit()
 % omega
 ? attach(new File("temp.ser"))
 ? x[0]
 100
 ? x[0] = 20
 ? y[0]
 20        // still reference to the same object as x.
This example also illustrates the convenience methods for serializing the database and attaching it.

Either persistence model can use compression. The GZIP output streams can be applied at an object level. A simple flag controls this in the PersistentObjectDatabase. For serializing the entire object, simply construct the ObjectOutputStream with a GZIPOutputStream and the input stream in a corresponding manner.

Programming support is provided by several different database classes. All databases can be attached as read-write or read-only. This status applies to the collection indicating whether new elements can be added or existing values removed from the collection. The class ReadWriteDatabase allows individual elements to be specified as read-write or read-only. This allows one to allow new elements be written to a database while ensuring that certain others are not removed or overwritten. (This is how final variables are implemented.)

The default class of database can be specified in the OmegaOptions file as defaultDatabaseClass

Databases are used extensively in the evaluation of statement blocks. The LazyFunctionDatabase is for use in evaluating functions with lazy arguments (as the name suggests) and hides the details of delayed evaluation of specific arguments. These evaluation frames are temporarily attached to the Evaluator's searchPath.

The different database classes implement the Database interface. One can substitute entirely different implementations of this interface into the evaluator's search path.

Databases also support listeners. Other objects can register to be notified of assignment and removal events. Additionally, they can receive notification of attachments and detachments.


Database Nesting

Databases now support the a.b style syntax for referencing the variable b in a Database a. This allows access to variables in nested Databases to be accessed easily For example,
[13] a = newDatabase("Foo")
{}
[14] a.b = 1
1
[15] a
{b=1}
[16] a.c = 2
2
[17] a.d = newDatabase("bar")
{}
[18] a.d.a = 1
1
[19] a
{c=2, b=1, d={a=1}}
[20] a.d.a
1
The primary motivation for this nesting is used in conjunction with CORBA NamingContexts to allow CORBA objects appear as local Omega variables. Specifically, the Omega NamingServiceDatabase class is a proxy for a NamingContext and can be attached to the SearchPath as a regular Database. Objects within a sub-NamingContext can then be accessed using the . (or field-access) notation This is explained more thoroughly in the CORBA documentation.

This is an example of the more general Omega DynamicFieldAccessInt interface which allows an object to reflect fields dynamically rather than supporting real Java Fields.


Options

The environment of the process and for each evaluator can be customized throug an Options object which is configured by the user (or directly by the programmer) via a simple ASCII file. See OmegaOptions in the Omega home directory. This allows one for example to control what classes are to be debugged, the default class of database to be used in the evaluator, the prompt string(s), etc. There is a simple mechanism to add new options/properties that control arbitrary aspects of the evaluator.

Initialization Script

The initialization script, $OMEGA_HOME/Scripts/Run/OmegaInit by default, is run when the application is run. A different file can be specified using the -s flag of the omega shell script. (Currently this is just for the console app, but we need to consolidate and abstract the notion of command line arguments, and the entire process into a class that can be shared by different applications. This fits with the notion of an Evaluator manager.) Also, the q() method checks the value of a property (OmegaFinalScript) for a file of Omega commands to run when the application is terminated.

Local Variables

It is relatively common that we may have a variable x somewhere in the search path and yet want to use it locally e.g within the body of loop. While we can declare it by providing a type such as
  for(i=0;i <k;i++) {
    Date x = 1
       ..
  }
this assigns a particular type to the variable or is irrelevant.

In order to handle this, we can use the "reserved word" "local" to identify that a variable should be created locally. This is much like the my in Perl. The above statement would look like

  for(i=0;i<k;i++) {
    local x = 1;
       ..
  }
but does not assign any type to the variable x.

The choice of the word local is not fixed as one can change the value of the field LocalDeclarator in the class Omega.Parser.Parse.LocalVariable


Constructors

A simplification of the code now allows two types of constructors.
 x = new StringBuffer()
 y = x.getClass()
 z = new y()
In other words, we can use an explicit class name or we can use a Class object stored in an Omega variable.

The evaluation model may make this understandable. We treat both as Name expressions. Evaluating a Name resolves an omega variable or a class name. The order of the search may lead to surprising results. Suppose, we have a variable named String. Then, new String() may fail because it will use the value of the variable String rather than treating it as a name of a class.
We may change this in the future to iron out this precedence. In the example above, to obtain an object of class String one can use x = new lang.String()


Scoping Rules

The scoping is such that there is a single search path that operates as a stack. The databases, frames for local blocks, functions, etc. are added to the ordered list. Objects are located by traversing this list and terminating when a the appropriate object is found. The searchPath() method displays the current list; find() allows one to locate an object, etc.

Optional Type Checking

In general, variables need not be declared. Statements such as
x = new File("foo");
automatically create an object named "x" in the appropriate database or overwrite the existing value of a previously created object named x. See also Scoping above.

The exception to the optional type declaration arises in cases such as

 static Object x = 2;
The grammar currently requires a type (default should be Object) if a modifier (e.g static, synchronized, static, public, etc.) is specified. Otherwise, a syntax error is reported. This needs to be changed.

In the near future, I will add the abilitity to optionally and dynamically require declaration of variables. We can do this without type checking using the local keyword introduced in omega. The dynamic aspect allows the feature to be applied to certain blocks, or expression trees, when debugging errors that may be caused by misspelling variable names, etc.


Lazy & Eager Evaluation

Lazy evaluation allows arguments not to be evaluated at the point a function is called but when that argument is used in the evaluation of a function. In some cases, it can be quite confusing and it is not a feature of the Java language. However, occassionally it is useful (e.g. a switch function) and there is support for this in Omega. Consistent with the spirit of other features in Omega, lazy evaluation is optional and implemented by instantiating classes that provide extensions to the basic evaluation mechanisms. By default, lazy evaluation is not used and function providers should seriously consider whether using it is appropriate. It adds complexity to understanding, automatic compilation, incompatability with evaluation of Java methods. (Additionally, if we attached the calling frame as it was being constructed, evaluation of successive arguments could use previous arguments as in the setq* of Lisp. Since this is a common form of lazy evaluation specification, this might suffice.)

When defining a function, a parameter name can be qualified not only by a type but also by the keyword lazy. This ensures that the value of this argument provided in a function call will not be evaluated until it is needed. For example,

function foo(x, y= x.size()) {
     ..
    show(y);
}
will defer the evaluation of y until the last statement is evaluated.

In some cases, (e.g. the switch function again), it is more convenient to declare all the arguments of a function as lazy, rather than having to qualify each one individually with lazy. This can be done by qualifying the function definition with the keyword lazy as in

 lazy function foo(x, y=x.size()) {
   ...
 }
The lazy keyword must preceed the optional return type.

Given the ability to specify the ``mode'' of an individual argument and a default mode as lazy, we have also introduced the keyword eager which is used in exactly the same places as lazy but has the opposite effect - namely, the argument is evaluated at the time of the function call. For example, in the following, all the arguments will be evaluated only when they are accessed except the first one.

 lazy function Switch(eager condition, ...) {
   ..
 }

We can also introduce a deferred which might indicate to evaluate at the time of the function call but not at the time of the function definition. The Quote() method can be used for this.

For an alternative to interpreted functions, see Utility Functions

Compiled Lazy and Variable Number of Arguments Functions

Using objects and classes to provide compiled Java "functions" at the top-level is definitely useful. However, since they are Java methods, they do not support lazy arguments or variable argument number. With some restrictions, we can inelegantly provide support for these facilities. This is intended to be useful and practical rather than a picturesque feature of the language.

Classses that are to be used as entries in the function table list, either directly or indirectly via instances, can implement either or both of the interfaces LazyFunctionTable and VarArgsFunctionTable The first of these indicates that the arguments should not be evaluated, but passed directly to Java method as a list. Such methods must be declared

 public return type foo(org.omegahat.Environment.Parser.Parse.List args,
                               org.omegahat.Environment.Interpreter.Evaluator evaluator)
All the arguments in the interpreted call are passed unevaluated. The method itself is responsible for evaluating the arguments as necessary. The second argument provides the evaluator in which the evaluation should be performed. Methods that assist in the evaluation of these arguments (e.g. getting the frame correct, handling named argument assignments, matching names, etc.) can be provided in a base class that a class implementing either of these interfaces can extend. Many of the methods are already available in different forms from the expression classes in the Environment.Parser.Parse sub-package.

The second interface, VarArgsFunctionTable implies that the arguments should be evaluated (unless the object or class implements the LazyFunctionTable interface as well as this one). However, they are passed as a single List

For some examples of classes, see

This setup changes the search order for symbols. Now we search the function tables before the regular databases for functions.

We do not necessarily (or currently) return the reflect.Method associated with a method in a lazy or variable argument object.


Function Overloading

In Java, everything is an object and routines or functions are addressed as static methods in classes. In other words, there is no global space. However, in Omega, functions live on databases and can be invoked directly in much the same way as S, R, Xlisp, Matlab, etc. It is often convenient to overload these functions by defining functions with the same name but different argument types and number of arguments. These are methods and are differentiated from functions when on declares them by replacing the keyword function with method. For example, the two commands
 > function foo(x,y) {...}
 > method foo(Vector x, Vector y) {}
has the effect of merging the two function definitions into a collection of methods referred to by the "function" name foo. A call such as
  foo(a,b)
will search the collection of methods for the "most appropriate" match and then invoke that matching function.

Handling ... and lazy evaluation is not very well supported at present to say the least!


Compiled Classes

See Compilation Generic compilation of scripts, functions, classes, etc. coming later.

User Level Classes

While most of the development of Omega data structures/classes is done directly in Java, the ability to define new classes interactively and modify them easily (without recompiling, reloading, etc.) is useful. Thus, there is support for user-level interpreted classes in Omega. These are currently similar to their Java counterparts (for simplicity of a single model and ease of compilation of these classes into Java code) or indeed many object models in other languages.

There are now two implementations of these classes. See Interpreted Classes. The newest mechanism generates compiled Java classes to represent the user-level classes and implements the declared methods by calling the associated method. This neccesitated facilities for identifying unqualified method calls by looking in the instance itself rather than in the global list and

The entirely interpreted version does support multiple inheritance. Additionally, linkage between the parent classes and fields is supported so that changes to base classes are propagated to the children, by default, but with events which allow the right of veto.

Classes are represented by templates or prototypes which absorb the fields and methods from the parent classes. Instances transfers copies of these fields on instantiation (allowing changes to the prototypes between instantiation) for regular fields and references for the static fields shared across instances of the class. Similarly, method tables (named collections of Method objects) are shared across instances.

A class can be defined by directly instantiating `free' AbstractUserClass or UserClass objects and adding fields to it. In this case, they are not visible to the evaluator and cannot be used for method and field dispatching. To introduce the class to the system, one can use the evaluator methods defineClass(AbstractUserClass) and defineClass(String name, Vector superclasses) and add fields with the return value. This has the effect of putting the AbstractUserClass into a table which manages packages of classes in a multi-way hierarchy. These tables are evaluator-specific but inherited from the parent evaluator when available.

As with most of the larger features added to the interpreter/evaluation model, support for User Level classes is optional and does not impact the performance of the basic evaluator which does not support it. This is done basically by providing classes that extend field and method access and instantiating those in an extended parser that overrides the factory methods for creating such objects.


Casting

Casting is useful/necessary in a few circumstances.
  1. modifying the method selection in a method lookup
  2. method dispatching for methods in ancestor classes
  3. accessing shadowed variables in ancestor classes
Unfortunately, the second of these is hard to implement using simple Java commands, even reflectance. The invoke() method of the reflect.Method class insists on invoking the method dynamically located within the object, not the class of the method object. Hence, we cannot invoke methods in the base class overridden by a derived class in an instance of the derived class.

In Java 1.1, we can get around this limitation of the JVM and implement casting of types for arbitrary method invocation. Consider the class A

public class A {
  public String foo()  {
   System.err.println("In A");
    return "A";
  }

  public int val() {
   return(1);
  }

  public boolean bar() {
   return(true);
  }
}
and the derived class B
public class B extends A {
  public String foo() {
    System.err.println("In B");
    return "B";
  }

  public int val() {
   return(-1);
  }

  public boolean bar() {
    return(false);
  }
}
Then, creating an instance of the class B, we can invoke each of its methods.
[] b = new B()
[] b.foo()
In B
B
[] ((A)b).foo()
In A
A
[] b.val()
-1
[] ((A)b).val()
1

Similarly, if we have a class C that extends B and doesn't override all of the methods will behave as expected.

The change to the semantics of the op code invokespecial does not allow us to do this in Java2 (at the moment!!!).


Casting Arguments

Consider the first case. Suppose we have an overloaded method A in a class called Cast with two definitions as follows.
  public void A(Hashtable tb) {
    System.err.println("In hashtable");
  }

  public void A(Omega.Utils.OrderedTable tb) {
    System.err.println("In ordered table");
  }
Then, the following illustrates the nature of casting the argument types to modify the method selection for a call.
>  t = new OrderedTable()  // extends Hashtable
{}
>  x = new Cast()
Cast@1dce4700
>  x.A(t)
In ordered table
null
>  x.A((Hashtable)t)
In hashtable
null
>
When a cast expression is evaluated, the result is checked against the class to which it is being typecast. This is done with the isAssignableFrom in the Class core Java class.

Casting for Element Access

Field Access
Here consider the example that CastA extends Cast and that both have a public field named foo of type String and assigned values Cast and CastA respectively.
 x = CastA();
 x.foo   // CastA
  CastA
 ((Cast)x).foo //
  Cast
This handles different types for shadowed variables. For example, if we introduce a class CastC that has a field int foo = 13; all behaves "as expected".

Casting also works in the very special case of a method call invoking getClass() and the qualifier being a CastExpression. In this case, the return value is the class to which the object has been cast.

 
 > x = new CastC();
 > ((Cast)x).getClass()
  class Cast
 > x.getClass().getField("foo")
  public int CastC.foo
 > ((Cast)x).getClass().getField("foo")
  public java.lang.String Cast.foo
Methods
This currently does not work and is unlikely to without a major change to the JVM. For methods, dynamic method lookup is done within the invoke() method of the Method class. Thus, the wrong method is actually invoked and the result is the same as ignoring the cast. For example, suppose CastA extends Cast and each has its own method named A that prints out the class in which it is defined (statically, not based on this.getClass())
 x = new CastA();
 ((Cast)x).A()
  CastA
We would like this to be Cast.

Note also that a call such as

x.super.foo()
does not parse in the current grammar. So instead, one must type
((SuperClass)x).foo()
. However, this requires that one know the super class of the object x. Of course, it is important that one does know this, but it is still inconvenient to type. Accordingly, one can now use the idiom
   ((super)x).f
So super is now recognized in a cast expression as a special word and implies the superclass of the object to which the expression being cast evaluates.

Class Name Completion

When one refers to a class, we look through the classpath elements and attempt to find a match. For example, new Vector() will create an instance of the class java.lang.Vector (assuming there is no other match). This essentially replaces the need for import statements with a more general mechanism. One difficulty is that it is harder to temporarily specify a different order for searching. This can be done by permuting the elements in the evaluator's classpath.

Dynamic Class Loading

The class loading in Omega has recently been extended to support In the first version of Omega, all classes that were to be used either directly by the Omega system classes, or by the user via the interpreter were loaded from the regular System classloader. This necessitated that the appropriate directories & jar files be in the classpath before the Omega process was started.

Now, system level classes (i.e. those used by the classes in the Omega distribution) must be in the classpath but classes referenced only by the user are loaded by a separate class loader. This is done by adding new directories or jar/zip files to the object returned by the method localClasses() as follows:

  localClasses().add(new File("/home/duncan/Java/extensionDirectory"));
  localClasses().add(new File("http://www.omegahat.org/extensions.jar"));
  localClasses().add("/home/duncan/Java/extensionDirectory");
  localClasses().add(new URL("http://www.omegahat.org/extensions.jar"));

  u = new URL("http://tigger/Java-SSH.zip");
  localClasses().add(new LocalURLClassList(u,new DeferredEagerURLClassLoader(u)))

  l = localClasses().add("http://www.omegahat.org/Java-SSH.zip")
From this point, the usual partially qualified class name will function by looking in 1) the system classes, 2) the user classes and 3) the dynamic classes. Exactly the same mechanism is used, except that the classes loaded implicitly by the user rather than the system are maintained by one or more different class loaders. This makes them invisible to the Omega classes!

URLs in this context are slightly different from regular files in that they are expected to be Zip or Jar files, rather than directories. When the URL is such a container, the ClassList mechanism reads the list of contained classes. The classes are now identified and can be referenced using partial qualification. There are three different URL-based class loaders which resolve the class references.

  1. The first (URLClassLoader) is entirely lazy and loads the bytes for a class when it is referenced. This incurs the cost of reestablishing the URL connection for each class and traversing the list of ZipEntrys until the appropriate one is found.
  2. A second version (EagerURLClassLoader) allows a single connection to be established and all the classes contained in the URL to be loaded immediately. This trades the speed of establishing the connection with the possibility that most of the classes will never be used and would not have been loaded.
  3. A third version (DeferredEagerURLClassLoader) changes the tradeoff by loading the bytes for each class but not actually defining the class. When the connection is first established to read the list of classes, the bytes are also retrieved and cached. When the class is referenced, the bytes are taken from this local table and converted to a class definition.
    The tradeoff here is again loading bytes that may never be used and also storing these. This class can be used to store all the byte arrays and then immediately after initialization the extraneous classes can be removed from the local byte table.

Dynamic classes are created and loaded by writing byte-code (via the Jas packages) and a special class loader. This allows the classes to be written to disk for cached use in future sessions or to be written into a buffer directly loaded. The latter means that the file does not have to be written into one of the elements in the classpath (local or system).

The following illustrates how to create a new class, named FunctionActionListener and

> dynamicClassLoader().defineClass("java.awt.event.ActionListener", "FunctionActionListener")
class FunctionActionListener
> v = new FunctionActionListener(function(x) {x++;})
> btn.addActionListener(v);

The different class loaders and locators are fields in the basic evaluator. These are shared across parent and children when new evaluators are created from the parent evaluator. They can of course be immediately overwritten or different constructors can be invoked to alter this behavior.


Array Initialization

JDK1.1 allows inline initialization of arrays such as x = new int[]{1,2,3};. The elements in the initializer must be constants however. We relax this due to our dynamic evaluation model. One can have arbitrary objects here. So x = new Integer[]{1,2,x,y,z}; works. This will be true even in the compiled code. There we can substitute a call to get("x") for an database object named x or a subsequent call to x[i] = x; where x is a field in a class

Subsetting

There is now support for some simple subsetting via an extensible mechanism for newly created classes. The parser recognizes simple array-like subscripting with the [ operator and now the [[ operator. In general, the former returns an object of the same class as the source and the latter returns discards this class if a single element is returned. These work differently from the S/R model in that the dimensions are in different [ rather than comma-separated. Both operators support the
  1. multi-index single dimension/subscripting
  2. single index, multi-dimension
  3. a hybrid of both
Thus to get the first element of an object contained in the second element of an object x, we use the expression
x[1][0]
Note that we are using 0 based counting. There is also support for assignments of the form
x[1][0] = 1
At present, the assignments do not support multi-index.

The mechanism works for some of the core Java classes (as special cases). These are currently Vector, Hashtable Enumeration. (Assignments are not done on Enumeration.) The mechanism is extensible in that new classes can be created that provide methods implement the subsetting and the assignment to subsets as is appropriate for the class. To offer this, a class implements one or either of the interfaces Subsettable and AssignableSubset (These methods here are selected/invoked in preference to the built-in methods for Hashtables, Vector, etc.)

The subsetting mechanism is implemented in the methods get and assign in ArrayAccess in the Parser.Parse package.

Examples of classes that implement these two subsetting interfaces are OrderedTable and DataList This mechanism illustrates the more general notion of operator overloading. This will be added for the different operators +, -, *, /, etc.

A new mechanism now allows the subsetting to be performed by functions or compiled utility functions. When we fail to find an appropriate method for the subsetting, we search for a function or compiled method in the internal function list of the evaluator for a method named subset which takes 3 arguments:

See SubsetFunctions for an example of how this works.

Database Element Access

An additional cube of syntactic sugar has been added to allow simple references to elements of Database objects. The user can now issue commands such as
  db = new ObjectDatabse()
  db.x = foo(z)
  db.x    
The last element of the name on the LHS of the assignment expression is used as a key in the Database (as a String) and its corresponding element returned or assigned a value. The retrieval and assignment use the appropriate methods in the Database so type, read-only, etc. attributes are handled correctly.

If the particular instance of the Database has a publically accessible field of the given name, the last component of the field access is not considered as a key. Instead, the field in the object is used and either returned or assigned the new value. (Of course, public access is not encouraged and should be provided via accessor functions.) One can explicitly assign an object to the Database using the assign() method or via the subsetting operators, rather than this syntactic shortcut.

This can be easily used to provide easy access to assignments to specific databases. When a database is attached, it can be assigned to an object in the default database. It currently provides its own name by default and can be provided one. Thus, having attached a database named "X", the command

  X.y = 1
would be shorthand for
  assign("y",1,getDatabase("X"))

Rather than polluting the name space of the default database, newly attached database do not have to be assigned to an Omega variable. Instead, the search mechanism for the FieldAccess can be modified to traverse the search path of Databases in the event that a variable is not found.

The current implementation allows arbitrary nesting of access to database elements. For example,

 d = new ObjectDatabase()
 d.table = new ObjectDatabase()
 d.table.element =  1
 d.table.element
 d.table
See getField() in FieldAccess and setField() in AssignExpression.

Optional new

For most classes, one can omit the new. So x = Vector(3) will call the appropriate constructor for Vector. It does this if there is no function object or method that could be called instead. This is an implicit new. This does not work for primitive types (int, short, etc.) This is a feature of the parser and throws a syntax error. Similarly, when declaring and initializing an array or declaring an array with an unspecified dimension, one needs to use new
  x = String[3]; // works
  x = String[3][]; // doesn't work
  x = String[]{"a","b","c"}; // doesn't
  x = new String[]{"a","b","c"}; // does
The simple statement
  x =  Vector
assigns the class Vector to x. This avoids the more cumbersome x = Class.forName("java.lang.Vector") or even findClass("Vector"); Of course, it is ambiguous in that if there is an object of that name on a database, it will be returned.

Method Invocation

There is a reasonably complete facility for call dispatching that takes care of
  1. function invocation
  2. method dispatching
  3. internal methods provided by the evaluator
  4. casting to an arbitrary class from which an object is derived.
The idea is that if the call is unqualified it refers either to a function or to a method in the implicit this which is the evaluator. The latter provides access to ``internal'' functions. In the case of functions, an object of class Function is searched for and if found, it is evaluated with the arguments in the method call. No lazy evaluation is used.

This mechanism has been extended to handle method dispatching on Omega functions.

show(i) maps to evaluator.show(i) i.show() is the way to invoke the method show() on i.

Additionally, the code is now used to implement the S/R to Java interface. Calls from S/R are made Java objects and classes and arguments are "converted" to Java objects.

A feature has been added that allows promoting an object to an array in order to match a method. The object is contained in an array of length 1 and the arrays component type is identified by the matching method. For example, consider the class

public class foo {
 static public int doFoo(String[] x) {
  System.err.println("In String[]");
  return(x.length);
 }

 static public int doFoo(int[] x) {
  System.err.println("In int[]");
  return(x.length);
 }

 static public int doFoo(int[][] x) {
  System.err.println("In doFoo int[][]");
  return(x.length);
 }

 static public int doBar(int[][] x) {
  System.err.println("In int[][] "  + " " + x[0].length);
  return(x.length);
 }
}
and the calls
 foo.doFoo("my string")
 foo.doFoo(1)
 foo.doBar(new int[]{1,2})
Ordinarily, none of these would match any of the methods. By elevating the value to an array of the type specified in the method, each call resolves to the corresponding entry in the class.

This also works in the same way for constructors. It is is especially useful in the S/R to Java interface where S and R have no actual scalars but vectors of length 1. Thus the automated conversion of such an object is ambiguous. By converting to simple primitives, the Omegahat method dispatching will perform the appropriate promotion to arrays.

Non-public (i.e. explicitly protected/private and "inner") classes provide complications for this dynamic reflected invocation. For example,

 > p = Runtime.getRuntime().exec("cat some-file")
 > p.getClass()
 class java.lang.UNIXProcess
 >  p.getOutputStream()
class is not public - class java.lang.UNIXProcessEvaluation Error: inaccessible method public java.io.OutputStream java.lang.UNIXProcess.getOutputStream()java.lang.IllegalAccessException: java/lang/UNIXProcess

In Java2, there is a mechanism to temporarily make a reflected object (Field and Method) accessible, in spite of the permissions. For cases like the one above, this is not a security or programming issue. The non-public nature of the class is a feature of the Java implementation, not a design of the class.

The facility is used in the following manner.

 Method m = findMethod(.....);
 m.setAccessible(true);
 m.invoke(....);

Gaining access to fields for write permission and internal methods which are intended to be called only by other methods of that class or object is a potentially a bad thing. As a result, we wait until there is an access error and only then do we test that the method is public but the class is not. This is not currently implemented for static methods as it is assumed they are either public or not by design.

Casting is now honored. Suppose we have an instance, named x of class C which is derived from a class B which in turn extends A. Then, calls of the form

  x.foo()
  ((B)x).foo()
  ((A)x).foo()
will call the appropriate instance of the method foo() in the different super-classes.

The implementation of this requires dynamically generating and loading a new Java class. As a result, this might be an expensive operation. In the future, we might cache these classes.

Graphical Interfaces

The following might be provided easily using Swing.
Omega can be ...
Thought has gone in to displaying results of commands in the JTextPane with different colors for the different components using an HTML-like language. There is a relatively abstract mechanism that requires a particular instance of a filter that will convert the Omega output into HTML.
VRML output could be cute in other contexts.
Class viewers
There are some very nice graph viewers for classes and methods.
Data editor
A DataGrid or spread-sheet like frame which allows one to
The Quote() operator which protects expressions. This will allow user-level thread creation and invocation.

Utilities

Variable expansion.
I have just added a parser that allows one to traverse a string and replace "terms" of the form ${VARIABLE} with the value in a Properties table indexed by the String VARIABLE. While this is used to expand environment variables in-line, it can be used for arbitrary purposes with any table.
Archive Files - ZIP and JARs
We can now read the contents of files that are themselves contained in zip or jar files. A particular instance of this is the way we read the default scripts that control the initialization and termination of an Omega session in the binary/JAR release. This is available as a source() method in the evaluator and via the class Omega.IO.ArchiveEntry.

Compilation

Listeners
In order to allow functions and expressions act as listeners in the event model (both AWT/Swing and class-specific events), we need a mechanism to generate a class that implements the listener interface but evaluates the function/expression in response to the event-handler method being invoked.
This is done in the GUITools/ package. A class there generates byte code dynamically which can be immediately loaded.
For distribution and portability purposes and the interest in speed, we want to be able to compile Omega scripts and functions.
This should be quite straightforward using Jas to generate the byte code from the data structures currently in the generated by the parser. The quality of the generated byte code will of course require a significant intellectual effort. But this does allow us to implement features such as tail recursion, etc. not only in the interpreter/functions but also in the byte compiled code.

Prompt Evaluation

Far from efficient, the prompt for an interactive evaluator can now be specified as an expression, rather than a string literal. This is cute, but not that useful you might think. However, it does illustrate several useful features of the Omega language. Firstly, it illustrates how the evaluator can be used in a variety of different circumstances and can effectively be embedded within itself. Additionally, it illustrates the configurabilty of Omega. The prompt is specified in the options configuration file as a String. Likewise, the default evaluator class is specified in these Java properties. These together illustrate an aspect of the adaptability of the Omega environment.

Now consider the prompt expression string. This can be a simple expression that access variables in the databases or in the evaluator itself. For example, we may choose to emit the task number as the prompt surround by [] pair. This might be done (depending on the evaluator class and features) as

"[" + taskList().size() + "]"
The method taskList() is provided by the evaluator and presumably is a (derivative of) Vector or Hashtable that contains the previously. (Alternatively, we might cache the previous k tasks but keep a running count of the number of total tasks processed accessible via the method numTasks().)

Alternatively, we may just keep a count of the number of prompts emitted. An expression of the form

_lineNumber++; "["+lineNumber+"] "
provides this functionality. Of course, we chose an unusual name for the line number count (_lineNumber) in an effort not to conflict with other names used during the Omega session. A better way to do this is to use a Closure to provide local scope via an anonymous function:
 function() { static int count = 0;
              return("[" +  ++count + "] ");
             }
Here, count is local to this function and we take care to turn this into an evaluable expression.

Multiple Evaluator Classes, Instances and Threads

There several different evaluator classes including support for embedded, interactive, function-enriched, signal handling use. The default class used to start omega is specified in the properties file. As well as offering different classes, one can create multiple evaluator objects or instances that are in existence simultaneously. This allows different environments - search paths, class loaders, etc. - to be maintained and used for different tasks without reconfiguring a single evaluator or re-creating an evaluator each time. In addition to being in existence at the same time, the evaluators can be running in different threads concurrently. On a multi-processor machine, and even on a single processor machine, we get potential performance improvement. Java2's use of native/kernel threads makes multiple processor use beneficial. Groups of evaluators are managed by the Evaluator Manager which behaves as a factory for creating new (or cached) evaluators and also brokering communication between them.

Signal Handlers

There is support for handling signals in a central signal handler provided by the JavaSignals library. This allows lengthy computations to be interrupted at the console, user-level signals indicating external events such as modification of control files, etc. The class SignallableEvaluator provides an evaluator that can be registered as a signal handler. The two methods it implements as a SignalListener can be modified or overwritten to provide handlers for different signals.

Users can provide their own signal handlers in the form of Omega expressions or functions for different signals and have these be evaluated when the signal is raised. This is done in the UserLevelSignalHanders class which is a separate Evaluator. Usually, a single instance of this class is created in the startup scripts and populated with the appropriate handlers.

In the near future, we will support specification via named properties of the form

INT: function(sig) { ......}
HUP: rereadInits()
This requires JNI support and so cannot be used in an applet, but it doesn't make sense there anyway! Using JNI

Miscellaneous

Sequence operator
The (current) sequence operator is :: It can be used something like 1::10, foo(x)::20 0::(x.length-1)
The expression objects each have a method apply() which takes an Omega function and applies it to each of the elements in the parse tree rooted at that expression. This is recursive and currently returns a java.util.Vector. In the future, we will support returning trees with the same structure and other features.

Coming to a cinema near you!

Features on the horizon:
Byte compilation
One of the most important aspects of this language is that we will be able to bridge the divide between easy-to-use interactive languages and compiled low-level languages used for performance. We should be able to use jas, jasmin, etc. to byte-compile the Omega code into Java byte code and realize these improvements in performance. Some modification to the interactive languages may be necessary to obtain optimal performance rather than leaving all types, etc. to be dynamically determined.

Lexers and Parsers

To support user-level classes and global functions and allow users access the latter within a user-class method, a new keyword - global - has been added to the grammar. (Please recompile the entire installation if working from the CVS repository!)

The class of the default lexer is now configured to be read from the OmegaOptions file. This allows different lexers to be easily instantiated without recompiling. In fact, setting the value in the options() object of the evaluator allows us to change the lexer dynamically.

The primary motivation for having two lexer classes (omegaJavaLexer and omegaNestedStringLexer) is to provide different ways of handling escaped strings. The orginal lexer (omegaJavaLexer) cannot handle escaped strings without some modification, and more importantly, the user interface for such nesting must be simple. So, the second and default lexer omegaJavaLexer now treats the " " and ' ' pairs as mutually-nestable. We still of course require balanced pairs (so a string cannot start with " and end with ' or vice-versa). Using this, the following works parse(" x= 'abc';") and gives an expression containing the literal string "abc".

In order to handle this nesting, we have to sacrifice the the Java syntax for speciying characters - e.g. 'a'. At the moment, I don't think this is a problem. However, there is a chicken-egg problem in that we don't use characters much in statistics because we use strings instead because the languages encourage this. To use this syntax, set the default lexer property defaultLexerClass to omegaJavaLexer. As mentioned above, this can be done dynamically and temporarily - for the duration of one command. For example,

[] x = 'a'
a
[] x.getClass()
class java.lang.String
[] options().put("defaultLexerClass","omegaJavaLexer")
omegaNestedStringLexer
[] x = 'a'
a
[] x.getClass()
class java.lang.Character
[] 

The current complicated framework (or maze) of calls to parse a string in the Evaluator class is intended to allow us to reuse previously initialized lexers and parsers. Currently, we have to instantiate a new lexer and parser for each line of input. If we want to handle split-lines of input and just be generally more efficient, we have to find a way to convince See ResettableLexer and the lexer methods in Environment/Parser/AntlrParser.


Substitute

One can parse strings via the parse() method of the evaluator. This returns a List of expressions. This is a tree of expressions and can be recursively processed. One such method is substitute which behaves like its S/R counterparts. It takes a Hashtable of values indexed by some key and recursively traverses the children of the tree replacing objects that "match" the key with the content. The current setup works for Name objects. The name is converted to a string and a corresponding entry in the Hashtable is used as a replacement (directly) for the Name object. For example,
[1] db = new Hashtable(); db["x"] = parse("foo(2);")[[0]]; p = parse("if(x > 1) { foo(x); }")
if(x > 1) {
	foo(x);
}
[2] p.substitute(db)
if(foo(2) > 1) {
	foo(foo(2));
}
[3] 
This does not currently work correctly in all cases. This is due to the the tree relationships needing to be simplified into one model.

Comments

Comments can be specified using the Java comment identifiers - //, /* */ and /** */. At present, these can be specified only at the statement level. For example
  /** Comment */
  for(i=0; i < 3; i++) {
     /* another comment */
   x = 1;
     // 
   y = foo(x);
  }
However,
 if(x == 1 // test for something
      && y < 2)
will generate a syntax error. This will change in the future.

Note that these statement-level comments may not be printed currently. The toString methods of the different expressions need to handle these. Note also that each expression class has two linked lists of comments - pre and post.


Language Changes

For the most part, the syntax of the Omega language is the same as Java's. In this section we note the changes.
Exponentiation is specified using expressions of the form 3^2
The ^ symbol in Java is the binary XOr operator. Currently, the precedence in Omega is the same as the same symbol in Java.
Subsetting [ now returns an object of the same class as the container object. [[ returns a scalar where appropriate.
This may change.

Input Continuation Separator

Input is read via the InputReader class which takes care of emitting/displaying the prompt and reading lines of input. The terminated input is passed to the evaluator and onto the parser via a listener model, thus anonymizing the consumers of the input. Currently, the parser throws an exception for incomplete expressions. Until it is modified, top-level expressions can be continued on other lines by terminating the current input line with the single-line comment character //. For example, we can define a function as follows
function foo(x) { //
    y = x+1;   //
    x*(y+x);   //
}
The parser will change relatively soon to communicate the incomplete input to the reader.

Internal/Compiled Functions

The evaluation of function-like calls of the form
  foo(x,y)
are now resolved in a different manner. Previously, the following steps were used:
  1. look for a function named foo
  2. if not found, look for suitable method named foo in are now resolved in a different mannerevaluator
  3. if not found, consider call as
    new foo(x,y)
To facilitate dynamically adding compiled/internal functions found in step 2, we have added an additional step before 2 which allows searching arbitrary objects and classes registered with the evaluator.

See Utility Functions for a more complete description and an example session.


Missing Features that ought to be added soon!

Method dispatching doesn't convert Integers, etc. to their corresponding primitives!
This is done now.
Operator overloading
The subset operators [ and [[ are done. Overloading of other operators such as +, *, /, etc. will work in much the same way and each requires an interface and recognition of objects implementing that interface in the expression class in Omega.Parser.Parse. See Addable, are now resolved in a different mannerSubtractable, etc.
Labels for continue, break and goto.
We won't miss the last one.

Other Utilities

File search mechanism via used defined search path.
Like the X-windows userfilesearchpath, Java's import statement, users should be able to say load,source, etc. which invokes a locateObject() method. This is done via the FileLocator class which takes a list of directories (or Jar files) and searches within these when a "file" is requested.
Help
The Evaluator class currently has a collection of help() methods which display the tree of HTML documents using the latest JHelp package from Sun. There is little content, but the structure is mapped.

We may generalize this to use XML based documentation which we can either process into HTML or use directly with specialized browsers. The latter allows us to offer "live" functionality in the browser such as interactive plots, drag and drop of objects onto plots, updating of results, etc. The JEditorPane in the Swing/JFC package is a well designed starting point for our customization. See the new documentation in S as a prototype model. The Java XML package from Sun provides the necessary tools for parsing and processing XML documents and trees.
Alternatively, we might use the flexible Doclet approach in JDK1.2 or merge the XML and Doclet approaches.

Templates, or Generics, as available in GJ are covenient. These may be added soon. They can of course be used to create Java byte-code that is used in Omega.

Duncan Temple Lang <duncan@research.bell-labs.com>
Last modified: Thu Mar 9 13:48:34 EST 2000