Tele-Presence Microscopy

Tele-Presence Microscopy & the ANL LabSpace (eLab) Project



ANL Principal Investigators:

Nestor J. Zaluzec - Materials Science Division

R. Stevens,R. Evard,T. Disz,R. Olson - Mathematics and Computer Science Division

T. Kuhfuss - Electronics and Computing Technology Division

TPM Executive Summary

Tele-Presence Microscopy (TPM) is an advanced concept in the integration of computers and high speed networks with scientific instruments for operation, control, communication and research which makes use of ANL's Advanced Analytical Electron Microscope (AAEM) and related instruments as development/testbed sites. The implementation of a Tele-Presence Microscopy Facility allows a user from a remote location to either observe and/or control state-of-the-art instrumentation in a real time interactive mode. This concept has multiple applications in government, industry and education. When operational, a TPM user will be able to actively participate in scientific investigations at unique government resources such as national user facilities without being physically present at that facility. Manufacturers would be able to configure demonstration equipment which are accessible via the TPM system and thus allow prospective customers to remotely evaluate instrumentation before purchase. After acquisition, implementation by the manufacturer of a TPM system would allow remote service/diagnostics by a systems engineer who resides at the manufacturing site. Finally in an educational environment, students can initiate tele-presence operation of instruments which may not be available at their host institution, allowing widest possible access to unique facilities, Alternatively, should students have access to local equipment, they will have the opportunity of consulting an advisor or non-local expert in the field in an on-line mode during their actual experimental session, thus freeing valuable time which would be otherwise wasted during unproductive experiments..

The generic TPM system is composed of both software and hardware, which operate in a client/server relationship. The local user will be able to control access to the instrument, via a secure login system granting either observation mode or control mode to a remote access request. The remote TPM user would be present at a workstation displaying system configured windows showing relevant experimental details ( images, spectra, instrument &/or control parameters...), which would be either active or passive, depending upon the operational mode. Hot links will be provided so that it will be possible to integrate special resources (such as massively parallel computers) needed for data processing and analysis, when they are available either locally or remotely. Using a software plug-in module an individual manufacturer will be able customize the generic TPM interface, to provide access to unique capabilities relative to their instruments and highlight special capabilities which will vary from instrument to instrument. The TPM workstation client software will be designed to be as platform independent as possible (i.e. capable of running on most high end workstations) and will therefore allow the user community widest access at minimum cost.

The program integrates resources and personnel from the Materials Science Division, the Mathematics and Computer Science Division and Electronics and Computing Technologies Division of ANL as well as the industrial partners in a multi-year program to develop a Tele-Presence Facility. Once the TPM project is completed, the system will be placed on line together with the AAEM and made a coherent part of the ANL AAEM facility. Generic TPM client software for this instrumentation will be distributed free of charge to prospective users of the facility, while server software and hardware will be distributed to industrial partners for further commercialization. When completed information concerning implementation of a TPM facility at any prospective site (Com, Edu, Gov...) will be distributed to the Scientific Community.The AAEM Project will maintain and upgrade the generic TPM product to keep the system current with technological advances and distribute the generic client/server modules to the user community through its extensive electronic communications network and will provide corresponding upgrades to it's industrial partners to further advance integration of the technology into the commercial environment.


LabSpace: The National Electronic Laboratory Infrastructure

R. Stevens,R. Evard,T. Disz,R. Olson - Mathematics and Computer Science Division

T. Kuhfuss - Electronics and Computing Technology Division

Nestor J. Zaluzec - Materials Science Division


Executive Summary

National laboratories. Advanced scientific instruments and data analysis resources. High performance computers. Scientists, technologists, professors, and students. The vision suggested in the call for proposals is to combine all of these over the National Information Infrastructure in such a way that makes distributed collaboration as useful to scientific experiments as being in the same location.

Distributed collaboration over the Internet is critical to the success of today's scientific ventures, which are far-reaching and often require specialists from around the world to synchronize. We suggest that it is possible to provide a solution so powerful that it changes the way that scientists and researchers think about work and working together.

In LabSpace, we propose the creation of a set of electronic laboratories, or elabs, which will be networked institutions housing interfaces to scientific equipment, meeting rooms, private offices, and interaction areas. The scientist's interface to these locations will be a browsing environment based on the familiar World Wide Web programs, and will be layered on a suite of groupware tools such as audio and video clients, as well as control mechanisms for remote scientific instruments and state of the art network transport and security.

The members of the LabSpace project are uniquely positioned in the diverse areas of networking, high performance computing, collaborative environments, innovative interfaces, and distributed scientific research. We propose a program, that, over a three year period, will create electronic laboratories, build the basic tools necessary for interaction, and study the effects of these environments for three testbeds: the Advanced Analytical Electron Microscope Laboratory, and the LabSpace project itself, which will be a distributed development effort.

The results of this project, a National Electronic Laboratory Infrastructure, will be available to other scientific institutions during the creation and after the conclusion of the effort. Electronic Laboratories will be built that will contain unique research opportunities, information repositories, and powerful computing and analysis equipment. This intuitive environment will bring together scientists and researchers from around the world.

Motivation and Need for Distributed Collaborative Tools

Today's scientific world is populated with fabulous resources, ranging from collider detectors to super computers, from advanced photon sources to virtual reality labs. Yet the most important resources are the scientists and technologists who design and use these tools. They travel all over the world for the opportunity to work together, moving to and from the places where the scientific instruments are located, all in the name of furthering scientific research.

Perhaps one of the most important potential tools that we have today is the global information network, the Internet. It has the capacity to bring together all of these resources. We simply need to put all the instruments on the network and let everyone work together remotely. This will cut down on travel and increase the amount of time that the people, our most valuable resource, can work together on experiments and research.

Yet the critical question remains: what is the most effective way to accomplish this? We believe that any solution must take into account the aspects of real laboratories that make collaboration effective. Beyond the obvious rooms with scientific instruments and libraries of information, labs provide meeting places with white boards, computing and media infrastructure, and other interaction tools. Most importantly, they create an environment where people can interact with each other professionally and casually in ways that make collaboration truly effective. We believe that all of these functions must be incorporated into any successful solution.

To this end, we propose the creation of a National Electronic Laboratory Infrastructure. Sites who wish to host distributed collaborations will create a networked interface to their facilities that will house their scientists, their tools, and their visitors. People will visit these laboratories and work in these laboratories, potentially spending as much or more time in the elab than in the lab. The natural interface provided by such an environment - looking at an instrument, talking to a fellow scientist, sharing a virtual white board - will promote natural collaboration without impeding the process with excessively complex technology.

These elabs will be available to any scientist with an ordinary workstation and an Internet connection. As more sophisticated tools such as CAVEs or high performance data engines become available, they will be integrated seamlessly into the environment, taking advantage of technology shifts while still promoting collaboration with conventional systems.

The benefits to the users are clear. They will be able to maintain contact with their fellow researchers regardless of their location. They will be able to set an experiment in motion within an elab, and return at a later date to check on the progress. They will have a permanent and natural interface to their scientific instruments and to information archives. The tools that they need to carry out such interactions will be united under a common interface, as natural and as as powerful as the WWW browser upon which it is based. As technical limitations are removed, the collaborations will be more productive than before, with less stress on the individuals involved. Researchers will meet people with similar interests while working in the elabs, and new partnerships will be formed. The electronic networked space will make a new form of people networking possible, promoting and enhancing the state of scientific research in this country. A paradigm shift will take place, and we will all enhance our research and collaborative efforts. By the completion of this project, a system will be in place that will allow all of the National Laboratories to be on-line, sharing their resources with scientists around the world.

Current systems show that this is indeed possible. Many previous projects, such as Xerox PARC's famed "MediaSpace", have investigated how people work together when given the opportunity to share an electronic place. The results of all these experiments have been encouraging and educational, yet their user population has been limited to those at the few sites participating in the experiment. In LabSpace, we propose to open up the electronic laboratories to any qualified personnel in any location. The scale of the project and the number of participants will necessarily be larger than previous collaborative projects.

Other large-scale projects do exist. The most notable of these is the ground-breaking MBone effort, which utilizes the multicast capabilities of TCP/IP to create audio and video sessions shared by users world-wide. However, these projects are focusing on the technical aspects of the multicast implementation and on specific tools to exploit the potential. These tools can be used for interaction between people, but they are unwieldy and have limited use. Their focus is not on the scientist and his environment, but rather on the features of the specific tool. These tools are essential, and have a place within LabSpace, but they don't solve the collaboration problem.

LabSpace focuses on the user. It creates a natural interactive environment capable of supporting any tools needed for joint research. It defines an open architectural framework that encompasses existing standards such as the MBone technology and the World Wide Web information structure, combining these in unique ways that will enhance natural collaboration.

In many ways, LabSpace is based on examples of MOO-based networked spaces that, while primitive, have been shown to be quite useful for researchers spread across time and space. BioMOO [ref] is one example of these environments, which brings together biologists into a text-based space. LabSpace goes beyond currently available MOO [ref] technology, combining the concept of a networked space with graphical tools, high-end interfaces, and the best available security mechanisms.

We propose to create a National Electronic Laboratory Infrastructure. We will start the work with several testbeds:

The participants in LabSpace bring together a uniquely strong set of abilities. The high performance computing environment, strong software development history, and visionary leadership of the Mathematics and Computer Science Division of Argonne National Laboratory will focus and organize the project. The Materials Science Division of ANL brings to the project world class facilities and expertise on electron microscopy and microanalysis and has existing local, national and international user base whose collaborative efforts span government labs, universities and industry as well as strong ties to the secondary education environment. Northeastern University brings several years of virtual environment development and design to the project. The Electronic Visualization Laboratory of the University of Illinois at Champaign/Urbana are acknowledged leaders in the field of revolutionary interface design. Together, these government and educational institutions have a unique set of skills and resources that will combine to make LabSpace, and distributed collaboration, a success.

Technical Approach

The fundamental goal of LabSpace is to promote and enhance open-ended networked collaboration among scientists and researchers. LabSpace consists of:

The following sections discuss each of these in detail.

Persistent Electronic Laboratories

People, not scientific instruments, are the most important part of any scientific project. If there weren't any people, there wouldn't be a project in the first place, or there would be no one to understand it. The same can be said of collaboration. The point of collaborating is to get several people to work together. Therefore, in order to have a successful distributed collaboration, the people and the communications between them must be of highest priority.

The key to these experiments is to help the scientists involved work together most effectively. They must be able to manipulate aspects of the experiment, locate information that they need, and work together using natural means of interaction. When working with a complex laboratory machine, one wants dials and buttons, gauges and readouts. When working with other people, one wants to turn to them and say something, or to walk into a meeting room with a set of documents and pass them around. One wants to sit around while the experiment runs, monitoring progress while exchanging idle yet informative chatter. Casual and natural activities like these are essential in making distributed collaboration an effective reality.

An Interaction Metaphor

LabSpace creates persistent virtual locations on the network where people can interact in natural ways. The spaces are organized around centers of interest such as the National Collaboratories. They are created in a flexible and extensible manner, allowing them to be modified according to the project's needs. A researcher is able to enter a location and interact with the people and objects located within the space.

This paradigm provides endless opportunity for creating natural environments which promote cooperation and interaction. Here are a few examples:

Organizing resources and communication patterns into a spatial paradigm creates an extremely powerful metaphor for interaction. A place is an association between people working together and the objects they're using. This knowledge can be used to coordinate the technical modules which carry out the electronic communication, minimizing the effort required by the scientist to initiate connections and control. More importantly, it emphasizes the relations between people and allows them to interact casually in a social setting. By simplifying the user interface and making the distributed communication more natural, virtual locations enhance collaborative potential.

Several of these types of spaces are already in existence. AstroVR [ref], a collaborative environment at CalTech [?], uses the MOO server from Xerox PARC to create a virtual laboratory for astronomers. BioMOO [ref] brings together biologists from around the world. The InfoPark [ref] is a similar construction for systems administrators from several different sites. Most of these environments are primarily text-based, yet they are already significantly promoting collaboration. The sociological aspects of these systems are currently being studied, but informal feedback from the user communities is extremely positive.

The eLab Server

A collaborative site or center runs a server, known in LabSpace as an "elab", that acts as the interface for that site. It controls access to the resources for which it is responsible and mediates connections between researchers. It is extensible by a powerful programming language, allowing spaces to be created and organized, as well as allowing new interfaces and capabilities to be developed within the space. The elab is up at all times, creating a permanent location on the network. Thus, the elab becomes a persistent presence on the net for information storage and personnel interaction.

Initially, the LabSpace project will build elabs using a MOO server from Xerox PARC, after having added extensions for powerful interfaces. The MOO server already provides much of the power needed for LabSpace, allowing us to create an elab environment very quickly. We anticipate that the national elab infrastructure will grow too large for MOO within several years, therefore we will develop an alternate system that is designed for more scalable and distributed operation, yet will provide compatible functionality to the user community.

LabSpace will provide the following deliverables to create the elab environment.

The User Interface

Each user wishing to interact with an elab runs a program called a "browser" that communicates with the server. The program takes on the role of the user's capability broker, describing to the server what types of services the user can support. Examples of these include an audio/video connection, multicast capacity, or more advanced interfaces such as a CAVE. It also invokes the user's local programs which implement these aspects of the interface, and it validates and authenticates connections. In these ways, it is acting as the user's "broker", which is described in more detail below.

A number of user interfaces are possible, ranging from a simple text module (appropriate for low-speed connections) to a modified graphical WWW browser, depending on the capabilities of the user's environment, Initially, development will focus on the WWW browser. A user will connect to an elab by navigating the Web with the browser. They'll interact with it by pointing and clicking much like the expected WWW interface. As other interfaces are needed, they'll be created by the brower. Walking into a room where other people are located will cause their images to be displayed on the screen, and any sounds heard in the room (typically conversation) will be played by the audio client. If audio and video interaction aren't possible or desirable, or if a more technical interaction with the elab is necessary, a text client will be invoked. The possibilities for the user interface are quite extensive, enabling one to take advantage of both the information organization capabilities of WWW and the interactive collaborative potential of a persistent location.

The deliverables that we will build for the user interface include:

Information Location

As many people are discovering, the Internet is a big place that's easy to get lost in. While most users will know exactly how to locate the elabs that they commonly work with, some may not have this information when they begin. We provide an information distribution mechanism that helps users locate the resources that they need.

An elab server will run a service called a "catalog" (which may be a part of the server or may be separate). The catalog is associated with a session or with a site and understands the local capabilities and resources. It responds to queries about resources in a number of formats, including HTTP, enabling elabs to send information to standard World Wide Web clients. This capability leverages the ubiquity and flexibility of WWW, providing a powerful network navigation mechanism familiar to most Internet users. The security layer built into the elab and the browser (as described later) allows the information to be selectively distributed.

With regards to information location, the LabSpace project will provide:

The Module Layer

In LabSpace, a scientist uses the browser as the primary user interface, as it provides navigation and elab communication capabilities. However, the interfaces that are tailored to specific applications are split among "modules" which implement the details of that type of interaction. These may be general teleconferencing applications, such as a window containing images of one or more co-workers, or very specialized interfaces, such as an electron microscope controller. Many different modules will often be active at one time, so that, for example, one may discuss the view of the microscope while working with it. The set of tools in use by one person, regardless of their type or use, is called a "session".

The principle role of a module is to work with one or more data streams, acting as the user interface for the data. The module for an audio source data stream may be a program that drives audio speakers. A data stream consisting of a series of white board modifications will probably be sent to a module that implements a white board, or may be sent to an archiving mechanism to be recalled later. Modules may be as simple as a clock or as complex as the interface to a collider detector. Several of these modules have special roles that help the session become a unit that works together rather than simply a set of tools vying for screen space.

The most important of the modules is called the "broker". It acts much like the mixing board in a sound studio. It mediates connections between modules, allows or disallows remote connections, and understands the capabilities of the user's collaborative environment. It works closely with the browser (and may in fact be part of the same program) to make remote communication as smooth and simple as possible.

All of the other modules can be thought of as "generators" and/or "receptors", which, respectively, are programs which create data streams or handle data streams. A video camera is a generator, while a window displaying the output of that camera is a receptor. Many programs, such as a white board interface, are both. Specialized receptors and generators can act as filters, which convert one form of data stream to another, or archivers, which store data streams for later access.

A collaboration between two parties is thus a conversation between two sessions. A generator on one end arranges with its broker to negotiate with the broker on the other side. Once the two brokers have verified that this connection is authorized to take place, the remote broker passes the connection on to the remote receptor, which proceeds to communicate directly with the generator. The data is passed in whatever format and by whatever mechanism is appropriate to those modules.

The specification of the protocol for communicating with the broker will be one product of the LabSpace research project. The individual modules' protocols will not be specified, but will be left open-ended, so that modules of arbitrary types may be added whenever appropriate. In this way, the LabSpace architecture is open-ended, allowing tools of any type to be developed that still work within the framework.

The rest of this section describes the broker, data streams, and the other modules in detail.

The Broker

The broker is the most important process at the module level of a LabSpace session. It acts as the central authority for connections, routing the right kinds of data to the right module. Each user will have a broker, as will each elab. The broker's roles include: There must be exactly one broker per session, where a session is defined as a set of associated modules, most likely under the control of one user. These need not all be executing on the same computer, nor even at the same site.

The modules trust the broker to provide them with correct connection information, thus the modules and the broker must have a secure communication mechanism.

The broker acts as a coordinating mechanism for a session, which helps to unite the potentially large number of modules into one perceived entity. This is important for both the user and the programmer interfaces.

The LabSpace project will provide:

The LabSpace project will develop specifications for these protocols as well as a sample implementation of a broker.

Data Streams

Modules in LabSpace view the world in terms of data streams. A data stream is any source of data that originates from a module and is targeted for another (or many other) modules. Examples of data streams include the output of an audio microphone, the data needed to drive an interface to an electron microscope, and the sequence of commands generated by a shared editor.

Data streams must be able to be combined, creating in essence a new type of stream. This allows one to associate related data, such as the audio and video produced by a telecommunications package. A more complicated multiprotocol data stream might combine the head tracking information from a head mounted display, the hand location and position of a dataglove, and the sound from a microphone into a complete model of someone doing working with virtual reality equipment.

LabSpace does not attempt to define all types of data streams. It leaves the details of the data streams up to the modules that will be producing or using that data. However, LabSpace must be able to define the protocols in ways that allow it to to combine streams, archive them in useful ways, and to sequence them as they were originally produced.

LabSpace will provide:

Receptors

Modules which can receive data streams are called receptors (noting that many will also generate data as well). Examples of receptors include: Receptors need not be limited to simply receiving and displaying data, however, any module with this capacity is a receptor.

The receptors that will be built by the LabSpace project include:

This list of modules may be updated somewhat over the duration of the project as certain features and aspects of wide-spread distributed collaboration become better understood. Where it makes sense, such as with the audio, video, and white board interfaces, the modules will be able to use existing MBone protocols in order to ensure compatibility.

Generators

Generators are modules which create a data stream. A few sample generators include: The LabSpace project will provide generators that correspond to each of the receptors listed above.

Filters

Modules which act as receptors for a data stream, transform it in some way, and then pass the data on are known as "filters". They can be used to turn one form of data into another, combine several data streams into one, extract one type of data from a complex data stream, or take one stream and send it to many places, to name a few possibilities.

The broker can invoke filters in order to transform data into a format that the local modules can understand, or the user may start one in order to explicitly manipulate the data as desired.

The filters delivered by the LabSpace project will depend enormously on the needs and specific hardware requirements of the testbed communities.

Playback Engines

A special type of receptor that has been designated to store data streams is known as an "playback engine". These engines form a critical part of the elab environment.

Playback engines have many intriguing possibilities:

  • - They may be used to index and search the data they store. (This is known to be a very difficult problem with some types of data).
  • - They can be used to maintain the state of various modules, such as virtual white boards, such that the next time the module is initiated, it can have the same state as it did the last time it was used.
  • - They may develop into permanent repositories for important data, recallable and accessible according to the modules needs and capabilities.
  • - As in the example above, they can act as virtual VCRs, essentially recording a collaborative session, portions of which could be replayed at a later time. Playback engines will need to have significant processing ability and off-line storage capability, and will most likely require high-end scalable I/O computers supporting the archiving and recall capability.

    The LabSpace project will provide:

    Module Layer Summary

    At the module layer, LabSpace carefully distinguishes roles of the modules, separating concerns as much as possible. The broker takes on the role of the session organizer, mediating connections and keeping track of the capabilities of the various modules.

    This organization allows one to develop both high- and low-end applications that can interoperate, and focuses the modules on their role - handling the communication data that is critical to a succesfull collaborative experience.

    The Communication Layer

    LabSpace expects, naturally, to utilize a network of some type. Most, if not all, of the people sharing the collaborative environment will be doing so over digital networks. An important goal of LabSpace is to operate independently of the particular network type that is used, yet to take advantage of features provided by the network when they are available.

    For example, not all network transport systems provide multicast support, but it is essential to some forms of large-scale communication. Therefore, multicast capability is supported within LabSpace, but is not required.

    At the network interface, LabSpace supports these ideals:

    LabSpace's design is not tied to a specific network implementation. It will work using TCP/IP or HIPPI, over wireless networks or through fiber optic cables.

    The initial implementation of LabSpace will be built on TCP/IP, but a library will be provided that abstracts away the details of the networking implementation, allowing for portability of programs that use this library.

    In order to accommodate existing code, the use of this library will not be required. Should a client wish to directly manipulate the network in order to exploit multicast, it is free to do so. However, the library will provide network independence that the client may not otherwise achieve.

    Scalability Issues

    This model is expected to scale quite well to approximately one hundred users. When the population involved in a collaboration grows beyond that, the elab server and the user's broker may become bottlenecks, although not for every application. For example, if the interaction is broadcast or multicast over an insecure channel, any number of people will be able to tune in and participate (much like MBone videos). Should the channel need to be limited to a thousand specific people, they could all be given a particular access key that the elab broker would require to authenticate the session, but even the state traffic in the elab server is likely to be excessive.

    We feel that a hundred users will be quite satisfactory for the first years of the project. However, as mentioned above, one goal of the project is to develop a more distributed, peer-to-peer elab architecture that will scale arbitrarily.

    We do not expect broker traffic to be a significant problem, as the broker is primarily involved with control information, not actual data. If broker traffic is too heavy and creates a bottleneck, however, methods of delegating responsibilities will be explored, including using several brokers as proxy agents, and dividing responsibilities across multiple processes or machines.

    In general, the scalability of the collaboration is more likely to be limited by human constraints rather than technical restrictions. A thousand people watching a presentation is not difficult to manage, but a thousand people having a conversation is not going to be a success (unless you consider total anarchy a success), regardless of how well the tools are built.

    Technical Summary

    LabSpace defines an open architecture for flexible and scalable collaboration. It consists of several levels of abstraction, each designed to simplify creation and take advantage of existing standards. The network layer is the foundation for the communication mechanisms. The module layer brings together a myriad of useful tools. On top of these is the laboratory layer, which, combined with the playback engine, creates a shared, permanent space with a sense of history. The result is a powerful and intuitive architecture focused on the user and the user's needs for distributed collaboration.

    Return to ANL Microscopy & Microanalysis WWW Server.


    Nestor J. Zaluzec / Zaluzec@aaem.amc.anl.gov