OpenSkies™ Training System

Where Technology meets Training Methodology

What does it do?

Many simulators have the capability to familiarize the student with simulations of the actual instruments and a few have the capability to create scripts for mission play. The OpenSkies Training System provides a training scenario authoring tool that allows for the training of both students and instructors in a Virtual Reality Environment for any type of simulation. The strength of the OpenSkies Training System relies on the close integration of the analysis and performance measurement capabilities directly within the simulation software.

Why do I need it?

OpenSkies is based on the training methodologies developed in the Naval Air Warfare Center Training Systems Division (NAWCTSD) for actual Navy training. OpenSkies was created with the following key ideas behind it:

• To improve the student’s performance beyond current training program capabilities.
• To provide a measurable performance standard.
• To provide the fast, simple creation of new training courses.
• To provide a low-cost solution to training on complex, expensive equipment.

How will this system improve the students performance?

This system provides capabilities for:

• Providing for a more quantitative approach to performance measurement.
• Showing students exactly where their deficiencies exist and allowing them to concentrate on those items.
• Providing an interface for creating personalized scenarios to address a student’s particular weaknesses.
• Supporting student proficiency at specific levels of performance.
• Providing a more structured environment for teaching instructors.

What is unique about this system?

• Applies a quantitative approach that allows for a better comparison of performance.
• Provides for tracking of class level of performance.
• Provides support for rating standardization of instructors.
• Provides for the development of training scenarios in virtual reality simulations in hours rather than days or weeks.
• This system may be customized to any domain for faster scenario development.

What are the payoffs/advantages of this system?

• It applies structured event-based training methodologies to a virtual environment.
• It allows for recording and playback of the entire training mission for later analysis.
• It provides automatic performance analysis feedback to the student.
• It has a low cost - requiring only a sub-$1000 PC.
• It has networked multi-participant capabilities for team training.

OpenSkies Scenario Development and Performance Measurement

Many systems have been created for the development of realistic simulations. However, systems for creating effective learning environments are only beginning to emerge. One method that has been developed at the NAWCTSD is the Event-Based Approach to training (EBAT). This method shows considerable promise and has been successfully used in a number of settings to establish effective learning environments. EBAT provides a systematic approach for developing learning objectives, generating scenarios, measuring performance, and providing feedback (Oser, Cannon-Bowers, Dwyer, and Salas, 1998).

Cybernet has worked with NAWCTSD through the Small Business Innovative Research program to create the OpenSkies Training System. This system has closely integrated the EBAT with an OpenGL based virtual environment toolkit to produce a training scenario authoring system. This authoring system employs the systematic approach to training developed by Fowlkes, Lane, Salas, Franz, and Oser (1994). This approach is diagramed below as an iterative procedure.

By using this approach, OpenSkies is able to provide an effective learning environment for students and instructors.

OpenSkies Scenario Authoring and Performance Measurement Interface

The OpenSkies interface contains tools for easily creating new scenarios and directly testing the student on the training exercises. The student's exercise is completely recorded for later analysis by the instructor as well as automatically analyzing which objectives the student has and has not met in each exercise. This interface includes:

• Point and click programmable scenarios, i.e. menus, filters, editable scripts.
• Dialog boxes to test the student during the scenario.
• Initial conditions of the actual scenario.
• The capability to track all of the student's responses (or lack thereof) to any event.
• Collection and playback capability for any exercise.
• Instructor analysis of student performance.
• Automated performance analysis available as a report.
• Capability for comparison and tracking of an individual's performance.
• A complete application programming interface for adding complex and new event types.
• Support for multiple configurations - student only, multiple students, student and instructor.

Employing these interfaces, OpenSkies is able to produce a virtual environment training system that provides realistic training and a complete performance analysis package for training both students and instructors.

Scenario Development

This capability provides the instructor with a simple interface for developing event based scenarios.

Scenario Purpose and Objectives
This interface allows the instructor to define the scenario objectives for the student, through a point and click dialog box. These objectives are entered into the objective database by the instructor and are generally defined for the specific type of training.

Scenario Events
This dialog box allows the instructor to select particular events to solicit responses from the student. This can be interactions with other objects, test questions, etc. This interface can be configured to be domain specific. For example, a flight training domain would set up pre-flight, takeoff, enroute, etc. sections for development and focus on communications, navigation and situational awareness skills for training the student. These events can also be weighted by the instructor in order to have the student focus on certain objectives.

Object Initialization
This interface provides a dialog box to set any initial conditions of the students vehicle/avatar interface. Examples include limited amounts of fuel for vehicles or an unfamiliar tool set for the avatar.

Briefing/ Debriefing
This interface provides the instructor with the capability to brief/debrief the student. This consists of a text and MIME interface for providing information. The MIME interface allows the instructor to attach any type of document to the briefing including such items as video, audio, word documents, html documents or any other information accessible on the computer system.

Environment
This interface allows the instructor to change the weather and other environmental conditions of the scenario.

Performance Analysis

This capability is provided by several pieces, recording and playback, marking of events and mission evaluation and mission summary.

Recording and Playback
The student’s mission is automatically recorded for later playback and analysis. This includes all interactions with the system, such as the simulation environment, pop-up questions and responses.

Marking of Events
The instructor may mark events in the system while monitoring it in real-time or playback for later analysis of particular events.

Mission Evaluation
This interface uses the recording and playback capabilities of the system. It allows the instructor to playback the student's scenario. The instructor may jump ahead to particular events and play them as well as rate the student's performance for any particular event or objective.

Mission Summary
This summary effectively scores the student on his/her performance. The system logs all events and objectives that the student did or did not make and gives the student's points for successful objectives. The scores for each objective are defined by the instructor.

Example Scenario Development

We present here a simple example scenario development for a pilot doing a pre-flight checkout. Assuming we have already developed the domain specific objectives and skill sets, the instructor can quickly develop a new scenario.

Scenario Purpose and Objectives
The instructor starts by selecting the appropriate objectives necessary to train the student for a specific skill set. For this specific flight domain, the objectives are defined with:

• The objective number.
• The objective description.
• The phase of flight (i.e. PreFlight, Takeoff, Landing).
• The skill set (i.e. Communications, Navigation and Situational Awareness).
• The standard NATOPS objective.

The instructor selects from the complete list in order to focus on a subset for the mission. The instructor may also provide a scenario description.

The objective database may easily be modified for any particular domain. A domain expert merely has to modify the Access 97 database. All of the fields are textual and simple to modify.

Scenario Events

The instructor continues by entering any data about the aircraft type and initial state, such as its amount of fuel it has. The instructor enters the main section of the scenario development, the Scenario Events interface, where he/she is able to create the actual script content.

The instructor develops the script content via the following steps:

• The instructor starts by selecting the initial ‘Phase of the Mission’ for the course. For this particular domain, this would be ‘Pre-Flight’.
• The instructor selects the skill set that the student will be trained on. The instructor may select from such items as ‘Instrument Checkout’, ‘Request Clearance’, or ‘Taxi to Runway’. Assuming the instructor selects ‘Request Clearance’, the interface provides a list of events available for these criteria for addition to the scenario.
• The event choices will now be limited to a few items such as ‘Tower Communication’, ‘Radio Traffic’, or ‘Change Frequency’, representing the events that may occur at this point in the scenario. For this example, the instructor selects ‘Tower Communication’ and adds it to the scenario. The interface presents the instructor with a list of sub-tasks that will occur.
  • The student initiates a call to the tower requesting clearance.
  • The tower acknowledges the call from the student and requests standby.
  • The tower provides clearance to the student.
  • The student acknowledges the clearance to the tower.

• The instructor selects each of these sub-tasks and defines the specific inputs and outputs of these sub-tasks. The following dialog defines inputs for the clearance from the tower event. These include:

• The objective.
• The radio station of the student.
• Time offset from the beginning of the scenario or from another event.
• The maximum score that the student may achieve for this event.
• The recording of the audio from the external source such as the control tower.
• A description of the actual message.

• Once the instructor has finished entering the events and their inputs, the scenario is saved.
• The instructor may then test the scenario by executing the application.

This system allows the instructor to quickly develop the course based on specific skills for each particular domain. Further, by defining the domain previous to the course development the instructor can easily understand and develop the course in a familiar manner.

Performance Analysis

The student is presented with the Mission Evaluation dialog upon completion of the scenario. This interface provides the following features:

• A map displaying the students intended flight path, the students actual flight path and markers for each event.
• A simple textual debriefing dialog.
• A results and scores dialog. This is described later.
• A playback timeline. This allows the student/instructor to jump to and playback specific events in the timeline.
• An event list. Instructors may modify the student scores from here.
• Filters for displaying the events.

Evaluation Results

This Mission Evaluation also provides an evaluation results page. This dialog shows the student/instructor:

• The event description.
• The objective tied to the event.
• Whether the event was triggered or not.
• The evaluation points given for the event.
• The total number of triggered events.
• The total points for the event.
• The maximum possible points for the event.

This interface provides a quantitative analysis of the student's performance and allows the student and the instructor to focus on the students deficiencies. The student may then quickly go back and try and correct his/her mistakes. This also allows the instructor to save time by only focusing on the points where the student needs help.

Running the Example Scenario

The interface for executing any scenario is simple and straightforward.

• The student logs in and loads the particular scenario.
• The student is then presented with the briefing for the scenario and starts the exercise in the aircraft with the initial conditions set by the instructor. This may include the flight path of the scenario if the instructor wishes.
• The student then executes the scenario and responds to events such as pop-up dialogs for situational awareness, radio calls from the tower and any other events programmed by the instructor. The system records the entire scenario watching for responses to all events.
• Once the student has completed the scenario, he/she is presented with the summary analysis of his/her performance. This provides the student with a rating and analysis of which objectives were met.
• The student may then go back and try the scenario again to correct any errors occurring in the first run.

This interface is all automatic, running the student through the scenario without the student needing any training on the software previous to executing the scenario. This provides a simple interface that a student may use at any time. Also, since the system runs on desktop PCs, the student may actually practice at home on a home desktop PC equipped with a joystick or other input device.

The system supports either the student flying with or without an instructor and performance analysis in either real-time or post exercise. The instructor rides along by sitting at another station that is connected via a network to the students station. This networked machine may be anywhere that it is possible to connect via the network. This enables an instructor monitor a student from across the country. The instructor may then critique the student's performance or even take the controls from the student in order to demonstrate the maneuver that he/she wishes the student to perform.

Once the student has performed the exercise the instructor may analyze the student's performance. The instructor can look at the summary analysis page and decide if there are any particular pieces to investigate. The instructor can use the Mission Evaluation screen to playback the entire scenario or move to any particular piece and playback that part of the scenario specifically. This playback includes all control by the student, any popups and radio calls as well as all of the student's responses.

In addition, the scenario may contain events that require the instructor to respond. These would be accessed on playback and may allow for such things as the instructor to rate the performance of a particular maneuver. Once the instructor has finished evaluating the student's performance he can print out the summary analysis and rate each of the student's performances.

In this manner, the instructor's grading habits may also be monitored, since any student's performance is then comparable to any other student performing the same scenario. Also instructors may be trained by having actual instructors run the scenarios and purposely make or not make mistakes for the student instructors to grade.

Conclusion

The OpenSkies Virtual Environment Training System provides a complete interface for desktop training and performance measurement using a virtual environment. This functionality provides a new level of performance measurement that will produce faster and better trained students and instructors.

References

Randall L. Oser, Janis Cannon-Bowers, Daniel J. Dwyer, and Eduardo Salas (1998), "ESTABLISHING A LEARNING ENVIRONMENT FOR JSIMS: CHALLENGES AND CONSIDERATIONS", Naval Air Warfare Center training Systems Division Orlando, FL.

Fowlkes, J.E., Lane, N.E., Salas, E., Franz, T., Oser, R. (1994) "Improving the Measurement of Team Performance: The TARGETs Methodology", Military Psychology, 6, 47-61.

Cybernet Systems Corporation is located at 727 Airport Blvd, Ann Arbor, MI 48108 and can be reached by phone at 734-668-2567. More information about the company is available at www.cybernet.com and information about OpenSkies is available at www.openskies.net.