New approaches to data visualization

Some of the new approaches to data visualization include, but are not limited to:

• GeoZui4D
• AUV Mission Planning
• Whale Tracking
• Flow Visualization
• Mid-water Fish

For more information on the Data Visualization projects, please see the Data Visualization Research Lab website.

Related Publications for Data Visualization

GeoZui-4D

We continue a very strong focus on the development of innovative approaches to data visualization and the application of these approaches to ocean mapping and other NOAA-related problems. The visualization team (Arsenault, Bogucki, Plumlee, Sullivan, Pineo and Schwehr) under the supervision of Lab Director Colin Ware has developed a novel and innovative 3-D visualization environment, GeoZui-3D. GeoZui-3D is a highly interactive 3-D visualization system designed to support a number of different research projects and ocean mapping applications. GeoZui3D was described in detail in previous progress reports. In 2005, GeoZui3D evolved to incorporate time-varying data opening up a world of new visualization possibilities evolving into what we now call GeoZui-4D. The GeoZui software has been made available to the public and more than 40 groups have downloaded the software.

This past year many important enhancements have been made to GeoZui-4D including:

• The package has been streamlined building on Linux and Windows using SCons.
• Release packages can now be assembled using SCons.
• Enhanced target object have been developed (Arsenault, Calder and Locke) for midwater targets that allow the automatic construction of smoothed point normals from multibeam sonar data. This allows point data to be correctly shaded and for the appropriate design of automatically oriented facets to provide "virtual hull" effects. This approach proved to be very valuable in a repeat survey that documented the degradation of wrecks of the WWI German fleet at Scapa Flow.
• Added support for visualizing acoustically tracked fish or other targets.
• Added for real-time AUV visualization enhancements in support of work with the FETCH/DOERRI AUV.
• File associations: .gzx files can now be associated with GeoZui-4D. This allows GeoZui-4D to launch and load a .gzx file by activating it in a file browser.
• A library has been developed and integrated (libgz4d). GeoZui-4D now uses functionality from libgz4d. This helps eliminate redundancy in code.
• Time referenced ogg file support. Allows compressed sound files to load and play at desired time without an external .gzx file. This is done by embedding the .gzx file in the .ogg file.
• Added capability to visualize mid-water target data (EM3002 and 7125) from real-time sources using network sockets.
• SRTM plus support.
• Added support for visualization clusters (e.g. GeoWall II).
• Added support for Nintendo Wii remote an input device under Linux.
• Added capability to visualize beam pattern of modeled acoustic arrays.
• Support dynamic display of photographs (scaling, priority, etc.).

As mentioned above a new, in-house library (libgz4D) has been developed to allow much greater functionality and flexibility in the further development of GeoZui-4D and other visualization applications. At present libgz4D have the following characteristics:

• Incorporates modified GLUT with spaceball support under Windows.
• Uses common SCons based cross platform build system.
• Supports a subset of X3D.
• Supports Wiimote devices
• Includes Swig wrappers which enable Python access to libgz4d.
• Uses gz4d namespace and includes directory subdirectory to help prevent name clashes.
• Includes Doxygen source code documentation.
• Designed to integrate with various frameworks and scene graphs.
• Grib data reader (Grib version 1).
• Offscreen rendering support.
• Option to build as static or shared library.
• Movie recording capabilities.
• Supports socket classes.
• Supports sound recording and playback devices.
• Fourier transform support.
• Threads support.
• Modeling of acoustic response of elements and array of elements.
• Distortion grids that allow display of warped, time changing images.
• Initial support for interfacing with Simrad ME70 multibeam sonar.

The GeoZui-4D task is blending more and more with our Chart of the Future (GeoNAV), Midwater Fish, and AUV tasks. Further developments of GeoZui-4D will be discussed under those headings.

Related Publications for GeoZui4D

AUV Mission Planning and Control and Simulation (Arsenault and Bogucki)

Whereas much of our past visualization efforts have been focused on the 3-D interactive display of static features like the seafloor, with the evolution to GeoZui-4D we have been able to add dynamic, time-varying systems. As part of this effort, and in support of our long-term goal of evaluating the potential of Automous Underwater Vehicles for the collection of hydrographic and seafloor characterization data, the visualization group has been developing 3-D tools for the planning, monitoring and review of AUV missions. In the past year much of this work focused on our close collaboration with Dr. Art Trembanis at the University of Delaware and his FETCH 3 AUV (called DOERRI). With our initial access to the DOERRI AUV we quickly learned that it suffered from serious problems with its control and operation software (e.g. it had difficulty flying a straight path, maintaining constant depth, etc.). To address these issues graduate student Robert Bogucki and Roland Arsenault worked in collaboration with University of Delaware and the AUV manufacturer (PRISM) to build new control software as well as a simulator to test their developments. As part of this effort Arsenault implemented a navigation module for DOERRI that reads serial sensors (GPS, DVL, Attitude, etc.) and provides integrated navigation fixes to AUV control modules and sensors. Real-time visualization tools were developed using GeoZui-4D for tracking DOERRI on the surface as well as for providing submerged navigation from a Linkquest Tracklink system (Figure 5.).

Figure 5. Example of real-time display of AUV navigation in GeoZui-4D

As the DOERRI vehicle was being tested it quickly became clear that it has severe problems with its control software. Addressing these issues, Bogucki developed and tested new C++ and LabView code and modified the existing software and operating system configuration. The vehicle was reconfigured to allow for a multiple codebase branch development with an independent operating system configuration used for each individual branch using a 'multi-boot' partition setup. The DOERRI LabView-based software was restructured for greater modularity, testability and understandability. Given the limited access to the vehicle as well as the danger of testing new control software on an AUV that is out of visible or communication range, a simulation module was developed based on a third party visualization library. This allowed for initial testing of the AUV's control code separately from the vehicle's hardware, using a simple kinematic model of the vehicle.

The approach fixed problems with the vehicle's actuator control thus improving the AUV's responsiveness to operator or control system originated commands. Joystick control of rudder, elevators and throttle were implemented and code was modified to streamline communications with a topside computer. These improvements translated to safer launch/recovery in real deployment conditions. Additionally data logging subroutines were developed for individual onboard instruments and modules created for new AUV behaviors such as multi-stage descent, waypoint seeking, station keeping and surface autopilot.

Software was also developed to better control the AUV while on the surface by providing visualization and telemetry to the AUV pilot. This visualization was integrated into a wireless AUV pilot setup based on a head mounted display and an ultra-portable computer. These software developments and implementations have been tested in a number of environments including the Center's test tank, Mendums Pond, NH; Pepper Creek, DE, and the mouth of Delaware Bay at Lewes, DE. A deep-sea deployment of the vehicle took place in the Black Sea, Ukraine as part of Byzantium 2007 underwater archaeological expedition in conjunction with Bob Ballard, Dwight Coleman, National Geographic and the NOAA Office of Ocean Exploration. Unfortunately the vehicle catastrophically failed during this expedition. This will be discussed later on in the report.

A particularly useful result of our AUV efforts has been the development of the AUV simulator toolkit which allows the testing of software changes on the behavior of the vehicle without risk to the AUV. This code has now been ported to Python, while integrating the visualization with a third-party physics engine library. Within the new framework, the vehicle's mass distribution and geometrical extents can be approximated with a collection of geometrical primitives - independently of the more complex 3D meshes used to present the vehicle to the user visually. The simulated AUV can be placed in a virtual environment alongside fish models and arbitrary bathymetry. Vehicles, fish and other objects can be assigned masses and moments of inertia, and subjected to the influence of specified forces and torques. Newtonian physics solutions and collision detection are provided by the physics engine, allowing for a more realistic specification of the simulated vehicle's behavior. The physics based approach is much more flexible and vehicle-agnostic, potentially allowing us to use the toolkit for vehicles with diametrically different propulsion and actuation schemes as long as some information about the physical forcing is available.

Deformable geometry objects have also been implemented in the simulator, with the purpose of modeling configurations of marine organisms with dynamically changing spatial extents (Figure 6.). The position of vertices specifying the geometry is tied to node objects with "physical" properties, allowing for their repositioning through user interactions (utilizing ray-casting techniques for picking objects and applying forces to them) or algorithmically (custom callback functions for position or force/torque application). The toolkit can generate a real-time simulated output of a simple single-beam echosounder based on the scene geometry and the vehicle's position and orientation relative to other objects in the environment. A virtual AUV-mounted camera provides imagery of the fish school as "seen" by the vehicle. User interaction with the simulation in real-time is now supported through mouse-picking and support for 3DConnexion Space Navigator six degree of freedom input device simultaneously.

Figure 6. AUV Simulator with seafloor and physics-based fish

The above innovations will ultimately allow for developing a suite of simulated AUV instruments of greater sophistication-incorporating more realistic models (sonar, water medium, and sound propagation) as well as more realistic vehicle and fish school models. Rapid development of new functionality and integration with other solutions are now feasible due to the transition to a high level scripting language. Increased degree of simulation realism through physics computation will allow the verification of the validity and robustness of simplified models used within the vehicle's onboard control code. While the loss of the DOERRI Fetch was a setback, NOAA's Southwest Fisheries Center in La Jolla owns another FETCH vehicle designed for fisheries research and operated under the supervision of David Demar and Center graduate Randy Cutter. Robert Bogucki is now working with this group to continue with these developments and to produce further developments focused specifically on their research needs and in particular, the adaptive mapping of fish schools.

Whale Tracking

Another particularly exciting aspect of GeoZui-4D has been its application to visualizing the underwater behavior of humpback whales supporting both basic science and policy as humpback whales are an endangered species whose decline are attributed to ship collisions and fishing gear entanglement. Understanding their underwater behavior is essential to mitigating both of these causes. NOAA and WHOI scientists have developed suction-cup-mounted tags that are attached to a whale and record depth, pitch, roll, and sound for as long as the tag remains on the whale (the record is now 22 hours). Our visualization team has taken these data and created fully georeferenced 4-D display of the whale's diving and swimming behavior in the context of the bathymetry, other vessels, and ambient sounds (Fig. 7). A vessel tracking component combines digital data from radar and AIS with visual sightings to better understand the effect of vessels on whale behavior. The result has provided unprecedented insight into the diving and feeding patterns of the whales as well as their response to the approach of vessels. Numerous papers on, and demos of, this technology have been presented at both scientific and policy meetings.

This past year, improvements made to the whale tracking software include:

• Obtained calibration files and .dtg files. This allows a deeper look into the raw data instead of relying only on processed Matlab files. l sequences.
• libDTAG: A library was built to directly access .dtg files. This allowed converters to be built to convert DTAG data to ogg, flac and other formats.

Ware, Arsenault, Weber and Schmidt participated in another Stellwagen Marine National Sanctuary organized whale tagging cruise in July of 2007. During this cruise acoustic pingers developed at the Center were deployed to allow the tracking of the tagged whales while submerged (see discussion of the work of Val Schmidt and Tom Weber later in the report) and exploring the behavior of multiple animals (tagged with hydrophones) in the same foraging group. This work, being developed by Schmidt into a Master's Thesis, is showing very promising initial results, having provided underwater whale tracks and important whale energetic metrics such as swimming speed (which in the past could only be estimated). We have preliminary but credible evidence of humpback whale speeds up to 12 knots. In addition we can corroborate whale feeding methods that have previously been hypothesized and can place them in the context of the local bathymetry and seafloor type. The results from these tags are being analyzed and incorporated into Trackplot - the software developed at the lab for the analysis of whale behavior. The development of improved hardware mounts for the pingers have turned into a senior project for a group of undergraduate mechanical engineering students (Tom Weber, advisor).

Related Publications for Whale Tracking

Flow Visualization

The incorporation of flow visualization models into the GeoZui-4D environment has opened of a range of applications and interest from ocean and current modelers both inside and outside of NOAA. Our goal is to provide tools that allow both researchers and members of the public to better understand the output from flow models. This is important to NOAA because of the increase in the number and quality of global, ocean, and estuarine flow models. These models are becoming critical to interpreting and generalizing physical oceanographic data, understanding marine ecologies, understanding weather and climate prediction. The flow visualization work is being carried out by Ware and graduate student Daniel Pineo; partial funding for this work has also been provided by NSF. Ware is also building the beginnings of a new flow visualization package to deal with sigma coordinate models. He is working closely with NOAA (and other) modelers and currently the new visualization package can load data from the following models:

• Cbofs (Chesapeake Bay)
• Gbofs (Galveston Harbor)
• Nyofs (New York Harbor)
• Gulf of Mexico

The flow visualization package can display salinity or temperature profiles and supports an exciting array of particle tracing methods. It also allows for 2D or 3D viewing.

Significant enhancements this year include the transformation from a desktop to server application and the development by Pineo of a method of automatic flow illustration using a biased advection technique. This method produces a simplified flow visualization that enhances the major flow features within the viewing area. The result is a visualization of complex flow in a form previously only achievable by human illustrators. The file loader of the flow visualization software has also been enhanced to allow time-varying, multi-layered atmospheric data, and internal data structures have been optimized resulting in significant improvements in performance.

There are now four sub-projects within the flow visualization initiative:

Museum Displays/ Smithsonian Global Flow Visualization

Following on our successful development of a kiosk-based interactive 3-D museum exhibit for Seacoast Science Center (GeoExplorer) which allows an interactive tour through an immersive 3-D environment up and under the Piscataqua River, stopping at interesting sights along the way, Ware has developed a prototype touch-screen display that incorporates flow models for the Pisquataqua River, Great Bay and Little Bay Estuary. The display shows the flow of water in the Estuary as a function of tides and currents. Wherever the screen is touched a bright dye is injected into the system and the observer can see the fate of the injected particles over several current and tidal cycles. In another museum-based effort, work with Kate Raisz of Northern Lights continues on the Smithsonian Science-on-a-Sphere visualization of global flow patterns for the new Smithsonian Oceans Hall. The design for this exhibit is near completion and will be released in 2008. The same code will be used with SkyScan to visualize ocean processes in planetaria through a recently acquired Granite State Technology Innovation Grant.

FlowVis2D/NOAA Nowcoast

Colin Ware and NOAA employee John Kelley continue to work on a project that will use the FlowVis2D software to create innovative and more effective ways of presenting NOAA flow model output (from the HYCOM system of models) to the general public. Briana Sullivan will be helping with evaluation and technology transfer. This effort has attracted some attention within NOAA. Carlos Lozano at NOAA's National Center for Environmental Prediction (NCEP) is also evaluating the FlowVis2D software as a tool for visualizing their models.

Visualization of Coupled Atmosphere and Ocean Model Output for Hurricane Forecasting

There is considerable interest in being able to visualize the atmosphere and the ocean data in a single visualization. While this project is just beginning, it has the support of Naomi Surgi who is in charge of hurricane modeling for forecasting at NCEP. Hurricane Katrina data has been obtained. The package being developed is being called FlowVis-4D. The SkyScan project (mentioned above) will also involve a visualization of Hurricane Katrina based on the same data.

Integrating Marine Mammal Data with Flow model output

This project is just beginning. The idea is to integrate the mammal tracking visualization capabilities with flow visualization capabilities using sigma coordinate models. This tool should be able to evaluate the behavior of tagged targets within the context of oceanographic processes and may be ideally suited for exploring time series data from ocean observatories.

Mid-water Fish

One of the most exciting recent advances has been our adaptation of a new generation of multibeam sonars to allow the real-time visualization of targets in the water column. While aspects of this fall under our visualization theme, you can read more about it on it's own Mid-Water Mapping webpage.