UI and Visualization

Table of Contents

• UI architecture overview
• QML overlay system
• GL Studio integration
• Camera and view management
• Role configuration for UI components
• Best practices

VR-Engage provides a layered UI architecture that combines Qt/QML for modern 2D overlays, GL Studio for high-fidelity cockpit instruments, and a flexible camera system for managing player perspectives. This section covers creating custom heads-up displays, binding UI elements to simulation state, integrating third-party instrument panels, and controlling camera views for different player roles.

UI architecture overview

The VR-Engage UI system renders multiple layers on top of the 3D scene, each serving distinct purposes. The base layer renders the 3D world including terrain, entities, and environmental effects. Above this, 2D overlay layers display HUD elements, instrument panels, and menus. The topmost layer handles cursor rendering and modal dialogs.

flowchart TB
   Cursor["Cursor Layer<br/>(Mouse, Interaction)"]
   Overlay["UI and Overlay Layer<br/>(HUDs, Menus, GL Studio Cockpits)"]
   Scene["3D Scene Layer<br/>(Terrain, Entities, Effects)"]

   Cursor --- Overlay --- Scene

Each layer operates independently but shares access to common data sources. The Player Attribute Store provides the primary mechanism for passing simulation data to UI elements. Components publish values like speed, heading, and system status to the attribute store, and UI elements bind to these values for automatic updates. This decoupling enables UI development without modifying simulation components, and allows multiple UI elements to display the same data without coordination. For a deeper discussion of attribute naming, lifetime, and performance patterns, see the Player Attribute Store section of the Player Station Framework page.

The rendering order produces correct compositing. The 3D scene renders first, establishing the world view. Overlay elements render with appropriate blending to appear on top of the scene. QML and GL Studio elements render within the overlay layer based on their z-order configuration. The cursor layer renders last to ensure it remains visible above all other content.

QML overlay system

QML provides the primary technology for creating VR-Engage HUDs and menus. Qt's declarative syntax enables rapid UI development with automatic property binding, while the underlying C++ integration provides access to simulation data and component state.

QML overlay configuration

QML overlays in VR-Engage are configured through the role definition Lua files rather than loaded programmatically. The role definition specifies which QML files to display and how to bind their properties to the Player Attribute Store. This configuration-driven approach allows UI customization without code changes.

-- filepath: data/simulationModelSets/examples/qmlHud/roles/driverOverlay.lua
 
components = {
   ["indicatorOverlay"] = {
      componentType = "DtQtQuickOverlay";
      priority = 10;
 
      -- Path to the QML file to load
      qmlFilename = "examples/exampleQmlHudDriver.qml";
 
      -- Name of the QML object that owns the bound properties
      qmlDataItemObjectName = "stateItem";
 
      -- Map QML state properties to attribute store paths
      bindQmlPropertyToAttribute = {
         mobilityKill = "vehicle-status.mobility-kill";
         firePowerKill = "vehicle-status.firepower-kill";
      };
   };
};

The QML file defines a state Item with properties that the framework updates from the attribute store based on the bindQmlPropertyToAttribute mapping and the qmlDataItemObjectName setting. This pattern decouples the QML layout from specific component implementations.

// filepath: data/UI/examples/exampleQmlHudDriver.qml
import QtQuick 2.15
import QtGraphicalEffects 1.15
 
Item {
   id: root
 
   // Item with properties updated from PlayerStation state attributes.
   // Properties are mapped to attributes in the bindQmlPropertyToAttribute
   // table of a DtQtQuickOverlay component in the role definition.
   Item {
      id: state
      objectName: "stateItem"
 
      // Properties updated from PlayerStation attributes
      property bool mobilityKill: false
      property bool firePowerKill: false
 
      // Called each frame after property updates
      signal tick(double dt)
 
      onMobilityKillChanged: {
         if (state.mobilityKill)
            mobilityKillFlash.restart();
         else
            mobilityKillFlash.complete();
      }
 
      onFirePowerKillChanged: {
         if (state.firePowerKill)
            firePowerKillFlash.restart();
         else
            firePowerKillFlash.complete();
      }
   }
 
   // Visual elements and animations react to the bound properties.
   // For example, mobilityKillFlash and firePowerKillFlash change
   // overlay colors when the corresponding attributes become true.
}

Publishing to the attribute store

Components publish data to the attribute store using template methods and hierarchical paths. The attribute store uses bracket notation to navigate the hierarchy and template methods for type-safe access.

flowchart LR
   Components["Player Components"]
   Attrs["Player Attribute Store"]
   Qml["QML Overlays"]
   Gls["GL Studio Cockpits"]

   Components --> Attrs
   Attrs --> Qml
   Attrs --> Gls

// filepath: src/vehicleStateComponent.cxx
#include "vrePlayerStation/playerStation.h"
#include "vreUtil/attribute.h"
 
bool DtVehicleStateComponent::postInitialize()
{
   if (!DtPlayerComponent::postInitialize())
   {
      return false;
   }
 
   // Create attribute hierarchy in postInitialize (after all components exist)
   // Use bracket notation to navigate and create the hierarchy
   myVehicleAttrs = playerAttributeStore()["vehicle"];
   myVehicleAttrs["speed"]->set<double>(0.0);
   myVehicleAttrs["heading"]->set<double>(0.0);
   myVehicleAttrs["engine-active"]->set<bool>(false);
 
   // Create damage status hierarchy
   myDamageAttrs = playerAttributeStore()["vehicle-status"];
   myDamageAttrs["mobility-kill"]->set<bool>(false);
   myDamageAttrs["firepower-kill"]->set<bool>(false);
 
   return true;
}
 
void DtVehicleStateComponent::tick(double deltaTime)
{
   // Update attributes - QML bindings automatically receive new values
   double speed = calculateSpeed();
   myVehicleAttrs["speed"]->set<double>(speed);
 
   double heading = calculateHeading();
   myVehicleAttrs["heading"]->set<double>(heading);
}

Reading from the attribute store uses similar syntax with get<T>() or getOr<T>() methods:

// filepath: src/damageDisplayComponent.cxx
void DtDamageDisplayComponent::tick(double deltaTime)
{
   // Read attribute values with default fallback
   bool mobilityKill = playerAttributeStore()["vehicle-status"]["mobility-kill"]->getOr<bool>(false);
   bool firepowerKill = playerAttributeStore()["vehicle-status"]["firepower-kill"]->getOr<bool>(false);
 
   // Use values to update display state
   updateDamageIndicators(mobilityKill, firepowerKill);
}

Handling input for UI toggles

Components can respond to user input to toggle HUD visibility or change UI modes. Input handling uses the DtInputLogic helper class with addActionHandler() to register callbacks for named actions.

// filepath: src/toggleableHudComponent.cxx
#include "vrePlayerStation/playerStation.h"
#include "vrePlayerStation/inputLogic.h"
#include "vreInput/vreInputManager.h"
 
bool DtToggleableHudComponent::initialize(DtPlayerStation* player, DtInitTable& config)
{
   if (!DtPlayerComponent::initialize(player, config))
   {
      return false;
   }
 
   // Get the input group name from config (defined in role's input mapping)
   std::string inputGroup = config.lookupString("inputGroup", "hud-controls");
 
   // Register action handler using input logic helper
   myInputLogic.addActionHandler(inputGroup, "toggle-hud",
      DtActionDelegate(this, &DtToggleableHudComponent::onToggleHud));
 
   myHudVisible = true;
 
   return true;
}
 
void DtToggleableHudComponent::onToggleHud(float value)
{
   // Action triggers on button press (value > 0.5)
   if (value > 0.5f)
   {
      myHudVisible = !myHudVisible;
 
      // Update attribute store - QML overlay can bind to this
      playerAttributeStore()["hud"]["visible"]->set<bool>(myHudVisible);
   }
}
 
void DtToggleableHudComponent::shutdown()
{
   // Input logic automatically cleans up registered handlers
   DtPlayerComponent::shutdown();
}

The corresponding input mapping in the role configuration binds keyboard or controller inputs to the action:

-- filepath: data/simulationModelSets/VR-Engage/roles/components/hudInputMapping.lua
 
inputMappings = {
   ["hud-controls"] = {
      ["toggle-hud"] = {
         keyboard = { key = "H"; };
      };
   };
};

GL Studio integration

GL Studio provides high-fidelity 2D and 3D cockpit instrumentation with realistic gauge rendering, dial movements, and warning indicators. VR-Engage leverages GL Studio integration through the VR-Vantage display engine, which provides the rendering infrastructure for GL Studio cockpit overlays.

GL Studio design workflow

The typical GL Studio workflow begins in the GL Studio editor, where artists create instrument designs with named variables for dynamic elements. These variables represent inputs like airspeed, altitude, or warning states. The editor exports the design as a runtime component that VR-Vantage loads and renders.

flowchart LR
    subgraph Design["GL Studio Editor"]
        Create["Create Instruments"]
        Variables["Define Variables"]
        Export["Export Runtime"]
    end

    subgraph Integration["VR-Vantage / VR-Engage"]
        Load["Load via Role Config"]
        Bind["Bind via Attribute Store"]
        Render["Render Overlay"]
    end

    subgraph Runtime["Simulation"]
        Components["Player Components"]
        Attrs["Attribute Store"]
        Tick["Frame Update"]
    end

    Create --> Variables
    Variables --> Export
    Export --> Load
    Load --> Bind
    Bind --> Render

    Components --> Attrs
    Attrs --> Bind
    Tick --> Render

Configuring GL Studio cockpits

GL Studio cockpits are packaged and configured as VR-Vantage HUDs rather than as VR-Engage components. VR-Engage roles do not instantiate a dedicated DtGlStudioOverlay component. Instead, VR-Engage components publish values such as flight/airspeed-kts, flight/altitude-ft, flight/heading-deg, and warnings/master to the Player Attribute Store. VR-Vantage's GL Studio integration (libraries like vrvGlStudio and vrvGlStudioShared) binds those attributes to GL Studio variables according to the cockpit definition; refer to the VR-Vantage GL Studio documentation for cockpit configuration details.

Updating cockpit instruments from components

Components update cockpit instruments by publishing to the attribute store. The GL Studio integration layer automatically reads bound attributes and updates the corresponding GL Studio variables each frame.

// filepath: src/flightInstrumentsComponent.cxx
#include "vrePlayerStation/playerStation.h"
#include "vreUtil/attribute.h"
 
bool DtFlightInstrumentsComponent::postInitialize()
{
   if (!DtPlayerComponent::postInitialize())
   {
      return false;
   }
 
   // Create attribute hierarchy for flight instruments
   myFlightAttrs = playerAttributeStore()["flight"];
   myFlightAttrs["airspeed-kts"]->set<double>(0.0);
   myFlightAttrs["altitude-ft"]->set<double>(0.0);
   myFlightAttrs["heading-deg"]->set<double>(0.0);
 
   myWarningAttrs = playerAttributeStore()["warnings"];
   myWarningAttrs["master"]->set<bool>(false);
 
   return true;
}
 
void DtFlightInstrumentsComponent::tick(double deltaTime)
{
   // Calculate flight parameters from entity state
   const DtEntityState* state = entityState();
   if (!state)
   {
      return;
   }
 
   // Update airspeed (m/s to knots)
   double airspeedKts = state->velocity().magnitude() * 1.94384;
   myFlightAttrs["airspeed-kts"]->set<double>(airspeedKts);
 
   // Update altitude (meters to feet)
   double altitudeFt = state->position().z() * 3.28084;
   myFlightAttrs["altitude-ft"]->set<double>(altitudeFt);
 
   // Update heading (radians to degrees, normalized 0-360)
   double headingDeg = state->orientation().heading() * 57.2957795;
   if (headingDeg < 0.0)
   {
      headingDeg += 360.0;
   }
   myFlightAttrs["heading-deg"]->set<double>(headingDeg);
 
   // Update warning state based on system conditions
   bool warningActive = checkWarningConditions();
   myWarningAttrs["master"]->set<bool>(warningActive);
}

GL Studio input handling

GL Studio cockpits can include interactive elements like buttons, switches, and knobs. When users interact with these elements, the GL Studio runtime generates events. VR-Vantage's GL Studio event integration (for example, through the vreGlsEventSignaler plugin) can publish these interactions into the Player Attribute Store under paths such as cockpit-events/master-arm-clicked. VR-Engage components then treat them as standard state attributes and can respond to cockpit interactions through attribute callbacks.

// filepath: src/cockpitInteractionComponent.cxx
bool DtCockpitInteractionComponent::postInitialize()
{
   if (!DtPlayerComponent::postInitialize())
   {
      return false;
   }
 
   // Watch for cockpit interaction events via attribute store callbacks
   myCockpitEvents = playerAttributeStore()["cockpit-events"];
 
   // Register callback for master arm switch clicks
   myCockpitEvents["master-arm-clicked"]->addCallback(
      [this](DtAttribute& attr) {
         if (attr.getOr<bool>(false))
         {
            handleMasterArmToggle();
            // Reset the event flag
            attr.set<bool>(false);
         }
      });
 
   return true;
}
 
void DtCockpitInteractionComponent::handleMasterArmToggle()
{
   myMasterArmed = !myMasterArmed;
 
   // Update state for cockpit visual feedback
   playerAttributeStore()["weapons"]["master-armed"]->set<bool>(myMasterArmed);
}

Camera and view management

VR-Engage provides camera control for managing player viewpoints through the VR-Vantage display engine. The view system supports multiple camera modes including cockpit views, external views, and sensor views. Camera behavior is typically configured through the role definition rather than programmatically controlled.

View system architecture

VR-Engage roles define view configurations that specify camera attachment points, field of view, and rendering parameters. The display engine manages view channels and camera positioning based on these configurations.

Configuring views in role definitions

View configurations specify which attachment point the camera uses, field of view settings, and other rendering parameters. Multiple views can be defined for different display channels or switchable camera modes.

-- filepath: data/simulationModelSets/VR-Engage/roles/tankCommander.lua
 
displayLayouts = {
   ["1 Screen Horizontal"] = {
      channelLayout = {
         { name = "main"; x = 0; y = 0; width = 1; height = 1; };
      };
   };
   ["2 Screen Horizontal"] = {
      channelLayout = {
         { name = "main"; x = 0; y = 0; width = 0.5; height = 1; };
         { name = "sensor"; x = 0.5; y = 0; width = 0.5; height = 1; };
      };
   };
};
 
views = {
   ["commanderView"] = {
      attachPoint = "commander_eye";
      fieldOfView = 60.0;
      nearClip = 0.1;
      farClip = 50000.0;
   };
   ["sensorView"] = {
      attachPoint = "sight_optic";
      fieldOfView = 10.0;  -- Narrow FOV for magnified view
      nearClip = 1.0;
      farClip = 20000.0;
   };
};

Look-around and head tracking

Many roles support look-around capability, allowing the player to rotate their view within the cockpit. This is typically handled by dedicated control logic components that process mouse or joystick input and adjust the view orientation relative to the attachment point.

// filepath: src/commanderControlLogic.cxx
#include "vrePlayerStation/playerStation.h"
#include "vrePlayerStation/inputLogic.h"
 
bool DtCommanderControlLogic::initialize(DtPlayerStation* player, DtInitTable& config)
{
   if (!DtEntityControlLogic::initialize(player, config))
   {
      return false;
   }
 
   // Read look-around configuration
   mySensitivity = config.lookupDouble("lookSensitivity", 1.5);
   myMaxPitch = config.lookupDouble("maxPitch", 80.0) * 0.0174533; // degrees to radians
   myMaxYaw = config.lookupDouble("maxYaw", 150.0) * 0.0174533;
 
   // Register look-around input handlers
   std::string inputGroup = config.lookupString("inputGroup", "commander");
   myInputLogic.addActionHandler(inputGroup, "look-yaw",
      DtActionDelegate(this, &DtCommanderControlLogic::onLookYaw));
   myInputLogic.addActionHandler(inputGroup, "look-pitch",
      DtActionDelegate(this, &DtCommanderControlLogic::onLookPitch));
   myInputLogic.addActionHandler(inputGroup, "reset-view",
      DtActionDelegate(this, &DtCommanderControlLogic::onResetView));
 
   return true;
}
 
void DtCommanderControlLogic::onLookYaw(float value)
{
   myYawInput = value;
}
 
void DtCommanderControlLogic::onLookPitch(float value)
{
   myPitchInput = value;
}
 
void DtCommanderControlLogic::onResetView(float value)
{
   if (value > 0.5f)
   {
      myCurrentYaw = 0.0;
      myCurrentPitch = 0.0;
   }
}
 
void DtCommanderControlLogic::tick(double deltaTime)
{
   DtEntityControlLogic::tick(deltaTime);
 
   // Apply look input with sensitivity
   myCurrentYaw += myYawInput * mySensitivity * deltaTime;
   myCurrentPitch += myPitchInput * mySensitivity * deltaTime;
 
   // Clamp to configured limits
   myCurrentYaw = std::clamp(myCurrentYaw, -myMaxYaw, myMaxYaw);
   myCurrentPitch = std::clamp(myCurrentPitch, -myMaxPitch, myMaxPitch);
 
   // Publish view orientation to attribute store for display engine
   playerAttributeStore()["view"]["yaw"]->set<double>(myCurrentYaw);
   playerAttributeStore()["view"]["pitch"]->set<double>(myCurrentPitch);
}

Switching between view modes

Roles can define multiple view modes that the player switches between using keyboard shortcuts or cockpit controls. View switching is typically handled by publishing the active view mode to the attribute store.

// filepath: src/viewSwitchComponent.cxx
bool DtViewSwitchComponent::initialize(DtPlayerStation* player, DtInitTable& config)
{
   if (!DtPlayerComponent::initialize(player, config))
   {
      return false;
   }
 
   std::string inputGroup = config.lookupString("inputGroup", "view-controls");
 
   myInputLogic.addActionHandler(inputGroup, "view-cockpit",
      DtActionDelegate(this, &DtViewSwitchComponent::onViewCockpit));
   myInputLogic.addActionHandler(inputGroup, "view-external",
      DtActionDelegate(this, &DtViewSwitchComponent::onViewExternal));
   myInputLogic.addActionHandler(inputGroup, "view-sensor",
      DtActionDelegate(this, &DtViewSwitchComponent::onViewSensor));
 
   return true;
}
 
void DtViewSwitchComponent::onViewCockpit(float value)
{
   if (value > 0.5f)
   {
      playerAttributeStore()["view"]["mode"]->set<std::string>("cockpit");
   }
}
 
void DtViewSwitchComponent::onViewExternal(float value)
{
   if (value > 0.5f)
   {
      playerAttributeStore()["view"]["mode"]->set<std::string>("external");
   }
}
 
void DtViewSwitchComponent::onViewSensor(float value)
{
   if (value > 0.5f)
   {
      playerAttributeStore()["view"]["mode"]->set<std::string>("sensor");
   }
}

Role configuration for UI components

UI components are configured through the standard role configuration system. The role definition specifies which HUD and cockpit components to load, and configuration parameters control their behavior and appearance.

Defining UI component groups

Role configuration files often use reusable Lua modules ("UI component groups") that contain HUD and cockpit components. Multiple such modules can be pulled into a role via the standard includes mechanism to create complete configurations with appropriate HUDs and instruments.

-- filepath: data/simulationModelSets/VR-Engage/roles/components/pilotHud.lua
 
components = {
   -- QML-based flight HUD overlay
   ["flightHud"] = {
      componentType = "DtQtQuickOverlay";
      qmlFilename = "UI/overlays/flightHud.qml";
      qmlDataItemObjectName = "stateItem";
      bindQmlPropertyToAttribute = {
         airspeed = "flight/airspeed-kts";
         altitude = "flight/altitude-ft";
         heading = "flight/heading-deg";
         masterWarning = "warnings/master";
      };
   };
 
   -- Component that publishes flight data to attribute store
   ["flightInstruments"] = {
      componentType = "DtFlightInstrumentsComponent";
   };
};

-- filepath: data/simulationModelSets/VR-Engage/roles/components/pilotCockpit.lua
 
components = {
   -- Look-around control
   ["pilotViewControl"] = {
      componentType = "DtPilotControlLogic";
      inputGroup = "pilot";
      lookSensitivity = 1.5;
      maxPitch = 80.0;
      maxYaw = 150.0;
   };
};

Complete role definition

The role definition assembles these UI-related components (whether defined locally or brought in via includes) into a complete player configuration. UI components typically load after core control components to ensure data sources are available.

-- filepath: data/simulationModelSets/VR-Engage/roles/fighterPilot.lua
 
displayLayouts = {
   ["1 Screen Horizontal"] = {
      channelLayout = {
         { name = "main"; x = 0; y = 0; width = 1; height = 1; };
      };
   };
};
 
components = {
   -- Core control components
   ["pilotControlLogic"] = {
      componentType = "pilotControlLogic";
   };
 
   ["flightModel"] = {
      componentType = "flightModel";
   };
 
   -- Weapon systems
   ["weaponSystem"] = {
      componentType = "weaponSystem";
   };
 
   ["targetingSystem"] = {
      componentType = "targetingSystem";
   };
 
   -- UI and visualization
   ["pilotHud"] = {
      componentType = "pilotHud";
   };
 
   ["pilotCockpit"] = {
      componentType = "pilotCockpit";
   };
 
   -- Sensors and auxiliary displays
   ["radarDisplay"] = {
      componentType = "radarDisplay";
   };
};

Best practices

Efficient UI development in VR-Engage requires attention to performance, maintainability, and user experience. The following guidelines reflect lessons learned from production deployments.

Performance optimization

UI rendering can significantly impact frame rate if not managed carefully. The most common performance issues involve excessive attribute store updates, unnecessary QML property evaluations, and inefficient data access.

Minimize attribute update frequency by using change detection. Rather than updating every attribute every frame, track previous values and only call set<T>() when values actually differ. For rapidly changing values like position, consider update rate limiting or dead-band filtering.

// filepath: src/efficientHudComponent.cxx
void DtEfficientHudComponent::tick(double deltaTime)
{
   // Accumulate time for rate-limited updates
   myUpdateAccumulator += deltaTime;
 
   // Update rapidly-changing values at reduced rate (e.g., 20 Hz instead of 60 Hz)
   if (myUpdateAccumulator >= myUpdateInterval)
   {
      myUpdateAccumulator = 0.0;
      updateDynamicValues();
   }
 
   // Update slow-changing values only on actual change
   updateStaticValues();
}
 
void DtEfficientHudComponent::updateDynamicValues()
{
   double newSpeed = calculateSpeed();
 
   // Dead-band filter: only update if change exceeds threshold
   if (std::abs(newSpeed - myLastSpeed) > 0.1)
   {
      myLastSpeed = newSpeed;
      mySpeedAttr->set<double>(newSpeed);
   }
}
 
void DtEfficientHudComponent::updateStaticValues()
{
   // Check warning conditions (these change infrequently)
   bool warning = checkWarningConditions();
   if (warning != myLastWarningState)
   {
      myLastWarningState = warning;
      myWarningAttr->set<bool>(warning);
   }
}

Use QML best practices including avoiding complex JavaScript expressions in bindings, preferring property bindings over imperative updates, and using Loader elements to defer creation of hidden UI sections.

Maintainability

Structure QML files for maintainability by separating concerns into focused components. Create reusable QML components for common UI elements like gauges, status indicators, and warning displays. Use Qt's resource system to organize QML files and ensure they deploy correctly.

// filepath: resources/components/Gauge.qml
import QtQuick 2.15
 
// Reusable gauge component
Item {
    id: gauge
 
    property real value: 0.0
    property real minValue: 0.0
    property real maxValue: 100.0
    property string label: "Gauge"
    property string units: ""
    property color normalColor: "white"
    property color warningColor: "yellow"
    property color criticalColor: "red"
    property real warningThreshold: 80.0
    property real criticalThreshold: 95.0
 
    width: 120
    height: 60
 
    property color currentColor: {
        if (value >= criticalThreshold) return criticalColor
        if (value >= warningThreshold) return warningColor
        return normalColor
    }
 
    Column {
        anchors.centerIn: parent
        spacing: 4
 
        Text {
            text: gauge.label
            color: "gray"
            font.pixelSize: 12
            anchors.horizontalCenter: parent.horizontalCenter
        }
 
        Text {
            text: gauge.value.toFixed(1) + " " + gauge.units
            color: gauge.currentColor
            font.pixelSize: 24
            font.bold: true
            anchors.horizontalCenter: parent.horizontalCenter
        }
    }
}

Accessibility and readability

Design HUDs for readability under varying conditions. Use sufficient contrast between text and background, provide configurable text sizes, and consider color-blind users when selecting indicator colors. Test UI layouts at different resolutions and aspect ratios to ensure elements remain visible and properly positioned.

See also

• Player Station Framework for component lifecycle
• Input and Control for input-driven UI updates
• Inter-Process Communication for backend data binding
• Role Configuration for overlay configuration in roles

Return to Toolkit Development