Date of Award

5-25-2026

Document Type

Thesis

Degree Name

Master of Science (MS)

Department

Computer Science

First Advisor

Aryabrata Basu

Abstract

Eye tracking in modern XR head-mounted displays can capture high-frequency gaze data at 90–120 Hz, generating thousands of samples within a single 30-second interaction. At the same time, the XR market has grown to millions of active users worldwide, with major platforms such as Meta investing heavily in eye-tracking-enabled devices. Despite this large-scale adoption and data availability, most applications still rely on hand-controller input, with gaze either processed offline or reduced to a simple pointing signal. As a result, the majority of temporally rich gaze information remains unused in real-time interaction, limiting system responsiveness, reducing interaction fidelity, and preventing effective utilization of user attention in immersive environments. This research presents Virtual Intelligence for Gaze Observation and Representation(VIGOR), a real-time, gaze-aware XR framework that synchronizes eye tracking with head-based interaction. The system was developed in Unity and features a head-controlled cursor interface for structured tasks (fixation, saccade, and smooth pursuit). These tasks are designed to elicit dominant eye-movement behaviors. During interaction, binocular gaze data are continuously logged and temporally aligned with events such as dwell activation, button selection, and scene transitions. The raw eye-tracking signals are transformed into a compact binocular gaze direction representation, which serves as the input for temporal modeling. After evaluating multiple neural network architectures (i.e., LSTM, BiLSTM, TCN), a Temporal Convolutional Network was selected for its ability to model high-frequency gaze sequences efficiently. Temporal features are learned directly by the TCN through causal 1D convolutions, rather than using any explicit dimensionality reduction technique. The model achieves consistent performance in 80% training, 78% validation, and 78% test accuracy, with a minimal generalization gap, indicating stable learning. Higher per-class performance is observed for fixation ( 93%), while dynamic classes such as saccade and smooth pursuit remain more challenging. Despite minor confusion between dynamic classes, the system provides reliable real-time inference under XR constraints, as evidenced by consistent F1-score and recall and statistically stable bootstrap confidence intervals. To ensure scalability and privacy, the framework is designed to support federated learning, where each device performs local training and shares only model updates with a central server. Raw gaze data remain on the device, reducing privacy risks. The outcome is a unified XR framework that combines gaze logging in real-time, interaction synchronization, and on-device learning, supporting attention-aware applications and privacy-preserving behavioral analytics.

Download

Share

COinS