Online Bayesian Goal Inference
UCLA · Tao Gao Lab · 2023–2024Abstract
We model and infer agent goals within grid-based environments using Bayesian inference and reinforcement learning. This project explores how observers can predict goals from observed actions using structured probabilistic reasoning. We develop a real-time online visualization system that demonstrates goal inference as agents navigate toward unknown targets.
Figure 1: Overview of the Online Bayesian Goal Inference system. An agent navigates a grid world while a Bayesian observer updates goal posteriors in real time.
Method
The system consists of three main components:
- MDP Solver: Uses Value Iteration with Bellman updates to pre-compute optimal value functions for multiple hypothetical goals across the grid environment.
- Policy Extractor: Implements a gamma-discounted softmax policy with a temperature parameter for controlled action stochasticity, allowing the model to account for sub-optimal agent behavior.
- Bayesian Inference Engine: Performs real-time posterior updates over possible goals based on observed agent actions, using the extracted policies as likelihood functions.
Figure 2: System architecture showing the MDP solver, policy extractor, and Bayesian inference pipeline.
Results
Key achievements of this project include:
- Foundational MDP solver using Value Iteration and Bellman updates for pre-computing optimal value functions across multiple hypothetical goals
- Gamma-discounted softmax policy extractor with temperature parameter for controlled action stochasticity
- Robust Bayesian goal inference mechanism for real-time posterior updates based on observed actions
- Full-stack, low-latency online visualization system using Python (RL backend) and Node.js/Socket.io (frontend) for real-time demonstration
Interactive Demo
The full-stack visualization system demonstrates real-time goal inference as agents navigate grid environments. The backend runs reinforcement learning computations in Python, while the frontend provides an interactive visualization via Node.js and Socket.io.
Citation
@article{tang2024goalinference,
title = {Online Bayesian Goal Inference in Grid Environments},
author = {Tang, Yuer and Gao, Tao},
year = {2024},
institution = {UCLA}
}