The recent workshop on Behavioral Machine Learning provided a pivotal platform for researchers focused on applying sophisticated algorithms to real-world health challenges. A presentation led by mDOT Center Researchers from the University of Massachusetts Amherst captured significant attention, introducing StepCountJITAI, a crucial new simulation environment.

The central challenge discussed was the persistent issue of data scarcity in real physical activity adaptive intervention studies. Deploying Reinforcement Learning (RL) methods in mobile health (mHealth) apps is costly and time-consuming, often resulting in limited interaction opportunities with participants over weeks or months. This constraint renders many standard, data-hungry RL algorithms ineffective for policy learning in this domain. The solution presented was the creation of an environment whose dynamics are highly relevant to adaptive interventions, unlike existing benchmarks.

Deep Dive into Behavioral Dynamics

The core of the discussion centered on how StepCountJITAI accurately models the complexities inherent in messaging-based physical activity Just-in-Time Adaptive Interventions (JITAIs). The environment goes far beyond simple step counts by incorporating two critical behavioral variables:

1. Habituation Level ($H$): This models how accustomed a participant becomes to receiving messages. As $H$ increases, the motivational messages become less effective, which consequently reduces the Step Count ($S$) reward.

2. Disengagement Risk ($D$): This variable measures the probability of a participant abandoning the study early. Crucially, if the intervention system sends messages that are poorly tailored (not matching the True Context, $C$), the disengagement risk increases. If $D$ exceeds a preset threshold ($\text{Dthreshold}$), the simulation episode terminates.

Furthermore, researchers highlighted the modeling of context uncertainty. In real studies, researchers must rely on inferred probabilities ($P$) about a participant’s state (e.g., ‘stressed’ vs. ‘not stressed’) rather than the true context ($C$). StepCountJITAI allows RL agents to train in an environment where context uncertainty ($\sigma$) is a selectable parameter, forcing them to learn policies that mitigate the risk of sending incorrectly contextualized messages.

The environment’s actions were detailed, showing the four possible intervention options the RL agent can choose: sending no message ($a=0$), sending a non-contextualized message ($a=1$), or sending a message customized to context 0 ($a=2$) or context 1 ($a=3$).

The presentation and associated work drew on foundational expertise in adaptive intervention policy learning. Although the primary authors are from UMass Amherst, the overall research initiative is supported by grants from the National Institutes of Health, including those often tied to mobile health optimization and behavioral machine learning centers, such as the mDOT Center research community.

Perceived Outcomes and Takeaways

The subsequent discussion focused heavily on the experimental results, which used StepCountJITAI to test various RL methods, including DQN, REINFORCE, and PPO, against Thompson Sampling (TS).

Key Outcomes of the Discussion:
• RL Superiority Confirmed (Under Ideal Conditions): The experiments demonstrated that the tested RL methods could effectively learn policies, achieving a high average return of approximately 3000, significantly outperforming standard TS (which achieved returns around 1500) in the tested configuration.

• The Crucial Role of Stochasticity: The environment’s capability to model stochasticity and between-person variability via Uniform or Beta distributions proved essential. Additional experiments showed that when using higher context uncertainty (e.g., $\sigma=0.8$) or higher stochasticity parameters in the dynamics (controlled by $a_{hd}, \sigma_s, a_{de}$), the average returns of all RL methods dropped significantly, confirming that the environment successfully simulates the real-world complexity that makes policy learning challenging.

• Accelerating Future Research: The consensus was highly positive regarding StepCountJITAI‘s implementation. By utilizing a standard API for RL (i.e., gymnasium), the simulation environment is immediately compatible with existing RL research workflows. This open-source implementation is poised to accelerate the development of new RL algorithms that are better tailored to address data scarcity in adaptive intervention optimization.

The general outcome of the session was clear: StepCountJITAI offers the research community a vital, standardized, and realistic platform to test the next generation of data-efficient RL algorithms for personalized mHealth interventions.