The mDOT Center

Transforming health and wellness via temporally-precise mHealth interventions
mDOT@MD2K.org
901.678.1526
 

TR&D2: Optimization

mDOT Center > Research Projects > TR&D2: Optimization

Dynamic Optimization of Continuously Adapting mHealth Interventions via Prudent, Statistically Efficient, and Coherent Reinforcement Learning

Designing Reinforcement Learning Algorithms for Digital Interventions: Pre-implementation Guidelines
Authors:
Publication Venue:

Algorithms in Decision Support Systems

Publication Date:

July 22, 2022

Keywords:

reinforcement learning, online learning, mobile health, algorithm design, algorithm evaluation

Related Project:
Abstract:
Online reinforcement learning (RL) algorithms are increasingly used to personalize digital interventions in the fields of mobile health and online education. Common challenges in designing and testing an RL algorithm in these settings include ensuring the RL algorithm can learn and run stably under real-time constraints, and accounting for the complexity of the environment, e.g., a lack of accurate mechanistic models for the user dynamics. To guide how one can tackle these challenges, we extend the PCS (predictability, computability, stability) framework, a data science framework that incorporates best practices from machine learning and statistics in supervised learning to the design of RL algorithms for the digital interventions setting. Furthermore, we provide guidelines on how to design simulation environments, a crucial tool for evaluating RL candidate algorithms using the PCS framework. We show how we used the PCS framework to design an RL algorithm for Oralytics, a mobile health study aiming to improve users’ tooth-brushing behaviors through the personalized delivery of intervention messages. Oralytics will go into the field in late 2022.
TL;DR:

Online RL faces challenges like real-time stability and handling complex, unpredictable environments; to address these issues, the PCS framework originally used in supervised learning is extended to guide the design of RL algorithms for such settings, including guidelines for creating simulation environments, as exemplified in the development of an RL algorithm for the mobile health study Oralytics aimed at enhancing tooth-brushing behaviors through personalized intervention messages.

Statistical Inference with M-Estimators on Adaptively Collected Data
Authors:
Publication Venue:

Advances in Neural Information Processing Systems

Publication Date:

December 2021

Keywords:

contextual bandit algorithms, confidence intervals, adaptively collected data, causal inference

Abstract:
Bandit algorithms are increasingly used in real-world sequential decision-making problems. Associated with this is an increased desire to be able to use the resulting datasets to answer scientific questions like: Did one type of ad lead to more purchases? In which contexts is a mobile health intervention effective? However, classical statistical approaches fail to provide valid confidence intervals when used with data collected with bandit algorithms. Alternative methods have recently been developed for simple models (e.g., comparison of means). Yet there is a lack of general methods for conducting statistical inference using more complex models on data collected with (contextual) bandit algorithms; for example, current methods cannot be used for valid inference on parameters in a logistic regression model for a binary reward. In this work, we develop theory justifying the use of M-estimators — which includes estimators based on empirical risk minimization as well as maximum likelihood — on data collected with adaptive algorithms, including (contextual) bandit algorithms. Specifically, we show that M-estimators, modified with particular adaptive weights, can be used to construct asymptotically valid confidence regions for a variety of inferential targets.
TL;DR:

We develop theory justifying the use of M-estimators—which includes estimators based on empirical risk minimization as well as maximum likelihood—on data collected with adaptive algorithms, including (contextual) bandit algorithms.

Batch Policy Learning in Average Reward Markov Decision Processes
Authors:
Publication Venue:

The Annals of Statistics

Publication Date:

December 21, 2022

Keywords:

average reward, doubly robust estimator, Markov Decision Process, policy optimization

Related Project:
Abstract:

We consider the batch (off-line) policy learning problem in the infinite horizon Markov decision process. Motivated by mobile health applications, we focus on learning a policy that maximizes the long-term average reward. We propose a doubly robust estimator for the average reward and show that it achieves semiparametric efficiency. Further, we develop an optimization algorithm to compute the optimal policy in a parameterized stochastic policy class. The performance of the estimated policy is measured by the difference between the optimal average reward in the policy class and the average reward of the estimated policy and we establish a finite-sample regret guarantee. The performance of the method is illustrated by simulation studies and an analysis of a mobile health study promoting physical activity.

TL;DR:

We consider batch policy learning in an infinite horizon Markov Decision Process, focusing on optimizing a policy for long-term average reward in the context of mobile health applications.

Data-driven Interpretable Policy Construction for Personalized Mobile Health
Authors:
Publication Venue:

IEEE International Conference on Digital Health (ICDH)

Publication Date:

July 10, 2022

Keywords:

learning systems, optimized production technology, behavioral sciences, electronic healthcare, decision trees

Related Project:
Abstract:

To promote healthy behaviors, many mobile health applications provide message-based interventions, such as tips, motivational messages, or suggestions for healthy activities. Ideally, the intervention policies should be carefully designed so that users obtain the benefits without being overwhelmed by overly frequent messages. As part of the HeartSteps physical-activity intervention, users receive messages intended to disrupt sedentary behavior. HeartSteps uses an algorithm to uniformly spread out the daily message budget over time, but does not attempt to maximize treatment effects. This limitation motivates constructing a policy to optimize the message delivery decisions for more effective treatments. Moreover, the learned policy needs to be interpretable to enable behavioral scientists to examine it and to inform future theorizing. We address this problem by learning an effective and interpretable policy that reduces sedentary behavior. We propose Optimal Policy Trees + (OPT+), an innovative batch off-policy learning method, that combines a personalized threshold learning and an extension of Optimal Policy Trees under a budget-constrained setting. We implement and test the method using data collected in HeartSteps V2N3. Computational results demonstrate a significant reduction in sedentary behavior with a lower delivery budget. OPT + produces a highly interpretable and stable output decision tree thus enabling theoretical insights to guide future research.

TL;DR:

Online RL faces challenges like real-time stability and handling complex, unpredictable environments; to address these issues, the PCS framework originally used in supervised learning is extended to guide the design of RL algorithms for such settings, including guidelines for creating simulation environments, as exemplified in the development of an RL algorithm for the mobile health study Oralytics aimed at enhancing tooth-brushing behaviors through personalized intervention messages.

Just-in-Time Adaptive Interventions for Suicide Prevention: Promise, Challenges, and Future Directions
Authors:
Publication Venue:

Psychiatry: Interpersonal and Biological Processes

Publication Date:

July 18, 2022

Keywords:

suicide, self-injury, just-in-time adaptive interventions

Abstract:

The suicide rate (currently 14 per 100,000) has barely changed in the United States over the past 100 years. There is a need for new ways of preventing suicide. Further, research has revealed that suicidal thoughts and behaviors and the factors that drive them are dynamic, heterogeneous, and interactive. Most existing interventions for suicidal thoughts and behaviors are infrequent, not accessible when most needed, and not systematically tailored to the person using their own data (e.g., from their own smartphone). Advances in technology offer an opportunity to develop new interventions that may better match the dynamic, heterogeneous, and interactive nature of suicidal thoughts and behaviors. Just-In-Time Adaptive Interventions (JITAIs), which use smartphones and wearables, are designed to provide the right type of support at the right time by adapting to changes in internal states and external contexts, offering a promising pathway toward more effective suicide prevention. In this review, we highlight the potential of JITAIs for suicide prevention, challenges ahead (e.g., measurement, ethics), and possible solutions to these challenges.

TL;DR:

In this review, we highlight the potential of JITAIs for suicide prevention, challenges ahead (e.g., measurement, ethics), and possible solutions to these challenges.

Engagement in Digital Interventions
Authors:
Publication Venue:

American Psychologist

Publication Date:

March 17, 2022

Keywords:

engagement, digital interventions, affect, motivation, attention

Related Project:
Abstract:

The notion of “engagement,” which plays an important role in various domains of psychology, is gaining increased currency as a concept that is critical to the success of digital interventions. However, engagement remains an ill-defined construct, with different fields generating their own domain-specific definitions. Moreover, given that digital interactions in real-world settings are characterized by multiple demands and choice alternatives competing for an individual’s effort and attention, they involve fast and often impulsive decision making. Prior research seeking to uncover the mechanisms underlying engagement has nonetheless focused mainly on psychological factors and social influences and neglected to account for the role of neural mechanisms that shape individual choices. This paper aims to integrate theories and empirical evidence across multiple domains to define engagement and discuss opportunities and challenges to promoting effective engagement in digital interventions. We also propose the AIM-ACT framework, which is based on a neurophysiological account of engagement, to shed new light on how in-the-moment engagement unfolds in response to a digital stimulus. Building on this framework, we provide recommendations for designing strategies to promote engagement in digital interventions and highlight directions for future research.

TL;DR:

This paper focuses on defining and understanding engagement in digital interventions by combining various theories and evidence from different domains. It introduces the AIM-ACT framework, which explains how engagement happens in response to digital stimuli based on neurophysiological principles and offers suggestions for designing effective engagement strategies in digital interventions.

The Microrandomized Trial for Developing Digital Interventions: Experimental Design and Data Analysis Considerations
Authors:
Publication Venue:

Psychological Methods

Publication Date:

January 13, 2022

Keywords:
Micro-randomized trial (MRT), health behavior change, digital intervention, just-in-time adaptive intervention (JITAI), causal inference, intensive longitudinal data
Related Project:
Abstract:
Just-in-time adaptive interventions (JITAIs) are time-varying adaptive interventions that use frequent opportunities for the intervention to be adapted-weekly, daily, or even many times a day. The microrandomized trial (MRT) has emerged for use in informing the construction of JITAIs. MRTs can be used to address research questions about whether and under what circumstances JITAI components are effective, with the ultimate objective of developing effective and efficient JITAI.

The purpose of this article is to clarify why, when, and how to use MRTs; to highlight elements that must be considered when designing and implementing an MRT; and to review primary and secondary analyses methods for MRTs. We briefly review key elements of JITAIs and discuss a variety of considerations that go into planning and designing an MRT. We provide a definition of causal excursion effects suitable for use in primary and secondary analyses of MRT data to inform JITAI development. We review the weighted and centered least-squares (WCLS) estimator which provides consistent causal excursion effect estimators from MRT data. We describe how the WCLS estimator along with associated test statistics can be obtained using standard statistical software such as R (R Core Team, 2019). Throughout we illustrate the MRT design and analyses using the HeartSteps MRT, for developing a JITAI to increase physical activity among sedentary individuals. We supplement the HeartSteps MRT with two other MRTs, SARA and BariFit, each of which highlights different research questions that can be addressed using the MRT and experimental design considerations that might arise.
TL;DR:
Throughout we illustrate the MRT design and analyses using the HeartSteps MRT, for developing a JITAI to increase physical activity among sedentary individuals.
Translating Strategies for Promoting Engagement in Mobile Health: A Proof-of-Concept Microrandomized Trial
Authors:
Publication Venue:
Health Psychology
Publication Date:

December 2021

Keywords:

engagement, mobile health (mHealth), Micro-Randomized Trial (MRT), reciprocity, reinforcement

Related Project:
Abstract:
Objective: Mobile technologies allow for accessible and cost-effective health monitoring and intervention delivery. Despite these advantages, mobile health (mHealth) engagement is often insufficient. While monetary incentives may increase engagement, they can backfire, dampening intrinsic motivations and undermining intervention scalability. Theories from psychology and behavioral economics suggest useful nonmonetary strategies for promoting engagement; however, examinations of the applicability of these strategies to mHealth engagement are lacking. This proof-of-concept study evaluates the translation of theoretically-grounded engagement strategies into mHealth, by testing their potential utility in promoting daily self-reporting.

Method: A microrandomized trial (MRT) was conducted with adolescents and emerging adults with past-month substance use. Participants were randomized multiple times daily to receive theoretically-grounded strategies, namely reciprocity (the delivery of inspirational quote prior to self-reporting window) and nonmonetary reinforcers (e.g., the delivery of meme/gif following self-reporting completion) to improve proximal engagement in daily mHealth self-reporting.

Results: Daily self-reporting rates (62.3%; n = 68) were slightly lower than prior literature, albeit with much lower financial incentives. The utility of specific strategies was found to depend on contextual factors pertaining to the individual’s receptivity and risk for disengagement. For example, the effect of reciprocity significantly varied depending on whether this strategy was employed (vs. not employed) during the weekend. The nonmonetary reinforcement strategy resulted in different outcomes when operationalized in various ways.

Conclusions: While the results support the translation of the reciprocity strategy into this mHealth setting, the translation of nonmonetary reinforcement requires further consideration prior to inclusion in a full scale MRT.
TL;DR:
A microrandomized trial (MRT) was conducted with young adults with past-month substance use. Participants were randomized multiple times daily to receive theoretically-grounded strategies to improve proximal engagement in daily mHealth self-reporting.
The Mobile Assistance for Regulating Smoking (MARS) Micro-Randomized Trial Design Protocol
Authors:
Publication Venue:

Contemporary Clinical Trials

Keywords:

engagement, Micro-randomized trial (MRT), mobile health (mHealth), self-regulatory strategies, smoking cessation

Publication Date:

November 2021

Related Project:
Abstract:

Smoking is the leading preventable cause of death and disability in the U.S. Empirical evidence suggests that engaging in evidence-based self-regulatory strategies (e.g., behavioral substitution, mindful attention) can improve smokers’ ability to resist craving and build self-regulatory skills. However, poor engagement represents a major barrier to maximizing the impact of self-regulatory strategies. This paper describes the protocol for Mobile Assistance for Regulating Smoking (MARS) – a research study designed to inform the development of a mobile health (mHealth) intervention for promoting real-time, real-world engagement in evidence-based self-regulatory strategies. The study will employ a 10-day Micro-Randomized Trial (MRT) enrolling 112 smokers attempting to quit. Utilizing a mobile smoking cessation app, the MRT will randomize each individual multiple times per day to either: (a) no intervention prompt; (b) a prompt recommending brief (low effort) cognitive and/or behavioral self-regulatory strategies; or (c) a prompt recommending more effortful cognitive or mindfulness-based strategies. Prompts will be delivered via push notifications from the MARS mobile app. The goal is to investigate whether, what type of, and under what conditions prompting the individual to engage in self-regulatory strategies increases engagement. The results will build the empirical foundation necessary to develop a mHealth intervention that effectively utilizes intensive longitudinal self-report and sensor-based assessments of emotions, context and other factors to engage an individual in the type of self-regulatory activity that would be most beneficial given their real-time, real-world circumstances. This type of mHealth intervention holds enormous potential to expand the reach and impact of smoking cessation treatments.

TL;DR:

This paper describes the protocol for Mobile Assistance for Regulating Smoking (MARS) – a research study designed to inform the development of a mobile health (mHealth) intervention for promoting real-time, real-world engagement in evidence-based self-regulatory strategies. 

Designing Reinforcement Learning Algorithms for Digital Interventions: Pre-implementation Guidelines
Statistical Inference with M-Estimators on Adaptively Collected Data
Authors:
Publication Venue:

Advances in Neural Information Processing Systems

Publication Date:

December 2021

License:
Language:

Python

Shell

The Mobile Assistance for Regulating Smoking (MARS) Micro-Randomized Trial Design Protocol

Mobile health (mHealth) interventions have typically used hand-crafted decision rules that map from biomarkers of an individual’s state to the selection of interventions. Recently, reinforcement learning (RL) has emerged as a promising approach for online optimization of decision rules. Continuous, passive detection of the individual’s state using mHealth biomarkers enables dynamic deployment of decision rules at the right moment, i.e., as and when events of interest are detected from sensors. RL-based optimization methods that leverage this new capability created by sensor-based biomarkers, can enable the development and optimization of temporally-precise mHealth interventions, overcoming the significant limitations of static, one-size-fits-all decision rules. Such next generation interventions have the potential to lead to greater treatment efficacy and improved long-term engagement.

However, there exist several critical challenges to the realization of effective, real-world RL-based interventions including the need to learn efficiently based on limited interactions with an individual while accounting for longer-term effects of intervention decisions, (i.e., to avoid habituation and ensure continued engagement), and accommodating multiple intervention components operating at different time scales and targeting different outcomes. As a result, the use of RL in mHealth interventions has mostly been limited to very few studies using basic RL methods.

To address these critical challenges, TR&D2 builds on more precise biomarkers of context, including TR&D1 risk and engagement scores, to develop, evaluate, and disseminate robust and data efficient RL methods and tools. These methods continually personalize the selection, adaptation and delivery timing decision rules for core intervention components so as to maximize long-term therapeutic efficacy and engagement for every individual.

Sayma Akther, PhD

Assistant Professor


Soujanya Chatterjee, PhD

Applied Scientist II


Satya Shukla, PhD

Senior Research Scientist


Md Azim Ullah, PhD

Applied Scientist


  1. P. Liao, Z. Qi, R. Wan, P. Klasnja, S. Murphy Batch Policy Learning in Average Reward Markov Decision Processes. Annals of Statistics. 2022 Sept 17.
  2. Trella AL, Zhang KW, Nahum-Shani I, Shetty V, Doshi-Velez F, Murphy SA. Designing Reinforcement Learning Algorithms for Digital Interventions: Pre-Implementation Guidelines. Algorithms. 2022; 15(8). NIHMSID: NIHMS1825651.
  3. Bertsimas D., Klasnja, P., Murphy, S., & L. Na (2022) Data-driven Interpretable Policy Construction for Personalized Mobile Health. 2022 IEEE International Conference on Digital Health. 2022, pp. 13-22.
  4. Coppersmith DDL, Dempsey W, Kleiman EM, Bentley KH, Murphy SA, Nock MK. Just-in-Time Adaptive Interventions for Suicide Prevention: Promise, Challenges, and Future Directions. Psychiatry. 2022 Jul 18;:1-17. doi: 10.1080/00332747.2022.2092828. [Epub ahead of print] PubMed PMID: 35848800; NIHMSID:NIHMS1821198.
  5. Nahum-Shani I, Shaw SD, Carpenter SM, Murphy SA, Yoon C. Engagement in digital interventions. Am Psychol. 2022 Mar 17;. doi: 10.1037/amp0000983. [Epub ahead of print] PubMed PMID: 35298199; NIHMSID:NIHMS1800077.
  6. Qian T, Walton AE, Collins LM, Klasnja P, Lanza ST, Nahum-Shani I, Rabbi M, Russell MA, Walton MA, Yoo H, Murphy SA. The microrandomized trial for developing digital interventions: Experimental design and data analysis considerations. Psychol Methods. 2022 Jan 13;. doi: 10.1037/met0000283. [Epub ahead of print] PubMed PMID: 35025583; PubMed Central PMCID: PMC9276848.
  7. Nahum-Shani I, Rabbi M, Yap J, Philyaw-Kotov ML, Klasnja P, Bonar EE, Cunningham RM, Murphy SA, Walton MA. Translating strategies for promoting engagement in mobile health: A proof-of-concept microrandomized trial. Health Psychol. 2021 Dec;40(12):974-987. doi: 10.1037/hea0001101. Epub 2021 Nov 4. PubMed PMID: 34735165; PubMed Central PMCID: PMC8738098.
  8. Zhang KW, Janson L, Murphy SA. Statistical Inference with M-Estimators on Adaptively Collected Data. Adv Neural Inf Process Syst. 2021 Dec;34:7460-7471. PubMed PMID: 35757490; PubMed Central PMCID: PMC9232184.
  9. Nahum-Shani I, Potter LN, Lam CY, Yap J, Moreno A, Stoffel R, Wu Z, Wan N, Dempsey W, Kumar S, Ertin E, Murphy SA, Rehg JM, Wetter DW. The mobile assistance for regulating smoking (MARS) micro-randomized trial design protocol. Contemp Clin Trials. 2021 Nov;110:106513. doi: 10.1016/j.cct.2021.106513. Epub 2021 Jul 24. PubMed PMID: 34314855; PubMed Central PMCID: PMC8824313 .

Presentations by Susan Murphy

  1. Harnessing real world behavior data to optimize treatment delivery. ABCT 56th Annual Convention, New York, NY November 20, 2022
  2. Acceptance Lecture, 2021 Van Wijngaarden Award, Centrum Wiskunde & Informatica, Amsterdam, Netherlands, November 3, 2022
  3. Inference for Longitudinal Data After Adaptive Sampling. 2022 Al-Kindi Distinguished Statistics Lecture, King Abdullah University of Science and Technology, Saudi Arabia, October 20, 2022
  4. We used Reinforcement Learning; but did it work? 2022 Al-Kindi Distinguished Statistics Lecture, King Abdullah University of Science and Technology, Saudi Arabia, October 19, 2022
  5. Data, Personalization, Digital Health! Quantitative Science Grand Rounds, Moffitt Cancer Center, Tampa, FL. October 5, 2022
  6. Inference for Longitudinal Data After Adaptive Sampling. Wellner Lecture, University of Idaho, Moscow, ID September 29, 2022
  7. Panelist, launch event for the Kempner Institute for the Study of Artificial and Natural Intelligence, Harvard University, September 22, 2022
  8. Inference for Longitudinal Data After Adaptive Sampling. Operations Research & Financial Engineering Department Colloquium, Princeton University, Princeton NJ, September 13, 2022
  9. We used RL; but did it work? S.S. Wilks Memorial Lecture, Princeton University, Princeton NJ, September 12, 2022
  10. We used RL; but did it work? Workshop on Reinforcement Learning at Harvard, Center for Brain Science, Harvard University, August 30, 2022
  11. Data, Personalization, Digital Health! Keynote. European Health Psychology Society Conference, Bratislava, Slovakia, August 26, 2022
  12. Inference for longitudinal data after adaptive sampling, Keynote, ICSA 2022 Applied Statistics Symposium, Gainesville, FL, June 22, 2022
  13. Data, Personalization, Digital Health! Keynote (virtual). Society for Ambulatory Assessment Conference, June 14, 2022
  14. Inference for Longitudinal Data After Adaptive Sampling. Keynote, 35th New England Statistics Symposium (NESS), University of Connecticut, Storrs, CT, May 24, 2022
  15. Assessing Personalization in Digital Health. Charles L. Odoroff Memorial Lecture, University of Rochester Medical Center, Rochester, NY, May 19, 2022
  16. Optimizing Your Digital Health JITAI using a Micro-Randomized Trial, ECNP Digital Health Network, online, Get Digital Talk, May 10, 2022
  17. Assessing Personalization in Digital Health. Invited virtual talk, Deutsche ArbeitsGemeinschaft Statistik (DAGStat 2022), Hamburg, Germany, March 29, 2022
  18. We used Reinforcement Learning, but did it work? Virtual keynote, AI for Behavior Change workshop, AAAI 2022, February 28, 2022
  19. We used Reinforcement Learning; But Did It Work?​​, AI Talk (virtual), Chalmers University of Technology, Gothenburg, Sweden, January 19, 2022
  20. We used Reinforcement Learning; but did it work? Virtual presentation, CIS Colloquium, EPFL, December 13, 2021
  21. 2020 H.O. Hartley Award Lecture at Texas A&M (lecture was postponed; presented December 6, 2021)
  22. Assessing Personalization in Digital Health, Virtual presentation, Australian Trials Methodology Conference 2021, December 6, 2021
  23. Data, Personalization, Digital Health, Virtual presentation, Distinguished Speaker Series, Research Center for Child Well-Being, University of South Carolina, December 3, 2021

Presentations by Raaz Dwivedi (postdoctoral researcher)

  1. Counterfactual inference in sequential experimental design. INFORMS 2022, Indianapolis, Indiana, October 17, 2022
  2. Near-optimal compression in near-linear time. Poster. Royal Statistical Society International Conference (2022), Aberdeen, Scotland, September 13, 2022
  3. Counterfactual inference in sequential experimental design. Poster. Royal Statistical Society International Conference (2022), Aberdeen, Scotland, September 13, 2022
  4. Generalized Kernel Thinning, JSM 2022, Washington DC, August 10, 2022
  5. Counterfactual inference for sequential experimental design. IMS 2022 Annual Meeting, London, UK, June 2022
  6. Generalized kernel thinning. Virtual poster, International Conference on Learning Theory, April 2022.
  7. Distribution compression in near-linear time. Contributed poster. Tenth International Conference on Learning Representations (Virtual), April 2022
  8. Revisiting minimum description length complexity in overparameterized models. Symposium on Algorithmic Information Theory & Machine Learning, Alan Turing Institute, London, UK, July 4.
  9. Counterfactual inference for sequential experimental design. Poster. Synthetic Control Methods Workshop, Princeton NJ, June 2.
  10. Counterfactual inference for sequential experimental design. Poster. 2022 American Causal Inference Conference, Berkeley, CA, May 24.
  11. Near-optimal compression in near-linear time. Talk at the Kernel Methods for Numerical Integration mini-symposium, 2022 SIAM Conference on Uncertainty Quantification, April 14, 2022
  12. Near-optimal compression in near-linear time. Invited talk at the workshop Foundations of Stable, Generalizable and Transferable Statistical Learning, Mathematical Sciences Research Institute, Berkeley, CA, March 8
  13. Generalized kernel thinning. Virtual poster, Advances in Approximate Bayesian Inference, February 2022.
  14. Counterfactual inference in sequential experimental design. Talk at the Department of Statistics, Harvard University, February 24.
  15. Counterfactual inference in sequential experimental design. Simons workshop on Learning from Interventions (video) Berkeley, California, February 14
  16. Imputation with nearest neighbors for adaptively collected data. Talk at the Foundations of Data Science Institute Advisory Board meeting 2022, February
  17. Distribution compression in near-linear time. Contributed poster. 4th Symposium on Advances in Approximate Bayesian Inference, February 2
  18. Near-optimal Compression in Near-linear Time. 27th Annual LIDS Student Conference, Machine Learning and Statistics Session at MIT
  19. Imputation with nearest neighbors for adaptively collected data, Foundations of Data Science Institute Retreat 2022 (virtual), January 6

Presentation by Kyra Gan (postdoctoral researcher)

  1. Greedy Approximation Algorithms for Active Sequential Hypothesis Testing. Simons Institute Workshop Quantifying Uncertainty: Stochastic, Adversarial, and Beyond, Berkeley CA, September 12, 2022.

Presentation by Shuangning Li (postdoctoral researcher)

  1. Network Interference in Micro-randomized Trials. INFORMS 2022, Indianapolis, IN, October 17, 2022.

Presentations by Kelly Zhang (graduate student)

  1. Statistical Inference After Adaptive Sampling for Longitudinal Data. INFORMS 2022, Indianapolis, IN, October 17
  2. Statistical Inference After Adaptive Sampling for Longitudinal Data. Invited talk in the Prediction and Inference in Statistical Machine Learning session, JSM 2022, Washington, DC, August 11
  3. Statistical Inference After Adaptive Sampling for Longitudinal Data. Talk in the invited session Inference Methods for Adaptively Collected Data, 2022 IMS Annual Meeting, London, UK, June 29
  4. Statistical Inference After Adaptive Sampling for Longitudinal Data, Department of Psychology, University of Toronto, Toronto, CA, June 2022
  5. Statistical Inference for Bandit Data, Virtual presentation, Department of Statistics, EPFL, December 3, 2021

Presentations by Anna Trella (graduate student)

  1. Designing Reinforcement Learning Algorithms for Digital Interventions: Pre-implementation Guidelines. mDOT webinar (virtual), Nov 21, 2022
  2. Designing Reinforcement Learning Algorithms for Digital Interventions: Pre-implementation Guidelines. Workshop on Reinforcement Learning at Harvard, Center for Brain Science, Harvard University, August 30, 2022
  3. Designing Reinforcement Learning Algorithms for Digital Interventions: Pre-implementation Guidelines. Poster, SEAS Summertime Seminar Series, Harvard, July 6, 2022 https://sites.google.com/g.harvard.edu/seas-seminar-series/home?authuser=0
  4. Designing Reinforcement Learning Algorithms for Digital Interventions: Pre-implementation Guidelines. International Chinese Statistics Association Applied Stats Symposium. Talk in the session Precision Digital Health Care via Machine Learning, Gainesville, FL, June 21, 2022
  5. Designing Reinforcement Learning Algorithms for Digital Interventions: Pre-implementation Guidelines. Extended abstract and oral presentation, RLDM 2022, June 8, Brown University, Providence, RI
  6. Designing Reinforcement Learning Algorithms for Digital Interventions: Guidelines before Going into the Field. Poster and Lightning Talk, Women in Data Science, Cambridge, MA, March 9, 2022

Presentations by Xiang Meng (graduate student)

  1. An Algorithm to Determine Treatment Timing in Mobile Health: Design and Evaluation. INFORMS 2022, Indianapolis, IN, October 2022
  2. Assessing the Effectiveness of a Sampling Algorithm for Just-in-Time Intervention Delivery. Virtual talk in the contributed session Adaptive Design/Adaptive Randomization, ENAR 2022, March 28, 2022
  3. An Algorithm to Determine Treatment Timing in Mobile Health: Potential, Design and Evaluation, CMStatistics2021, virtual presentation, December 18, 2021

Presentation by Prasidh Chhabria (undergraduate student)

  1. RL Digital Interventions Under User Heterogeneity: A Bayesian Nonparametric Approach. Poster, RLDM 2022, Providence, RI, June 9, 2022.

Santosh Kumar, PhD

Lead PI, Center Director, TR&D1, TR&D2, TR&D3


Benjamin Marlin, PhD

Co-Investigator, TR&D1, TR&D2



Sameer Neupane

Doctoral Student


Mithun Saha

Doctoral Student


Karine Karine

Doctoral Student


Hui Wei

Doctoral Student


Research and development by TR&D2 will significantly advance RL methodology for personalizing decision rules; in particular, with regards to online algorithms that personalize interventions for each user by appropriately pooling across multiple users.