Ambiguous Dynamic Treatment Regimes: A Reinforcement Learning Approach

10/27/22 | 4:15pm | E51-145


 

 

 

 

Soroush Saghafian

Associate Professor
Harvard University


Abstract: A main research goal in various applications is to use an observational data set and provide a new set of counterfactual guidelines that can yield causal improvements. When using available methods, researchers often have to rely on assumptions that are violated in real-world applications (e.g., medical decision-making or public policy), especially when (a) the existence of unobserved confounders cannot be ignored, and (b) the unobserved confounders are time-varying (e.g., affected by previous actions). When such assumptions are violated, one might face ambiguity regarding the underlying causal model.  This ambiguity is inevitable, since the dynamics of unobserved confounders and their causal impact on the observed part of the data cannot be understood from the observed data. Motivated by a case study of finding superior treatment regimes for patients who underwent transplantation in our partner hospital (Mayo Clinic) and faced a medical condition known as New Onset Diabetes After Transplantation (NODAT), we introduce a new methodology termed Ambiguous Dynamic Treatment Regimes (ADTRs), in which the causal impact of treatment regimes is evaluated based on a “cloud” of potential causal models. We then connect ADTRs to Ambiguous Partially Observable Markov Decision Processes (APOMDPs) proposed by Saghafian (2018), and consider unobserved confounders as latent variables but with ambiguous dynamics and causal effects on observed variables. Using this connection, we develop two Reinforcement Learning methods termed Direct Augmented V-Learning (DAV-Learning) and Safe Augmented V-Learning (SAV-Learning), which enable using the observed data to efficiently learn an optimal treatment regime. We establish theoretical results for these learning methods, including (weak) consistency and asymptotic normality. We further evaluate the performance of these learning methods both in our case study (using clinical data) and in simulation experiments (using synthetic data). We find promising results for our proposed approaches, showing that they perform well even compared to an imaginary oracle who knows both the true causal model and the optimal treatment regime under that model. Finally, we highlight that our approach allows for a two-way personalization: optimal treatment regimes are personalized based on both patient characteristics and physicians’ preferences.

Bio: Dr. Soroush Saghafian is interested in using and developing operations research and management science techniques that can have significant public benefits. He is the founder and director of the Public Impact Analytics Science Lab (PIAS-Lab) at Harvard, which is devoted to advancing and applying the science of analytics for solving societal problems that can have public impact. His current teaching focuses on Machine Learning and Big Data Analytics tools for solving societal problems.  His current research focuses on the application and development of operations research methods in studying stochastic systems with specific applications in healthcare and operations management. He has been collaborating with a variety of hospitals to improve their operational efficiency, patient flow, medical decision-making, and more broadly, healthcare delivery policies. He also serves as a faculty affiliate for the Harvard Ph.D. Program in Health Policy,  the Harvard Center for Health Decision Science, the Harvard Mossavar-Rahmani Center for Business and Government (M-RCBG), the Harvard Data Science Initiative, the Belfer Center for Science and International Affairs, and is an associate faculty member at the Harvard Ariadne Labs (Health Systems Innovation).

Dr. Saghafian's research has appeared in the news, including in pieces and interviews by The Hill, National Academy of Medicine, New-Meical.net, Industry Global News 24, Global Health News Wire, Eureka Alert (American Association for the Advancement of Science), Managed Healthcare Executive, and INFORMS. He has won various awards for his research, including the INFORMS MSOM Young Scholar Prize (2021) “[for] outstanding contributions to scholarship in operations management," the Inaugural INFORMS 2020 Mehrotra Research Excellence Award "for significant contributions to the practice of health applications through operations research (OR) and management science (MS) modeling and methodologies,"  POMS 2019 College of Healthcare Best Paper Award (first place), INFORMS 2018 Public Sector Best Paper Award (second place),  POMS 2017 College of Healthcare Best Paper Award (second place), INFORMS MSOM (Manufacturing & Service Operations Management) Journal 2016 Best Paper Award, INFORMS  MSOM Society 2016 Best Paper Award of Service Special Interest Group (SIG), INFORMS 2015 Junior Faculty Interest Group (JFIG) Best Paper Competition (Honorable Mention), INFORMS 2012 MSOM (Manufacturing and Service Operations Management Society) Best Student Paper Award, 2012 IOE Richard Wilson Prize, 2010 INFORMS Pierskalla Award "for the best research paper in healthcare", University of Michigan College of Engineering Outstanding Ph.D. Research Award, Production and Operations Management (POMS) 2011 College of Healthcare Operations Management best  paper award (second place), 2010 Murty Prize for best research paper in optimization, and the 2007 IOE Bonder Fellowship award for applied operations research, INFORMS 2011 Doing Good with Good OR Award  (Honorable Mention), and POMS College of Supply Chain 2013 and 2009 Best Student Paper Award (Honorable Mention).

Dr. Saghafian serves  on the editorial board of a few journals including Operations Research, Production and Operations Management, Service Science, and IISE Transactions. He also serves as an AE or referee for various journals including Operations Research, Management Science, Manufacturing and Service Operations Management, Health Care Management Science, Mathematical Reviews (American Mathematical Society), Operations Research Letters, Naval Research Logistics, Service Science, and  Production and Operations Management.

Dr. Saghafian has also served as a chair/co-chair or a review panel member in various committees including: INFORMS Pierskalla Award for the Best Paper in Healthcare, INFORMS Healthcare Applications Society Best Student Paper Award,  INFORMS MSOM Healthcare Special Interest Group, INFORMS MSOM Best Student Paper review panel, POMS College of Healthcare Operations Management Best Paper Award, and the review panel for International Conference on Health Care Systems Engineering. Dr. Saghafian's research has been supported through various grants from National Science Foundation (NSF), Mayo Clinic, and Harvard University (HKS), among others.

Event Time: 

2022 - 16:15