Optimization of multi-stage dynamic treatment regimes utilizing accumulated data

Xuelin Huang, Sangbum Choi, Lu Wang, Peter F. Thall

Research output: Contribution to journalArticlepeer-review

13 Citations (Scopus)


In medical therapies involving multiple stages, a physician's choice of a subject's treatment at each stage depends on the subject's history of previous treatments and outcomes. The sequence of decisions is known as a dynamic treatment regime or treatment policy. We consider dynamic treatment regimes in settings where each subject's final outcome can be defined as the sum of longitudinally observed values, each corresponding to a stage of the regime. Q-learning, which is a backward induction method, is used to first optimize the last stage treatment then sequentially optimize each previous stage treatment until the first stage treatment is optimized. During this process, model-based expectations of outcomes of late stages are used in the optimization of earlier stages. When the outcome models are misspecified, bias can accumulate from stage to stage and become severe, especially when the number of treatment stages is large. We demonstrate that a modification of standard Q-learning can help reduce the accumulated bias. We provide a computational algorithm, estimators, and closed-form variance formulas. Simulation studies show that the modified Q-learning method has a higher probability of identifying the optimal treatment regime even in settings with misspecified models for outcomes. It is applied to identify optimal treatment regimes in a study for advanced prostate cancer and to estimate and compare the final mean rewards of all the possible discrete two-stage treatment sequences.

Original languageEnglish
Pages (from-to)3424-3443
Number of pages20
JournalStatistics in Medicine
Issue number26
Publication statusPublished - 2015 Nov 20
Externally publishedYes


  • Backward induction
  • Multi-stage treatment
  • Optimal treatment sequence
  • Q-learning
  • Treatment decision-making

ASJC Scopus subject areas

  • Epidemiology
  • Statistics and Probability


Dive into the research topics of 'Optimization of multi-stage dynamic treatment regimes utilizing accumulated data'. Together they form a unique fingerprint.

Cite this