Deep reinforcement learning with a critic-value-based branch tree for the inverse design of two-dimensional optical devices

In optical engineering, designing a device or a system with the desired property is an important but challenging task. It is relatively straightforward to compute the physical properties of a given design; however, there is no general method for the reverse, i.e., designing with desired properties. To address this problem with a computational method, this paper proposes a deep reinforcement learning-based inverse design framework consisting of two methods: Inverse DEsign Agent (IDEA) and Critic-Value-based Branch Tree (CVBT) algorithm. IDEA is a deep reinforcement learning model based on the Advantage Actor–Critic (A2C) method using a deep learning simulator as a replacement for a real environment, significantly reducing training time compared to conventional methods. The CVBT algorithm suggests several design candidates using the critic values of IDEA, while conventional methods propose only one candidate. In this study, the proposed framework was applied to a two-dimensional optical device design problem. Experimental results with untrained target properties demonstrated that the proposed model, IDEA-CVBT, achieved state-of-the-art performance in terms of accuracy and stability. For instance, in a scenario with a binary type design, IDEA-CVBT exhibited an accuracy of 91.5%, while conventional deep reinforcement models showed 36.1% and 7.4% accuracies. Extensive analyses verified that IDEA-CVBT can be employed as an assistance system for engineers since IDEA-CVBT suggests several appropriate candidates that satisfy target properties.