List sphere decoding with a probabilistic radius tightening

In this paper, we present a low-complexity list sphere search algorithm for achieving near-optimal a posteriori probability (APP) detection in iterative detection and decoding (IDD). Motivated by the fact that the list sphere decoding searching a fixed number of lattice points is inefficient in many scenarios, we design a criterion to search lattice points with non-vanishing likelihood and derive the optimal sphere radius satisfying this requirement. Further, in order to exploit the sphere constraint as it is instead of using necessary conditioned versions, we incorporate a probabilistic tree pruning strategy into the list sphere search. Through simulations on realistic IDD systems, we show that the proposed method provides considerable complexity savings while maintaining near-optimal performance.