Accurate and consistent segmentation of infant brain MR images plays an important role in quantifying the early brain development, especially in longitudinal studies. However, due to rapid maturation and myelination of brain tissues in the first year of life, white-gray matter contrast undergoes dramatic changes. In fact, the contrast inverses around 6 months of age, where the white and gray matter tissues are isointense and hence exhibit the lowest contrast, posing significant challenges for segmentation algorithms. In this paper, we propose a novel longitudinally guided level set method for segmentation of serial infant brain MR images, acquired from 2 weeks up to 1.5 years of age. The proposed method makes optimal use of T1, T2 and the diffusion weighted images for complimentary tissue distribution information to address the difficulty caused by the low contrast. A longitudinally consistent term, which constrains the distance across the serial images within a biologically reasonable range, is employed to obtain temporally consistent segmentation results. The proposed method has been applied on 22 longitudinal infant subjects with promising results.