Graph matching is a robust correspondence detection approach which considers potential correspondences as graph nodes and uses graph links to measure the pairwise agreement between potential correspondences. In this paper, we propose a novel graph matching method to augment its power in establishing anatomical correspondences in medical images, especially for the cases with large inter-subject variations. Our contributions have twofold. First, we propose a robust measurement to characterize the pairwise agreement of appearance information on each graph link. In this way, our method is more robust to ambiguous matches than the conventional graph matching methods that generally consider only the simple geometric information. Second, although multiple correspondences are allowed for robust correspondence, we further introduce the sparsity constraint upon the possibilities of correspondences to suppress the distraction from misleading matches, which is very important for achieving accurate one-to-one correspondences in the end of the matching procedure. We finally incorporate these two improvements into a new objective function and solve it by quadratic programming. The proposed graph matching method has been evaluated in the public hand X-ray images with comparison to a conventional graph matching method. In all experiments, our method achieves the best matching performance in terms of matching accuracy and robustness.