HAETAE: In-domain Table Pretraining with Header Anchoring

Understanding structured table data with language models is crucial for various downstream tasks in information retrieval. However, transformer-based table embedding models struggle to consistently represent headers across varying entity contexts. This inconsistency undermines the generalizability of embeddings across in-domain tables that share universal semantics. To address this gap, we propose a novel pretraining method for in-domain tables, HAETAE, that explicitly separates header embeddings from contextual entity embeddings. Our method introduces a dedicated header encoder and learnable alignment mechanisms, built upon header-aware serialization. Experimental results demonstrate that HAETAE enhances generalization and stability in predicting headers and values of in-domain tables, achieving higher accuracy than baselines while showing the notable potential of knowledge transfer in cross-domain tables. The source code of HAETAE is available at https://github.com/woojoonjung/HAETAE.