DPP-ViT: Dynamic Patch Pruning for Low Complexity Vision Transformer Accelerator

While vision transformers excel in various computer vision tasks, their high computational cost limits use on resource-constrained devices, highlighting the need for complexity reduction. In this paper, we present dynamic patch pruning for low complexities of vision transformers (DPP-ViT). To identify relatively more important patches, the column-wise accumulations of attention maps are computed and those are used as importance scores. Through DPP-ViT with block-wise importance score thresholds, our approach considers image-wise difficulties and block-wise sensitivities, removing the patches that contribute the least to accuracies. Additionally, we present a reconfigurable accelerator that dynamically changes dataflow and PE structure by applying pruning-aware row-level reconfiguration. DPP-ViT achieves 47% computation reduction with a minor -0.25% degradation on DeiT-B model in ImageNet top-1 accuracy. The proposed reconfigurable accelerator also achieves 47.96×/ 4.36×/ 1.47× speed-ups compared to EdgeCPU, EdgeGPU, and vision transformer accelerator ViTCoD.