A Case Study on the Critical Role of Geometric Regularity in Machine Learning