Learning to Transform for Generalizable Instance-wise Invariance
ICCV 2023


We predict a distribution of spatial transformations for any input image. This can be used for data augmentation, aligning instances before classification, and adapting to out-of-distribution poses.



Computer vision research has long aimed to build systems that are robust to spatial transformations found in natural data. Traditionally, this is done using data augmentation or hard-coding invariances into the architecture. However, too much or too little invariance can hurt, and the correct amount is unknown a priori and dependent on the instance. Ideally, the appropriate invariance would be learned from data and inferred at test-time.
We treat invariance as a prediction problem. Given any image, we use a normalizing flow to predict a distribution over transformations and average the predictions over them. Since this distribution only depends on the instance, we can align instances before classifying them and generalize invariance across classes. The same distribution can also be used to adapt to out-of-distribution poses. This normalizing flow is trained end-to-end and can learn a much larger range of transformations than Augerino and InstaAug. When used as data augmentation, our method shows accuracy and robustness gains on CIFAR 10, CIFAR10-LT, and TinyImageNet.



This work was supported, in part, by the BAIR/Google fund.

The website template was borrowed from the Fourier Feature Networks project page and Michaƫl Gharbi.