Masksembles for Uncertainty Estimation
by Nikita Durasov, Timur Bagautdinov, Pierre Baque, and Pascal Fua Published at the Conference on Computer Vision and Pattern Recognition (CVPR), 2021
Abstract
Deep neural networks have amply demonstrated their prowess but estimating the reliability of their predictions remains challenging. Deep Ensembles are widely considered as being one of the best methods for generating uncertainty estimates but are very expensive to train and evaluate. MC-Dropout is another popular alternative, which is less expensive, but also less reliable. Our central intuition is that there is a continuous spectrum of ensemble-like mod els of which MC-Dropout and Deep Ensembles are extreme examples. The first one uses an effectively infinite number of highly correlated models, while the second relies on a finite number of independent models.
To combine the benefits of both, we introduce Masksembles. Instead of randomly dropping parts of the network as in MC-dropout, Masksemble relies on a fixed number of bi nary masks, which are parameterized in a way that allows changing correlations between individual models. Namely, by controlling the overlap between the masks and their size one can choose the optimal configuration for the task at hand. This leads to a simple and easy to implement method with performance on par with Ensembles at a fraction of the cost. We experimentally validate Masksembles on two widely used datasets, CIFAR10 and ImageNet.
Presenter
Nikita Durasov is currently a PhD student at EPFL where he works under supervision of Prof. Pascal Fua at Computer Vision Laboratory on problems connected with Deep Learning and Computer Vision.
Links