Learning with per-sample side information

Learning from few samples is a major challenge for parameter-rich models such as deep networks. In contrast, people can learn complex new concepts even from very few examples, suggesting that the sample complexity of learning can often be reduced. We describe an approach to reduce the number of samples needed for learning using per-sample side information. Specifically, we show how to speed up learning by providing textual information about feature relevance, like the presence of objects in a scene or attributes in an image. We also give an improved generalization error bound for this case. We formulate the learning problem using an ellipsoid-margin loss, and develop an algorithm that minimizes this loss effectively. Empirical evaluation on two machine vision benchmarks for scene classification and fine-grain bird classification demonstrate the benefits of this approach for few-shot learning.