The Anatomize deep learning with information theory is a nice summary of the bottleneck theory and deep learning talk by Professor Tishby. The blog also has a nice reference to a two part write up on traditional learning theory that doesn't actually explain the success the deep learning: Tutorial on ML theory, part1and part 2.
Update: A really interesting article on To Remember, the Brain Must Actively Forget This is very similar to the idea of the bottleneck theory and deep learning models.
