Wednesday, January 24, 2018

Information Theory of Deep Learning. Bottleneck theory



The Information Bottleneck Method by Tishby, Pereira, and Bialek is an interesting way to look at what is happening in a deep neural network.    You can see the concept explained in following papers


  1. Deep Learning and the Information Bottleneck Principle , and
  2. ON THE INFORMATION BOTTLENECK THEORY OF DEEP LEARNING 


Professor Tishby also does a nice lecture on the topic in

Information Theory of Deep Learning. Naftali Tishby

With a follow-on alk with more detail:

Information Theory of Deep Learning - Naftali Tishby

Deep learning aside, there are other interesting applications of the bottleneck method.  It can be used to categorize music chords:

Information bottleneck web applet tutorial for categorizing music chords

and in this talk,  the method is used to quantify prediction in the brain

Stephanie Palmer: "Information bottleneck approaches to quantifying prediction in the brain"


I found the following talk also interesting simplified version of the concept applied to deterministic mappings:

The Deterministic Information Bottleneck




Sunday, January 21, 2018

Visualization

Nice visualization of Hilbert Curves   and Linear Algebra.


I like this comment by "matoma" in Abstract Vector Space:

Furthermore, why does the exponential function appear everywhere in math? One reason is that it (and all scalar multiples) is an eigenfunction of the differential operator. Same deal for sines and cosines with the second-derivative operator (eigenvalue=-1).

Noting the definition of Eigenfunction


Affine Transformation

Nice visuals on Affine Matrix Transformation

Data science, machine learning, and artificial intelligence....

I like this explanation:

What's the difference between data science, machine learning, and artificial intelligence?

Monday, January 15, 2018

Nice Lecture series on Causality

In Reinforcement Learning you need to define functions for rewards and state transitions.   Big data is an good source of modeling external actors to a deep learning system.  What is needed is a causality model for the data to be able to simulate the external world correctly.

This set of lectures are quick introduction to the field.

part 1

part 2

part 3

part 4

Friday, April 21, 2017

On back propagation algorithm



Nice detailed explanation of back propagation in neural networks: