I see the trend of Neural network approaching the network in network in the future. As some fully end-to-end trained network can be fine tuned at the connection of the ‘parent’ network, reaching a small well functioning part of the ‘brain’.

These ideas are coming from GoogLeNet, fast and faster -RCNN.