(193b) How Does a Deep Neural Network Learn to Approximate Functions?

Neural networks, particularly deep neural nets, have achieved remarkable success and widespread acknowledgement as an effective tool for addressing classification and function approximation tasks. Their input-output structure and their ability to manipulate feature space towards a given objective make them amenable to a variety of applications such as process control, fault-diagnosis, and materials design.

However, their inherent black-box nature, and hence their inability to provide mechanistic explanations of their recommendations, make them difficult to trust in critical applications. This lack of a clear understanding (if not a theory) of the working of neural networks has led to the criticism that such machine learning techniques are more like alchemy, rather than chemistry, replete with trial-and-error attempts with no insights.

In this paper, we report our progress on a systematic study of how a neural network learns the underlying patterns in input-output data by carefully exploring its features space. We use the Shekel function as a test bed to probe into the functioning of a neural network, node by node and layer by layer. We perform controlled experiments with the number of nodes in a layer and the number of layers in a network to understand what nodes and layers in a network achieve. In other words, we make a systematic effort towards the understanding of how the number of nodes (width) and layers (depth) in a network determine the capabilities of the network in approximating an arbitrary function - i.e., how deep neural networks learn to represent arbitrary functions. Our approach gives us novel and useful insights into the internal workings of a deep neural network, thereby shedding some light into the black box.