Understanding art with neural networks: art restoration meets AI (artificial intelligence)

I have always been curious about the recent buzz on ‘deep learning’ and recently got a chance to explore this area while researching methods of ‘art restoration and origins of paintings’. A lot of the technical notes are the words of the speaker of the TED talk linked below in the bibliography.

Let me begin with an introduction on the method of neural networks and then give some examples of how this powerful tool is being used to understand paintings and artistic creativity. Of course, this is a layman perspective (me being an electrical engineer!) so I value all feedback to know whether this post was useful in advancing your understanding or to fuel your curiosity.

Neural networks came into existence when people started thinking of brain as computer (Fig. 1a) and tried to make a computing architecture with very simple units to imitate neurons (Fig. 1b). The architecture of a typical neural network is as follows: It has neuron units each of which can do simple calculations (such as a function of addition) and have tunable connections with other neurons. Each neurons can receive information from many other neurons and sends it downstream to other neurons. Sheets of neurons were connected in a sort of network with layers as shown in Fig. 1b. Note that each layer only connected to the next layer. The information (such as a pixel in Fig. 1b) enters an input layer, is processed in the intermediate layer and the result is obtained at the deepest layer. Connections between the neural layers have weights (random at first) which evolve as the network learns. At the far /deep side of the network there are fewer and fewer neurons.

Fig 1a. Simplistic model of brain neuronal network, (left) sensory neuronal outer layers (right) deep neuronal network in brain
Fig 1b. Computer architecture to imitate brain neural network. This architecture tries to identify an image as that being of a cat or dog

So how does the network ‘learn’? With training. Let me explain. For an image learning network for example, we need to train this network with thousands of images and we tell the network the answer over and over iteratively so as to strengthen (increase the weight) the right connections. The computer can finally identify even from pictures it has never seen before and this is the key (what is called ‘learning’). The neural network has hence learnt to generalize what makes a cat and dog.

In the structure of the network, the initial layers are reactive to features such as edges, in the inner layers the layers can extract higher order features such as like an eye and final layers can understand it is a cat. Our brain is seen to work similarly in a very very very simplistic interpretation.

Now, people are using these networks creatively. The reverse of the neural network learning should also work, it is called a generative network/ recurrent network. Someone came up with an algorithm where you show a network that is trained on a say a picture, run the network forward, and from whatever the network saw they adjusted the pixels in the picture towards this interpretation. They observed that early layers cared about simple features, but as we move through the network we have higher order features.


Fig 2. Five successive layers of a network interpreting different features

The above interpretation makes me wonder if some networks of the brain perceive images in a similar fashion of matching certain patterns (edges, boxes etc.) and then matches it to its stored library of memories to interpret the object. One thus arrives at the idea of ‘perception’. What I mean is: you may thing that the images in Fig 3. are turtle but not really: it is the brain which interprets it as turtle.

Screen Shot 2017-02-19 at 3.59.06 PM.png

Similar to visual perception we can imagine neural networks to interpret text, music etc. (One of the central idea being extracting patterns from large amounts of data)

How can these networks be used for arts or to inspire arts? With a technique which is an extension of neural networks called ‘deep’ neural networks.

Deep neural networks is an extension of neural networks where the computer goes one step further and each neuron figures out its own function as well. In one of the recent papers (Video 1.) treating art with deep networks the author takes a photograph, choose a painting style that he would like to interpret and the network applies the artistic style of this painting to the image and transforms it as shown below in Fig 3.

Screen Shot 2017-02-19 at 4.12.09 PM.png
Fig 3. (right) Input painting (left) Output painting which learnt style from inset painting (left bottom)


Video 1. Deep neural networks learn painting styles

In the slide dock below, I have also collected many other paper titles on deep networks for art restoration and also some other papers on machine learning (this is the broad umbrella under artificial intelligence and neural networks are a sub-category) and computer vision (field of computer science which makes computers “see”) techniques applies to understand art. Enjoy!!



What are neural networks?



Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

Create a free website or blog at WordPress.com.

Up ↑