What is style? How is a style defined in art?
An interesting neural style transfer technique seems to be able to capture styles in art and transfer those styles onto another image.
Neural style transfer is an optimization technique used to take two images, a content image and a style reference image, and blend them together such that the input image is transformed to seem to be “painted” in the style of the style image. The algorithm extracts the texture of the style reference image through a technique called Convolutional Neural Network.
The algorithm separates the image into many layers and calculates how “correlated” these layers are to each other. For example, let the first two channels in an image be Red and Yellow. Suppose, the red channel captures some simple feature (say, vertical lines) and if these two channels were correlated then whenever in the image there is a vertical line that is detected by the Red channel then there will be a Yellow-ish effect of the second channel. Mathematically, dot-product between the vectors is used to calculate a correlation between different filters or channels. If the dot-product across two filters is large then two channels are said to be correlated and if it is small then the images are un-correlated.
The image $p$ is fed through the CNN(Convolutional Neural Network). Let $C(p)$ be the resulting output sample, called the 'content' of the input $ p$. The style image $a$ is then fed through the same CNN. These activations are encoded into a Gramian matrix representation, call it $S(a)$ to denote the 'style' of a. The goal of neural style transfer is to synthesize an output image $x$ that exhibits the content of p applied with the style of $a$, i.e. $\displaystyle C(x)=C(p)$ and $\displaystyle S(x)=S(a)$.
However, you first need to explicitly train a network to reproduce the style of your desired image. Once the network is trained, you can then apply it to any content image you wish. But training a network may take hours for a computer to compute. You can find some pre-trained models from this GitHub page.
Anyhow, I took some code from the internet, set up the environment required for this project, and played around with some images. Below are some pre-trained models, or style references, that I used.
La Muse by Pablo Picasso
Starry Night by Vincent van Gogh
The Scream by Edvard Munch
I applied these to some images on my phone and got these results:
# import the necessary packages
import argparse
import imutils
import time
import cv2
# construct the argument parser and parse the arguments
ap = argparse.ArgumentParser()
ap.add_argument("-m", "--model", required=True,
help="neural style transfer model")
ap.add_argument("-i", "--image", required=True,
help="input image to apply neural style transfer to")
args = vars(ap.parse_args())
# load the neural style transfer model from disk
print("[INFO] loading style transfer model...")
net = cv2.dnn.readNetFromTorch(args["model"])
# load the input image, resize it to have a width of 600 pixels, and
# then grab the image dimensions
image = cv2.imread(args["image"])
image = imutils.resize(image, width=600)
(h, w) = image.shape[:2]
# construct a blob from the image, set the input, and then perform a
# forward pass of the network
blob = cv2.dnn.blobFromImage(image, 1.0, (w, h),
(103.939, 116.779, 123.680), swapRB=False, crop=False)
net.setInput(blob)
start = time.time()
output = net.forward()
end = time.time()
output = output.reshape((3, output.shape[2], output.shape[3]))
output[0] += 103.939
output[1] += 116.779
output[2] += 123.680
output /= 255.0
output = output.transpose(1, 2, 0)
print("[INFO] neural style transfer took {:.4f} seconds".format(
end - start))
# show the images
cv2.imshow("Input", image)
cv2.imshow("Output", output)
cv2.waitKey(0)
The code uses a library called torch and Opencv to blend the pre-trained models of style references to any desired photos.
If you are an IB DP student, then you must be familiar with the areas of knowledge in TOK - art being one of them. How is art classified as knowledge? What determines styles? In this example, the neural network extracts and compares how "correlated" each layer is and matches them with the corresponding layer on the output image. As a result, the brushstrokes, colors, and shapes are transferred to the final image and that image seems to inherit the styles of the reference image. But again, what is style? And what does it mean for an algorithm to capture style, a human perception that differentiates an artwork from another? It is truly breathtaking to live in the digital age in which artificial intelligence is infused into the everyday life.
https://www.pyimagesearch.com/2018/08/27/neural-style-transfer-with-opencv/ https://arxiv.org/pdf/1508.06576.pdf https://towardsdatascience.com/how-do-neural-style-transfers-work-b76de101eb3
