Fast neural style transfer II.

Monday. July 06, 2020 - 2 mins

Updates on previous posts

I wrote about neural style transfer previously in this and this post and I finally corrected my code for the latter to work properly. :)

Fast neural style transfer

Since I already wrote a blog post about fast neural style transfer I’ll just sum up the main idea here quickly. Taking a transform network and a loss network jointly where only the weights of the transform network is updated. After convergence the transfrom model can be used in a feed-forward manner to generate stilizied images.

Feature loss

feature-loss

\[L_{content}^{\phi, j} (\hat{y}, y) = \frac{1}{C_j H_j W_j} || \phi(\hat{y}) - \phi{y} || ^{2}_{2}\]

To make the content resamble the output image we simple take the Euclidian squared distance between the extracted feautres from the original image \(y\) and the transformed image \(\hat{y}\).

The style loss is a bit trickier. Since we want to keep the style of the style image and not its spatial features we drop spatial structure in a way that was introduced in the original neural style transfer paper the Gram matrix:

gram matrix

Where \(phi_j\) is a feature map extracted at some intermediate layer from the loss network. Given these Gram-metrices from several intermediate layers we can take the Forbenius-norm of the difference between the Gram-matrix of the style image through the loss net and the transformed image thourgh the loss net.

style loss

Results

Van Gogh	Picasso

To train the network with the imagenette dataset:

pipenv shell
pipenv install
pipenv run python train.py --starry_night --width=<width_int> --height=<height-int>

Examples

Heidelberg - images

Original	Van Gogh	Picasso

Videos

Outside scene with friends

Van Gogh	Picasso

Mount Vesuvio’s inside

Van Gogh	Picasso

There are some burn-out probably due to the original images I’ve taken because the transfer model cannot handle the extreme pixel values. In the Van Gogh stilizied images the whites are overexposed while in the Picasso stilizied images the sky seems to mess with algorithm.

My implementation is available here

@Regards, Alex

References

[1] Original implementation by J. C. Johnson.

[2] A residual PyTorch implementation by E. Linder-Norén

[3] The research paper that introduced fast neural style transfer by J. C. Johnson

Alex Olar

Christian, foodie, physicist, tech enthusiast