A collaborative effort between GPT-3, Midjourney, and myself.
If you follow my work, you may have noticed that I’ve started to use cover images created with AI.
For those of you who don’t know about it, Midjourney is an independent research lab that came out of stealth a few months ago with the intention of “exploring new mediums of thought [and] expanding the imaginative powers of the human species.” Midjourney is also the name of the text-to-image model I used to create the images.
I’ve been looking for new ideas to explore with AI visual generators for a few weeks now. This experiment provides a way to visually contrast the creative mastery of some of the best human painters in history with the ability of these AI models. I created an almost automated end-to-end process with GPT-3 and Midjourney — although I’m in the loop to help them communicate.
I chose famous landscape paintings for three reasons. They’re very recognizable, GPT-3 can easily provide a visual description to design a prompt, and Midjourney excels at painting landscapes.
Once I selected the paintings, I had to prompt GPT-3 to give me the descriptions. I used a generic simple prompt: “Describe with visual details [name of painting] by [name of author] in one paragraph.” I took the first or second completions without editing, even if I knew GPT-3 had missed some elements.
I then adapted the description to a prompt that Midjourney could understand. I added four elements to all prompts apart from GPT-3’s description. In the beginning, I included the medium (e.g. oil on canvas), style (e.g. impressionism), and genre (e.g. landscape). In the end, I added the aspect ratio of the original to make a clearer comparison.
I didn’t prompt the author or the name of the painting to force Midjourney to take only indirect inspiration — as conceived by GPT-3 — from the originals.
I disclosed the prompts for you to try yourself. I’ve realized it’s becoming a common practice among digital artists (and users of text-to-image models in general) to not disclose the prompts. I don’t intend to sell these paintings or gatekeep the skills that are required to create art with AI, so I’ve shared the prompts unedited.
If you want to know more about prompting, I recommend PromptoMANIA as a starting point. If you want to create great prompts without having to learn prompting skills, try this prompt generator hosted on Hugging Face.
If you want to improve beyond that, you’ll have to practice and experiment. I’m not anywhere near the level of some AI artists that have been honing their craft since the beginning of this new AI art scene.
After sending the prompts to Midjourney, it gave me four images to choose from. I upscaled the ones that reminisced me of the original and then chose one.
Each section below comprises the original landscape painting on the left and Midjourney’s recreation on the right. Above the images, I’ve written GPT-3’s description of the original painting and my translation to a syntactically correct prompt for Midjourney.
Enjoy, and let me know in the comments what you think!
The painting is set at night, with a bright moon in the sky. The stars are shining brightly, and the landscape is bathed in a blue light. The trees and houses in the distance are silhouetted against the sky, and the whole scene has a peaceful, dreamlike quality.
oil on canvas painting + post-impressionism + cloudscape + night + bright moon in the sky + shining stars + landscape bathed in a blue light + trees and houses in the distance are silhouetted against the sky + a peaceful dreamlike quality, — ar 5:4.
The painting features a hay wain, or wagon, being pulled by two horses as it crosses a river. The scene is set against a backdrop of mountains, trees, and clouds. The painting is known for its simple and idyllic depiction of rural life in England.
oil on canvas painting + romanticism + landscape + a hay wain pulled by two horses as it crosses a river + a backdrop of mountains, trees, and clouds in the background + simple and idyllic depiction of rural life in England, — ar 30:21.
The painting is of the sun just peeking over the horizon, with the sky a beautiful orange and red. The water is a deep blue, and there are some small boats in the distance. The brushstrokes are thick and textured, making the painting look almost like a photograph.
oil on canvas painting + impressionism + cityscape + sun over the horizon + beautiful orange and red sky + deep blue water + small boats in the distance + thick and textured brushstrokes, — ar 21:16.
The painting is a winter landscape painting that depicts a group of men returning from a hunting trip. The men are tired and their dogs are barking. The snow is coming down and the trees are covered in snow. The men are carrying their dead prey on their shoulders.
oil on panel painting + northern renaissance + genre + winter landscape + a group of men returning from a hunting trip + tired men + barking dogs + snowing + trees are covered in snow + men carry their dead prey on their shoulders, — ar 34:25
The painting is a landscape painting that features a man standing on a rocky outcropping with his back to the viewer. The man is looking out over a foggy valley with mountains in the distance. The painting has a somber tone and conveys a sense of loneliness.
oil on canvas painting + romanticism + allegorical painting + landscape painting + a man standing on a rocky outcropping + back view + The man is looking out over a foggy valley + mountains in the distance + somber tone + a sense of loneliness, — ar 63:79.
In the painting, the city of Toledo is shown in the distance, with a few buildings and mountains visible. The sky is a deep blue, and the light reflects off of the river in the foreground. The colors are very sharp and bright, making the painting seem almost surreal.
oil on canvas painting + mannerism + late renaissance + cityscape + landscape + city of Toledo in the distance + a few buildings and mountains + deep blue sky + river in the foreground + almost surreal painting, — ar 70:63.
The painting is set in the town of Estaque, near Marseille, France. The town is built on a hill, and the houses are painted in a variety of colors. The rooftops are red, the walls are white, and the doors and windows are blue. The sky is a deep blue, and the sun is shining. The painting is very brightly colored and has a lot of movement.
oil on canvas + fauvism + landscape + town of Estaque, near Marseille, France + the town is built on a hill + houses painted in a variety of colors + red rooftops + white walls + blue doors and windows + deep blue sky + sun shining + very brightly colored painting + a lot of movement, — ar 99:80
In “Mont Sainte-Victoire,” Paul Cézanne depicts a massive mountain perched above the French countryside. The artist has simplified the forms in the painting, distilling the scene down to its essentials. The result is a work that conveys the grandeur and stillness of the mountain. The muted colors and soft light add to the sense of calm and peace in the painting.
oil on canvas painting + post-impressionism + landscape + a massive mountain perched above the French countryside + simplified forms + the grandeur and stillness of the mountain + muted colors + soft light + sense of calm and peace, — ar 5:4
Water Lilies by Claude Monet is a beautiful painting of lilies in a pond. The lilies are different shades of pink and white, and they are surrounded by green leaves. The water is calm and still, and the sky is reflected in the surface of the pond.
oil on canvas + impressionism + flower painting + lilies in a pond + lilies are shades of pink and white + surrounded by green leaves + water is calm and still + sky is reflected in the surface of the pond, — ar 4:2
In The Storm on the Sea of Galilee, Rembrandt depicts the Biblical story of the disciples in a small boat on the Sea of Galilee being terrified by a storm. The disciples are shown huddled together in the boat, their faces filled with fear. The waves are shown crashing against the boat, and the sky is filled with dark, menacing clouds.
oil on canvas + baroque + religious painting + Biblical story of the disciples in a small boat on the Sea of Galilee + terrified by a storm + disciples huddled together in the boat + faces filled with fear + waves crashing against the boat + sky filled with dark, menacing clouds, — ar 4:5
As a general note, I have to say that most paintings, although very well painted, are far from recreations of the originals. One reason is that GPT-3 can’t give precise details unless those are very representative of the painting (it tends to be generic).
If I used human-made detailed descriptions of the paintings, I’d have gotten better results. Still, text-to-image models are generally pretty bad at compositionality. That means they have a hard time understanding “X is above Y” or “X is at the right of Y.” They don’t get the relative positions of elements and those are often key distinctive features of the artworks.
Another reason is that I’ve constrained GPT-3 and Midjourney for this experiment. The prompt to GPT-3 is simple and I took the first or second generations. From Midjourney I chose the painting from the first batch of results without editing or inpainting.
One key takeaway from this experiment is that both GPT-3 and Midjourney can give you impressive results as long as you don’t intend to guide them to a predefined result. They’re great with generic creations but not with specific ones (DALL·E 2 is better at this).
In summary, I wanted to see how the GPT-3+Midjourney combo — with me in the loop — would reimagine the original human-made artworks. It was a communication experiment between humans and AIs.
As a final reminder: This isn’t an experiment to prove anything about these AIs’ abilities. It’s just to open a door to their otherwise opaque minds.