It combines cutting-edge text-to-image generation with hand-drawn sketch input. AI algorithms that generate images from input text are currently very popular. OpenAI’s Craiyon (formerly DALL-E mini) and Google’s Imagen AI have unleashed a wave of wonderfully weird procedurally ge

combines cutting-edge text-to-image generation with hand-drawn sketch input.

AI algorithms that generate images by inputting text are currently very popular. OpenAI's Craiyon (formerly DALL-E mini) and Google 's Imagen AI have unleashed a wave of wonderfully weird procedurally generated art synthesized by human and computer imagination. On Tuesday, Meta revealed that it has also developed an artificial intelligence image generation engine, which it hopes will help build immersive worlds in the Metaverse and create advanced digital art.

When using the artificial intelligence generation engine, it takes a lot of work to create an image from just the sentence "There is a horse in the hospital". First, the sentence itself is fed into a transformer model, which is an neural network that parses the words in the sentence and develops a contextual understanding of the relationships between them. Once it has the gist of the user description, the AI ​​will synthesize a new image using a set of GAN (Generative Adversarial Networks).

Thanks to efforts in recent years to train ML models on ever-expanding sets of high-definition images and carefully curate textual descriptions, today's state-of-the-art artificial intelligence can create realistic images no matter what nonsense you feed them. The specific creation processes of different artificial intelligences are different.

Meta AI

For example, Google's Imagen uses a diffusion model, according to a keyword blog in June this year: "It learns to convert patterns of random points into images. These images start with low resolution and then gradually increase the resolution." Google's Parti AI, on the other hand, "first converts a set of images into a sequence of code entries, similar to puzzle pieces. It then translates a given text prompt into these code entries and creates a new image." .

Although these systems can create most things described to them, the user does not have any control over specific aspects of the output image. "To realize the potential of artificial intelligence to drive creative expression, people should be able to shape and control the content generated by the system," Meta CEO Mark Zuckerberg said in a blog post on Tuesday.

The company's "exploratory artificial intelligence" The smart research concept, called Make-A-Scene, works by incorporating user-created sketches into its text-based image generation, outputting a 2,048 x 2,048 pixel image. This combination allows users to not only describe what they want in an image, but also determine the overall composition of the image. "It shows how people can use words and simple drawings to communicate their vision more concretely, using a variety of elements, form, arrangement, depth, composition and structure," Zuckerberg said.

In testing, a human evaluation panel overwhelmingly chose the text-plus-drawing image over the text-only image because it was more consistent with the original sketch (99.54% of the time) than with the original text description. Consistency is better (66% of the time). To further develop the technology, Meta is sharing its Make-A-Scene demo with renowned AI artists including Sofia Crespo, Scott Eaton, Alexander Reben and Refik Anadol, who will use the system and provide feedback. There's no word yet on when the AI ​​will be available to the public.