Transforming normal images into expressive portrait videos using AI tools; check here

The Artificial Intelligence (AI) race is rapidly transforming every industry, and Alibaba has recently introduced new AI tools to support merchants on its platforms. These tools boast impressive features that are truly mind-blowing.

One of the remarkable capabilities of these AI tools is their ability to make images talk, sing, and come to life from any audio file with astonishing accuracy.

Additionally, they are equipped with image and video generation capabilities for promotional purposes.

Introducing EMO: Emote Portrait Alive - Generating Expressive Portrait Videos with Audio2Video Diffusion Model under Weak Conditions by Alibaba

Keywords: Diffusion Models Video Generation Talking Head pic.twitter.com/xCRyCAKICH

— Miles AI-Chan (@the_milesinfo) March 3, 2024

Not only do these tools promise to save merchants a significant amount of money, but they also aim to enhance their productivity and efficiency on the e-commerce platform.

Alibaba's Emote Portrait Alive is a prime example of these advancements. This innovative technology uses artificial intelligence to animate portraits, infusing them with dynamic and nuanced emotional expressions. Users can now generate a range of emotions and showcase their creativity with ease.

According to reports, Alibaba provides a Python SDK that allows developers to seamlessly integrate Emote Portrait Alive functionalities into their applications.

ALIBABA's EMO AI (Emote Portrait Alive) is amazing!

Single photo or even drawing to an emotive, expressive. video that looks real!

From their GitHub:
Portrait Alive - Generating Expressive Portrait Videos with Audio2Video Diffusion Model under Weak Conditions (Institute for… pic.twitter.com/frWrvHmFjT

— Connecting the Dots⚡🚗 to Disruption🌎🚀🔴✨🏄‍♀️🌟 (@ConnectingODots) March 3, 2024

#EMO (Emote Portrait Alive)
Img Input: AI Mona Lisa generated by dreamshaper XL
Vocal Input: Miley Cyrus - Flowers. Covered by YUQI pic.twitter.com/eIiWo1B0Z6

— Laurent Lequien (@laurentlequien) March 4, 2024

Esto es ALUCINANTE.

¡Esta IA puede hacer que una sola imagen cante, hable y rapee a partir de cualquier archivo de audio de forma expresiva!

Presentamos EMO: Emote Portrait Alive de Alibaba.

10 ejemplos salvajes ↓

1. AI Lady de Sora cantando Dua Lipa pic.twitter.com/qzWuWcmLy3

— Adam (@Adamaestr0_) March 3, 2024

The above video demonstration of the Alibaba AI tools showcases its ability to generate various facial expressions and head poses, surpassing previous methods used by studios like D-ID or HeyGen.

The research paper by Alibaba highlights the use of two attention mechanisms: Reference-Attention and Audio-Attention.

Reference-Attention ensures that the animated face retains the likeness of the person in the reference image, maintaining their identity.

On the other hand, Audio-Attention helps synchronise the movements of the animated face with the sounds or words in the audio clips.

For further details about these AI tools, you can explore their research paper here.