Modernising silent movies with AI

22 Feb 2023


“Silent pictures are the true motion picture form,” said Alfred Hitchcock, one of the most influential filmmakers of the 20th century. While silent films are now no longer as popular as they once were, they are still an art form with unique aesthetics and storytelling techniques. These early motion pictures also offer an important window into our collective past.

To enable modern moviegoers to better appreciate silent films, a team of researchers at HKBU are using artificial intelligence (AI) technologies to develop novel ways that can rejuvenate classical silent films.

Renewing silent movies through modern technology

“Silent movies are important archives of our history, and their roles are just as crucial as cultural relics around the world. However, few people can appreciate their value due to film degradation over time,” says Dr Wan Renjie, Assistant Professor of the Department of Computer Science, who is leading the research project. Drawing on his expertise in image restoration, Dr Wan aspires to help contemporary audiences understand human experiences in the past by giving silent movies a second life.

The two-year research project harnesses the latest AI technologies to enhance the viewing experience while preserving the films’ aesthetic values. Collaborators of the project include Dr Xue Wei, Assistant Professor, and Professor Yuen Pong-chi, Associate Dean of Science (Teaching and Learning) and Chair Professor of the Department of Computer Science, as well as Dr Kelvin Lee, Associate Head and Associate Professor of Practice of the Department of Interactive Media. The project has been recently awarded the Blue Sky Research Fund, which is a funding scheme set up by HKBU to encourage curiosity-driven research to drive the development of science, humanity and the arts.

Reconstructing an audio-visual experience

For audiences who are accustomed to the immersive audio-visual feasts produced by new technologies, they might be less interested to see black and white silent films. Therefore, one of the focuses of the research project is to add plausible colour information to the monochrome images.

The research team is training machine learning algorithms to generate colours to the early motion pictures. Dr Wan says, “We are applying image processing techniques and computer vision to colourise and enhance low-quality monochrome video frames. This transforms the frames into colourful content that is more aesthetically appealing and relevant to modern audiences.”

Generating audible dialogue is another key aspect of the project. Silent films often used intertitles, which are filmed text edited into the midst of the video frames, to convey plot elements or the dialogue between the characters. Based on the information on the intertitles, the HKBU team is developing audio processing technology that can transform the text into audio. In addition, the project will analyse the oral movement of the actors in the video frames to generate spoken dialogue.

Due to film deterioration, some video frames of the silent movies are lost. To complement the lost information, the research team is deploying generative models to restore the frames and generate new content based on the descriptions in the intertitles.

A new way of preserving history

While the project is still underway, Dr Wan believes the technologies being developed by the research team will have great potential for broad practical applications. “Our research can provide a new way of preserving historical relics. Silent movies are digital resources of history in our current society, and we must make them relevant and accessible. The best way to preserve this type of digital archive is to rejuvenate it,” he says.

He further points out that the project can open new prospects in the research of image processing technologies, as well as creating new opportunities in their application.

Dr Wan’s other research interests include computational photography and 3D vision, and he has previously worked on research projects which focused on developing image processing technologies for reflection removal, low-light image enhancement and image restoration. A fan of photography, he is working with researchers at Peking University to deploy AI algorithms into newly developed cameras and examining further applications of AI in the arts.