26 Jan 2023
What if we could bask in history and art in the caves of Dunhuang for one second, and stand in front of Buddhist monuments in India the next? What if we could watch a ballet performance where the dance keeps evolving to our every whim? What if we could erase geographical boundaries and share viewing experiences with people from across the world at the same time?
Some of these novel experiences have already become reality at HKBU’s Visualisation Research Centre. Home to the world’s first 360-degree immersive LED Visualisation Cinema and the 180-degree iDome Cinema, the newly established facility will provide innovative platforms for theatre, dance, music and sports, and transform multimedia archives into post-cinematic encounters that people can explore and experience.
Taking art appreciation to a new level
The Visualisation Research Centre is one of the key deliverables under the two-year art tech project “Future Cinema Systems: Next-Generation Art Technologies”, which is led by HKBU in collaboration with the City University of Hong Kong (CityU) and the École Polytechnique Fédérale de Lausanne (EPFL) in Switzerland. The pioneering project has been awarded HK$35.4 million in funding from the Innovation and Technology Support Programme under the Innovation and Technology Commission of the HKSAR Government.
“With the new appreciation of the relationship between art and technology, there has been a global demand for new forms of cultural experience and digital entertainment. At a time when technical capabilities are developing rapidly, our challenge is to create new artistic platforms and new creative opportunities,” says Professor Jeffrey Shaw, Chair Professor of the Academy of Visual Arts at HKBU, who leads the project.
A ground-breaking visionary in new media art, Professor Shaw has long been experimenting with new technologies in his works. Through the Future Cinema Systems project, he is working with artificial intelligence experts at HKBU led by Professor Liu Jiming, Associate Vice-President (Research Development) and Dean of Science, to leverage technological breakthroughs in the fields of visualisation, human-computer interaction and co-evolutionary narrative to explore the future experience of archive, place and performance.
New forms of immersive experience can now be enjoyed at the 360-degree cinema at the Visualisation Research Centre, which showcases the unique outcomes of the Future Cinema Systems. Walking into this first-of-its-kind cinema, which is eight metres wide, four metres tall and consisting of over 400 LED panels (26 million LED pixels), the audience can fully immerse themselves in a three-dimensional environment. Viewers are not merely looking at panoramic movies or photographs; they also get the sense of stepping inside a spectacular virtual landscape, whether it is an exhibition hall at a museum, historical monuments across the ocean, prominent heritage sites or even the setting of video games. This expansive visual experience is amplified by a 32.4 channel surround sound system that further substantiates this total sense of immersion.
Using high-resolution stereoscopic 3D technologies, the cinema lets viewers interactively engage with the visualised content by enlarging the focus on their areas of interest. This in turn adds new layers of understanding and meaning to the materials they see.
Expanding the limits of cinema
Besides creating unprecedented viewing experiences, the new cinema at HKBU further challenges the conventional passive relationship between the audience and the screen.
Professor Shaw says when watching movies or a theatrical performance, people usually just sit tight in their chairs in the dark. With the Future Cinema Systems, interactivity will play an important role. “It is very important that the new cinematic space invites social interaction between the viewers,” he says.
This is made possible by the use of LED in place of a typical projection system. The bright light-emitting features of LED allow the viewing environment to be fully illuminated, which is similar to a daylight situation where people are fully cognizant of each other’s presence. In addition, the images on the screen are not hampered by the audiences’ shadows. “We are developing technologies that will enhance interactivity between individuals and groups of people. In our cinematic environment up to 30 visitors can interact with the imagery and with each other,” says Professor Shaw.
Interactivity and connectivity can also take place on a global scale. With 360-degree projection systems installed at both CityU and EPFL, the local audiences and those telematically connected between multiple locations can share content and interact with each other simultaneously. Professor Shaw continues, “Our cinematic environment at HKBU will be connected to the others in Hong Kong and in Switzerland, and audiences at each location will share its imagery and interact with each other in real time. This is the essence of a globally shared immersive experience.”
AI-powered evolving interactions with the audience
To further innovate and explore the new scope of future cinema, Professor Shaw and his team are utilising technologies that can record human situations, responses and actions to enable human-computer interaction. “In a traditional movie the narrative is fixed as it is set by the director and screenwriters. In our project, we are more interested in developing a cinematic experience that is fully interactive and where the narrative becomes emergent,” he says.
Using wearable sensors and biometric devices, this new cinematic platform can for example track the viewers’ physical position, behavior, gesture, facial expression, body temperature and heart rate to understand what they are looking at, how they are behaving with respect to other viewers, and how they are responding to the content being presented to them. A co-evolutionary narrative engine, which comprises artificial intelligence (AI) and machine learning, will enable the audiovisual manifold to react and respond to the sensory prompts provided by the participants.
The goal is to provide a natural and seamless mixed reality experience where machine agents have purposeful and evolving interactions with the audiences, according to Professor Shaw. “The computer that is controlling the imagery can understand the interaction with the participants, and it is able to dynamically modify the content in relation to what is happening live with the audience,” he says.
Taking the example of a dance piece, he explains that the performance can continuously evolve and develop new dance expressions in response to the interaction with the audience. This will result in complex and engaging narratives where the virtual world is influenced by and in turn influencing the interactions with the public in the real world.
Boundless applications in the arts and creative industries
The other innovative facility at the Visualisation Research Centre is the iDome Cinema, which comprises a laser DLP projector, a vertical hemispherical screen that is four metres in diameter, and surround sound audio equipment.
Ideally suited for museological exhibitions, the iDome Cinema uses a fisheye lens to project spherical photos and videos that can be interactively rotated. Visitors to the iDome Cinema can almost literally walk inside its 360-degree representations to enjoy this immersive interactive experience. The facility can also be used as an educational tool for visualising scientific data.
With the new facilities in place on campus, Professor Shaw and his team are working to produce scaleable outcomes that can be experienced by the general public. The deliverables of the Future Cinema Systems project will first be deployed at selected facilities in Hong Kong, such as the Hong Kong International Airport, M+ Museum and Tai Kwun.
“We are developing the project in particular relation to Hong Kong’s cultural contexts. It will give local artists and institutions opportunities to explore the next generation creative potentials of art and technology,” says Professor Shaw. “We are about to experience something truly transformative, as the Future Cinema Systems project opens the door to boundless new horizons in the arts and creative industries.”