AR Friend: Controlling High Fidelity Character Models in an AR Environment

Professor Komura, Taku   (Principal Investigator (PI))

24

2022-08-01

2779576

AR Friend: Controlling High Fidelity Character Models in an AR Environment

AR Friend , Controlling High Fidelity Character Models , AR Environment

Computing Science and Information Technology

ITS/319/21FP

Innovation and Technology Support Programme

2022

On-going

The project provides an augmented reality (AR) platform for controlling virtual agents in aroom, and let them talk to the users as friends.  The objective is to control the agents in arealistic manner as if they are physically located in the room.  Travels are restricted under the Covid outbreak and chances to physically meet and talkto relatives or friends have significantly dropped.  This has led to significant increase inmental disorders, suicide rate and even dimensia.  It also reveals social communicationplays a significant role for mental health.  Although online communication has significantlyincreased in the last few years, the immersion of interaction is poor compared to realphysical communication.In this research, we develop AR Friend, a framework to allow users to communicate withtheir friends in an AR environment.   The AR friend will appear in front of the user whowears Augmented Reality (AR) devices such as HoloLense.  The user can talk andinteract with the agent as if it is in their room.  As the user moves around in the room, theAR friend will respond and walk towards the friend, so that it stays visible to the user.  The AR friend will be controlled either automatically or by another human remotely.  Thefull body motion will be generated automatically, based on the location of the agent withrespect to the user, the environment, and the conversation between the user and theagent.  The motion will be learned based on supervised learning using motion capturedata collected for training.  The AR friend has a significant impact on the society where people suffer from the lack ofcommunication with others due to restrictions.  It will provide an opportunity for peoplelocked in homes to have natural pseudo-physical communications with others.  It also provides a framework for various other applications, such as AR guidance in virtual museums, remote teaching and virtual pets.