Interacting with Pepper

A series of deployments

2018-2019

Human-Robot Interaction, Voice Interaction, Software, Observation, Deployment, COSMOTE-DT.

Pepper is a customisable humanoid Robot provided by Softbank. Pepper supports programming intuitive interactions that involve natural language, gestures, face recognition and screen interaction. In COSMOTE-DT IT Innovation Team, we customized the Robot to engage, entertain and assist customers in retail shops. In collaboration with Athena Research center, our team was working towards integrating Pepper services with a custom NLP to improve speech recognition and support a wider range of conversations in Greek language with diverse users, as well as improving gestural and voice response sequences. Our aim was ultimately the long term deployment of Pepper in key COSMOTE stores to support and entertain customers - e.g. advertise products, provide instructions on where to find a product and answer questions on prices and special offers.

Through a series of demo deployments in a COSMOTE flagship stores at Athens, our team collected observations to support research activities, engaging customers in interacting with the Robot. Initially, these demo deployments were mostly entertaining: for example, we programmed Pepper to provide real time weather forecast for the next hours or tell jokes. Interactions grew complex over time: for instance, scanning someone’s face, identifying age and gender, and providing targeted advertisement of products through voice and visual feedback (tablet screen).

My role in the project was twofold. I worked with the rest of the IT Innovation team in designing and programming apps for demo deployments. This involved designing dialog flows, ext to speech/speech to text conversions, front end development in python integrating various APIs, designing interactions for tablet (content, UI), testing and debuging.

Second, I collected field observations to support research and development activities. My particular contribution was in improving people’s response behaviour to the robo through combining both screen and voice interaction loops. A sub-area of interest included playfull interactions with children, studying children’s behaviour around the robot in the store and designing games combining voice and screen interactions, as well as imitating body movements.