Sans Hands Touchless, Gesture Based Interactions

With the purpose of allowing musicians to perform uninterrupted, today we are proud to announce that Sans Hands, a mixed reality application that allows you to turn a page on a document using facial gestures, has launched on the Apple App Store and is available on iPhone, iPad and MacOS. Requires iOS 14+ / Mac OS 11+.

We would like to personally thank Ioannis Dimitroulas for his help in the branding, development, copywriting and marketing of the application.

In April 2020, Blanca Azcarraga and Jacqueline Bowler invited our founder over for a lovely evening of music, food and wine. Jackie, who teaches music as adjunct faculty at NYU Steinhardt School of Culture, Education, and Human Development, is an extraordinarily talented pianist that was performing a soothing piece of Jazz so powerful that it captured the entire room.

Performing on a keyboard that held an iPad to display the music score, Jackie’s flow was fluid and mesmerizing; only to be interrupted every time she needed to turn a page. This periodic cavity within her elegant, serene performance brought up a conversation between Blanca, Jackie and Oscar whom questioned what could allow musicians to perform uninterrupted.

Jackie and Blanca brought up that technology, in the form of a foot pedal, does exist that helps musicians play music on an iPad without their hands but they cost between $50-$100. The group considered that this proved that there was a market need for a solution, which currently held an elevated price for something as simple as turning a page on a document. Blanca and Oscar then suggested to Jackie that she could do this with her face - via an inclination or a gesture, such as a smile or a frown - for $5.

Having been educated in Service Design by Steven Dean, we saw this as an opportunity to create a product that reduced an obvious Bio-cost.

Bio-cost is a concept introduced by Paul Pangaro, Hugh Dubberly, and CJ Maupin in their article 'Bio-cost: An Economics of Human Behavior,' which discusses the biological cost involved in human actions and decisions.

In this case, the cost that is reduced for musicians; whilst producing a product that demonstrated the quality and performance of the confidential projects that delasign had done at our time at Firstborn and R/GA for Apple, Nike and Beats.

Codename Marco, after Oscar’s childhood friend whom is an extraordinary musician, we began to prototype in Flutter, to begin to explore the potential of the application. Equipped with Firebase and Google ML Kit we put together a universal app, destined for Android and iOS, that allowed users to add and delete documents as well as to turn a page by inclining their head to the left or right.

In June 2020, we distributed the application via Microsoft App Center to 15 individuals, that ranged from musicians to students and professionals, all whom expressed that there was a need for this kind of technology but felt that there was a greater potential if they could also use their hands from a distance. A moment that coincided with Apple’s announcement of that Hand detection was now available on iOS and MacOS.

Having demonstrated Marco’s potential, we proceeded to carry out a Branding exercise to align on our vision for the platform which resulted in Sans Hands; a suite of applications that would allow users to perform their tasks uninterrupted via touchless gestures that did not require their hands to touch the screen.

Around this time we made the tough decision to drop the Flutter application, which was ready to launch, to develop a future proof application on iOS. This decision was based on the belief that Flutter is extraordinary for regular applications, such as standard applications, social networks or marketplaces but their dependencies may not be recommended for the production of state of the art applications where you want to rely purely on ones creative technical ability.

With this new branding and vision in mind we applied to Apple’s Universal App Quick Start Program, and were thrilled when in August 2020 we were honored with being one of the first to write the future of the Mac.

Our first objective was to demonstrate that all the touchless technologies were available on the Mac (Silicon or Intel) and iPhone and iPad. A pursuit that drove us to demonstrate that one could use their hand to interact with an Apple Computer, iPhone or iPad from a distance and raised the excitement around the company and the staff of the NeueHouse whom had interacted with the technology.

It did raise the necessity to produce something that could visualize the human hand on a screen. A mission that pushed us to dive deep into the Metal Shading Language, Apple’s Core ML & Core Image as well as the sensors that are available across Apple devices. A series of experiments that led us to create a visualizer that is agnostic of skin color and which accurately demonstrates the hand in real time.

Although we will deliver hand based gesture technologies in the future, we evaluated the potential of the hand based gesture technology at a production level and made the decision to create a product that could be used universally across Apple Devices with a subscription model that allows users to access specific features which will only be available on certain devices.

We aim to launch the subscription model along with more facial gestures by the end of Q2 2021.