Using Machine Learning to leverage the power of Augmented Reality

Augmented Reality (AR) in its current definition is the overlay of digital information on a real-world view. In practical terms, it is the process of recognising specific physical objects in a device’s camera and superimposing digital content such as video, audio or 3D models.

Visual recognition is one aspect of AR which encompasses image, object, scene and facial recognition. Computer vision technology is used to identify shapes and patterns through a complicated set of mathematical models. These models and processes are all facets of Machine Learning (ML) that drive Artificial Intelligence (AI).

ML is the science of “teaching” the system to look for commonalities and patterns and assessing the probability that a match is found. Effectively, with a set of mathematical models in place, the system is fed a collection of information that represents a positive match. For instance, if we want to teach the system to identify a cat, we provide thousands of images of cats and let the system process and find common visual patterns across all the images.

This is known as deep learning where the outcome is a system that can recognise and track almost any pattern. With this capability, we can inject a virtual projection into the area that is being recognised and tracked to deliver, what is called, an augmented reality experience.

The power of AI and ML is being able to make decisions based on the real-world scenario. Let’s consider its application in a security surveillance system. A machine that has been trained to detect weapons, such as knives and guns, can be used to observe CCTV security vision. In real-time, it can look for patterns in the scene that resemble its definition of a weapon. If identified, a notification alarm could be raised for someone to act.

Pattern recognition is not limited to visual only. Auditory, gesture and other data patterns can also be “taught” using ML. Continuing with our security surveillance example, a trained machine could be used to listen to sounds in the environment and detect patterns of shouting or offensive language being used.

The challenges

One of the hurdles in training a machine to identify patterns is sourcing enough material that is deemed a “positive match”. In these cases, systems are designed with feedback loops to allow machines to “learn by experience”. If for some reason the machine fails to detect what it is supposed to, it can be taught what was missing in the initial dataset and be trained to act on it the next time it occurs. All this is supported by an aspect of ML called “convolutional neural networks”. Different nodes that perform specific mathematical functions on the dataset are interconnected to achieve the specified outcome.

The opportunities

In a time when vast amounts of information is available at our fingertips, being able to recognise the world around us and decipher what is relevant will become critical. Whether at work, at home or in a social setting, successful real-world augmentation will rely on AI and ML observing and recognising our environment and adapting information to match our situation.

As hardware technology improves and wearable, handsfree devices become a reality, ML and AR will become an integral, yet ambient part of our lives.

Simon Galanakis is a passionate advocate of effective AR experiences and is currently Appearition’s Platform Architect and Senior Solution Designer.

The changing face of a workplace

Disruptive technologies like Augmented Reality (AR), Virtual Reality (VR) and Artificial Intelligence (AI) are redefining the workplace. Several jobs’ skill sets are constantly evolving, and several others’ scope has been widening.

Appearition and its team of researchers are always working towards solving tomorrow’s problems, thereby enabling others’ success. One such area of our research includes a project on drawing on immersive games experiences to develop tools for workplace support in the digital economy.

It is a collaborative research project between Appearition Pty Ltd and the Swinburne University of Technology. The project combines AR and AI informed by decades of game design, to develop a support platform for future modes of collaboration, workplace learning, and decision making.

‘Appearition is very committed to this research partnership with Swinburne University. Research and Development is one of the three main arms of our company. All our platforms and services have been developed after years of rigorous research and testing. We are also constantly working towards upgrading them to fit evolving technology and people needs,’ shares Vivek Aiyer, Founder, CEO, Appearition.

A transdisciplinary research team from Games and Interactivity, Immersive Experiences, and Data Visualisation are working with us, and the Swinburne Smart Cities Research Institute, to develop a prototype platform for supporting future workplace models in urban environments rich in IoT technologies.

The why

As technologies reshape our workplace, it will require a new understanding of work to support these changes. Ventures such as Airtasker, Uber, and Airbnb are challenging traditional workplace models. They are predicated on fast-paced real-time interactions between employees, often geographically remote from each other. Although relatively simple, the core concepts of these new business structures can be applied within enterprises and between enterprise and external clients. Network connectivity, rapid exchange of data, and mobile display technologies are some of the well-established characteristics of smart city environments in which this work takes place. New platforms are needed to support these new work models, because they require employees to share and respond to real-time data quickly and collaboratively, in an improvisational construct.

The How

Online multiplayer and location-based AR games provide a readymade platform for investigating how to achieve new interactions between employees. Multiplayer games host complex ecosystems of thousands of players with distinctive attributes and experience levels. Location-based AR games embed information within urban environments ensuring data is ready-to-hand. ‘Our project aims to develop a prototype platform to support the needs of workers in the digital economy using proven methods from these game design platforms. Collaborative task attainment, group formation, decision support, environmental information overlays, role management, inventory and setting of goals are key features that the prototype system will benefit from in drawing on game design to enable smart citizens to respond to change in the workplace,’ explains John McCormick, Lecturer in Interactive Media, Swinburne University.

Games are familiar to a large percentage of next-generation work seekers and therefore do not pose a huge cultural shift. The innovation in this project is to integrate widely understood methods for player organisation and interaction with a new approach to integrating AR and AI. These technologies are changing the nature of smart cities to work beyond data collection and IoT networks to explore how these can be made tangible and relatable to those living and working in urban environments.

AR does this through the real-time integration of data with the user’s environment and context. AI filters and connects the rich data of the IoT environments of smart cities to manage interactions between players, non-player characters, and the urban environment. While integrating AI and AR is not in itself new, our project will address how the relationships between users within a company can be supported and facilitated within this type of platform. However, these technologies can be disruptive in the workplace. Drawing upon game design methodologies this project will generate a new approach to the management of dynamic, timely information displayed fluently in the workspace.

Watch this space to know more on Appearition’s research projects. Get in touch with us to know more on using AR for your business!

Augmented Reality – The future of retail industry

Home delivery, telephone order and e-commerce. What do these things have in common? They changed the way we shopped. Introduction of these technology/ideas, enhanced our shopping process. Similarly, Augmented Reality (AR) is set to alter the face of retail industry.

AR is highly scalable and has multiple uses across this industry. A look at some interesting ways in which AR is expected to grow:

Trial/testing of a product:

AR helps to bridge the gap between a customer’s perception of the product and the reality. This is the quintessential need for physical shopping or trial for a product.

Furniture: In 2013, IKEA – the Swedish retailer tested the ropes of AR, by launching an AR-based app. It used AR-overlay of 3D models of IKEA’s products on the real-time feed of the camera. It helps visualise how a product would look at a given space.

Shoes: Sneaker-maker Converse created an AR app in 2010. When one points a camera at their feet, the app overlays a projection of the shoe on their feet.

Clothing: Ever stood in a never-ending queue ahead of the trial room only to find the dress doesn’t fit you? Well, AR is about to change the game in clothing trials using overlay.

Japanese retail store – Uniqlo tried to address this concern by enabling an augmented trial room. This room had a mirror with an LCD screen that let you choose the apparel you wish to try. The app then overlays different colors of the clothing to help you make the best choice possible.

Virtual makeup trial: Sephora, the makeup retailer launched an app – ‘Sephora Virtual Artist’ that enabled overlaying different makeup looks. Once the customer scans the face using a camera, the app detects the different organs like nose, lips, eyebrows, skin etc and allows you to try a shade.

 

Product information:

When shopping at a busy super-market, we are often sceptical of a product’s ingredients. The aisles are long and nearest assistant is pre-occupied. What if one had a personal assistant to share further information on every product? AR does just that, acting as your personal shopping assistant. Several retail big-wigs like Walmart, Tesco, Carrefour have been testing and experimenting with AR in shopping.

Chinese e-commerce retailer – Yihaodian, opened virtual stores across the country. This enabled consumers to shop on the go. The app has virtual shopping aisles; customers can select and arrange for a home delivery.

AR is already booming in several such retail verticals. Add the magic of AR to your business. To know more, drop us a message.

This blog is a re-written excerpt from an article first published in the Augment.com

Image Courtesy (in the order of images used): 1, 2, 3, 4

AR Digital Transformation and the Indian Market

Ravichandran Lakshminarayanan, Member, Board of Advisors, Appearition India speaks on Augmented Reality, Digital Transformation trends and the Indian market and more:

Your take on the AR/VR market in India.

Though AR/VR arrived, so to speak, maybe about three years back, significant impact or adoption (usage) is yet to be seen, be it in the consumer space (though we have VR centres in some malls or VR headgear sold with smartphones) or in the enterprise space.

AR/VR is certainly going to be a game changer for industries like retail, real estate, entertainment, and tourism in India.

Education is another area where we will see significant AR/VR adoption. (I foresee not only schools & colleges taking advantage of the power of AR/VR, but many other formal and non- formal fora including corporate/industrial training will take advantage of these frontier technologies).

The mobile market in India – Smartphones undergo constant metamorphosis. What features do you think they need to adapt to stay abreast with the AR/VR tech?

India is a price sensitive market. Today there are more than 300 million mobile phone connections, in India. Mobile phone sales literally skyrocketed, only because device prices came down drastically.

Mobile phones are going to drive AR/VR usage and it is very important that the AR/VR capable phones are priced right for the Indian market.

Next is content in local languages like Tamil, Hindi, Telugu, Kannada. For AR/VR to drive mass usage/adoption, local language content will be a key.

Last but not least, the devices(phones) will have to be more user-friendly than they are now.

 Your take on digital transformation for businesses these days.

Today there is no business that can shun or stay away from digital transformation. The degree of transformation or rather what percentage of the business is digital versus non-digital may vary from business to business but there’s not a single business that’s 100% non-digital.

In the Indian context, this is so particularly after the introduction of GST.

Though it has taken time, businesses have understood the benefits of digital transformation.

How has the Indian market reacted to digital transformation so far? What are some factors that seem to aid it/ factors that act as a barrier?

The Indian market was quite reluctant to accept/embrace digital transformation, initially.

The reasons were:

  • the market did not understand what digital transformation is.
  • the market could not see value in digital transformation immediately
  • digital transformation was taking firms/players out of their comfort zone
  • firms were unsure of the payback period if they opted for digital transformation.

Slowly but surely as a few early adopters took hesitant steps towards digital transformation and started ‘seeing’ the benefits, more and more firms followed. Today, I dare say, there are very few firms, if any, that have not been touched at all by digital transformation.

Today everybody understands that digital transformation is not only inevitable but benefits all. Firms are still evaluating their ROI and hence the pace of adoption is not ideal yet.

There are obstacles and challenges, the foremost being connectivity not only in terms of speed and price but in terms of reach. There are still many areas in the country where connectivity is very poor or non-existent.

Technologies like 5G and other indigenous technologies and solutions, will hopefully, address this issue.

What are some barriers that companies face while venturing to a new geography?

In my opinion, the most important barrier is the culture. The culture in each geography is unique and the sooner companies understand the local culture the better for them. The next important challenge is to understand the local laws, business environment. There could be challenges in effective communication besides language differences. The pace at which things move, including the pace of business negotiations, can vastly vary. Distance and time could be a challenge, too. Finding and hiring people who are trustworthy and competent can be a challenge. Establishing franchises, signing up agents and other business associates may take longer than what companies ‘back home’ is used to.

What sort of obstacles did you face while setting up your e-commerce venture and how did you overcome them?

The very first challenge was not being able to register a domain name, from India.

(Internic was the only share registry and for registering any domain one had to pay $100; the only way to pay was online; this was not possible from India. I had to take the help of my sister in the US, who paid this $100 and we registered chennaionline.com on Aug 15, 1997.)

One of the foremost challenges for a startup in a sunrise industry, which the online space was, way back in 1997, was getting the right people to come on board. We leaned on friends and other contacts to get the initial few hires.

The next challenge was funding –VC/Angel funding culture, back then, was literally nonexistent. Raising capital as equity was very difficult if not impossible. Debt funding or in Indian parlance, a bank loan was the only option.

Banks lend against tangible assets – land, building, plant & machinery – and also insist on security for the loan. Here was a business that had no tangible assets (except some PCs and servers and switches and modems – assets that depreciated faster than the mercury rising in Chennai summer) and a business model that was at best vague (actually from the bank perspective, it was all Latin and Greek).

The Bank Manager professed he understood nothing but said he was impressed with my sincerity of purpose and sanctioned the loan. (“As a breed, we are risk averse, but if we do not take some risk to support highly qualified technocrats, when we get an opportunity, then we do not deserve to be sitting here”, he said!)

Though the potential of the web/online was fairly well established globally, India was slow to adopt or embrace this ‘new economy’ and hence every ‘pitch’ had to be from ‘ground zero’. One of our (founding team) important roles was to evangelise Internet & e-commerce; evangelise we did with passion and enthusiasm. This not only got us noticed but became a competitive advantage, too.

There were many more challenges including getting the right office space, etc.

There were so many Foreign Exchange rules & regulations that receiving or sending money out of India was a huge task. We did $ transactions wrestling with a plethora of forms and multiple agencies.

Do you see similar obstacles for firms these days, despite a bridge that filled the knowledge gap?

Certainly not. Thanks to information/knowledge available in one click, for anyone, and the more business-friendly environment, many of the earlier day business challenges do not exist. However, each new firm (or old firm) has a unique challenge. It is much like a baby’s growth – from conception to infancy to childhood to adolescence to adulthood – at every stage in the life of a firm, there are unique challenges. A successful firm is one which understands this and also understands every challenge is a possible opportunity.

Shaping the role of educators with technology

 

Digital transformation will impact job roles in future. Hence, it is important to shape the role of an educator. Market leaders predict that Augmented Reality and Virtual Reality (AR/VR) will change the basics of how teachers teach.

E-Learning

Immersive technology will enter e-learning and classroom learning in near future, making teachers mentors first, says Michio Kaku, Physicist and Author. A recent Technavio study predicts that online higher education market in the US is set to grow at a compound annual growth (CAGR) rate of 20% until 2021.

If teachers start using technology as a medium of instruction whilst teaching, it will enable students to learn the ropes of this at an early stage.

Some ways in which teachers can use AR/VR technology in classrooms:

  1. Smart class: Moving away from traditional projectors or picture projects, teachers can use AR to overlay an animal or forest picture for a social science class or produce a square or hexagonal shape for geometry.
  2. Presentation skills: Adding fun to Powerpoint, teachers can promote the use of AR/VR and prompt students to learn further by pointing at an object.
  3. Beyond classrooms: AR/VR can be used anywhere with just a small, handheld device. This can enable teachers to make field trips and lab visits even more interesting with prompters and overlays.

The Technavio report further says AR in education market will grow steadily at a CAGR of more than 82% by 2021. Educationists say, students are increasingly interested in learning things that are aided with an augmented overlay or e-vision. In such an environment, the students are completely engrossed in the space around them. These enhance students’ cognitive and interactive skills.

Based on an article published in Edtech and EdSurge.