spotsize AI helps shoe shoppers determine size, improves experience

spotsize has developed a solution which turns online shops into the shoe-shopping experience customers are familiar with from brick-and-mortar stores. Using an augmented reality (AR) solution, the spotsize application takes over the shoe-fitter’s role. With the help of machine learning, the program uses a smartphone camera to measure customers’ feet to determine the appropriate shoe size. This eliminates the need for trying on multiple sizes and reduces the number of returns which involve costs and effort for shops and customers. spotsize was looking for a partner to implement the ML components of the app − and they found it in inovex.

Digital 3D modelling for feet

Before sizing can take place, however, a 3D model of each customer’s foot must be created. In order to ensure reliable measuring and sizing, inovex integrated an image classification solution into the spotsize program. As object recognition via smartphone is still a new application area, however, this challenge required additional research. Firstly, the technology used for 3D object classification had not yet been extensively tested with end-users, and secondly, there are not yet many use cases for three-dimensional depth measurement using smartphones.

The proof of concept (PoC) was completed after three weeks, and the solution was integrated less than four months later. Images of customers’ feet are analysed by a machine learning algorithm, while deep-learning models trained to identify foot shapes are used to exclude distracting elements, such as objects in the background of the images, from the measurement process.

Use of technology

Due to the comparatively new technology used, there were initial issues with the training data which needed to be resolved. Instead of using existing training data, custom models were trained. This ensured better object recognition.

MLFlow was used for managing the models and administering the experiments, while Apple CoreML was used to convert the models for use by smartphones. Evaluations were carried out using Jupyter notebooks. spotsize ultimately captured and labelled the datasets and integrated the models into the app, and integration into the SAP Cloud followed.

The app enables a digital 3D model of each customer’s foot to be created and its length, width, and height to be calculated. Additional data can be read from the 3D model. The biometric data collected using an AR interface also flows into a big data solution, which will help to detect and recommend shoe sizes even faster in the future. The system is self-learning and becomes more effective with usage.

Conclusion

The use of new technologies, such as 3D object classification using smartphones, has created a unique AI solution which harnesses the full potential of smartphones to create a state-of-the-art e-commerce experience. This project required a flexible response to technological limitations, and multiple approaches were researched.

The solution ultimately selected was so successful that more features are now available for end-users than were initially planned. Instead of having users take individual photos, images of their feet can now be captured using AR scanning. This improves the user experience even more.