-
Phoemo (pronounced "fo-mo")
-
Realtime emotional analysis done from a remote recording device
-
Interactive crowd analytics displayed
Inspiration
An hour before the hackathon, we had the perfect idea. 5 minutes after check in, we threw it out the window. Talking to the Microsoft Team exposed us to machine learning technologies that we never even imagined being able to use. Intrigued by Microsoft’s facial and emotional recognition APIs, we wanted to combine our passion for creative solutions with the novelty of machine learning to develop a unique, integrated solution to help improve people’s experiences in a variety of contexts.
We envisioned a product that would promote public safety by detecting emotional distress in a group, enhance people’s experiences in crowded areas by providing real-time feedback on their moods so that an appropriate action may be taken, and enhance product evaluation by automatically gathering data on an audience’s reactions.
What it does
Phoemo, a concatenation of “Photo Emotion” (pronounced “fo-mo”), is an integrated solution that observes and performs statistical analysis on the emotions displayed in a crowd environment. It provides a modular, interactive interface that is suitable for use in virtually any crowd situation. An external recording device can wireless connect from anywhere to a portable, central processing unit and instantly calculate statistics on the emotional ambience of a group of people’s surroundings to provide a holistic evaluation of the scene. Check out an example here
How we built it
We leveraged the power of Microsoft’s centralized cloud computing resources — the Cognitive Services API — to integrate emotional recognition into our project. We used a centrally-located Raspberry Pi running a minimalist node.js HTTP and web socket server to communicate with input devices, serve static content and a dynamic data aggregation model, and query the Microsoft API for emotion data. Both the input devices and the front end communicated with the server using sockets, allowing for real-time updates to the emotional analysis.
Challenges we ran into
One of the major setbacks of Phoemo was the delay between how quickly we received video data from an input device versus how quickly we could query the Microsoft API (limited to 20 requests per minute), tabulate the data, send it to the client, and render it. We attempted to solve it by writing an asynchronous worker queue to prevent overloading the already-weak Raspberry Pi and overwhelming the client, consequently causing a delay.
Figuring out how to effectively implement Data-Driven Documents (d3.js) was difficult, especially with its incompatibility with angular.js.
Accomplishments that we're proud of
Although we faced hardware and network constraints, we were able to communicate relatively smoothly across multiple devices simultaneously. We took on the challenge of working with a completely new field of computer science, and the fact that we were able to complete a machine learning task in and of itself was a major accomplishment for us.
What we learned
Before coming to LA Hacks, we had been mostly been aware and immersed in the social network aspects of computer science and had never thought that we would be able to write a machine learning application. We had always heard the phrase machine learning thrown around, but we could never really physically understand what that entailed, or the unchartered possibilities of machine learning. Talking to the Microsoft Team and learning about their APIs shifted the paradigm of the problems we wanted to solve.
What's next for @217 Phoemo
Get a better camera, test the limits of the microsoft API by sending more images of larger groups of people. Faster hardware to create a smoother data analytic experience Create an integrated, portable, stand-alone solution that houses the entire system (high resolution internal camera, Raspberry Pi) in a case the size of an Apple TV with ports for power and a network connection. — uses are for analysis in static locations. Use improved hardware and networking to allow for remote data collection that is transferred to a central, modular system which can be set up anywhere to allow for mobile applications such as field testing, drones, and more.
Log in or sign up for Devpost to join the conversation.