-
-
Room view in editor mode
Our project is on the healthcare track
Inspiration
About 350,000 house fires happen in the US each year. One teammate recently had a house fire, and thanks to the quick response from the fire department, many parts of the house were salvaged. In lieu of this recent event, we wondered how to help support the fire department through the use of technology to save others from the same experience. By allowing them to review their performance in a more immersive manner to prevent people like our teammate from going through such events.
After speaking with our friend who works as a firefighter near our college’s campus, as well as extensive background research, we found out that the current system for performance reviews relies on personal experience and video footage. In order to enhance these reviews, we decided to use 3D modeling and virtual reality to transform the process. The lossy act of remembering one’s decisions in a life-or-death moment can then be replaced by interactive 3D reconstructions of mission history. So, an immersive replay of the fire response can thus prove to improve performance and save more lives in the future.
What it does
Our solution uses VR technology to create an immersive replay of the fire response. The reviewer puts on the headset and sees an information rich graph, alerting them to key moments in the fire response (special points of interest). Then they select these points in time and are teleported to the actions of the firefighters during this time. The firefighter's exact actions are captures using 3D reconstruction technology.
Using monocular video data (single camera, like a firefighter helmet cam) we generated a point cloud using SLAM technology and then a 3D mesh through point meshing algorithm to detail the environment the fighter went through. We then implemented a graph on the user’s sidebar to examine the room’s temperature change, modeled from a realistic dataset created by simulation, over the fire’s duration. This is used to identify points of interest through the fire. Additionally, we simulated room temperature data spatially using a meshed plot. Our 3D model depicts how the temperature of a room could be displayed for a user to examine specific areas of the room that had high temperature peaks, and this used to identify areas of interest within the room for training exercises and performance reviews.
How we built it
Used Unity3D VR, MATLAB, Colmap
Challenges we ran into
In MATLAB, we had to overcome one technical challenge of exporting a 3D graph with interactive features in order to make a meaningful connection with the VR technology. We were able to make this important connection by creating a pipeline to convert the simulation to an interactive file that could be implemented in VR. This was very important for making the system efficient and user friendly.
Accomplishments that we're proud of
Figured out intricacies of Unity VR, found out how 3D room reconstruction works, getting seasoned with MATLAB
What we learned
Learned 3D room reconstruction, deepened VR skills, learned MATLAB (!!!!)
Both members of our team learned more about areas of technology we were relatively unfamiliar with, including 3D plotting/mapping with MATLAB and generating direct SLAM point clouds. Generating the mapped environments involved learning about the computer vision ecosystem, especially when it comes to depth mapping and meshing together point clouds. We looked through many different pipelines, including OpenSLAM, multiSLAM, and more before finally settling on Colmaps.
In addition, while our team was familiar with using Unity, we were new to integrating it with VR. Once the initial environment setup was done, figuring out how to code interactions pushed our understanding of VR. We developed new knowledge about the usages of VR in healthcare. An interesting avenue of education was choosing between various interaction technique (OpenXR vs OVR) and evaluating which was best for our hack.
What's next for Firefighter SLAM
Open source this project Addition of complete pipeline for multi-video to 3D mesh creation under the hood of a Unity-based app Adding more data streaming features into app 3D temperature visualizations Improve interaction methods with the data Features to Pause, Play, Rewind operations
Log in or sign up for Devpost to join the conversation.