Batlytics

Inspiration

Compared to the professional level, college-level baseball lacks the complex analytical tools to further improve the training and performance of the players. With the advances in the field of computer vision along with the ease of recording video data of practices and games, it feels like an obvious choice to leverage these techniques to analyze the actions of players. Furthermore, technical schools can leverage their high-performance clusters for a very detailed analysis, and also have a lightweight version ready to be used by coaches on their laptops or phones. These insights provide a significant advantage over the other university teams.

What it does

Analytics is a light-weight web app running in the cloud, that offers the following baseball analytics features, that can be used for both hitters and pitchers:

Tracks limbs, joints positions in a given video recording - this data is then visualized in the original video
Tracks arm angles in a given recording - this data is then visualized in the original video
Tracks a baseball bat in a given recording - this data is then visualized in the original video
Tracks a baseball ball in a given recording - this data is then visualized in the original video
It uses past match data and generates interactive 3D plots, where a user can filter the data as per PitchCall, ExitSpeed, etc. It also displays the Key Performance Indicator (KPI) metrics such as average relative speed and average spin axis values

As described above, the app takes an input video and produces an output video with detailed information visualized in the original video. In addition, this analytical information can be accessed and used in other ways suitable for the coaches or other team members.

How we built it

We leveraged the video recordings provided by the Georgia Tech baseball team. We used both video recordings of a hitter and a pitcher to test out our proof of concept.

The pose estimation used to track limbs and joint positions are detected using a pre-trained MediaPipe BlazePose yielding very accurate results while minimizing the computational requirements. By replacing the current (“full”) architecture with the “lite” version, the analysis can be easily run locally on mobile devices. The limb and joint positions are then used to compute arms angles.

The baseball bat and ball object detection are performed by a pre-trained YOLOv5 architecture (namely a smaller, more efficient “yolov5s” architecture) which finds object bounding boxes for 80 classes and we extract only ones relevant to us - baseball bat and ball.

After gathering all the data we visualize the limbs and joint positions over the video frame and baseball bat/ball bounding boxes if they were detected in the frame.

For track_1, we leveraged the CSV data provided by the Georgia Tech baseball team, we used PitchTrajectory columns to generate a 3d interactive visualization, we also added filters to modify the plot colors with existing columns like SpinAxis, ExitSpeed, PlayResult.

The front-end part is serving this analysis backend (models and visualizations) via the Streamlit web app framework.

Challenges we ran into

Challenge #1: Data quality In order to properly track features in the video, we need a lot of high-quality frames. This can be achieved by using a camera capturing the video with high Frames Per Second (FPS) rate (“high-speed cameras”). Namely for the tracking of the ball, having enough frames capturing the ball is essential for the models to pick up on it - because some frames might be distorted and therefore the ball might not be successfully detected.

Challenge #2: Camera calibration By calibrating a camera with a set of calibration recordings we could get a so-called calibration matrix from which we could derive the projection matrix. The projection matrix would allow us to project the 2D points (i.e. points from the image) to 3D points (i.e. real-world coordinates). This would allow us to do a more precise analysis and also possibly allow us to develop a decent players’ eyes focus estimation.

Challenge #3: Understanding baseball CSV data It took us a lot of time to understand the dataset and what all columns refer to. After understanding the features, our team again stumbled upon how to create a meaningful visualization from the provided data. After resolving these issues, finally, we were able to construct a 3D plot using the plotly library.

Accomplishments that we're proud of

A specific problem statement formulation in video analysis of players
Putting together data visualization web app using Streamlit
Putting together a rather complex computer vision ensemble with pose estimation along with bat and ball tracking
Additionally, figuring a large part of the player's field location tracking (i.e. the other problem - mentioned at the end of this document)

Aside from technical aspects of the project

Not losing ourselves while communicating among the team members that are 10 hours apart Provided a good foundation for a robust analytical toolkit and pointers to features that can be implemented in the future.

What we learned

As a developer:

Working with OpenCV module, MediaPipe BlazePose, YOLOv5
Development using Deepnote environment
Developing a web app via Streamlit
Understanding baseball data

What's next for Batlytics

Collect higher quality data, possibly labeled video recordings that would allow us to further improve currently used models.
Gather camera calibration data to allow us to project 2D points to 3D points, enabling us to generate additional and more precise analytics data
Develop a parallel, batch-processing versions version that could work in a map-reduce fashion on high-performance clusters (i.e. process parts of the input parallelly and join them in the end)
Develop a light, locally run, real-time version for mobile devices (requires only very small changes - namely changing the model's architectures to the lite versions)
Test with the latest architectures for video processing - such as liquid neural networks.
Develop a visualization dashboard with dynamic data reading - i.e. a user uploads CSV files, and the app constructs a set of plots