The Heck

Inspiration

When I see someone streaming microscopy on twitch I often want to know more about the microorganisms under the microscope! But often there is no expert around to ask questions to, and reading wikipedia by yourself is not a very community-building activity.

It's hard to describe in words something you are still learning about; hence why the vision aspect is so important.

What it does

It adds a new command to the chatbot that allows viewers to ask chatgpt about what is it that they see on the screen. It sends a screenshot along with an optional prompt that includes their question.

How I built it

I expanded the existing infrastructure of a previous project that is also related to the expansion of engagement on microbiology topics on twitch: It uses OpenAI's python library to make the API integration easier.

Challenges I ran into

Twitch does not provide an API for taking a screenshot of a live stream, so I had to hack a way of getting the screen data.

One option would be to have a program to be installed on the streamer's computer. Maybe an OBS plugin, or something standalone. But that would raise the barrier of entry too much.

So the solution I decided to go with is to make use of the twitch clipping API to make a clip of the stream in the moment the chat command is issued. The bot then proceeds to download the clip, and extract an image frame of it, taking into account the delay between the API call and its effect.

Accomplishments that I'm proud of

Done! It is done! It's a project to help with science communication and nurtures curiosity, and it is done! ;)

What I learned

OpenAI's API is easy. Integrating a feature into an already existing project is much easier than making everything from scratch. Vision capabilities of LLMs are fun. LLMs are smart but you still have to sanitize their output for things such as "keep your text shorter than X characters".