Skip to content
This repository was archived by the owner on Jan 7, 2025. It is now read-only.

Support ASR evaluation#53

Merged
teelisyys merged 17 commits intomasterfrom
feature/updated_evaluate_and_annotate_cmds
Nov 2, 2022
Merged

Support ASR evaluation#53
teelisyys merged 17 commits intomasterfrom
feature/updated_evaluate_and_annotate_cmds

Conversation

@teelisyys
Copy link
Copy Markdown
Contributor

No description provided.

Copy link
Copy Markdown

@thisismatu thisismatu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few things:

The docs for evaluate sub-commands are missing. These need to be generated.

The transcribe command complains about some JSON error when adding a single WAV file. In the design doc the idea was that it should accept both single files or JSON Lines files if you wanted to send multiple files. As a user, i don't have any idea of what format this command expects and this renders the command useless to me.

The error message after the progress is missing a line break making the whole thing a bit hard to read. Usually the first characters are on the previous line:

image

@teelisyys teelisyys requested a review from thisismatu October 26, 2022 13:08
@teelisyys teelisyys force-pushed the feature/updated_evaluate_and_annotate_cmds branch from d9533d0 to 280d6ed Compare October 26, 2022 13:49
@teelisyys teelisyys force-pushed the feature/updated_evaluate_and_annotate_cmds branch from 280d6ed to e651d59 Compare October 26, 2022 13:53
@teelisyys teelisyys dismissed thisismatu’s stale review October 26, 2022 13:57

Docs added, single WAV support added, improved progress bar behaviour on error

Copy link
Copy Markdown

@thisismatu thisismatu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great, thanks!

The only thing i'm struggling with at the moment is that when transcribing a single file i always get the same Closing file failed: invalid argument error.

./bin/speechly transcribe ./ndgt.wav -a 962f1ea7-9970-47ab-b7e6-bc9c88ed21d0

@thisismatu
Copy link
Copy Markdown

thisismatu commented Oct 28, 2022

Noticed that annotate, evaluate asr and evaluate nlu all take APP ID as positional argument. For the sake of unity and better UX, i would love that transcribe would follow suit. I'm ok with the input being second positional argument or flag, whatever makes your life easier :)

@thisismatu
Copy link
Copy Markdown

thisismatu commented Oct 28, 2022

As noted in todays demos, instead of printing all the results when using evaluate asr, perhaps it could work in the same fashion as evaluate nlu i.e. only print out the things that were wrong. Like this:

Audio: podcast1.wav
Ground truth: WELCOME TO ANOTHER EPISODE OF THE SPEECHLY PODCAST WHERE YOU CAN EXPECT CONVERSATIONS
Prediction:   WELCOME TO ANOTHER EPISODE OF THE SPEECHLY PODCAST WHERE YOU CAN EXPECT CONVERSATIONS EXPRESS

Audio: podcast3.wav
Ground truth: THIS CONCEPT OF VOICE BEING AN EXPERT UI COULD YOU MAYBE UNPACK THAT CONCEPT A LITTLE BIT WHEN I FIRST MET STEVE JOBS THAT VERY FIRST
Prediction:   THIS CONCEPT OF VOICE BEING AN EXPERT UI COULD YOU MIGHT BE UNPACK THAT CONCEPT A LITTLE BIT WHEN I FIRST MET STEVE JOBS THAT VERY FIRST

Word Error Rate (WER): 0.04 (3/68)

thisismatu
thisismatu previously approved these changes Nov 1, 2022
Copy link
Copy Markdown

@thisismatu thisismatu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@teelisyys teelisyys merged commit d72fe6a into master Nov 2, 2022
@teelisyys teelisyys deleted the feature/updated_evaluate_and_annotate_cmds branch November 2, 2022 07:33
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants