Skip to content
This repository was archived by the owner on Jan 7, 2025. It is now read-only.

Improve evaluate output format#55

Merged
thisismatu merged 4 commits intomasterfrom
format-evaluate-output
Nov 9, 2022
Merged

Improve evaluate output format#55
thisismatu merged 4 commits intomasterfrom
format-evaluate-output

Conversation

@thisismatu
Copy link
Copy Markdown

@thisismatu thisismatu commented Nov 8, 2022 •
edited
Loading

  • Update progress bar characters and width and clear it on finish
  • Group both ASR and NLU output better
  • Use our fork of Needleman-Wunsch alignment for ASR and NLU output. This is a rather simple tool that shows the where the ground truth and prediction differs.

Progress bar

Transcribing  50%  █████████▌░░░░░░░░░░  (2/4, 36 utt/min) [5s:3s]

NLU output

speechly evaluate nlu <app_id> test.txt

Line: 2
└─ Ground truth: *add_to_cart i want [two|2](amount) of those
└─ Prediction:   *add_to_cart i want *two*********** of those

Line: 4
└─ Ground truth: *set_delivery_date delivery [tomorrow|2022-01-*01](delivery_date)
└─ Prediction:   *set_delivery_date delivery [tomorrow|2022-11-10*](delivery_date)

Accuracy: 0.50 (2/4)

ASR output

speechly evaluate asr <app_id> test.jsonl 
                                                                  
Audio: podcast1.wav
└─ Ground truth: WELCOME TO ANOTHER EPISODE OF THE SPEECHLY PODCAST
└─ Prediction:   WELCOME TO ANOTHER EPISODE OF THE SPEECH** PODCAST

Audio: podcast3.wav
└─ Ground truth: THIS CONCEPT OF VOICE BEING AN EXPERT UI COULD YOU MAY***BE UNPACK THAT CONCEPT
└─ Prediction:   THIS CONCEPT OF VOICE BEING AN EXPERT UI COULD YOU MIGHT BE UNPACK THAT CONCEPT

Word Error Rate (WER): 0.04 (3/68)

Known issues

Long lines will wrap, but at least with these changes they are easier to distinguish. This PR does not try to make changes to that since it's a rabbit hole, believe me. I was in that hole for a brief moment of time before i managed to pull myself out of it...

Mathias Lindholm added 2 commits November 8, 2022 22:37
Update progress style and clear on finish
30dfed9
Improve evaluate accuracy output
0ba8cea 
Group items better and use Needleman-Wunsch alignment for asr evaluation output
@bigdatabaracus
Copy link
Copy Markdown
Contributor

bigdatabaracus commented Nov 9, 2022 •
edited
Loading

Thanks @thisismatu Nice! One comment right of the bat. I think using the Needleman-Wunsch alignment also for NLU to highlight differences would make a lot of sense. As an example.

Line: 2
└─ Ground truth: *add_to_cart i want -two----------- of those
└─ Prediction:   *add_to_cart i want [two|2](amount) of those

@bigdatabaracus
Copy link
Copy Markdown
Contributor

bigdatabaracus commented Nov 9, 2022

Thanks @thisismatu for the update. Out of curiosity does the second NLU example in the PR description look now like this now?

Line: 4
└─ Ground truth: *set_delivery_date delivery [tomorrow|2022-10--09](delivery_date)
└─ Prediction:   *set_delivery_date delivery [tomorrow|2022-1-1-09](delivery_date)

@thisismatu
Copy link
Copy Markdown
Author

thisismatu commented Nov 9, 2022 •
edited
Loading

@bigdatabaracus it varies, hence i didn't add it in the first place. As there's no customization, our options are to remove it for nlu or live with this.

Few example outputs with ground truth on top and prediction on bottom:

[tomorrow|2022-10-11]
[tomorrow|2022-11-10]

[tomorrow|2021-01--01]
[tomorrow|2022-11-10-]

[tomorrow|1998-12-21-]
[tomorrow|2022-11--10]

@bigdatabaracus
Copy link
Copy Markdown
Contributor

bigdatabaracus commented Nov 9, 2022

The hyphen symbol is an unfortunate choice of alignment character for the go library. We are just about to add that character to our ASR output as well. 🤔

teelisyys
teelisyys previously approved these changes Nov 9, 2022
View reviewed changes
@thisismatu thisismatu requested a review from teelisyys November 9, 2022 11:05
@thisismatu thisismatu merged commit bb8c46b into master Nov 9, 2022
@thisismatu thisismatu deleted the format-evaluate-output branch November 9, 2022 11:39
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

@teelisyys teelisyys Awaiting requested review from teelisyys

1 more reviewer

@bigdatabaracus bigdatabaracus bigdatabaracus approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@thisismatu @bigdatabaracus @teelisyys