Pull requests must include at least one of the required labels: internal, highlight, enhancement, bug, deprecation, documentation. Except for internal, pull requests must also include a description in the release notes section.

I updated the editing instructions to include our style guide:

    editing_instructions = """
    Proofread the following release notes text to be clear, concise, and error-free. 
    Use sentence-style capitalization. Return comments as-is if included in the original text. 
    If the original text is a short line with no punctuation at the end, do not add punctuation. 
    Please adhere to our style guide, which can be found here: https://docs.validmind.ai/about/style-guide.html
    """

Hm, I just tried this with an OpenAI API key I generated but I think I don't have the right permissions on my account to be able to run the script correctly. (I also got the "exceeds token length" one when I tried with our shared API key saved in 1Password.) Again, might be a user error day 😓

Hm, I just tried this with an OpenAI API key I generated but I think I don't have the right permissions on my account to be able to run the script correctly. (I also got the "exceeds token length" one when I tried with our shared API key saved in 1Password.) Again, might be a user error day 😓

'Exceeds token length' is not so much an error as a feature, it just means you've passed more text to ChatGPT than it can digest in one go. There are a few ways around this (chunk, sequential input) but the current version doesn't handle this. Generally, release notes should be short and for cases like really long ones, you probably want a human to take a closer look anyway.

I can demo this when we meet but basically the script should run like it did before, only it now also edits the release notes text and PR titles with calls to a new function that feeds the content to ChatGPT with a prompt and your style guide.

FYI, I broke something in this script when I added more editing instructions. Please hold off on reviewing until it's fixed!

OK, I have a fix working locally — somewhere along the way of cleaning up the script to get it ready for review, I broke the logic for edited content and didn't notice. 🙃 My apologies, especially to @validbeck!

I'm making some additional improvements to the script now to handle longer release notes that exceed the maximum token length and then I will ask you to review again when ready.

[Elevator music ...]

OK, so after spending several more hours on this, I've come to the conclusion that the script works as intended and might always have worked that way. Every once in a while, ChatGPT simply returns what it was given but it seem to be an issue with ChatGPT rather than the script. It is possible this is what was happening earlier today, but I have no way of proving this.

I have removed the style guide from the editing instructions as it seems to make the output less predictable and might have contributed to the issue. Instead, I have added some more short editing instructions right into the script.

I've also reworked how we pass the prompt to ChatGPT a bit, experimenting with several different options. We continue to pass system and user prompts separately, as in the original script, after an interim experiment of combining the two.

Some editing examples:

ORIGINAL TITLE: Improvements to create_new_test

EDITED TITLE: Improvements to create_new_test

--

ORIGINAL TITLE: Fix python dependencies

EDITED TITLE: Fix Python dependencies

--

ORIGINAL TITLE: Custom Metric Function Decorator

EDITED TITLE: Custom metric function decorator

--

ORIGINAL RELEASE NOTES TEXT: ValidMind now supports the ability to "compose" multiple Unit Metrics into complex outputs. These composite metrics can be logged as a single result that can be used as a content block in your documentation and this result will be updated automatically when re-running the documentation test suite.

EDITED RELEASE NOTES TEXT: ValidMind now supports the ability to compose multiple unit metrics into complex outputs. These composite metrics can be logged as a single result that can be used as a content block in your documentation. This result will be updated automatically when re-running the documentation test suite.

--

ORIGINAL RELEASE NOTES TEXT: We've introduced a new metric decorator that's designed to turn any function into a ValidMind Metric. Custom metrics offer added flexibility by allowing users to extend the library of default metrics provided by ValidMind, enabling you to document any type of model or use case. This new decorator makes the process of creating and using custom metrics much simpler by almost completely eliminating the boilerplate code required to define and register a custom metric. Check out the new notebook to see how you can start using custom metrics in your documentation!

EDITED RELEASE NOTES TEXT: We've introduced a new metric decorator designed to turn any function into a ValidMind Metric. Custom metrics offer added flexibility by allowing you to extend the library of default metrics provided by ValidMind. This lets you document any type of model or use case.

This new decorator simplifies the process of creating and using custom metrics. It almost completely eliminates the boilerplate code required to define and register a custom metric.

Check out the new notebook to see how you can start using custom metrics in your documentation.

Also, I was able to figure out the maximum token length issue: the issue was with the API call itself and how max_tokens was determined originally. We now hardcode a value rather than calculating one based on the input lines which could exceed what the model actually supports even though the text passed to it didn't.

Since the script can take a minute or two to run, I also added some additional processing output.

Here's a clean run that edited the release notes with no maximum token errors from the OpenAI API:

❯ make release-notes
Enter a full GitHub release URL (leave empty to finish): https://github.com/validmind/developer-framework/releases/tag/v2.0.7
Enter a full GitHub release URL (leave empty to finish): https://github.com/validmind/frontend/releases/tag/v1.19.0
Enter a full GitHub release URL (leave empty to finish): https://github.com/validmind/documentation/releases/tag/v2.0.7
Enter a full GitHub release URL (leave empty to finish): 
Enter the release date (Month Day, Year) [May 13, 2024]: 
Generating & editing release notes ...
  Processing validmind/developer-framework/#31 ...
  Processing validmind/developer-framework/#35 ...
  Processing validmind/developer-framework/#19 ...
  Processing validmind/developer-framework/#9 ...
  Processing validmind/developer-framework/#30 ...
  Processing validmind/developer-framework/#21 ...
  Processing validmind/developer-framework/#17 ...
  Processing validmind/developer-framework/#28 ...
  Processing validmind/developer-framework/#33 ...
  Processing validmind/developer-framework/#23 ...
  Processing validmind/developer-framework/#29 ...
  Processing validmind/developer-framework/#26 ...
  Processing validmind/developer-framework/#37 ...
  Processing validmind/developer-framework/#36 ...
  Processing validmind/developer-framework/#34 ...
  Processing validmind/developer-framework/#32 ...
  Processing validmind/developer-framework/#27 ...
  Processing validmind/developer-framework/#24 ...
  Processing validmind/developer-framework/#18 ...
  Processing validmind/developer-framework/#25 ...
  Processing validmind/frontend/#696 ...
  Processing validmind/frontend/#709 ...
  Processing validmind/frontend/#711 ...
  Processing validmind/frontend/#706 ...
  Processing validmind/frontend/#690 ...
  Processing validmind/frontend/#710 ...
  Processing validmind/frontend/#699 ...
  Processing validmind/frontend/#703 ...
  Processing validmind/documentation/#165 ...
  Processing validmind/documentation/#172 ...
  Processing validmind/documentation/#170 ...
  Processing validmind/documentation/#171 ...
  Processing validmind/documentation/#166 ...
  Processing validmind/documentation/#169 ...
Added release notes to _quarto.yml, line 97
Files to commit:
 M _quarto.yml
?? releases/2024-may-13/

@validbeck in case it's helpful, I just created a quick pull request with this PR here: #185

@validbeck in case it's helpful, I just created a quick pull request with this PR here: #185

Ah, just saw you approved — thank you! Yes, this works but let's tweak the editing instructions as we gain more experience with ChatGPT. Thank you so much for re-reviewing!