@digitalmoksha i'm just noticing now -- you have a status indicating that you were out of office between April 16 and 25. thank you for being responsive on this issue, but in the future, please, please prioritize your time away over responding to issues (especially lower priority ones like this).

i'll close this and open an issue for the specific feature request that's been formed from our conversation here.

thanks a ton for your input, as well, @engwan.

... the format is not just any Markdown, but specifically GitLab Flavored Markdown (GLFM)

that makes perfect sense; i assumed the syntax was specific to gitlab, i just didn't go hunting for GLFM :)

Issue notes really refer to the entire class of "notes", which primarily consist of user comments. A subclass of those, called system notes, are used for displaying such messages as changed title, etc - they inform the user of certain activities that have happened.

with this refined description and understanding, i can now adjust my earlier statement to say that what i'd like to see are a "better" (for my use case) message representing system notes. given your earlier question and statement:

Should there be a more defined format for querying these type of system events? I think this issue demonstrates that would be beneficial for some API clients, and we should probably open a specific issue detailing what that means and what the requirements are.

i'd be happy to close this issue, and open a feature request for this after ideating out what a potential implementation might look like a bit more (from the client's perspective). does that align with the process your team would like to see?

their format is determined by what is placed in the body of the note

this tracks with my understanding of their current implementation, but it's good to clarify.

Should there be a more defined format for querying these type of system events? I think this issue demonstrates that would be beneficial for some API clients, and we should probably open a specific issue detailing what that means and what the requirements are.

i think this is ultimately what i'd like to see. i'm not sure how unique my use case is (it doesn't feel like it would be that unique?), but abstractly, it would be a fantastic improvment to have those sorts of events be first class citizens - if i'm not mistaken, issue notes are effectively "someone changed one or more attributes about the issue" (be it the title, labels, description, what-have-you -- i do not have an understanding of the exhaustive list of the different note types). what i mean by first class citizens is that the /projects/:id/issues/:iid/notes endpoint returned an IssueNote, some struct with a shape like:

{
  kind: "TITLE_CHANGED",
  old: "foobar",
  new: "baz quux",
  diff: "(the current body value)"
  created: ...
  ...
}

internally, i'd presume that this was represented by some IssueNote interface or class that had proper subclasses/implementations (i don't know the first thing about ruby's vernacular, sorry) representing the other event types -- and the fields could be unique for each event type.

said differently, and more succinctly: it doesn't currently seem like issue notes are differentiated in their current implementation, and i'm of the opinion that they could and possibly should be.

It is generally recognized that the content returned is markdown

is it? in the issue notes api documentation, the second object in the example response doesn't appear to be valid markdown:

{
    ...
    "body": "Text of the comment\r\n",
    ...
}

and i'm not entirely sure if the previous syntax was valid markdown, either:

changed title from **some title** to **some title{+ edited+}**

well, to clarify, i'm pointing at the {+ edited +} bit, which in any markdown rendering lib i've ever seen or used, would be interpreted literally as part of the string, which is not what was rendered on your UI. i took this as a gitlab-specific-syntax for representing the "title change" events, and is why i called it "markdown-esque" in my original comment. this understanding is the basis of my earlier statement that i'd personally consider the format of the value returned for the body attr to be part of the API contract.

maybe i'm being pedantic. that isn't my goal, and as i've mentioned i'm really not familiar with gitlab's API as of yet, so the sum of my comments here could simply be interpreted as "new user feedback", if that makes sense.

i think, ideally, that what i'd like to see is changes in a changelog associated with a given endpoint, so that i can monitor a feed (e.g. the changelog) and programmatically determine if something relevant to my interests/concerns is on it. given the nature of a changelog, this makes it reactionary and wouldn't avoid the issue i encountered, but it would at least let me respond to these sorts of changes more expediently (and not spend the bulk of a day hunting down the issue).

i recognize the challenges associated with doing this, but thought voicing it might be useful if a discussion about changing the changelog's format (or messages, at least) ever popped up.

But historically, we don't treat these content changes as breaking changes. Still, we could've probably added a note in the release notes? 🤔

i don't have historical context for this endpoint (i joined as a maintainer of the project i mentioned not too long ago), but it seems to be simply returning the raw value for whatever the source event was (the notes on issue comment threads, e.g. title change, external mentions, etc). if that's a correct understanding, i think that the value emitted there (delineated by each event type, i guess) should definitely be considered part of the API contract. but perhaps i'm biased due to this issue :P

thanks for the reply (and for correcting my typo in the title -- that escaped me, heh).

1. We do have a changelog, https://gitlab.com/gitlab-org/gitlab/-/raw/master/CHANGELOG.md, in which we try to track bug fixes and feature addition/changes. I think it only gets updated when a new release is created. If you search that file for title changed, you'll see then entry for the change. ... I would certainly recommend monitoring the https://about.gitlab.com/releases/categories/releases/ page - the release posts that are put together have a lot of detail in them. They don't list every single change, but do link out to the various changelogs for the complete list. This specific bug fix was not mentioned in the 17.11 post, but it is in the changelog. ... It's also possible to get an RSS feed for when the changelog file is updated - https://gitlab.com/gitlab-org/gitlab/-/commits/master/CHANGELOG.md - in the upper right you can see a feed icon. It doesn't show the details of what was changed, just that it was indeed changed.

taking a look at the changelog, my immediate thought is to write something to grab it periodically and parse it, pinging me (email, or whatever) when the project issue notes api has any sort of change. however, that doesn't seem like a plausible solution, because there isn't a specific reference to the API that was changed in the changelog -- what's in there for this specific change is insufficient, as it's human-generated (it seems like that at first glance, at least):

Fix how title changed system notes rendered

i'm making this assumption because i think this is more of a free-form changelog, and "title changed" or "system notes" doesn't seem like enough information to programmatically determine changes were made to the of the values sent out for this specific endpoint -- is this a valid concern/assumption?

I would certainly recommend monitoring the https://about.gitlab.com/releases/categories/releases/ page - the release posts that are put together have a lot of detail in them. They don't list every single change, but do link out to the various changelogs for the complete list. This specific bug fix was not mentioned in the 17.11 post, but it is in the changelog.

one of the first things i did was hunt through the recent release notes, but i wasn't too sure what to look for -- even now, searching for "title", "changed", or "notes" yields no relevant results (manually reviewing it provides the same result) -- on any of the linked, as well. to be honest, it's almost too dense for skimming through -- the only place it seems to exist is the changelog, as you've pointed out, and at the moment i believe the concern i note above is valid.

just to chime in here -- i think something like this would be awesome for "some field's value was changed" events:

{
    "field_name": "foo",
    "previous_value": "bar",
    "current_value": "baz",

    # if we need the diff:
    "diff": "..."
}

(also, thanks @engwan!)

i left a comment on the MR, but copying it here to avoid needing to click the link.

i understand the reason that the MR was submitted and merged, however, i'd like to leave this feedback:

this was a breaking change to any downstream consumer that parses the body field. it resulted in spurious test failures that blocked our pipeline, seemingly randomly, until i hunted down the issue. it has resulted in a multi-day outage and inability to merge (or at least, a reduction in the confidence of our e2e tests that hit your API), and broke the ability for anywhere from one up to thousands of users (we don't collect usage metrics) to interact with gitlab's API through our tool. it would have been fantastic to be notified of this ahead of time so that this could have been avoided.

i could very well be incorrect, but i don't see a single method which i could have used to have been notified about this breaking change. is there an obvious combination of labels i can watch for, a changelog, or some other such feed that would allow me to -- if not actively, at least passively or retroactively -- find (changes) related to the issue notes api?

coming from #536827 (closed). i understand the reason this was submitted and merged, however, i'd like to leave this feedback:

this was a breaking change to any downstream consumer that parses the body field. it resulted in spurious test failures that blocked our pipeline, seemingly randomly, until i hunted down the issue. it has resulted in a multi-day outage and inability to merge (or at least, a reduction in the confidence of our e2e tests that hit your API), and broke the ability for anywhere from one up to thousands of users (we don't collect usage metrics) to interact with gitlab's API through our tool. it would have been fantastic to be notified of this ahead of time so that this could have been avoided.

i could very well be incorrect, but i don't see a single method which i could have used to have been notified about this breaking change. is there an obvious combination of labels i can watch for, a changelog, or some other such feed that would allow me to -- if not actively, at least passively or retroactively -- find (breaking) changes related to the issue notes api?

tangent: it would be fantastic if the golang SDK was improved to map these to a Note that had some predetermined category and new/old fields instead of needing to parse these (so that your API and SDK could be updated atomically, or at least in as close to tandem as your teams can get).

would that sort of change be welcome? if i'm better off asking in another project for this or my original issue, please feel free to point me in that direction.

sudoforge (e09ce114) at 16 Jul 04:59

If you would still like me to work on creating a test file that I can share, let me know, and I'll get that to you tomorrow.

Sorry for the delay; yep, that patch confirmed it:

Decrypting a chunk of length 0
Unhandled Java Exception in create_output():
java.lang.NullPointerException
        at pdftk.com.lowagie.text.pdf.StandardDecryption.finish(StandardDecryption.java:112)
        at pdftk.com.lowagie.text.pdf.PdfEncryption.decryptByteArray(PdfEncryption.java:571)
        at pdftk.com.lowagie.text.pdf.PdfReader.getStreamBytesRaw(PdfReader.java:2234)
        at pdftk.com.lowagie.text.pdf.PdfReader.getStreamBytesRaw(PdfReader.java:2250)
        at pdftk.com.lowagie.text.pdf.PRStream.toPdf(PRStream.java:241)
        at pdftk.com.lowagie.text.pdf.PdfIndirectObject.writeTo(PdfIndirectObject.java:182)
        at pdftk.com.lowagie.text.pdf.PdfWriter$PdfBody.add(PdfWriter.java:384)
        at pdftk.com.lowagie.text.pdf.PdfWriter$PdfBody.add(PdfWriter.java:363)
        at pdftk.com.lowagie.text.pdf.PdfWriter$PdfBody.add(PdfWriter.java:355)
        at pdftk.com.lowagie.text.pdf.PdfWriter.addToBody(PdfWriter.java:2069)
        at pdftk.com.lowagie.text.pdf.PdfCopy.copyIndirect(PdfCopy.java:271)
        at pdftk.com.lowagie.text.pdf.PdfCopy.copyObject(PdfCopy.java:343)
        at pdftk.com.lowagie.text.pdf.PdfCopy.copyDictionary(PdfCopy.java:299)
        at pdftk.com.lowagie.text.pdf.PdfCopy.copyObject(PdfCopy.java:341)
        at pdftk.com.lowagie.text.pdf.PdfCopy.copyDictionary(PdfCopy.java:299)
        at pdftk.com.lowagie.text.pdf.PdfCopy.copyObject(PdfCopy.java:341)
        at pdftk.com.lowagie.text.pdf.PdfCopy.copyDictionary(PdfCopy.java:296)
        at pdftk.com.lowagie.text.pdf.PdfCopy.addPage(PdfCopy.java:567)
        at com.gitlab.pdftk_java.cat.create_output_page(cat.java:121)
        at com.gitlab.pdftk_java.cat.create_output_cat(cat.java:294)
        at com.gitlab.pdftk_java.TK_Session.create_output(TK_Session.java:1444)
        at com.gitlab.pdftk_java.pdftk.main_noexit(pdftk.java:192)
        at com.gitlab.pdftk_java.pdftk.main(pdftk.java:163)

Hey @marcvinyals, thanks for the quick reply. I'm unable to share the files themselves, as they contain a mix of (my own) highly sensitive PII and (a company's) IP which I am not allowed to share. For context, these files were received from Adobe Sign after they were signed through the platform -- I may be able to recreate this by creating an Adobe Sign account and a tester file; I'll do this when I have time this evening (UTC-0700) and update this comment thread.

Passing in an input_pw wasn't necessary to encounter the exception. The command executed was:

pdftk /tmp/signed-files/*.pdf cat output /tmp/signed.pdf

The stacktrace was (for files part-1.pdf through part-9.pdf; only one exception shown for brevity):

Error: Unexpected Exception in open_reader()
java.lang.NoClassDefFoundError: org/bouncycastle/crypto/BlockCipher
        at pdftk.com.lowagie.text.pdf.StandardDecryption.update(StandardDecryption.java:94)
        at pdftk.com.lowagie.text.pdf.PdfEncryption.decryptByteArray(PdfEncryption.java:568)
        at pdftk.com.lowagie.text.pdf.PdfString.decrypt(PdfString.java:273)
        at pdftk.com.lowagie.text.pdf.PdfReader.readDecryptedDocObj(PdfReader.java:723)
        at pdftk.com.lowagie.text.pdf.PdfReader.readDocObj(PdfReader.java:1135)
        at pdftk.com.lowagie.text.pdf.PdfReader.readPdf(PdfReader.java:508)
        at pdftk.com.lowagie.text.pdf.PdfReader.<init>(PdfReader.java:172)
        at pdftk.com.lowagie.text.pdf.PdfReader.<init>(PdfReader.java:161)
        at com.gitlab.pdftk_java.InputPdf.add_reader(InputPdf.java:73)
        at com.gitlab.pdftk_java.TK_Session.add_reader(TK_Session.java:60)
        at com.gitlab.pdftk_java.TK_Session.add_reader(TK_Session.java:67)
        at com.gitlab.pdftk_java.TK_Session.open_input_pdf_readers(TK_Session.java:80)
        at com.gitlab.pdftk_java.TK_Session$Parser.parse_state_page_seq(TK_Session.java:503)
        at com.gitlab.pdftk_java.TK_Session$Parser.parse(TK_Session.java:186)
        at com.gitlab.pdftk_java.TK_Session.parse(TK_Session.java:1098)
        at com.gitlab.pdftk_java.pdftk.main_noexit(pdftk.java:186)
        at com.gitlab.pdftk_java.pdftk.main(pdftk.java:163)
Caused by: java.lang.ClassNotFoundException: org.bouncycastle.crypto.BlockCipher
        at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:602)
        at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
        ... 17 more
Error: Failed to open input PDF file:
   /tmp/signed-files/part-9.pdf

This looked like it might be a classpath issue, so I installed the commons-lang library using my system package manager and ensured the pdftk executable set the appropriate classpath URI. That didn't help (I later discovered that I needed bcprov), so I downloaded and built from source, which made some progress, but still failed (note: installing bcprov brings the system executable in line with what we see below).

$ gradle shadowJar
<snip>

$ java -jar build/libs/pdftk-all.jar /tmp/signed-files/*.pdf input_pw "" cat output /tmp/signed.pdf
WARNING: The creator of the input PDF:
   /tmp/signed-files/part-1.pdf
   has set an owner password (which is not required to handle this PDF).
   You did not supply this password. Please respect any copyright.
WARNING: The creator of the input PDF:
   /tmp/signed-files/part-2.pdf
   has set an owner password (which is not required to handle this PDF).
   You did not supply this password. Please respect any copyright.
WARNING: The creator of the input PDF:
   /tmp/signed-files/part-3.pdf
   has set an owner password (which is not required to handle this PDF).
   You did not supply this password. Please respect any copyright.
WARNING: The creator of the input PDF:
   /tmp/signed-files/part-4.pdf
   has set an owner password (which is not required to handle this PDF).
   You did not supply this password. Please respect any copyright.
WARNING: The creator of the input PDF:
   /tmp/signed-files/part-5.pdf
   has set an owner password (which is not required to handle this PDF).
   You did not supply this password. Please respect any copyright.
WARNING: The creator of the input PDF:
   /tmp/signed-files/part-6.pdf
   has set an owner password (which is not required to handle this PDF).
   You did not supply this password. Please respect any copyright.
WARNING: The creator of the input PDF:
   /tmp/signed-files/part-7.pdf
   has set an owner password (which is not required to handle this PDF).
   You did not supply this password. Please respect any copyright.
WARNING: The creator of the input PDF:
   /tmp/signed-files/part-8.pdf
   has set an owner password (which is not required to handle this PDF).
   You did not supply this password. Please respect any copyright.
WARNING: The creator of the input PDF:
   /tmp/signed-files/part-9.pdf
   has set an owner password (which is not required to handle this PDF).
   You did not supply this password. Please respect any copyright.
Unhandled Java Exception in create_output():
java.lang.NullPointerException
        at pdftk.com.lowagie.text.pdf.StandardDecryption.finish(StandardDecryption.java:111)
        at pdftk.com.lowagie.text.pdf.PdfEncryption.decryptByteArray(PdfEncryption.java:571)
        at pdftk.com.lowagie.text.pdf.PdfReader.getStreamBytesRaw(PdfReader.java:2234)
        at pdftk.com.lowagie.text.pdf.PdfReader.getStreamBytesRaw(PdfReader.java:2250)
        at pdftk.com.lowagie.text.pdf.PRStream.toPdf(PRStream.java:241)
        at pdftk.com.lowagie.text.pdf.PdfIndirectObject.writeTo(PdfIndirectObject.java:182)
        at pdftk.com.lowagie.text.pdf.PdfWriter$PdfBody.add(PdfWriter.java:384)
        at pdftk.com.lowagie.text.pdf.PdfWriter$PdfBody.add(PdfWriter.java:363)
        at pdftk.com.lowagie.text.pdf.PdfWriter$PdfBody.add(PdfWriter.java:355)
        at pdftk.com.lowagie.text.pdf.PdfWriter.addToBody(PdfWriter.java:2069)
        at pdftk.com.lowagie.text.pdf.PdfCopy.copyIndirect(PdfCopy.java:271)
        at pdftk.com.lowagie.text.pdf.PdfCopy.copyObject(PdfCopy.java:343)
        at pdftk.com.lowagie.text.pdf.PdfCopy.copyDictionary(PdfCopy.java:299)
        at pdftk.com.lowagie.text.pdf.PdfCopy.copyObject(PdfCopy.java:341)
        at pdftk.com.lowagie.text.pdf.PdfCopy.copyDictionary(PdfCopy.java:299)
        at pdftk.com.lowagie.text.pdf.PdfCopy.copyObject(PdfCopy.java:341)
        at pdftk.com.lowagie.text.pdf.PdfCopy.copyDictionary(PdfCopy.java:296)
        at pdftk.com.lowagie.text.pdf.PdfCopy.addPage(PdfCopy.java:567)
        at com.gitlab.pdftk_java.cat.create_output_page(cat.java:121)
        at com.gitlab.pdftk_java.cat.create_output_cat(cat.java:294)
        at com.gitlab.pdftk_java.TK_Session.create_output(TK_Session.java:1444)
        at com.gitlab.pdftk_java.pdftk.main_noexit(pdftk.java:192)
        at com.gitlab.pdftk_java.pdftk.main(pdftk.java:163)
There was a problem with pdftk-java. Please report it at
https://gitlab.com/pdftk-java/pdftk/issues
including the message above, the version of pdftk-java (3.2.2), and if possible steps to reproduce the error.

I tracked this new exception down to one file in particular, part-6.pdf, which happens to be a U.S. tax document (you likely now understand why I am unable to share this). I followed the stacktrace to this if check, added a few System.out.println debugging statements to confirm some assumptions, (I do not use Java and don't have a super useful developer environment set up for it at the moment), and made this change (which then allowed me to successfully generate my output file).

The patch in this MR avoids the exceptions, but, as a note, still generates that warning about not supplying a password.

Platform and version information:

Platform: Linux
Kernel: 5.9.14
pdftk-java version: 3.2.2

For context, I'm using the community/pdftk package, which uses this repository and installs an executable at /usr/bin/pdftk which invokes the jar as such:

#!/bin/sh
CP='/usr/share/java/bcprov.jar:/usr/share/java/commons-lang/commons-lang.jar:/usr/share/java/pdftk/pdftk.jar'
exec /usr/bin/java -cp "$CP" com.gitlab.pdftk_java.pdftk "$@"