@klaudworks Please add the following content to your PR description and select a checkbox:

- [ ] `doc-needed` 
- [ ] `doc-not-needed` <!-- Your PR changes do not impact docs -->
- [ ] `doc-included` 

The failed test is a flaky test. It relies on Ollama to do some calculation which is non-deterministic. You can rerun it and it should be fine.

@klaudworks, thank you for reporting and fixing this bug. This is indeed a valid problem, and the fix sounds good. @wenjin272, could you please help check the details?

The 0.2.0 version is already in RC vote, and will soon be releases. Given that there's a workaround to this issue (placing user-defined resource in flink/lib), I'd suggest not to withdraw the RC and fix this in the next patch release. WDYT?

Additionally, I think we should:

• Create an issue describing the problem, workaround and track the fix version. This is helpful for other people who run into the same problem, because people usually tend to search in issues rather than PRs when they run into problems. @klaudworks, would you like to do that?
• We are planning to add some e2e tests that actually submit jobs into a flink standalone cluster, instead of currently running in a mini-cluster. I think user-defined resource should be included as one of the test cases. @wenjin272

Hi, @klaudworks, thanks for your detailed bug report and fix. Since you only verified with custom resource, I verify the fix with custom event. It works well. The fix looks good to me.

Additionally, I investigated when we set the TCCL to the UserCodeClassLoader.

1. For agent plan deserialization, the UserCodeClassLoader is set by UserDefinedObjectsHolder in Flink StreamGraph.
2. For resource creation, the UserCodeClassLoader is set in JavaActionTask before execute action.
3. For event log deserialization, there is no place to set the UserCodeClassLoader because deserialization does not actually occur.

Thanks for looking into this!

@xintongsong I'll write a proper issue for this in the next few minutes.

@xintongsong Depending on the deployment scenario the current workaround might be quite inconvenient. In our case for example, I am running the platform providing flink and can't anticipate what kind of jobs users will try run. I'd have to rebuild the flink Docker image whenever someone modifies a custom resource.

@wenjin272 thanks for further validating the fix!

@klaudworks,

The workaround is admittedly not perfect. There will soon bee a patch release, likely in late Feb or early Mar, which should also carry fixes for other issues discovered during this period. Does it sound good to you?

The reason I don't want to withdraw the RC is that, we are trying to finalize the release before the Chinese New Year vacation. If we can make it, people who don't take the vacation can start trying out the 0.2.0 release and report issues during this time, so that when we are back we can resolve the issues immediately and ship 0.2.1 around the time mentioned above. Otherwise, we'll have to postpone the release to after the vacation, which means everything is paused during this time, and we'll probably have a stable patch release around end of March. And if we withdraw the RC now, it's very likely we won't make it before the vacation. It takes more than one week, building another RC, testing, voting, finalizing the artifacts, etc.

@xintongsong that sounds good. Thank you for considering me but don't worry about it. I am running a custom version of flink-agents with the patch included.

Ported to release-0.2 in b8559a3