netomi

Automatically adding comments to GitHub PRs, safely and reliably

Wed, 21 Aug 2024 18:00:00 +0000

Using GitHub Actions offers a lot of possibilities to automate your development processes. Imagine you would like to get a coverage report for any PR that gets created for your repository and add a coverage summary directly to the PR as a comments to make allow reviewers to immediately see if this PR meets the requirements wrt code coverage.

There are a lot of convenient actions available in the GitHub Marketplace that perform some action and add a comment to the PR. That usually works perfectly unless you come across the notorious error:

HttpError: Resource not accessible by integration

and your perfectly working workflow failed to add a comment to the PR that you just created. After some initial investigation you find out that it is due to a PR created from a fork. Further digging reveals that GitHub treats PRs from forks differently for security reasons, namely that in such case any workflow triggered by a pull_request trigger will not have access to any secrets or write tokens.

Trying to be smart

GitHub takes “secure by default” seriously, so you don’t easily get pwned by a malicious actor, however there are still ways to shoot yourself in the foot. There is an additional workflow trigger pull_request_target that will run a workflow from a PR in the context of the target repository, and as a consequence, have access to secrets and write tokens.

There are cases where doing something like that is acceptable and can be considered safe (see Keeping your GitHub Actions safe):

Generally speaking, when the PR contents are treated as passive data, i.e. not in a position of influence over the build/testing process, it is safe.

A very nice writeup about utilizing the pull_request_target trigger with some additional user permission checks can be found in this blog entry. However, the downside of that approach is that you need to think hard about the attack surface of your workflows to prevent any unexpected side effects, something we ultimately want to avoid.

Git (sic!) better

Some investigation of available workflow triggers reveals the workflow_run trigger, that allows to react once a workflow has completed, and workflows triggered by that run in the context of the repository itself. That would allow us to do the following:

run a workflow with pull_request trigger to perform some action, e.g. calculate the code coverage of the code in a PR
store the result of this workflow as an artifact attached to the workflow run
trigger another workflow upon completion of the first workflow using the workflow_run trigger with action completed
download the artifact attached to the completed workflow run
add a comment to the associated PR based on the artifact data

We have implemented this in a real-world example for the up-java repo of the Eclipse uProtocol project.

Gen(-AI+Mundane)

The first workflow will generate the relevant data for the PR, and upload the generated data as an artifact. See the snippet below for the relevant steps:

name: Java Test and Coverage

...

- name: Generate coverage comment
  uses: actions/github-script@v7
  with:
    script: |
      const fs = require('fs');
      const COVERAGE_PERCENTAGE = `${{ env.COVERAGE_PERCENTAGE }}`;
      const COVERAGE_REPORT_PATH = `https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }}/`;
        
      fs.mkdirSync('./pr-comment', { recursive: true });
        
      var pr_number = `${{ github.event.number }}`;
      var body = `
        Code coverage report is ready! :chart_with_upwards_trend:
        
        - **Code Coverage Percentage:** ${COVERAGE_PERCENTAGE}%
        - **Code Coverage Report:** [View Coverage Report](${COVERAGE_REPORT_PATH})
      `;
        
      fs.writeFileSync('./pr-comment/pr-number.txt', pr_number);
      fs.writeFileSync('./pr-comment/body.txt', body);

- uses: actions/upload-artifact@5d5d22a31266ced268874388b861e4b58bb5c2f3 # v4.3.1
  with:
    name: pr-comment
    path: pr-comment/

The comment that shall be added to the PR is stored in a file body.txt while the associated PR number is stored in a file pr-number.txt to know to which PR that comment should be added in the second workflow.

Any comment?

Now that we have a workflow to calculate the code coverage of the PR and generate a comment that we would like to add to it, we just need a workflow that gets triggered when the first workflow (filtered by its name) is completed:

...

on:
  workflow_run:
    workflows: ["Java Test and Coverage"]
    types:
      - completed

jobs:
  add-coverage-comment:
    runs-on: ubuntu-latest
    permissions:
      pull-requests: write
    if: >
      github.event.workflow_run.event == 'pull_request' &&
      github.event.workflow_run.conclusion == 'success'
    steps:
      - name: 'Download artifact'
        uses: actions/github-script@60a0d83039c74a4aee543508d2ffcb1c3799cdea # v7.0.1
        with:
          script: |
            var artifacts = await github.rest.actions.listWorkflowRunArtifacts({
               owner: context.repo.owner,
               repo: context.repo.repo,
               run_id: ${{github.event.workflow_run.id }},
            });
            var matchArtifact = artifacts.data.artifacts.filter((artifact) => {
              return artifact.name == "pr-comment"
            })[0];
            var download = await github.rest.actions.downloadArtifact({
               owner: context.repo.owner,
               repo: context.repo.repo,
               artifact_id: matchArtifact.id,
               archive_format: 'zip',
            });
            var fs = require('fs');
            fs.writeFileSync('${{github.workspace}}/pr-comment.zip', Buffer.from(download.data));
      - run: unzip pr-comment.zip
      - name: 'Comment on PR'
        uses: actions/github-script@60a0d83039c74a4aee543508d2ffcb1c3799cdea # v7.0.1
        with:
          github-token: ${{ secrets.GITHUB_TOKEN }}
          script: |
            var fs = require('fs');

            const issue_number = Number(fs.readFileSync('./pr-number.txt'));
            const body = fs.readFileSync('./body.txt', { encoding: 'utf8', flag: 'r' });

            await github.rest.issues.createComment({
              owner: context.repo.owner,
              repo: context.repo.repo,
              issue_number: issue_number,
              body: body
            });

The workflow will download the artifact attached to the workflow run that triggered it and add a comment to the associated PR with the attached content.

NOTE: There are pre-made 3rd party actions to download artifacts from previous workflow runs that could be used to further simplify such a workflow.

Closing words

GitHub.com is a very powerful platform to host open-source projects, but you need to be aware that its open and collaborative model also attracts malicious actors to attack or infiltrate (popular) repositories. Keep your repositories and workflows safe by following secure development best practices.

More information can be found in the Security Handbook @ Eclipse Foundation.

Controlling access to macOS large runners for GitHub Actions

Tue, 06 Aug 2024 10:00:00 +0000

In 2023, GitHub introduced new powerful macOS runners for GitHub Actions. These runners have a considerable higher amount of processors / memory and disk space allocated to them to speed up the execution of workflows. This advantage comes at a cost though, as billing per minute of executed workflow time is considerably higher as compared to normal runners (see billing for runners), on top of the usual minute multiplier for macOS runners (each minute of executed workflow time on a macOS runner counts as 10 minutes for billing purposes).

Going to speed up

In order to use such a macOS large runner, you can simply add a runs-on: <runner-type> to your job definition, e.g. using macos-latest-large as runner type:

name: learn-github-actions-testing
on: [push]
jobs:
  build:
    runs-on: macos-latest-large
    steps:
      - uses: actions/checkout@v4
      - name: Build
        run: swift build
      - name: Run tests
        run: swift test

Additionally, your organization needs to have a GitHub Team or GitHub Enterprise Cloud plan to be able to use such a macOS large runner (which is now the case for all Eclipse projects hosted on GitHub as of 2024), otherwise workflows using such a runner will fail to run. Once your organization is eligible to use large runners, you probably want to control the access to such runners for the repositories in your organization to avoid surprises when you receive your next invoice. GitHub offers a convenient way to define runner groups to control which repositories can access such large runners.

What the …

Unfortunately, such runner groups can only be defined for linux and windows runners, there is simply no way to prevent that macOS large runners are being used by any of your repositories once their use is configured in a workflow as described above. This poses a problem for non-profit organizations (like the Eclipse Foundation) that host a lot of projects and their associated repositories on GitHub as it might result in higher than expected billing expenses as some projects try using such large runners to speed up their workflows without realizing the consequences.

While it is possible to monitor the incurred costs of using GitHub Action minutes, this is a tedious and manual task and requires communication with projects to change their workflows if occurrences have been identified.

Gaming the system

The idea was born to add some automation to prevent the execution of workflows on such macOS large runners unless the project / repository is entitled to use such a runner.

After studying the available GitHub Rest API and preliminary testing, we figured out the following logic reliably prevents the execution of workflows on macOS large runners:

listen to workflow_job events with action queued
check whether the included workflow_job object has labels that indicate that the job is supposed to run on a macOS large runner
if the above evaluates to true and the repository is not eligible to use such a runner, cancel the workflow_run

To receive the necessary webhook events from GitHub in case a workflow is being queued to run, you have to set up an organization or repository webhook, listen for the event and apply the logic.

All good

At the Eclipse Foundation we are operating an open-source project called Otterdog in order to configure our numerous organizations and repositories hosted on GitHub at scale. This tool is effectively a GitHub App and is installed for all our projects / organizations on GitHub and already can listen to various events sent from GitHub. So naturally we added the above logic to this tool and allowed to define which organizations are allowed to use such large runners via a configuration file (see this example).

This allows us to control the use of macOS large runners which unfortunately is not yet possible through any of the administration consoles at GitHub. On the other hand, our implemented workaround showcases the power of GitHub Apps on how you can utilize them to adjust your GitHub experience to your organizational needs.

If you are member of an Eclipse project and would like to utilize macOS large runners for your workflows, reach out to us via the HelpDesk.

Bat - library to process and analyze byte code

Mon, 20 Jul 2020 13:00:00 +0000

The last couple of years I was busy with processing byte code with an amazing tool called ProGuard / DexGuard. After deciding to step back and take a break, I used the available spare time to work on my own tool to process any kind of byte code (class file, dex file) and perform some useful analysis on them.

You can access my initial take on this in my GitHub repo bat which is an acronym for byte code analysis toolkit.

The first useful utility that I created is a tool that copies the behavior of dexdump with some additional options, e.g. ability to filter the output for specific classes only:

java -cp /path/to/commands-1.0-SNAPSHOT.jar com.github.netomi.bat.DexDumpCommand -c '**Hello**' classes.dex

...
    
Class #0 header:
class_idx           : 10
access_flags        : 1 (0x0001)
superclass_idx      : 2
interfaces_off      : 0 (0x0000)
source_file_idx     : 0
annotations_off     : 0 (0x0000)
class_data_off      : 2209 (0x08a1)
static_fields_size  : 3
instance_fields_size: 0
direct_methods_size : 3
virtual_methods_size: 1

Class #0
  Class descriptor  : 'Lcom/example/HelloWorldActivity;'
  Access flags      : 0x0001 (PUBLIC)
  Superclass        : 'Landroid/app/Activity;'
  Interfaces        -
  Static fields     -
...

The next thing to work on will be an explainer tool, which is intended as a utility to help with the shrinking capability of tools like ProGuard or R8. In general these tools work amazingly well, but it is quite difficult to understand why something is actually kept from being shrunk or which specific rule is responsible for that. The explainer will trace the exact reason why a certain class / method / field is being kept. Something that might also be very useful is the other way around: what items will be kept because of a specific method or class. A tool like that might help to reduce dependencies or complexity in the code.

Feel free to leave comments on other useful things that you would like to see in the near future.

Backporting lambda expressions

Tue, 07 Jul 2020 12:00:00 +0000

Let’s take a look into lambda expressions as part of the Java language specifiction, what they are, how they are represented, and how existing bytecode can be backported to earlier class files versions without losing any functionality.

Introduction

A lambda expressions is basically syntactic sugar that allows you to define inline anonymous classes that implement specific interfaces. Take a look at the following example:

...
Button button = (Button) findViewById(R.id.button);
button.setOnClickListener((view) -> Toast.makeText(this, "Hello World!", Toast.LENGTH_LONG).show());
...

In this example we set a click listener for a specific button using a lambda expression. This is equivalent to writing this code:

...
Button button = (Button) findViewById(R.id.button);
button.setOnClickListener(new OnClickListener() {
    @Override
    public void onClick(View view) {
        Toast.makeText(this, "Hello World!", Toast.LENGTH_LONG).show();
    } 
});
...

The use of lambda expressions makes the whole code a bit more concise and less bloated. I dont want to go into more details on how to write lambda expressions in the java language. There are many resources out there for this purpose. In this blog post I would like to focus on how such lambda expressions are represented in byte code, and how it can be backported to earlier class file versions, which is important when targeting platforms like Android.

Representation

Let’s analyse how the javac compiler translates this java code into the corresponding class file format:

  public void onCreate(android.os.Bundle);
    descriptor: (Landroid/os/Bundle;)V
    flags: (0x0001) ACC_PUBLIC
    Code:
        ...
        23: invokedynamic #29,  0             // InvokeDynamic #0:onClick:(Lxxx/yyy/HelloWorldActivity;)Landroid/view/View$OnClickListener;
        28: invokevirtual #33                 // Method android/widget/Button.setOnClickListener:(Landroid/view/View$OnClickListener;)V
        ...

We can see that an instruction of type invokedynamic is executed which returns an object of type OnClickListener and passes this on to the setOnClickListener method of the Button object. The invokedynamic instruction calls the BootstrapMethod #0 as defined in the class:

BootstrapMethods:
  0: #71 REF_invokeStatic java/lang/invoke/LambdaMetafactory.metafactory:(Ljava/lang/invoke/MethodHandles$Lookup;Ljava/lang/String;Ljava/lang/invoke/MethodType;Ljava/lang/invoke/MethodType;Ljava/lang/invoke/MethodHandle;Ljava/lang/invoke/MethodType;)Ljava/lang/invoke/CallSite;
    Method arguments:
      #78 (Landroid/view/View;)V
      #79 REF_invokeSpecial xxx/yyy/HelloWorldActivity.lambda$onCreate$0:(Landroid/view/View;)V
      #78 (Landroid/view/View;)V

The definition of this BootstrapMethod contains information about the actual method to be called, its parameter and return types. The JVM will then dynamically create at runtime a CallSite targeting the specified target method and execute it.

In our case the target method is also stored in the class file itself as private method (with the name as specified in the BootstrapMethod):

  private void lambda$onCreate$0(android.view.View);
    descriptor: (Landroid/view/View;)V
    flags: (0x1002) ACC_PRIVATE, ACC_SYNTHETIC
    Code:
        ...
        aload_0
        ldc           #10                 // String: "Hello World!"
        iconst_1
        invokestatic  #41                 // Method android/widget/Toast.makeText:(Landroid/content/Context;Ljava/lang/CharSequence;I)Landroid/widget/Toast;
        invokevirtual #45                 // Method android/widget/Toast.show:()V
        return

Now this is the most simple example of a lambda expression. There are different cases to consider (e.g. capturing or non-capturing, references to constructors, …). I don’t want to go into too much detail here, lets dive into ways to backport these constructs in a way that they can be represented in earlier class file versions.

Backporting

There is a very popular and amazing tool called retrolambda which does exactly that, backport any lambda expression to java version 5, 6 or 7. For various technical reasons, I had the need to implement such a mechanism myself as the product I have been working on (ProGuard / DexGuard) also had to work in full standalone mode and is generally self-contained, thus has no external dependencies.

So the basic idea is to extract the generated private method into a separate lambda class that implements the necessary interface(s). Then replace the invokedynamic instruction calls in a way to create an instance of this lambda class.

The code to perform this is open-source and accessible at the ProGuard github repo.

Applied on the above example, the result would look like in the snippets below. An additional accessor method has been added to avoid modifying the visibility of the lambda methods, but that is not a technical necessity its rather a design choice.

  static void accessor$HelloWorldActivity$lambda0(xxx.yyy.HelloWorldActivity, android.view.View);
    descriptor: (Lxxx/yyy/HelloWorldActivity;Landroid/view/View;)V
    flags: (0x1008) ACC_STATIC, ACC_SYNTHETIC
    Code:
      stack=2, locals=2, args_size=2
         aload_0
         aload_1
         invokespecial #15                 // Method lambda$onCreate$0:(Landroid/view/View;)V
         return

  private void lambda$onCreate$0(android.view.View);
    descriptor: (Landroid/view/View;)V
    flags: (0x1002) ACC_PRIVATE, ACC_SYNTHETIC
    Code:
      stack=3, locals=2, args_size=2
         aload_0
         ldc           #10                 // String: "Hello World!"
         iconst_1
         invokestatic  #28                 // Method android/widget/Toast.makeText:(Landroid/content/Context;Ljava/lang/CharSequence;I)Landroid/widget/Toast;
         invokevirtual #31                 // Method android/widget/Toast.show:()V
         return

  public void onCreate(android.os.Bundle);
    descriptor: (Landroid/os/Bundle;)V
    flags: (0x0001) ACC_PUBLIC
    Code:
      stack=4, locals=2, args_size=2
         0: aload_0
         1: aload_1
         2: invokespecial #35                 // Method android/app/Activity.onCreate:(Landroid/os/Bundle;)V
         5: aload_0
         6: ldc           #36                 // int 2130968576
         8: invokevirtual #40                 // Method setContentView:(I)V
        11: aload_0
        12: ldc           #41                 // int 2130903040
        14: invokevirtual #45                 // Method findViewById:(I)Landroid/view/View;
        17: checkcast     #47                 // class android/widget/Button
        20: new           #49                 // class xxx/yyy/HelloWorldActivity$$Lambda$0
        23: dup
        24: aload_0
        25: invokespecial #52                 // Method xxx/yyy/HelloWorldActivity$$Lambda$0."<init>":(Lxxx/yyy/HelloWorldActivity;)V
        28: invokevirtual #56                 // Method android/widget/Button.setOnClickListener:(Landroid/view/View$OnClickListener;)V

and the corresponding generated lambda class:

  private final xxx.yyy.HelloWorldActivity arg$0;
    descriptor: Lxxx/yyy/HelloWorldActivity;
    flags: (0x0012) ACC_PRIVATE, ACC_FINAL

  public xxx.yyy.HelloWorldActivity$$Lambda$0(xxx.yyy.HelloWorldActivity);
    descriptor: (Lxxx/yyy/HelloWorldActivity;)V
    flags: (0x0001) ACC_PUBLIC
    Code:
      stack=2, locals=2, args_size=2
         0: aload_0
         1: invokespecial #13                 // Method java/lang/Object."<init>":()V
         4: aload_0
         5: aload_1
         6: putfield      #15                 // Field arg$0:Lxxx/yyy/HelloWorldActivity;
         9: return

  public void onClick(android.view.View);
    descriptor: (Landroid/view/View;)V
    flags: (0x0001) ACC_PUBLIC
    Code:
      stack=2, locals=2, args_size=2
         0: aload_0
         1: getfield      #15                 // Field arg$0:Lxxx/yyy/HelloWorldActivity;
         4: aload_1
         5: invokestatic  #24                 // Method xxx/yyy/HelloWorldActivity.accessor$HelloWorldActivity$lambda0:(Lxxx/yyy/HelloWorldActivity;Landroid/view/View;)V
         8: return

Of course, to fully support all possible variants of lambda expressions (and its sibling method references) some effort has to be put forward, but the necessary code is quite limited, take a look at the referenced source code, especially the class LambdaExpressionConverter. ProGuard offers a lot of functionality to make any kind of byte code engineering quite simple and effective.

Addendum

In this blog post I am focusing on lambda expressions, but ProGuard is capable of backporting various other language features of Java, like static interface methods, default methods, try-with-resources, string concatenations introduced in Java 9 and even the use of Java 8 APIs like the popular stream and time APIs.

Take a look at the source code, its very clean and can easily be read and understood imho.

Analysing string encryption of stringer

Fri, 05 Jun 2020 16:00:00 +0000

Stringer is one of many commercial tools to apply obfuscation on java byte code level. It supports various obfuscation techniques, but in this blog post I would like to analyse its ability to encrypt strings and explore ways to automatically deobfuscate jars obfuscated by stringer.

Let’s take a look at how the actual string encryption applied by stringer looks like. The following snippets are disassembled bytecode in jasmin notation:

ldc "\u5d5a\ub1c7\u0712"
invokestatic xxx/yy/z(Ljava/lang/Object;)Ljava/lang/String;

As you can see, an encrypted string, is pushed onto the stack and a method with a revealing signature is called, returning the decrypted string a runtime. My first attempt is to call the decrypt method myself with the same argument in the hope that it returns the decrypted string.

Unfortunately, this does not lead to a properly decrypted string, so there must be some additional mechanisms in place in order to prevent code lifting and to easily decrypt encrypted strings inside the jar file.

Looking at the decrypt method (xxx/yy/z), I can see the following instructions:

invokestatic java/lang/Thread/currentThread()Ljava/lang/Thread;
invokevirtual java/lang/Thread/getStackTrace()[Ljava/lang/StackTraceElement;
...
invokestatic sun/misc/SharedSecrets/getJavaLangAccess()Lsun/misc/JavaLangAccess;
aload 1
iconst_2
aaload
invokevirtual java/lang/StackTraceElement/getClassName()Ljava/lang/String;
invokestatic java/lang/Class/forName(Ljava/lang/String;)Ljava/lang/Class;
invokeinterface sun/misc/JavaLangAccess/getConstantPool(Ljava/lang/Class;)Lsun/reflect/ConstantPool; 1
invokevirtual sun/reflect/ConstantPool/getSize()I
...
invokevirtual java/lang/StackTraceElement/getClassName()Ljava/lang/String;
invokevirtual java/lang/StringBuilder/append(Ljava/lang/String;)Ljava/lang/StringBuilder;
...
invokevirtual java/lang/String/hashCode()I

It looks like the decrypt method analyses the calling stack and uses information from this class/method to setup its decryption key. Translating the same snippet to java code makes it quite clear:

StringBuilder sb = new StringBuilder();
StackTraceElement[] stackTraceElements = Thread.currentThread().getStackTrace();
JavaLangAccess javaLangAccess = sun.misc.SharedSecrets.getJavaLangAccess();
int constantPoolSize = javaLangAccess.getConstantPool(Class.forName(stackTraceElements[2].getClassName())).getSize();
sb.append(constantPoolSize);
...
sb.append(stackTraceElements[2].getClassName());
...
sb.append(stackTraceElements[2].getMethodName());
...
int key = sb.hashCode();
...

So various information from the calling class/method is being used to generate a key that is needed for the actual decryption. If the key does not match the expected values, only garbage is returned, making code lifting more challenging.

Let’s try to use proguard-core to modify the byte code in a way that we can easily execute it and get the decrypted strings during static analysis. For readability I will not post the full source code, but the relevant pieces of code to understand how this can be done.

We know that the decrypt method deduces its decryption key from the calling method. Now lets try to find all places in the jar file that seems to decrypt a string:

private static final String DECRYPT_METHOD_TYPE = "(Ljava/lang/Object;)Ljava/lang/String;";

private final Constant[] CONSTANTS = new Constant[]
{
    new MethodrefConstant(1, 2, null, null),
    new ClassConstant(X, null),
    new NameAndTypeConstant(Y, 3),
    new Utf8Constant(DECRYPT_METHOD_TYPE),
};
        
private final Instruction[] INSTRUCTIONS = new Instruction[]
{
    new ConstantInstruction(Instruction.OP_LDC, Z),
    new ConstantInstruction(Instruction.OP_INVOKESTATIC, 0),
};

private final InstructionSequenceMatcher matcher =
    new InstructionSequenceMatcher(CONSTANTS, INSTRUCTIONS);

...

public void visitAnyInstruction(Clazz clazz, Method method, CodeAttribute codeAttribute, int offset, Instruction instruction) {
    
    instruction.accept(clazz, method, codeAttribute, offset, matcher);
        
    // did we find a match?
    if (matcher.isMatching()) {
        InstructionSequenceMatcher matcher = matcher1.isMatching() ? matcher1 : matcher2;
        
        int classIndex = matcher.matchedConstantIndex(X);
        int nameIndex = matcher.matchedConstantIndex(Y);
        int stringIndex = matcher.matchedConstantIndex(Z);
            
        ...
    }
        
    ...

With the code snippet above, we look for code patterns that load a constant string onto the stack and invoke a static method which takes an Object as input and return a String. In the analysed samples of jars obfuscated with stringer all encrypted strings were encrypted using the same pattern, and its also quite uncommon for generic code to use similar patterns, thus we can easily find such encrypted strings in this case.

Now that we know which classes and methods contain encrypted strings, we can prepare the referenced decrypt method in such a way that the protection mechanism is not effective. For this purpose, we replace the aforementioned code in the decrypt method with the values of the actual calling method:

InstructionSequenceBuilder ____ = new InstructionSequenceBuilder();
            
instructions = new Instruction[][][]
{
    {
        ____.invokestatic("sun/misc/SharedSecrets", "getJavaLangAccess", "()Lsun/misc/JavaLangAccess;")
            .aload(1)
            .iconst_2()
            .aaload()
            .invokevirtual("java/lang/StackTraceElement", "getClassName", "()Ljava/lang/String;")
            .invokestatic("java/lang/Class", "forName", "(Ljava/lang/String;)Ljava/lang/Class;")
            .invokeinterface("sun/misc/JavaLangAccess", "getConstantPool", "(Ljava/lang/Class;)Lsun/reflect/ConstantPool;")
            .invokevirtual("sun/reflect/ConstantPool", "getSize", "()I")
            .invokevirtual("java/lang/StringBuilder", "append", "(I)Ljava/lang/StringBuilder;").__(),

        ____.pushInt(constantPoolSize)
            .invokevirtual("java/lang/StringBuilder", "append", "(I)Ljava/lang/StringBuilder;").__(),
    },

....
    
public void visitCodeAttribute(Clazz clazz, Method method, CodeAttribute codeAttribute) {
    CodeAttributeEditor codeAttributeEditor = new CodeAttributeEditor();

    clazz.accept(
        new AllMethodVisitor(
        new AllAttributeVisitor(
        new PeepholeEditor(codeAttributeEditor,
        new InstructionSequencesReplacer(constants,
                                         instructions,
                                         null,
                                         codeAttributeEditor,
                                         new InstructionCounter())))));
}

As you can see in this code snippet, we replace certain instruction patterns with known values so that the decryption works regardless from which method it is being called. Now the only thing left to do is to copy the original code to a new class, modify it as described, load the generated class and execute the method with the given encrypted string:

ProgramClass duplicatedClass =
    new ProgramClass(originalClass.u4version,
                     1,
                     new Constant[1],
                     originalClass.u2accessFlags,
                     0,
                     0);

ConstantPoolEditor constantPoolEditor = new ConstantPoolEditor(duplicatedClass);

duplicatedClass.u2thisClass =
    constantPoolEditor.addClassConstant(originalClass.getName(), null);
duplicatedClass.u2superClass =
    constantPoolEditor.addClassConstant(ClassConstants.NAME_JAVA_LANG_OBJECT, null);

// Copy over the class members.
MemberAdder memberAdder = new MemberAdder(duplicatedClass);

originalClass.fieldsAccept(memberAdder);
originalClass.methodsAccept(memberAdder);
    
modifiedClass.accept(
    new ProtectionRemover(ClassUtil.externalClassName(clazz.getName()),
                          method.getName(clazz),
                          constantPoolLength));

modifiedClass.accept(new ProgramClassWriter(os))
byte[] content = os.toByteArray();
...
    
// load the class contents with a custom ClassLoader which effectively calls defineClass.
ClassLoader classLoader =
    new ByteClassLoader(new URL[] { inputURL  },
                        CodeLifter.class.getClassLoader(),
                        Collections.singletonMap(externalClassName, content));

Class<?> loadedClass = Class.forName(externalClassName, true, classLoader);

...

Now that we have the modified decrypt method, we can just call it via reflection using the encrypted string as parameter. The returned result can be used to replace the original code in the obfuscated jar with a simple ldc instruction.

After we have implemented all this logic we see that the decrypted string still results in garbage, so there must be another protection mechanism against code lifting. Some more debugging reveals that the CodeSource of the ProtectionDomain of the class containing the decrypt method is also verified. If this does not return the expected value, the decryption is unsuccessful. Luckily this protection mechanism can easily be avoided by modifying the CodeSource of a given class using reflection:

URL inputURL = ... // original file name of the obfuscated jar
Class<?> modifiedClass = ....

CodeSource codeSource = new CodeSource(inputURL, (CodeSigner[]) null);
ProtectionDomain domain = modifiedClass.getProtectionDomain();

java.lang.reflect.Field field = domain.getClass().getDeclaredField("codesource");
field.setAccessible(true);
field.set(domain, codeSource);

So we change the CodeSource of the modified class to the one that the decrypt method expects, and finally, we can decrypt any strings in the obfuscated jar simply by the means of static code analysis.

The full source code to decrypt jars obfuscated by stringer will be made available at a later time. More to come!

Minimizing shaded dependencies with gradle and R8

Mon, 20 Apr 2020 13:00:00 +0000

Sometimes you need or want to include 3rd party dependencies into your own jar file e.g. to provide your users with an uber jar. These dependencies can additionally be repackaged to avoid any potential classpath issues. There exist various plugins for build systems like gradle or maven to achieve this:

Both plugins also offer a simple minimization feature, which tries to shrink the 3rd party dependencies as much as possible. For this purpose a small and nice library is being used that is able to analyse transitive dependencies of class files.

This already leads to some good results. Lets take a simple example. In my project Units of Mesurement I have the need to include some code from other projects, namely guava and hibernate validator. These two projects contain some nice utility classes that I would like to use in my own library:

In case only such a limited number of classes / methods are being used, copy&pasting the relevant code to your project is a viable option. But I wanted to explore if the minimize feature of the shadow jar plugin can help me without copying the code which is tedious and makes it more difficult to integrate potential bugfixes (apart from the testing coverage which might drop for your own code).

So I added a configuration like that to my project to add the 2 libraries as dependency, adding a relocate config, and enable the standard minimize feature:

shadowJar {
    exclude 'META-INF/**'
    exclude '**/**.properties'
    exclude '**/*.swp'

    relocate 'com.google.common', 'org.netomi.uom.util.shadow.guava'
    relocate 'org.hibernate.validator.internal', 'org.netomi.uom.util.shadow.hibernate'
    minimize()
}

dependencies {
    compile('com.google.guava:guava:28.2-jre')
    compile('org.hibernate.validator:hibernate-validator:6.1.3.Final')

    ...
}

The resulting jar contains the following classes from the 2 libraries:

tn@proteus:~/workspace/netomi/uom/build/libs$ unzip -l uom-1.0-SNAPSHOT-all.jar  | grep shadow
        0  2020-04-20 15:43   org/netomi/uom/util/shadow/
        0  2020-04-20 15:43   org/netomi/uom/util/shadow/guava/
        0  2020-04-20 15:43   org/netomi/uom/util/shadow/guava/annotations/
      616  2019-12-26 22:08   org/netomi/uom/util/shadow/guava/annotations/Beta.class
      670  2019-12-26 22:08   org/netomi/uom/util/shadow/guava/annotations/GwtCompatible.class
      678  2019-12-26 22:08   org/netomi/uom/util/shadow/guava/annotations/GwtIncompatible.class
      304  2019-12-26 22:08   org/netomi/uom/util/shadow/guava/annotations/VisibleForTesting.class
        0  2020-04-20 15:43   org/netomi/uom/util/shadow/guava/base/
     4184  2019-12-26 22:08   org/netomi/uom/util/shadow/guava/base/Absent.class
      851  2019-12-26 22:08   org/netomi/uom/util/shadow/guava/base/AbstractIterator$1.class
     1388  2019-12-26 22:08   org/netomi/uom/util/shadow/guava/base/AbstractIterator$State.class
     2381  2019-12-26 22:08   org/netomi/uom/util/shadow/guava/base/AbstractIterator.class
     1068  2019-12-26 22:08   org/netomi/uom/util/shadow/guava/base/CharMatcher$1.class
     2104  2019-12-26 22:08   org/netomi/uom/util/shadow/guava/base/CharMatcher$And.class
....

in total 131 classes:

tn@proteus:~/workspace/netomi/uom/build/libs$ unzip -l uom-1.0-SNAPSHOT-all.jar  | grep shadow | wc -l
131

This would definitely be too much, and I would prefer to copy&paste the relevant classes to my own library and update my NOTICE.txt file accordingly. I have worked for several years on a tool called ProGuard, so I was pretty sure that we can do better than that. Due to license problems (ProGuard is licensed under GPL) I decided to use R8 to extend the shadow jar plugin to use a full blown shrinker to improve the results of its minimization feature. R8 is the default minimization tool of the Android SDK and licensed under Apache v2.0 license.

The result of this proof-of-concept can be found in this pull request for the shadow jar plugin. Using R8 instead of just simply collecting transitive class dependencies results in a shaded jar that contains only this number of shaded classes:

tn@proteus:~/workspace/netomi/uom/build/libs$ unzip -l uom-1.0-SNAPSHOT-all.jar  | grep shadow 
        0  2020-04-20 15:52   org/netomi/uom/util/shadow/
        0  2020-04-20 15:52   org/netomi/uom/util/shadow/guava/
        0  2020-04-20 15:52   org/netomi/uom/util/shadow/guava/base/
      408  2019-12-26 22:08   org/netomi/uom/util/shadow/guava/base/Preconditions.class
      194  2019-12-26 22:08   org/netomi/uom/util/shadow/guava/base/Supplier.class
     1760  2019-12-26 22:08   org/netomi/uom/util/shadow/guava/base/Suppliers$MemoizingSupplier.class
     1812  2019-12-26 22:08   org/netomi/uom/util/shadow/guava/base/Suppliers$NonSerializableMemoizingSupplier.class
      850  2019-12-26 22:08   org/netomi/uom/util/shadow/guava/base/Suppliers.class
        0  2020-04-20 15:52   org/netomi/uom/util/shadow/hibernate/
        0  2020-04-20 15:52   org/netomi/uom/util/shadow/hibernate/util/
      520  2020-04-10 11:22   org/netomi/uom/util/shadow/hibernate/util/ConcurrentReferenceHashMap$Option.class
     9488  2020-04-10 11:22   org/netomi/uom/util/shadow/hibernate/util/ConcurrentReferenceHashMap.class
     2082  2020-04-10 11:22   org/netomi/uom/util/shadow/hibernate/util/ConcurrentReferenceHashMap$EntrySet.class
     1214  2020-04-10 11:22   org/netomi/uom/util/shadow/hibernate/util/ConcurrentReferenceHashMap$KeyIterator.class
     1689  2020-04-10 11:22   org/netomi/uom/util/shadow/hibernate/util/ConcurrentReferenceHashMap$EntryIterator.class
    10076  2020-04-10 11:22   org/netomi/uom/util/shadow/hibernate/util/ConcurrentReferenceHashMap$Segment.class
     1158  2020-04-10 11:22   org/netomi/uom/util/shadow/hibernate/util/ConcurrentReferenceHashMap$SoftKeyReference.class
     1225  2020-04-10 11:22   org/netomi/uom/util/shadow/hibernate/util/ConcurrentReferenceHashMap$SoftValueReference.class
      221  2020-04-10 11:22   org/netomi/uom/util/shadow/hibernate/util/ConcurrentReferenceHashMap$KeyReference.class
     3650  2020-04-10 11:22   org/netomi/uom/util/shadow/hibernate/util/ConcurrentReferenceHashMap$HashEntry.class
     2885  2020-04-10 11:22   org/netomi/uom/util/shadow/hibernate/util/ConcurrentReferenceHashMap$HashIterator.class
     2168  2020-04-10 11:22   org/netomi/uom/util/shadow/hibernate/util/ConcurrentReferenceHashMap$SimpleEntry.class
     1225  2020-04-10 11:22   org/netomi/uom/util/shadow/hibernate/util/ConcurrentReferenceHashMap$WeakValueReference.class
      624  2020-04-10 11:22   org/netomi/uom/util/shadow/hibernate/util/ConcurrentReferenceHashMap$ReferenceType.class
     1494  2020-04-10 11:22   org/netomi/uom/util/shadow/hibernate/util/ConcurrentReferenceHashMap$Values.class
     1395  2020-04-10 11:22   org/netomi/uom/util/shadow/hibernate/util/ConcurrentReferenceHashMap$WriteThroughEntry.class
     1222  2020-04-10 11:22   org/netomi/uom/util/shadow/hibernate/util/ConcurrentReferenceHashMap$ValueIterator.class
     1158  2020-04-10 11:22   org/netomi/uom/util/shadow/hibernate/util/ConcurrentReferenceHashMap$WeakKeyReference.class
     1676  2020-04-10 11:22   org/netomi/uom/util/shadow/hibernate/util/ConcurrentReferenceHashMap$KeySet.class

total 29

Most of them are inner classes of the ConcurrentReferenceHashMap which are simply needed and can not be shrunk. This result is certainly much better and would allow me to go the route of shading off-the-shelf dependencies into my own library rather than copy&pasting code.

If you would like to see this feature in the shadow jar plugin, star this issue.

The test that I made with my own library can be found here.

Note: I am of course perfectly aware that under normal circumstances it is not advised to shade any 3rd party dependency as it might result in duplicated code being loaded by consumers of your library. In this specific use-case I think a technique like this makes sense as I want to include a very small subset of a larger dependency without cutting the ties to upstream releases of these dependencies (and potentially using more features).

Managing multiple jdk versions using sdkman

Mon, 20 Apr 2020 12:00:00 +0000

It is a common use-case to have multiple versions of various tools installed on your computer. Notable examples are different JDK versions or gradle. For a long time I installed them manually and added aliases to my shell configuration like that:

alias gradle48="~/bin/gradle-4.8.1/bin/gradle"

This works well for tools like gradle, but installing and maintaining multiple JDK versions was always a bit of a pain. Recently I found a very neat and amazing solution to this problem: sdkman.

It is a small tool that manages multiple versions of development kits on your computer. Installation is very easy:

$ curl -s "https://get.sdkman.io" | bash

and afterwards add something like this to your .bashrc or respective profile script:

[[ -s "~/.sdkman/bin/sdkman-init.sh" ]] && source "~/.sdkman/bin/sdkman-init.sh"

Now you should be able to use the sdk command to install different SDKs or change the current version in use:

sdk list java

will print something like this:

================================================================================\n
Available Java Versions
================================================================================
 Vendor        | Use | Version      | Dist    | Status     | Identifier
--------------------------------------------------------------------------------
 AdoptOpenJDK  |     | 14.0.0.j9    | adpt    |            | 14.0.0.j9-adpt      
               |     | 14.0.0.hs    | adpt    |            | 14.0.0.hs-adpt      
               |     | 13.0.2.j9    | adpt    |            | 13.0.2.j9-adpt      
               |     | 13.0.2.hs    | adpt    |            | 13.0.2.hs-adpt      
               |     | 12.0.2.j9    | adpt    |            | 12.0.2.j9-adpt      
               |     | 12.0.2.hs    | adpt    |            | 12.0.2.hs-adpt      
               |     | 11.0.6.j9    | adpt    |            | 11.0.6.j9-adpt      
               |     | 11.0.6.hs    | adpt    | installed  | 11.0.6.hs-adpt      
               |     | 8.0.242.j9   | adpt    | installed  | 8.0.242.j9-adpt     
               | >>> | 8.0.242.hs   | adpt    | installed  | 8.0.242.hs-adpt     
...

You can see that I have installed different JDK versions and version 8.0.242.hs-adpt is currently the default. To switch to another version by default you can use for example sdk use java 11.0.6.hs-adpt.

Its an amazing tool and I can recommand it for anyone using a unix-based workstation.

Note: in case you are wondering about the suffixes: hs and j9. They refer to the actual JVM in use:

good old Hotspot
OpenJ9

The suffix fx indicates whether the JDK comes pre-packages with the JavaFX / OpenJFX toolkit (which is not part of the Oracle / OpenJDK distributions anymore).

Handling default methods with dynamic proxies in java

Fri, 17 Apr 2020 10:00:00 +0000

Java has a very powerful feature to create dynamic proxies for an object implementing a given set of interfaces at runtime. This can be used to modify some implementations or adding some aspects the the object, like logging, access checks or caching. The dynamic proxy pattern is used by a number of popular libraries to implement the core of their functionality. Some well-known libraries in this regard are:

spring (for its aspect oriented features)
retrofit

There is a runtime cost associated with the use of dynamic proxies, but it is usually not that bad, several benachmarks have been performed to prove their suitability for practical use-cases:

As a rule of thumb and based on my own experiments a dynamic proxy impacts performance such that an invocation of a proxied method is roughly 1.6 times slower compared to direct invocation of the same method. Which is not too bad when you consider what happens under the hood to make this working and comparing it with the performance of general reflection calls.

Let’s get to some code. In order to create a dynamic proxy for a given interface, one has to code the following:

Object delegate = ...
MyInterface impl =
    (MyInterface) Proxy.newProxyInstance(MyInterface.class.getClassLoader(), new Class[] { MyInterface.class }, new InvocationHandler() {
        public Object invoke(Object proxy, Method method, Object[] args) throws Throwable {
            try {
                return method.invoke(delegate, args);
            } catch (InvocationTargetException ex) {
                // this is needed to throw the original exception to the caller.
                throw ex.getCause();
            }
        }
    });

This will create a dynamic proxy for the interface MyInterface and delegate all the method calls it receives to the Object delegate. To make this work, the delegate object must be able to process the delegated calls, i.e. it must implement the requested interface.

Now we do not want to go into more details of dynamic proxies and what you can do with them, but rather take a look into a problem introduced with some new feature in Java 8. This version of java added support for default methods in interfaces. But what has this to do with dynamic proxies? If you take the example from above and use it with an interface containing default methods, you will get a runtime exception like this:

Exception in thread "main" java.lang.IllegalArgumentException: object is not an instance of declaring class
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.netomi.uom.util.Proxies$1.invoke(Proxies.java:52)
	at com.sun.proxy.$Proxy0.getSystemUnit(Unknown Source)
	...

This happens because the default method will be delegated to the actual object which usually does not have an implementation for this method (unless it has overridden it of course). In order to support such interfaces with dynamic proxies there is a workaround available for Java 8:

import java.lang.invoke.MethodHandles;
...

Constructor<MethodHandles.Lookup> constructor = MethodHandles.Lookup.class.getDeclaredConstructor(Class.class);
if (!constructor.isAccessible()) {
    constructor.setAccessible(true);
}

Object result =
    constructor.newInstance(method.getDeclaringClass())
        .unreflectSpecial(method, method.getDeclaringClass())
        .bindTo(proxy)
        .invokeWithArguments(args);
...

This will access a private constructor of the MethodHandles.Lookup class via reflection and use it to get a MethodHandle object for the default method, bind it to the actual proxy instance and invoke the method. A bit hacky but the only way to support that in Java 8.

Unfortunately, this approach fails with Java 9, but the good news is that there is supported API that we can use for the same purpose and does not required the use of reflection:

MethodHandles.Lookup lookup =
    MethodHandles.privateLookupIn(method.getDeclaringClass(), MethodHandles.lookup())

MethodHandle handle = null;
MethodType methodType =
    MethodType.methodType(method.getReturnType(), method.getParameterTypes());
      
if (Modifier.isStatic(method.getModifiers())) {
    handle = lookup.findStatic(method.getDeclaringClass(), method.getName(), methodType);
} else {        
    handle = lookup.findSpecial(method.getDeclaringClass(), method.getName(), methodType, method.getDeclaringClass());
}

Object result = handle.bindTo(proxy).invokeWithArguments(args);
...

This is a bit more complex but it is supported with all versions since Java 9+. To wrap things up we have to modify our initial code for a delegating proxy to something like that:

Object delegate = ...
MyInterface impl =
    (MyInterface) Proxy.newProxyInstance(MyInterface.class.getClassLoader(), new Class[] { MyInterface.class }, new InvocationHandler() {
        public Object invoke(Object proxy, Method method, Object[] args) throws Throwable {
            if (!method.isDefault()) {
                try {
                    return method.invoke(delegate, args);
                } catch (InvocationTargetException ex) {
                    // this is needed to throw the original exception to the caller.
                    throw ex.getCause();
                }
            }
     
            return DefaultMethodHandler.getMethodHandle(method).bindTo(proxy).invokeWithArguments(args);
        }
    });

The class DefaultMethodHandler wraps up the different mechanisms to get a method handle for a default method depending on the current JVM, the code can be found here. It is a derived work from the spring framework, released under Apache v2.0 license.

Taking default methods correctly into account opens up a lot of possibilities for creating dynamic proxies. In my own project Units of Measurement I use proxies to create concrete quantity implementations for different datatypes (double, BigDecimal) at runtime. This is a very flexible approach which can also easily be extended in the future with only a minimal performance impact.