The Worker API provides the ability to break up the execution of a task action into discrete units of work and then to execute that work concurrently and asynchronously. This allows Gradle to fully utilizes the resources available and complete builds faster. This guide will walk you through the process of converting an existing custom task to use the Worker API.

This guide assumes that you understand the basics of writing Gradle custom tasks. Please consider working through Writing Gradle Tasks first.

What you’ll create

You’ll start by creating a custom task class that generates MD5 hashes for a configurable set of files. Then, you’ll convert this custom task to use the Worker API. Then we’ll explore running the task with different levels of isolation. In the process, you’ll learn about the basics of the Worker API and the capabilities it provides.

What you’ll need

  • About

  • A text editor or IDE

  • A Java Development Kit (JDK), version 1.7 or better

  • A Gradle distribution, version 4.10-rc-2 or better

Create a custom task class

First, you’ll need to create a custom task that generates MD5 hashes of a configurable set of files.

In a new directory, create a buildSrc/build.gradle file.

buildSrc/build.gradle
repositories {
    jcenter()
}

dependencies {
    compile "commons-io:commons-io:2.5"
    compile "commons-codec:commons-codec:1.9" (1)
}
1 Your custom task class will use Apache Commons Codec to generate MD5 hashes.
If you are not familiar with buildSrc, this is a special directory that allows you to define and build custom classes that should be available for use in your build script. See the user manual for further information.

Now, create a custom task class in your buildSrc/src/main/java directory. You should name this class CreateMD5.

buildSrc/src/main/java/CreateMD5.java
import org.apache.commons.codec.digest.DigestUtils;
import org.apache.commons.io.FileUtils;
import org.gradle.api.tasks.*;

import javax.inject.Inject;
import java.io.File;
import java.io.FileInputStream;
import java.io.InputStream;

class CreateMD5 extends SourceTask { (1)
    @OutputDirectory
    File destinationDir; (2)

    @Inject
    public CreateMD5() {
        super();
    }

    @TaskAction
    public void createHashes() {
        for (File sourceFile : getSource().getFiles()) { (3)
            try {
                InputStream stream = new FileInputStream(sourceFile);
                System.out.println("Generating MD5 for " + sourceFile.getName() + "...");
                Thread.sleep(3000); (4)
                File md5File = new File(destinationDir, sourceFile.getName() + ".md5"); (5)
                FileUtils.writeStringToFile(md5File, DigestUtils.md5Hex(stream), (String) null);
            } catch (Exception e) {
                throw new RuntimeException(e);
            }
        }
    }
}
1 SourceTask is a convenience type for tasks that operate on a set of source files.
2 The output of the task will go into a configured directory.
3 The task iterates over all of the files defined as "source files" and creates an MD5 hash of each.
4 Insert an artificial sleep to simulate hashing a large file (the sample files won’t be that large).
5 The MD5 hash of each file is written to the output directory into a file of the same name with an "md5" extension.

Next, create a build.gradle that implements your new CreateMD5 task.

build.gradle
plugins { id 'base' } (1)

task md5(type: CreateMD5) {
    destinationDir = file("${buildDir}/md5") (2)
    source file("src") (3)
}
1 Apply the base plugin so that you’ll have a clean task to use to remove the output.
2 MD5 hash files will be written to build/md5.
3 This task will generate MD5 hash files for every file in the src directory.

Now, you’ll need some source to generate MD5 hashes from. Create 3 files in the src directory:

src/einstein.txt
Intellectual growth should commence at birth and cease only at death.
src/feynman.txt
I was born not knowing and have had only a little time to change that here and there.
src/oppenheimer.txt
No man should escape our universities without knowing how little he knows.

At this point, you can give your task a try:

$ gradle md5

You should see output similar to:

> Task :md5
Generating MD5 for einstein.txt...
Generating MD5 for feynman.txt...
Generating MD5 for oppenheimer.txt...


BUILD SUCCESSFUL in 12s

In the build/md5 directory, you should now see corresponding files with an md5 extension containing MD5 hashes of the files from the src directory. Notice that the task takes at least 9 seconds to run because it hashes each file one at a time (i.e. 3 files at ~3 seconds a piece).

Converting to the Worker API

Although this task processes each file in sequence, the processing of each file is independent of any other file. It would be really nice if this work was done in parallel and could take advantage of multiple processors. This is where the Worker API can help.

First, you’ll need to refactor the part of your custom task that does the work for each individual file into a separate class. This class is your "unit of work" implementation and should implement java.lang.Runnable.

buildSrc/src/main/java/GenerateMD5.java
import org.apache.commons.codec.digest.DigestUtils;
import org.apache.commons.io.FileUtils;

import javax.inject.Inject;
import java.io.File;
import java.io.FileInputStream;
import java.io.InputStream;

public class GenerateMD5 implements Runnable {
    private final File sourceFile;
    private final File md5File;

    @Inject (1)
    public GenerateMD5(File sourceFile, File md5File) { (2)
        this.sourceFile = sourceFile;
        this.md5File = md5File;
    }

    @Override
    public void run() {
        try {
            InputStream stream = new FileInputStream(sourceFile);
            System.out.println("Generating MD5 for " + sourceFile.getName() + "...");
            Thread.sleep(3000);
            FileUtils.writeStringToFile(md5File, DigestUtils.md5Hex(stream), (String) null);
        } catch (Exception e) {
            throw new RuntimeException(e);
        }
    }
}
1 This class must have a constructor annotated with javax.inject.Inject.
2 The parameters to the constructor are the parameters of the individual unit of work.

Now, you should change your custom task class to submit work to the WorkerExecutor instead of doing the work itself.

buildSrc/src/main/java/CreateMD5.java
import org.gradle.api.Action;
import org.gradle.api.tasks.*;
import org.gradle.workers.*;

import javax.inject.Inject;
import java.io.File;

class CreateMD5 extends SourceTask {
    private final WorkerExecutor workerExecutor; (1)

    @OutputDirectory
    File destinationDir;

    @Inject
    public CreateMD5(WorkerExecutor workerExecutor) { (2)
        super();
        this.workerExecutor = workerExecutor;
    }

    @TaskAction
    public void createHashes() {
        for (File sourceFile : getSource().getFiles()) {
            File md5File = new File(destinationDir, sourceFile.getName() + ".md5");
            workerExecutor.submit(GenerateMD5.class, new Action<WorkerConfiguration>() { (3)
                @Override
                public void execute(WorkerConfiguration config) {
                    config.setIsolationMode(IsolationMode.NONE); (4)
                    config.params(sourceFile, md5File); (5)
                }
            });
        }
    }
}
1 You’ll need to have the WorkerExecutor service in order to submit your work.
2 To get a WorkerExecutor, create a constructor annotated with javax.inject.Inject. Gradle will inject the WorkerExecutor at runtime when the task is created.
3 When submitting the unit of work, specify the unit of work implementation, in this case GenerateMD5.
4 For now, use an isolation mode of NONE. We’ll talk more about isolation modes later.
5 For each unit of work, configure the params property appropriately. These values should match up to the values passed to the constructor of GenerateMD5.

At this point, you should be able to try your task again.

$ gradle clean md5

> Task :md5
Generating MD5 for einstein.txt...
Generating MD5 for feynman.txt...
Generating MD5 for oppenheimer.txt...


BUILD SUCCESSFUL in 4s

The results should look the same as before, although the MD5 hash files may be generated in a different order due to the fact that the units of work are executed in parallel. One thing you should notice, however, is that the task runs much faster. This is because the Worker API executes the MD5 calculation for each file in parallel rather than in sequence.

Changing the isolation mode

The isolation mode controls how strongly Gradle will isolate items of work from each other as well as from the rest of the Gradle runtime. IsolationMode.NONE is the lowest level of isolation and will prevent a unit of work from changing the project state. NONE is the fastest isolation mode because it requires the least overhead to set up the work item to execute, so you’ll probably want to use this for simple cases. However, it will use a single shared classloader for all units of work. This means that every unit of work must use the same classes and can potentially affect each other through shared, static class state. It also means that every unit of work must use the same version of libraries that are on the buildscript classpath. But what if you wanted the user to be able to configure the task to run with a different (but compatible) version of the Apache Commons Codec library? The Worker API allows you to do that, too.

First, you’ll want to change the dependency in buildSrc/build.gradle to be compileOnly. This tells Gradle that it should use this dependency when building the classes, but should not put it on the build script classpath.

buildSrc/build.gradle
repositories {
    jcenter()
}

dependencies {
    compile "commons-io:commons-io:2.5"
    compileOnly "commons-codec:commons-codec:1.9"
}

Next, you’ll want to change the CreateMD5 task to allow the user to configure the version of the codec library that they want to use. It’ll resolve the appropriate version of the library at runtime and configure the workers to use this version. The isolation mode CLASSLOADER tells Gradle to run this work in a thread with an isolated classloader.

buildSrc/src/main/java/CreateMD5.java
import org.gradle.api.Action;
import org.gradle.api.file.FileCollection;
import org.gradle.api.tasks.*;
import org.gradle.workers.*;

import javax.inject.Inject;
import java.io.File;
import java.util.Set;

class CreateMD5 extends SourceTask {
    private final WorkerExecutor workerExecutor;

    @OutputDirectory
    File destinationDir;

    @InputFiles
    FileCollection codecClasspath; (1)

    @Inject
    public CreateMD5(WorkerExecutor workerExecutor) {
        super();
        this.workerExecutor = workerExecutor;
    }

    @TaskAction
    public void createHashes() {
        for (File sourceFile : getSource().getFiles()) {
            File md5File = new File(destinationDir, sourceFile.getName() + ".md5");
            workerExecutor.submit(GenerateMD5.class, new Action<WorkerConfiguration>() {
                @Override
                public void execute(WorkerConfiguration config) {
                    config.setIsolationMode(IsolationMode.CLASSLOADER);
                    config.params(sourceFile, md5File);
                    config.classpath(codecClasspath); (2)
                }
            });
        }
    }
}
1 Expose an input property for the codec library classpath.
2 Configure the classpath on the WorkerConfiguration when submitting the work item.

Next, you’ll need to configure your build so that it has a repository to look up the codec version at task execution time. We’ll also create a dependency to resolve our codec library from this repository.

build.gradle
plugins { id 'base' }

repositories {
    jcenter() (1)
}

configurations {
    codec (2)
}

dependencies {
    codec "commons-codec:commons-codec:1.10" (3)
}

task md5(type: CreateMD5) {
    destinationDir = file("${buildDir}/md5")
    source = file("src")
    codecClasspath = configurations.codec (4)
}
1 Add a repository to resolve the codec library - this can be a different repository than the one used to build the CreateMD5 task class.
2 Add a configuration to hold our codec library version.
3 Configure an alternate, compatible version of Apache Commons Codec.
4 Configure the md5 task to use the configuration as its classpath. Note that the configuration will not be resolved until the task is actually executed.

Now, if you run your task, it should work as expected using the configured version of the codec library:

$ gradle clean md5

> Task :md5
Generating MD5 for einstein.txt...
Generating MD5 for feynman.txt...
Generating MD5 for oppenheimer.txt...


BUILD SUCCESSFUL in 9s

Creating a Worker Daemon

Sometimes it is desirable to create even further isolation when executing items of work. For instance, external libraries may rely on certain system properties to be set which may conflict between work items. Or a library might not be compatible with the version of JDK that Gradle is running with and may need to be run with a different version. The Worker API can accommodate this with an isolation mode of PROCESS that causes the work to execute in a separate "worker daemon". These worker daemon processes will persist across builds and can be reused during subsequent builds. If system resources get low, however, Gradle will stop any unused worker daemons.

To utilize a worker daemon, you can simply change the isolation mode on the work items. You may also want to configure custom settings for the new process.

buildSrc/src/main/java/CreateMD5.java
import org.gradle.api.Action;
import org.gradle.api.file.FileCollection;
import org.gradle.api.tasks.*;
import org.gradle.process.JavaForkOptions;
import org.gradle.workers.*;

import javax.inject.Inject;
import java.io.File;
import java.util.Set;

class CreateMD5 extends SourceTask {
    private final WorkerExecutor workerExecutor;

    @OutputDirectory
    File destinationDir;

    @InputFiles
    FileCollection codecClasspath;

    @Inject
    public CreateMD5(WorkerExecutor workerExecutor) {
        super();
        this.workerExecutor = workerExecutor;
    }

    @TaskAction
    public void createHashes() {
        for (File sourceFile : getSource().getFiles()) {
            File md5File = new File(destinationDir, sourceFile.getName() + ".md5");
            workerExecutor.submit(GenerateMD5.class, new Action<WorkerConfiguration>() {
                @Override
                public void execute(WorkerConfiguration config) {
                    config.setIsolationMode(IsolationMode.PROCESS); (1)
                    config.forkOptions(new Action<JavaForkOptions>() {
                        @Override
                        public void execute(JavaForkOptions options) {
                            options.setMaxHeapSize("64m"); (2)
                        }
                    });
                    config.params(sourceFile, md5File);
                    config.classpath(codecClasspath);
                }
            });
        }
    }
}
1 Change the isolation mode to PROCESS.
2 Set up the JavaForkOptions for the new process.

Now, you should be able to run your task, and it will work as expected but using worker daemons instead:

$ gradle clean md5

> Task :md5
Generating MD5 for einstein.txt...
Generating MD5 for feynman.txt...
Generating MD5 for oppenheimer.txt...


BUILD SUCCESSFUL in 10s

Note that the execution time may be somewhat high. This is because Gradle has to start a new process for each worker daemon, which is expensive. However, if you run your task again, you’ll see that it runs much faster. This is because the worker daemon(s) started during the initial build have persisted and are available for use immediately during subsequent builds.

$ gradle clean md5

> Task :md5
Generating MD5 for einstein.txt...
Generating MD5 for feynman.txt...
Generating MD5 for oppenheimer.txt...


BUILD SUCCESSFUL in 5s

Summary

In this guide you learned how to:

  • Introduce the Worker API to an existing custom task to execute work in parallel with minimum isolation

  • Use the Worker API classloader isolation mode to execute work using an isolated classpath

  • Configure your task to isolate work even further in a separate "worker daemon" process

Help improve this guide

Have feedback or a question? Found a typo? Like all Gradle guides, help is just a GitHub issue away. Please add an issue or pull request to gradle-guides/using-the-worker-api and we’ll get back to you.