Kotlin/Library Package Maintainer Guide

This page contains useful information for any maintainers of the Kotlin packages' ebuilds.

Kotlin library build process overview
The upstream project uses Gradle as the build system, which handles the building process of the Kotlin libraries automatically. However, as of July 2021, Gradle is not integrated with Portage, and there is not an eclass that can be used to build and package artifacts for a Gradle project easily. Invoking Gradle directly in an ebuild seems to be a feasible solution, but Gradle downloads project dependencies from the Internet on its own, requiring  to be disabled in Portage's FEATURES and preventing any dependencies installed by Portage to be reused, so it is against the best practices.

Instead of relying on Gradle to automatically perform the build steps, the Kotlin library ebuilds directly call the Kotlin compiler and with equivalent compiler options, arguments and list of sources Gradle would use when the upstream project is being built by it. The sequence in which the modules are built and the order of compiler invocations are the same as the upstream project too. Those details about the build process are found through reverse engineering efforts, which will be described in this page below.

A Kotlin library component that is intended to be used with JVM may contain Java sources in addition to Kotlin sources. When such a component is being built, the Kotlin sources will be compiled first, and files will be generated. Then, any Java sources will be compiled by with those  files in the classpath. The files and the class files compiled from Java sources are all the contents in the JAR for the component in most cases. A few components may also have resource files that need to be included in the JAR as well.

Eclasses
The following eclasses are provided to facilitate creation of ebuilds for Kotlin libraries:


 * : The eclass for general Kotlin packages with Kotlin sources to compile, with support for compiling additional Java sources, JAR generation, source archive creation, JUnit 4 tests, installing pre-built binary JAR, and checks based on and .  This eclass is mainly designed for building sources in the main Kotlin programming language project; it might be possible to use it to build other Kotlin projects such as kotlinx libraries and third-party Kotlin libraries, but this has not been tested yet.
 * : An eclass based on for building modules under the directory in the upstream project's source tree.  The directory consists of reflection.jvm, which is the major part of kotlin-reflect, and various other modules which are both required by reflection.jvm during build time and parts of kotlin-reflect.  The modules correspond to the   subprojects in the main Gradle project.  This is a specialized eclass which is not intended to be used by a general Kotlin package's ebuild.

Obtaining compiler options used by Gradle
To build the Kotlin library packages properly and correctly, it is necessary to know how Gradle would compile them. Gradle will print the compiler options it uses to the output when its  option is enabled. As a side effect, this will make the output very verbose, so saving the output to a file might make searching in it easier. For example, the following command runs Gradle task  and copies the output to both standard output and, overwriting any existing contents in the file should it already exist:

Whenever Gradle invokes the Kotlin compiler, it prints the Kotlin compiler class, compiler classpath and compiler arguments to the debug output. These messages can be searched by querying occurrences of string, such as:

Gradle also prints the arguments it uses for. These arguments can be found by searching for occurrences of string, like:

For a few modules, Gradle may also call a command to start an external process directly. For example, it calls the command directly to build the   subproject. Such commands can be captured by searching for string  in the debug log:

Kotlin compiler classes and wrappers
The standalone version of the Kotlin compiler packaged and distributed by the upstream contains the following components for the compiler proper:


 * , the main compiler module
 * Various compiler wrapper executables
 * Other libraries used by the compiler

A compiler wrapper may set the KOTLIN_COMPILER environment variable to the name of the compiler class to be used, and the environment variable will be read by , the main Kotlin compiler wrapper.

Here is a list of classes in that are used when Gradle builds the Kotlin libraries:


 * org.jetbrains.kotlin.cli.jvm.K2JVMCompiler: Takes Kotlin sources as input, and produces Java class files as output. This is the default compiler class if KOTLIN_COMPILER is unset, and it is also the compiler class used in building all the JVM libraries.
 * org.jetbrains.kotlin.cli.js.K2JSCompiler: Takes Kotlin sources as input, and produces JavaScript files as output. This is the compiler class for.
 * org.jetbrains.kotlin.cli.js.internal.JSStdlibLinker: Processes JavaScript sources. Only used in building kotlin-stdlib-js.
 * org.jetbrains.kotlin.serialization.builtins.RunKt: The built-in serializer, which takes Kotlin sources as input and produces files as output.  Only used in building.

Compiler class used in kotlin-libs.eclass
does not have a variable for setting KOTLIN_COMPILER before is invoked, so the default compiler class for, which is org.jetbrains.kotlin.cli.jvm.K2JVMCompiler, will be used. However, ebuilds using can still specify a different compiler class by directly setting the KOTLIN_COMPILER environment variable before the eclass's   is called. For example, the following code changes the Kotlin compiler class used by to org.jetbrains.kotlin.cli.js.K2JSCompiler:

This method only works for compiler classes that comply with the common Kotlin compiler options because the  function of  sets some of the options in the arguments to. Some compiler classes listed above are known to be incompatible with those options, and this method does not work on them:


 * org.jetbrains.kotlin.cli.js.internal.JSStdlibLinker
 * org.jetbrains.kotlin.serialization.builtins.RunKt

When Gradle uses one of these compiler classes, it would run these classes directly with like. More details can be found in the following sections.

To see if a compiler class complies with the common Kotlin compiler options, try to set KOTLIN_COMPILER to that class and run, then see what options the compiler class accepts. If no common compiler option is shown at all in the help message or an exception is even thrown, then the compiler class is likely to not comply with the common compiler options.

{{Cmd|KOTLIN_COMPILER{{=}}org.jetbrains.kotlin.cli.js.K2JSCompiler kotlinc -help|collapse-output=true|output= Usage: kotlinc-js,   and   phase functions work similarly as {{Path|java-pkg-simple.eclass}} too. also has some additional variables for controlling how Kotlin sources are compiled, tested and installed.

Allow pre-built binary JAR to be used
The upstream hosts pre-built binary JAR artifacts for many libraries on Maven Central. To allow the pre-built JAR to be installed for an ebuild, specify the URI to the JAR in KOTLIN_LIBS_BINJAR_SRC_URI before inheriting kotlin-libs, and set JAVA_BINJAR_FILENAME to the base name of the JAR anywhere in the ebuild.

Specifying a non-empty value for KOTLIN_LIBS_BINJAR_SRC_URI has the following effects:


 * A  USE flag will be automatically added for the ebuild.  When this USE flag is enabled, the package will not be compiled and installed from source; the pre-built JAR pointed by KOTLIN_LIBS_BINJAR_SRC_URI will be installed instead.


 * If the  USE flag is disabled, then during , the JAR created by the ebuild will be compared with the pre-built JAR pointed by KOTLIN_LIBS_BINJAR_SRC_URI for non-trivial difference in JAR contents and incompatibility in API.  This is equivalent to specifying   for.


 * Another variable, KOTLIN_LIBS_SRCJAR_SRC_URI, will be recognized by . If KOTLIN_LIBS_SRCJAR_SRC_URI has a non-empty value that is set before kotlin-libs is inherited, then the   USE flag can be set when the   USE flag is enabled.  With  , the ebuild will pull the file pointed by KOTLIN_LIBS_SRCJAR_SRC_URI and install it as the source archive for the package at.
 * If KOTLIN_LIBS_SRCJAR_SRC_URI has a non-empty value, then another variable, KOTLIN_LIBS_SRCJAR_FILENAME, must be set to the base name of the file pointed by KOTLIN_LIBS_SRCJAR_SRC_URI.

Dependencies
A Kotlin package can have both build dependencies and runtime dependencies on other Java or Kotlin packages. Build dependencies are to be added to DEPEND. Runtime dependencies that should present in the classpath when the package is being used should go into both RDEPEND and CP_DEPEND. CP_DEPEND is used by to generate a runtime classpath for the package accordingly.

Checking build dependencies
The list of build dependencies for a Kotlin package can be extracted from the value for the  option in the Kotlin compiler arguments Gradle uses to build the package, which can be obtained by using instructions in the  section.

For example, the following JARs are in the classpath when kotlin-test-junit 1.5.20 is built by Gradle:



Not all JARs in the classpath are really required to build the package: the package can still compile without them. In general, JARs meeting any of the following criteria are not required:


 * JARs that contain only files and no  files.  Generally, all  files have this property.
 * JARs which are a dependency of another JAR in the classpath but are not directly used by the Kotlin package. For instance,  is a dependency of, but it is not used by the package itself, nor is its presence in the classpath required for  to work.

After applying these criteria, the list of JARs from the classpath can be filtered so only the ones that are really necessary are left:



Now, the value of DEPEND in the ebuild can be derived from this list:

If the ebuild supports the  USE flag, then the build dependencies from the classpath are unnecessary when that USE flag is enabled. For such kind of ebuilds, the build dependencies can be pulled conditionally:

Checking runtime dependencies
Not all build dependencies are needed when the Kotlin package is used during runtime, so CP_DEPEND is typically a subset of DEPEND. The actual list of runtime dependencies can be obtained with the tool shipped with JDK, which is installed under. The tool can list Java packages required by a JAR, and for each package, it either gives the file name of the JAR in the  option's value that provides the package or states that no such JAR can be found.

To obtain the list of runtime dependencies, first run against the the JAR created by the ebuild:

Without any dependency specified with the  option, all packages that are not a part of Java SE API will be reported as. Now, try to use the package names as a clue to determine the Portage package that provides each Java package, and add them to the classpath to see if every Java package's provider can be found:

When no Java package's provider is not found, the list of packages that should be added to CP_DEPEND (and thus RDEPEND ) can be derived:

In case CP_DEPEND is a subset of DEPEND, the value of DEPEND can be simplified by replacing the common dependencies with , such as:

Generating full build-time classpath
Any package declared in CP_DEPEND will be automatically added to the value of  in the arguments to  when the ebuild is being merged. However, some packages in DEPEND might need to be included in the classpath too during the build time. These packages can be added to the classpath via the JAVA_CLASSPATH_EXTRA variable.

Compilation
provides variables that can be used to control the command-line arguments to and. For the simplest case where a package can be compiled with just one normal invocation of (and possibly one additional invocation of, it is possible to write an ebuild for the package which does not define   at all, by inheriting kotlin-libs and setting the relevant eclass variables.

KOTLIN_LIBS_COMMON_SOURCES_DIR
This variable accepts an array of directories in which all files under it and its subdirectories should be added to the value of 's   option. will automatically transform this variable's value into a comma-separated list of files found under the specified directories and pass the list to the   option.

KOTLIN_LIBS_JAVA_SOURCE_ROOTS
This variable accepts an array whose elements should be added to the value of 's  option. will automatically join those elements to a string with a comma as the separator and pass the string to the  option.

KOTLIN_LIBS_KOTLINC_ARGS
This variable accepts an array of any extra options and arguments to.

There is no need to include the following options in this variable because they will be added by automatically, and/or they are overridable with other variables for the eclass:


 * and : These options are set to the feature release version by default and can be overridden with KOTLIN_LIBS_WANT_TARGET.
 * : This option is set to PN by default and can be overriden with KOTLIN_LIBS_MODULE_NAME.
 * : This option is controlled by KOTLIN_LIBS_COMMON_SOURCES_DIR.
 * : This option is controlled by KOTLIN_LIBS_JAVA_SOURCE_ROOTS.
 * : This option is controlled by CP_DEPEND, JAVA_CLASSPATH_EXTRA , and other variables in that affect the build-time classpath.
 * : This option will be set by the eclass to an appropriate path.

Redundant compiler options
In general, every compiler option Gradle would use to build a package should be set in the ebuild for the sake of consistency with the upstream. However, the following options can be safely omitted if certain conditions are met:


 * is unnecessary if  is set because   implies.
 * is unnecessary if the JDK to be used for compilation is pointed by the JAVA_HOME environment variable.
 * Options that merely control what messages prints to standard output, such as   and , can be omitted if the information shown as a result of adding them to the command-line arguments does not need to be printed.

KOTLIN_LIBS_SRC_DIR
This variable accepts an array of directories that contain the Kotlin sources to compile. will find all files under those directories and their subdirectories, and specify those files as the input files for.

KOTLIN_LIBS_JAVAC_ARGS
This variable accepts an array of any extra options and arguments to. If any Java sources need to be compiled as a result of having a non-empty value for KOTLIN_LIBS_JAVA_SOURCE_ROOTS, then these arguments will be passed to ; otherwise, they will be unused.

KOTLIN_LIBS_RUNTIME_COMPONENT
This variable manipulates the content of file in the produced JAR. Whether or not this variable should be set and the proper value for it can be determined by inspecting in the JAR pre-built by the upstream. If it contains a Kotlin-Runtime-Component entry, then this eclass variable should be set to the value for that entry (which should be either  or  ); this will cause an equivalent  to be created and copied into the JAR generated by the ebuild. If there is no such entry, or does not exist in the pre-built JAR, or there is even not a pre-built JAR for the package, then this variable needs not be set.

Verification
supports the following types of verification on the JAR built by the ebuild:


 * JAR comparison against the upstream's pre-built binary JAR, based on pkgdiff and japi-compliance-checker. This check will be automatically enabled for all ebuilds defining a non-empty value for KOTLIN_LIBS_BINJAR_SRC_URI when the   USE flag is disabled.
 * JUnit 4 test suite execution. All upstream test suites are based on JUnit 4.

Details about JAR comparison have already been introduced in the section. This section focuses on testing with JUnit 4.

Enabling JUnit 4
To enable JUnit 4 support of, add  to the value of KOTLIN_LIBS_TESTING_FRAMEWORKS before inheriting kotlin-libs:

This will automatically add dev-java/junit:4 to DEPEND when the  USE flag is enabled. However, it does not add dev-java/kotlin-test-junit to DEPEND, so if a Kotlin package's tests need it, the dependency needs to be explicitly declared in the ebuild.

Adding test dependencies
Test dependencies of a package are generally not mandatory if the test suite for the package will not be compiled and executed at all. Therefore, they should be pulled only if the  USE flag is enabled. Plus, test dependencies are only needed on during the build time because their sole purpose is to support the   phase function, so they should go into DEPEND.

In addition, any test dependencies that are Java packages and should present in the classpath when the test suite is being compiled and run should be added to JAVA_TEST_GENTOO_CLASSPATH, just like JAVA_CLASSPATH_EXTRA :

Obtaining compiler options for test classes
The Kotlin compiler options Gradle would use to compile the test classes can be obtained using the same method described in the section. Note that the Gradle task for test source compilation is. For example, the following command compiles the test suite for kotlin-stdlib:

Not all Kotlin core library components have a test suite. For instance, there is no test for kotlin-reflect. The  task can still be executed for those components, and the debug output will indicate that no Kotlin source is found:

2021-07-15T11:04:29.214-0700 [DEBUG] [org.gradle.api.Task] [KOTLIN] No Kotlin files found, skipping Kotlin compiler task

Eclass variables for test class compilation
The following variables of can be used to set the Kotlin compiler options for compilation of test sources. Their usage is the same as their counterpart introduced in the section without the word TEST in their name.


 * KOTLIN_LIBS_TEST_COMMON_SOURCES_DIR
 * KOTLIN_LIBS_TEST_KOTLINC_ARGS
 * KOTLIN_LIBS_TEST_SRC_DIR

Excluding classes from JUnit 4 invocation
Excluding a class from the test execution might be necessary for any of these reasons:


 * The class does not contain any JUnit 4 test method. Some classes in the test sources are just utility classes that solely provide helper methods for test cases, and passing them to JUnit 4 will cause a failure.
 * Some tests in the class would always fail due to unmet preconditions that cannot be satisfied within Portage. For example, the test.io.ReadWrite.testURL test case needs Internet access to establish a connection to http://kotlinlang.org, but with   enabled in Portage's FEATURES by default, the connection cannot be established, hence the test case would always fail.
 * Some tests in the class are known to have trouble with passing such that even the upstream has disabled them from the Gradle build scripts.

To prevent a class from being passed to JUnit 4, add the class's name to the JAVA_TEST_EXCLUDES variable's value (which should be an array). The variable is declared by, but honors it too.