User:Leo3418/Kotlin/Package Maintainer Guide

From Gentoo Wiki
Jump to:navigation Jump to:search

This page contains useful information for any maintainers of the Kotlin packages' ebuilds.

Kotlin library build process overview

The upstream project uses Gradle as the build system, which handles the building process of the Kotlin libraries automatically. However, as of July 2021, Gradle is not integrated with Portage, and there is not an eclass that can be used to build and package artifacts for a Gradle project easily. Invoking Gradle directly in an ebuild seems to be a feasible solution, but Gradle downloads project dependencies from the Internet on its own, requiring network-sandbox to be disabled in Portage's FEATURES and preventing any dependencies installed by Portage to be reused[1], so it is against the best practices.

Instead of relying on Gradle to automatically perform the build steps, the Kotlin library ebuilds directly call the Kotlin compiler and javac with equivalent compiler options, arguments and list of sources Gradle would use when the upstream project is being built by it. The sequence in which the modules are built and the order of compiler invocations are the same as the upstream project too. Those details about the build process are found through reverse engineering efforts, which will be described in this page below.

A Kotlin library component that is intended to be used with JVM may contain Java sources in addition to Kotlin sources. When such a component is being built, the Kotlin sources will be compiled first, and *Kt.class files will be generated. Then, any Java sources will be compiled by javac with those *Kt.class files in the classpath. The *Kt.class files and the class files compiled from Java sources are all the contents in the JAR for the component in most cases. A few components may also have resource files that need to be included in the JAR as well.

Eclasses

The following eclasses are provided to facilitate creation of ebuilds for Kotlin libraries:

  • kotlin-libs.eclass: The eclass for general Kotlin packages with Kotlin sources to compile, with support for compiling additional Java sources, JAR generation, source archive creation, JUnit 4 tests, installing pre-built binary JAR, and checks based on dev-util/pkgdiff and dev-util/japi-compliance-checker. This eclass is mainly designed for building sources in the main Kotlin programming language project; it might be possible to use it to build other Kotlin projects such as kotlinx libraries and third-party Kotlin libraries, but this has not been tested yet.
  • kotlin-core-deps.eclass: An eclass based on kotlin-libs.eclass for building modules under the core directory in the upstream project's source tree. The directory consists of reflection.jvm, which is the major part of kotlin-reflect, and various other modules which are both required by reflection.jvm during build time and parts of kotlin-reflect. The modules correspond to the kotlin:core:* subprojects in the main Gradle project. This is a specialized eclass which is not intended to be used by a general Kotlin package's ebuild.

Obtaining compiler options used by Gradle

Note
The instructions here assume use of the Graddle Wrapper located in the upstream project's source tree. As a result, in the instructions, the command to invoke Gradle is ./gradlew. Unless otherwise specified, this command should be executed when the working directory is the project root.

To build the Kotlin library packages properly and correctly, it is necessary to know how Gradle would compile them. Gradle will print the compiler options it uses to the output when its --debug option is enabled. As a side effect, this will make the output very verbose, so saving the output to a file might make searching in it easier. For example, the following command runs Gradle task kotlin-stdlib:jar and copies the output to both standard output and /tmp/gradle.log, overwriting any existing contents in the file should it already exist:

user $./gradlew --debug kotlin-stdlib:jar | tee /tmp/gradle.log

Whenever Gradle invokes the Kotlin compiler, it prints the Kotlin compiler class, compiler classpath and compiler arguments to the debug output. These messages can be searched by querying occurrences of string Kotlin compiler, such as:

user $grep "Kotlin compiler" /tmp/gradle.log
2021-07-10T19:23:42.477-0700 [DEBUG] [org.gradle.api.Task] [KOTLIN] Kotlin compiler class: org.jetbrains.kotlin.cli.jvm.K2JVMCompiler
2021-07-10T19:23:42.477-0700 [DEBUG] [org.gradle.api.Task] [KOTLIN] Kotlin compiler classpath: /home/leo/.nobackup/gradle/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-compiler-embeddable/1.5.20-dev-4401/b0096e9eed326f3694585663dc28f758585fb394/kotlin-compiler-embeddable-1.5.20-dev-4401.jar, ... (truncated)
2021-07-10T19:23:42.477-0700 [DEBUG] [org.gradle.api.Task] [KOTLIN] :kotlin-stdlib:compileKotlin Kotlin compiler args: -Xallow-no-source-files -classpath /home/leo/Projects/forks/kotlin/libraries/stdlib/common/build/libs/kotlin-stdlib-common-1.5.255-SNAPSHOT.jar:/home/leo/Projects/forks/kotlin/core/builtins/build/libs/builtins-1.5.255-SNAPSHOT.jar:/home/leo/.nobackup/gradle/.gradle/caches/modules-2/files-2.1/org.jetbrains/annotations/13.0/919f0dfe192fb4e063e7dacadee7f8bb9a2672a9/annotations-13.0.jar -d /home/leo/Projects/forks/kotlin/libraries/stdlib/jvm/build/classes/kotlin/main -Xjava-source-roots=/home/leo/Projects/forks/kotlin/libraries/stdlib/jvm/src,/home/leo/Projects/forks/kotlin/libraries/stdlib/jvm/runtime ... (truncated)

Gradle also prints the arguments it uses for javac. These arguments can be found by searching for occurrences of string NormalizingJavaCompiler, like:

user $grep "NormalizingJavaCompiler" /tmp/gradle.log
2021-07-10T19:24:31.485-0700 [DEBUG] [org.gradle.api.internal.tasks.compile.NormalizingJavaCompiler] Compiler arguments: -source 1.6 -target 1.6 -d /home/leo/Projects/forks/kotlin/libraries/stdlib/jvm/build/classes/java/main -encoding UTF-8 -h /home/leo/Projects/forks/kotlin/libraries/stdlib/jvm/build/generated/sources/headers/java/main ... (truncated)

For a few modules, Gradle may also call a command to start an external process directly. For example, it calls the java command directly to build the kotlin:core:builtins subproject. Such commands can be captured by searching for string Starting process in the debug log:

user $grep "Starting process" /tmp/gradle.log
2021-07-10T19:23:22.693-0700 [INFO] [org.gradle.process.internal.DefaultExecHandle] Starting process 'command '/usr/lib64/openjdk-8/bin/java''. Working directory: /home/leo/Projects/forks/kotlin/core/builtins Command: /usr/lib64/openjdk-8/bin/java -Didea.io.use.nio2=true -Dkotlin.builtins.serializer.log=true -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant ... (truncated)
Note
If some source files have already been processed by Gradle at least once before, then the Gradle debug output will not contain the compiler options for those files. This is because the upstream project is configured to use Gradle build cache to avoid recompiling the same set of source files twice. The build cache is located under ~/.gradle/caches/build-cache-1/ and is not subject to deletions done by ./gradlew clean. To completely clean everything so Gradle will start over and print compiler options again, run the following commands prior to Gradle invocation:
user $./gradlew clean
user $rm ~/.gradle/caches/build-cache-1/*
Note
The upstream project is configured to use Gradle Daemons, which can consume a lot of system memory. If no more Gradle commands would be executed for a while, stopping any running Daemons can free up plenty of memory. To stop all running Daemons, run ./gradlew --stop.

Kotlin compiler classes and wrappers

The standalone version of the Kotlin compiler packaged and distributed by the upstream (kotlin-compiler-1.x.y.zip) contains the following components for the compiler proper:

A compiler wrapper may set the KOTLIN_COMPILER environment variable to the name of the compiler class to be used[2], and the environment variable will be read by kotlinc, the main Kotlin compiler wrapper[3].

Note
The KOTLIN_COMPILER environment variable's value is effectively equivalent to any Kotlin compiler class in the Gradle debug output.

Here is a list of classes in kotlin-compiler.jar that are used when Gradle builds the Kotlin libraries:

  • org.jetbrains.kotlin.cli.jvm.K2JVMCompiler: Takes Kotlin sources as input, and produces Java class files as output. This is the default compiler class if KOTLIN_COMPILER is unset[4], and it is also the compiler class used in building all the JVM libraries.
  • org.jetbrains.kotlin.cli.js.K2JSCompiler: Takes Kotlin sources as input, and produces JavaScript files as output. This is the compiler class for kotlin-js[2].
  • org.jetbrains.kotlin.cli.js.internal.JSStdlibLinker: Processes JavaScript sources. Only used in building kotlin-stdlib-js[5].
  • org.jetbrains.kotlin.serialization.builtins.RunKt: The built-in serializer, which takes Kotlin sources as input and produces .kotlin_builtins files as output. Only used in building kotlin:core:builtins[6].

Compiler class used in kotlin-libs.eclass

kotlin-libs.eclass does not have a variable for setting KOTLIN_COMPILER before kotlinc is invoked, so the default compiler class for kotlinc, which is org.jetbrains.kotlin.cli.jvm.K2JVMCompiler, will be used. However, ebuilds using kotlin-libs.eclass can still specify a different compiler class by directly setting the KOTLIN_COMPILER environment variable before the eclass's src_compile is called. For example, the following code changes the Kotlin compiler class used by kotlin-libs.eclass to org.jetbrains.kotlin.cli.js.K2JSCompiler:

CODE Use org.jetbrains.kotlin.cli.js.K2JSCompiler as Kotlin compiler class for kotlin-libs.eclass
src_compile() {
	export KOTLIN_COMPILER=org.jetbrains.kotlin.cli.js.K2JSCompiler
	kotlin-libs_src_compile
}

This method only works for compiler classes that comply with the common Kotlin compiler options because the src_compile function of kotlin-libs.eclass sets some of the options in the arguments to kotlinc. Some compiler classes listed above are known to be incompatible with those options, and this method does not work on them:

  • org.jetbrains.kotlin.cli.js.internal.JSStdlibLinker
  • org.jetbrains.kotlin.serialization.builtins.RunKt

When Gradle uses one of these compiler classes, it would run these classes directly with java like java --classpath kotlin-compiler.jar org.jetbrains.kotlin.serialization.builtins.RunKt <args>. More details can be found in the following sections.

To see if a compiler class complies with the common Kotlin compiler options, try to set KOTLIN_COMPILER to that class and run kotlinc -help, then see what options the compiler class accepts. If no common compiler option is shown at all in the help message or an exception is even thrown, then the compiler class is likely to not comply with the common compiler options.

user $KOTLIN_COMPILER=org.jetbrains.kotlin.cli.js.K2JSCompiler kotlinc -help
Usage: kotlinc-js <options> <source files>
where possible options include:
  -libraries <path>          Paths to Kotlin libraries with .meta.js and .kjsm files, separated by system path separator
  -main {call|noCall}        Define whether the `main` function should be called upon execution
  -meta-info                 Generate .meta.js and .kjsm files with metadata. Use to create a library
  -module-kind {plain|amd|commonjs|umd}
                             Kind of the JS module generated by the compiler
  -no-stdlib                 Don't automatically include the default Kotlin/JS stdlib into compilation dependencies
  -output <filepath>         Destination *.js file for the compilation result
  -output-postfix <path>     Add the content of the specified file to the end of output file
  -output-prefix <path>      Add the content of the specified file to the beginning of output file
  -source-map                Generate source map
  -source-map-base-dirs <path> Base directories for calculating relative paths to source files in source map
  -source-map-embed-sources {always|never|inlining}
                             Embed source files into source map
  -source-map-prefix         Add the specified prefix to paths in the source map
  -target { v5 }             Generate JS files for specific ECMA version
  -Werror                    Report an error if there are any warnings
  -api-version <version>     Allow using declarations only from the specified version of bundled libraries
  -X                         Print a synopsis of advanced options
  -help (-h)                 Print a synopsis of standard options
  -kotlin-home <path>        Path to the home directory of Kotlin compiler used for discovery of runtime libraries
  -language-version <version> Provide source compatibility with the specified version of Kotlin
  -P plugin:<pluginId>:<optionName>=<value>
                             Pass an option to a plugin
  -progressive               Enable progressive compiler mode.
                             In this mode, deprecations and bug fixes for unstable code take effect immediately,
                             instead of going through a graceful migration cycle.
                             Code written in the progressive mode is backward compatible; however, code written in
                             non-progressive mode may cause compilation errors in the progressive mode.
  -script                    Evaluate the given Kotlin script (*.kts) file
  -nowarn                    Generate no warnings
  -verbose                   Enable verbose logging output
  -version                   Display compiler version
  @<argfile>                 Read compiler arguments and file paths from the given file

For details, see https://kotl.in/cli
user $KOTLIN_COMPILER=org.jetbrains.kotlin.serialization.builtins.RunKt kotlinc -help
Kotlin built-ins serializer

Usage: ... <destination dir> (<source dir>)+

Analyzes Kotlin sources found in the given source directories and serializes
found top-level declarations to <destination dir> (*.kotlin_builtins files)
user $KOTLIN_COMPILER=org.jetbrains.kotlin.cli.js.internal.JSStdlibLinker kotlinc -help
Exception in thread "main" java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.jetbrains.kotlin.preloading.Preloader.run(Preloader.java:87)
	at org.jetbrains.kotlin.preloading.Preloader.main(Preloader.java:44)
Caused by: java.lang.ArrayIndexOutOfBoundsException: 1
	at org.jetbrains.kotlin.cli.js.internal.JSStdlibLinker.main(JSStdlibLinker.kt:25)
	... 6 more

Creating an ebuild based on kotlin-libs.eclass

kotlin-libs.eclass is based on java-pkg-simple.eclass, so it recognizes and uses many variables of that eclass, supporting the same semantics. Its src_compile, src_install and src_test phase functions work similarly as java-pkg-simple.eclass too. kotlin-libs.eclass also has some additional variables for controlling how Kotlin sources are compiled, tested and installed.

Allow pre-built binary JAR to be used

The upstream hosts pre-built binary JAR artifacts for many libraries on Maven Central. To allow the pre-built JAR to be installed for an ebuild, specify the URI to the JAR in KOTLIN_LIBS_BINJAR_SRC_URI before inheriting kotlin-libs, and set JAVA_BINJAR_FILENAME to the base name of the JAR anywhere in the ebuild.

CODE Specify URI and file name for the pre-built JAR
EAPI=7

KOTLIN_LIBS_BINJAR_SRC_URI="https://repo1.maven.org/maven2/org/jetbrains/kotlin/${PN}/${PV}/${P}.jar"

inherit kotlin-libs

JAVA_BINJAR_FILENAME="${P}.jar"

Specifying a non-empty value for KOTLIN_LIBS_BINJAR_SRC_URI has the following effects:

  • A binary USE flag will be automatically added for the ebuild. When this USE flag is enabled, the package will not be compiled and installed from source; the pre-built JAR pointed by KOTLIN_LIBS_BINJAR_SRC_URI will be installed instead.
  • If the binary USE flag is disabled, then during src_test, the JAR created by the ebuild will be compared with the pre-built JAR pointed by KOTLIN_LIBS_BINJAR_SRC_URI for non-trivial difference in JAR contents and incompatibility in API. This is equivalent to specifying JAVA_TESTING_FRAMEWORKS="pkgdiff" for java-pkg-simple.eclass.
  • Another variable, KOTLIN_LIBS_SRCJAR_SRC_URI, will be recognized by kotlin-libs.eclass. If KOTLIN_LIBS_SRCJAR_SRC_URI has a non-empty value that is set before kotlin-libs is inherited, then the source USE flag can be set when the binary USE flag is enabled. With USE="binary source", the ebuild will pull the file pointed by KOTLIN_LIBS_SRCJAR_SRC_URI and install it as the source archive for the package at /usr/share/${PN}-${SLOT}/sources.
    • If KOTLIN_LIBS_SRCJAR_SRC_URI has a non-empty value, then another variable, KOTLIN_LIBS_SRCJAR_FILENAME, must be set to the base name of the file pointed by KOTLIN_LIBS_SRCJAR_SRC_URI.

Dependencies

A Kotlin package can have both build dependencies and runtime dependencies on other Java or Kotlin packages. Build dependencies are to be added to DEPEND. Runtime dependencies that should present in the classpath when the package is being used should go into both RDEPEND and CP_DEPEND. CP_DEPEND is used by java-utils-2.eclass to generate a runtime classpath for the package accordingly.

Checking build dependencies

The list of build dependencies for a Kotlin package can be extracted from the value for the -classpath option in the Kotlin compiler arguments Gradle uses to build the package, which can be obtained by using instructions in the #Obtaining compiler options used by Gradle section.

For example, the following JARs are in the classpath when kotlin-test-junit 1.5.20 is built by Gradle:

  • kotlin-test-annotations-common-1.5.255-SNAPSHOT.jar
  • kotlin-test-1.5.255-SNAPSHOT.jar
  • junit-4.12.jar
  • kotlin-test-common-1.5.255-SNAPSHOT.jar
  • kotlin-stdlib-1.5.255-SNAPSHOT.jar
  • kotlin-stdlib-common-1.5.255-SNAPSHOT.jar
  • hamcrest-core-1.3.jar
  • annotations-13.0.jar

Not all JARs in the classpath are really required to build the package: the package can still compile without them. In general, JARs meeting any of the following criteria are not required:

  • JARs that contain only .kotlin_metadata files and no .class files. Generally, all kotlin-*-common-*.jar files have this property.
  • JARs which are a dependency of another JAR in the classpath but are not directly used by the Kotlin package. For instance, hamcrest-core-1.3.jar is a dependency of junit-4.12.jar, but it is not used by the package itself, nor is its presence in the classpath required for junit-4.12.jar to work.

After applying these criteria, the list of JARs from the classpath can be filtered so only the ones that are really necessary are left:

  • kotlin-test-1.5.255-SNAPSHOT.jar
  • junit-4.12.jar
  • kotlin-stdlib-1.5.255-SNAPSHOT.jar
  • annotations-13.0.jar

Now, the value of DEPEND in the ebuild can be derived from this list:

CODE DEPEND variable definition
DEPEND="
	~dev-java/kotlin-test-${PV}:${SLOT}
	dev-java/junit:4
	~dev-java/kotlin-stdlib-${PV}:${SLOT}
	dev-java/jetbrains-annotations:13
"

If the ebuild supports the binary USE flag, then the build dependencies from the classpath are unnecessary when that USE flag is enabled. For such kind of ebuilds, the build dependencies can be pulled conditionally:

CODE DEPEND variable definition for an ebuild with the binary USE flag
DEPEND="
	!binary? (
		~dev-java/kotlin-test-${PV}:${SLOT}
		dev-java/junit:4
		~dev-java/kotlin-stdlib-${PV}:${SLOT}
		dev-java/jetbrains-annotations:13
	)
"
Note
If a Kotlin core library package depends on other core library packages, it is recommended to declare dependency on the same version of them.
Note
kotlin-libs.eclass automatically inserts dependency on dev-lang/kotlin-bin for the same Kotlin feature release as the package or above to DEPEND. If the ebuild supports the binary USE flag, then dev-lang/kotlin-bin will be pulled only if any Kotlin sources need to be compiled.

Checking runtime dependencies

Not all build dependencies are needed when the Kotlin package is used during runtime, so CP_DEPEND is typically a subset of DEPEND. The actual list of runtime dependencies can be obtained with the jdeps tool shipped with JDK, which is installed under ${JAVA_HOME}/bin. The tool can list Java packages required by a JAR, and for each package, it either gives the file name of the JAR in the -classpath option's value that provides the package or states that no such JAR can be found.

To obtain the list of runtime dependencies, first run jdeps against the the JAR created by the ebuild:

user $${JAVA_HOME}/bin/jdeps "$(java-config -p kotlin-test-junit-1.5)"
kotlin-test-junit.jar -> not found
kotlin-test-junit.jar -> /usr/lib64/openjdk-8/jre/lib/rt.jar
   kotlin.test.junit (kotlin-test-junit.jar)
      -> java.lang                                          
      -> kotlin                                             not found
      -> kotlin.jvm.functions                               not found
      -> kotlin.jvm.internal                                not found
      -> kotlin.test                                        not found
      -> org.junit                                          not found
   kotlin.test.junit.annotations (kotlin-test-junit.jar)
      -> java.lang                                          
      -> kotlin                                             not found

Without any dependency specified with the -classpath option, all packages that are not a part of Java SE API will be reported as not found. Now, try to use the package names as a clue to determine the Portage package that provides each Java package, and add them to the classpath to see if every Java package's provider can be found:

user $${JAVA_HOME}/bin/jdeps -classpath "$(java-config -p kotlin-stdlib-1.5,kotlin-test-1.5,junit-4)" "$(java-config -p kotlin-test-junit-1.5)"
kotlin-test-junit.jar -> /usr/share/junit-4/lib/junit.jar
kotlin-test-junit.jar -> /usr/share/kotlin-stdlib-1.5/lib/kotlin-stdlib.jar
kotlin-test-junit.jar -> /usr/share/kotlin-test-1.5/lib/kotlin-test.jar
kotlin-test-junit.jar -> /usr/lib64/openjdk-8/jre/lib/rt.jar
   kotlin.test.junit (kotlin-test-junit.jar)
      -> java.lang                                          
      -> kotlin                                             kotlin-stdlib.jar
      -> kotlin.jvm.functions                               kotlin-stdlib.jar
      -> kotlin.jvm.internal                                kotlin-stdlib.jar
      -> kotlin.test                                        kotlin-test.jar
      -> org.junit                                          junit.jar
   kotlin.test.junit.annotations (kotlin-test-junit.jar)
      -> java.lang                                          
      -> kotlin                                             kotlin-stdlib.jar
Note
If the upstream provides a pre-built JAR for the Kotlin package, it is recommended to repeat the same process on the pre-built JAR as an extra check for the sake of consistency with the upstream.

When no Java package's provider is not found, the list of packages that should be added to CP_DEPEND (and thus RDEPEND) can be derived:

CODE Definitions of CP_DEPEND and RDEPEND variables
CP_DEPEND="
	~dev-java/kotlin-stdlib-${PV}:${SLOT}
	~dev-java/kotlin-test-${PV}:${SLOT}
	dev-java/junit:4
"
RDEPEND="${CP_DEPEND}"

In case CP_DEPEND is a subset of DEPEND, the value of DEPEND can be simplified by replacing the common dependencies with ${CP_DEPEND}, such as:

CODE Definitions of CP_DEPEND, DEPEND and RDEPEND variables
CP_DEPEND="
	~dev-java/kotlin-stdlib-${PV}:${SLOT}
	~dev-java/kotlin-test-${PV}:${SLOT}
	dev-java/junit:4
"
DEPEND="
	!binary? (
		${CP_DEPEND}
		dev-java/jetbrains-annotations:13
	)
"
RDEPEND="${CP_DEPEND}"

Generating full build-time classpath

Any package declared in CP_DEPEND will be automatically added to the value of -classpath in the arguments to kotlinc when the ebuild is being merged. However, some packages in DEPEND might need to be included in the classpath too during the build time. These packages can be added to the classpath via the JAVA_CLASSPATH_EXTRA variable.

CODE Definitions of CP_DEPEND, DEPEND, RDEPEND and JAVA_CLASSPATH_EXTRA variables
CP_DEPEND="
	~dev-java/kotlin-stdlib-${PV}:${SLOT}
	~dev-java/kotlin-test-${PV}:${SLOT}
	dev-java/junit:4
"
DEPEND="
	!binary? (
		${CP_DEPEND}
		dev-java/jetbrains-annotations:13
	)
"
RDEPEND="${CP_DEPEND}"

JAVA_CLASSPATH_EXTRA="jetbrains-annotations-13"

Compilation

kotlin-libs.eclass provides variables that can be used to control the command-line arguments to kotlinc and javac. For the simplest case where a package can be compiled with just one normal invocation of kotlinc (and possibly one additional invocation of javac, it is possible to write an ebuild for the package which does not define src_compile at all, by inheriting kotlin-libs and setting the relevant eclass variables.

KOTLIN_LIBS_COMMON_SOURCES_DIR

This variable accepts an array of directories in which all *.kt files under it and its subdirectories should be added to the value of kotlinc's -Xcommon-sources option. kotlin-libs.eclass will automatically transform this variable's value into a comma-separated list of *.kt files found under the specified directories and pass the list to the -Xcommon-sources option.

KOTLIN_LIBS_JAVA_SOURCE_ROOTS

This variable accepts an array whose elements should be added to the value of kotlinc's -Xjava-source-roots option. kotlin-libs.eclass will automatically join those elements to a string with a comma as the separator and pass the string to the -Xjava-source-roots option.

Note
If kotlinc's -Xjava-source-roots option is needed to build a package, then the package will also require all Java sources under the directories specified in the option's value to be compiled by javac after the Kotlin sources have been compiled. This is handled by kotlin-libs.eclass automatically: if KOTLIN_LIBS_JAVA_SOURCE_ROOTS is not empty, then the eclass will call javac to compile all *.java files under each directory defined for the variable (including any subdirectories). In other words, there would normally be no need to call javac to compile Java sources under KOTLIN_LIBS_JAVA_SOURCE_ROOTS from the ebuild.

KOTLIN_LIBS_KOTLINC_ARGS

This variable accepts an array of any extra options and arguments to kotlinc.

There is no need to include the following options in this variable because they will be added by kotlin-libs.eclass automatically, and/or they are overridable with other variables for the eclass:

  • -api-version and -language-version: These options are set to the feature release version by default and can be overridden with KOTLIN_LIBS_WANT_TARGET.
  • -module-name: This option is set to PN by default and can be overriden with KOTLIN_LIBS_MODULE_NAME.
  • -Xcommon-sources: This option is controlled by KOTLIN_LIBS_COMMON_SOURCES_DIR.
  • -Xjava-source-roots: This option is controlled by KOTLIN_LIBS_JAVA_SOURCE_ROOTS.
  • -classpath: This option is controlled by CP_DEPEND, JAVA_CLASSPATH_EXTRA, and other variables in java-pkg-simple.eclass that affect the build-time classpath.
  • -d: This option will be set by the eclass to an appropriate path.
Redundant compiler options

In general, every compiler option Gradle would use to build a package should be set in the ebuild for the sake of consistency with the upstream. However, the following options can be safely omitted if certain conditions are met:

  • -no-reflect is unnecessary if -no-stdlib is set because -no-stdlib implies -no-reflect.
  • -jdk-home is unnecessary if the JDK to be used for compilation is pointed by the JAVA_HOME environment variable.
  • Options that merely control what messages kotlinc prints to standard output, such as -version and -verbose, can be omitted if the information shown as a result of adding them to the command-line arguments does not need to be printed.

KOTLIN_LIBS_SRC_DIR

This variable accepts an array of directories that contain the Kotlin sources to compile. kotlin-libs.eclass will find all *.kt files under those directories and their subdirectories, and specify those files as the input files for kotlinc.

KOTLIN_LIBS_JAVAC_ARGS

This variable accepts an array of any extra options and arguments to javac. If any Java sources need to be compiled as a result of having a non-empty value for KOTLIN_LIBS_JAVA_SOURCE_ROOTS, then these arguments will be passed to javac; otherwise, they will be unused.

Note
The -source and -target options for javac can be controlled with the KOTLIN_LIBS_JAVA_WANT_SOURCE_TARGET variable.

KOTLIN_LIBS_RUNTIME_COMPONENT

This variable manipulates the content of file META-INF/MANIFEST.MF in the produced JAR. Whether or not this variable should be set and the proper value for it can be determined by inspecting META-INF/MANIFEST.MF in the JAR pre-built by the upstream. If it contains a Kotlin-Runtime-Component entry, then this eclass variable should be set to the value for that entry (which should be either Main or Test); this will cause an equivalent META-INF/MANIFEST.MF to be created and copied into the JAR generated by the ebuild. If there is no such entry, or META-INF/MANIFEST.MF does not exist in the pre-built JAR, or there is even not a pre-built JAR for the package, then this variable needs not be set.

Verification

kotlin-libs.eclass supports the following types of verification on the JAR built by the ebuild:

  • JAR comparison against the upstream's pre-built binary JAR, based on pkgdiff and japi-compliance-checker. This check will be automatically enabled for all ebuilds defining a non-empty value for KOTLIN_LIBS_BINJAR_SRC_URI when the binary USE flag is disabled.
  • JUnit 4 test suite execution. All upstream test suites are based on JUnit 4.

Details about JAR comparison have already been introduced in the #Allow pre-built binary JAR to be used section. This section focuses on testing with JUnit 4.

Enabling JUnit 4

To enable JUnit 4 support of kotlin-libs.eclass, add junit-4 to the value of KOTLIN_LIBS_TESTING_FRAMEWORKS before inheriting kotlin-libs:

CODE Enable JUnit 4 support of kotlin-libs.eclass
EAPI=7

KOTLIN_LIBS_TESTING_FRAMEWORKS="junit-4"

inherit kotlin-libs

This will automatically add dev-java/junit:4 to DEPEND when the test USE flag is enabled. However, it does not add dev-java/kotlin-test-junit to DEPEND, so if a Kotlin package's tests need it, the dependency needs to be explicitly declared in the ebuild.

Adding test dependencies

Test dependencies of a package are generally not mandatory if the test suite for the package will not be compiled and executed at all. Therefore, they should be pulled only if the test USE flag is enabled. Plus, test dependencies are only needed on CHOST during the build time because their sole purpose is to support the src_test phase function, so they should go into DEPEND.

CODE Test dependencies declaration
DEPEND="
	test? (
		~dev-java/kotlin-test-${PV}:${SLOT}
		~dev-java/kotlin-test-junit-${PV}:${SLOT}
	)
"

In addition, any test dependencies that are Java packages and should present in the classpath when the test suite is being compiled and run should be added to JAVA_TEST_GENTOO_CLASSPATH, just like JAVA_CLASSPATH_EXTRA:

CODE Declare test dependencies and add them to classpath
DEPEND="
	test? (
		~dev-java/kotlin-test-${PV}:${SLOT}
		~dev-java/kotlin-test-junit-${PV}:${SLOT}
	)
"

JAVA_TEST_GENTOO_CLASSPATH="
	kotlin-test-${SLOT}
	kotlin-test-junit-${SLOT}
"

Obtaining compiler options for test classes

The Kotlin compiler options Gradle would use to compile the test classes can be obtained using the same method described in the #Obtaining compiler options used by Gradle section. Note that the Gradle task for test source compilation is testClasses. For example, the following command compiles the test suite for kotlin-stdlib:

user $./gradlew --debug kotlin-stdlib:testClasses | tee /tmp/gradle.log

Not all Kotlin core library components have a test suite. For instance, there is no test for kotlin-reflect. The testClasses task can still be executed for those components, and the debug output will indicate that no Kotlin source is found:

2021-07-15T11:04:29.214-0700 [DEBUG] [org.gradle.api.Task] [KOTLIN] No Kotlin files found, skipping Kotlin compiler task

Eclass variables for test class compilation

The following variables of kotlin-libs.eclass can be used to set the Kotlin compiler options for compilation of test sources. Their usage is the same as their counterpart introduced in the #Compilation section without the word TEST in their name.

  • KOTLIN_LIBS_TEST_COMMON_SOURCES_DIR
  • KOTLIN_LIBS_TEST_KOTLINC_ARGS
  • KOTLIN_LIBS_TEST_SRC_DIR

Excluding classes from JUnit 4 invocation

Excluding a class from the test execution might be necessary for any of these reasons:

  • The class does not contain any JUnit 4 test method. Some classes in the test sources are just utility classes that solely provide helper methods for test cases, and passing them to JUnit 4 will cause a failure.
  • Some tests in the class would always fail due to unmet preconditions that cannot be satisfied within Portage. For example, the test.io.ReadWrite.testURL test case needs Internet access to establish a connection to http://kotlinlang.org[7], but with network-sandbox enabled in Portage's FEATURES by default, the connection cannot be established, hence the test case would always fail.
  • Some tests in the class are known to have trouble with passing such that even the upstream has disabled them from the Gradle build scripts.

To prevent a class from being passed to JUnit 4, add the class's name to the JAVA_TEST_EXCLUDES variable's value (which should be an array). The variable is declared by java-pkg-simple.eclass, but kotlin-libs.eclass honors it too.

References

  1. Yuan Liao. Introducing ebuilds That Build Kotlin Core Libraries from Source, Leo3418's Personal Site, July 5th, 2021. Retrieved on July 10th, 2021.
  2. 2.0 2.1 compiler/cli/bin/kotlinc-js, JetBrains/kotlin GitHub repository, January 7th, 2021. Retrieved on July 10th, 2021.
  3. compiler/cli/bin/kotlinc (Line 95), JetBrains/kotlin GitHub repository, February 21st, 2021. Retrieved on July 10th, 2021.
  4. compiler/cli/bin/kotlinc (Line 84), JetBrains/kotlin GitHub repository, February 21st, 2021. Retrieved on July 10th, 2021.
  5. libraries/stdlib/js-v1/build.gradle, JetBrains/kotlin GitHub repository, April 17th, 2021. Retrieved on July 10th, 2021.
  6. core/builtins/build.gradle.kts, JetBrains/kotlin GitHub repository, March 5th, 2021. Retrieved on July 10th, 2021.
  7. libraries/stdlib/jvm/test/io/ReadWrite.kt, JetBrains/kotlin GitHub repository, April 6th, 2021. Retrieved on July 15th, 2021.