When you’re evaluating OpenJDK distributions, you have many factors to consider. One of the most fundamental is whether the distribution has gone through TCK testing.
TCK testing is a way of verifying that the distribution conforms to the Java SE specification, as defined by the JCP through individual JSRs (Java Specification Requests). This blog post explores what TCK compliance means (and doesn’t mean) and why you should care.
What is a TCK Test?
As noted above, the TCK is a way of verifying that the distribution you’re using conforms to the Java SE specification.
The TCK contains more than 100,000 individual tests, depending on Java version. For instance, Java 8 includes 126,000 tests, and Java 11 has around 139,000. The TCK contains a comprehensive set of test cases that each covers various aspects of Java specifications including APIs, libraries, language features, and other important components. For example, the TCK ensures the binary build of OpenJDK meets all the specifications of the individual JSRs for a given version of Java SE (e.g. Java 17).
For distribution providers, TCK compliance establishes technology credibility and ecosystem support with partners and customers. It enables them to guarantee interoperability, promote standardization, obtain commercial use certification, and meet legal or business requirements. TCK testing allows a provider to call an implementation “Java” or “Java-SE compliant.” Using an OpenJDK distribution which has passed the TCK tests provides third-party patent grants for the end user.
Do TCK tests verify performance?
TCK testing does NOT do performance testing (throughput, startup, warmup, latencies, footprint, etc.). It does not mean that all distributions are equal. Much like black box application testing, the TCK tests functionality (the JDK does something) but does not test how it does it. TCK testing ensures product quality via cross-platform testing to conform to Java standards (Java Language, Java Virtual Machine, and Java APIs) for all Java configurations supported by the distribution provider.
For more context, Azul Platform Prime is TCK tested, but the tests do not show that it delivers lower latency and higher throughput than regular OpenJDK (which it does). An application will exhibit the same functional behavior on both but will perform differently. The TCK test results will not reveal the difference in performance.
What is the threshold for passing TCK tests?
Every binary out of more than 1,100 that Azul updates and distributes passes all TCK tests. If a binary doesn’t pass the TCK, we don’t publish it or call it GA (ready for General Availability) quality.
A distribution must pass ALL the TCK tests to prove it conforms to the Java SE specification. So either it does or it does not. There is no “partial” passing or compliance with the Java SE specification.
How can you as an end user verify that a JDK/JVM distribution actually passed TCK testing?
There is no formal way. Individual providers license the TCK test suite from Oracle and can run it on their distributions. Ask the provider to confirm that they have run (and passed) all the TCK tests.
NOTE: Azul happily provides certificates for each binary that we distribute that confirms it passed all TCK tests relevant for a given Java SE version.
How long is TCK compliance valid?
Every distribution passes TCK only once before it’s released. Whenever a distribution changes anything, it must re-run and pass all TCK tests. For a given quarterly release which includes over 1,100 binaries that Azul provides, that’s a lot of TCK testing needed!
How long does TCK testing take?
It varies from several hours to several days, but on average it takes about a day to pass the full TCK cycle. It depends on what hardware the distribution provider uses to run the tests.
There are some manual tests. For example, printing must be verified as well as things like recording and playback of sound. There is no code analysis, just running the tests on the distribution’s binary because – as discussed earlier – a distribution passes or it doesn’t.
TCK tests are updated to reflect changes and ensure ongoing compliance testing, and keeping current with the latest TCK standards and specifications can be challenging. Triple-check that the JDK/JVM distribution you are using is always using and passing the latest TCK tests.
Does every distribution go through TCK testing?
Oracle provides the TCK for free under the Oracle TCK Community License Agreement (OCTLA). The TCK tests can only be used for OpenJDK distributions or those “substantially” derived from OpenJDK. Not every OpenJDK distribution undergoes TCK testing, and it’s important to know this when you choose a distribution.
What role does Azul play in TCK compliance?
Azul actively participates in the Java Community Process (JCP), the organization responsible for developing and maintaining Java standards. Each Azul JDK binary is verified compliant with Java SE specifications using the OpenJDK community TCK. This involves executing the TCK test suite against our JDKs in a controlled and reproducible test environment.
The TCK tests help identify any deviations or non-compliant behavior in a given JDK so we can make necessary adjustments to ensure compatibility. Running the TCK tests as part of the automated build and deployment process helps catch issues early and ensures that each code change is properly validated. This validation also minimizes the risk of compatibility issues, enhances software quality, and increases customer satisfaction.
Conclusion
All Java runtimes (JDKs/JVMs) have to behave the same way so applications produce the same results, no matter which OpenJDK you are using. To achieve this, every distribution has to comply with the Java SE specification as defined by the relevant Java Specification Request (JSR) through the Java Community Process (JCP). To verify this, distributions must pass all the tests of the Java Technology Compatibility Kit (TCK).
This doesn’t mean all distributions are implemented the same way, although most are based on OpenJDK. Azul provides two distributions that are TCK-compliant but very different:
- Azul Zulu Builds of OpenJDK (aka Zulu) are almost identical to vanilla OpenJDK, and it is the basis for our Azul Platform Core product.
- Azul Zing Builds of OpenJDK (aka Prime) are based on OpenJDK, with many significant enhancements, and is the basis for our Azul Platform Prime product
Whether you’re using vanilla OpenJDK or another distribution, we strongly encourage you to use only TCK-compliant distributions to ensure stability and compatibility for your Java applications and infrastructure.
You can find more information at these links:
https://openjdk.org/groups/conformance/JckAccess/
https://openjdk.org/groups/conformance/JckAccess/jck-access.html
Java Migration Made Simple
OpenJDK is the key. Visit the Azul Migration Learning Hub.
