License Certification (Mostly) Just Happens

The Eclipse Intellectual Property Policy defines two types of intellectual property (IP) due diligence for third party content. The so-called Type A Due Diligence is concerned exclusively with license certification; and Type B Due Diligence is concerned with license certification, provenance checking and a deep dive scan of the content for various anomalies.

Regarding the analysis of Type A third party content, the IP Policy makes this statement:

It will be the responsibility of the Eclipse Project to run and analyze the results of a scan tool provided by the EMO, using parameters provided by the EMO, to obtain the terms and conditions under which such Content would be distributed by the Eclipse Foundation, and ensure that such terms are consistent with the Project Licenses. The Eclipse Project will certify that the terms and conditions of its Non-Eclipse Content conform to the then-current licensing guidelines as provided by the EMO. No further approvals will be required from the EMO prior to the Eclipse Project placing the Non-Eclipse Content into the Repository.

In short, it is the project team’s responsibility to run a tool, analyse the results, and certify the content as being consistent with the various conditions. It took us a while to identify the scan tool.

The Eclipse IP Team has been using open source Fossology for a while. While Fossology is a very comprehensive tool that’s great for IP Analysts, our experience suggests that the learning curve is too steep for it to be used by committers. What we wanted to provide was a tool that could generate a simple report (both human and machine readable) containing a manifest and corresponding licensing.

We found what we needed in ScanCode, which is produced by some old friends at NexB. ScanCode has the ability to generate the manifest and summarize findings in a number of different formats, including HTML, JSON, and SPDX RDF and Tag/Value. Using ScanCode is pretty easy. In fact, it’s so easy that the Webmaster integrated its use into an Eclipse Genie script. So… for a project team to run the tool, all they really need to do is create a CQ. The rest just happens automatically.

license-certification-workflow

In cases where the third party content has a single white listed license, committers only need to create the CQ and then add the content to their builds. If Eclipse IP Team review is required, the committer may need to participate in that review.

To leverage this automatic scan, a project committer creates a Type A contribution questionnaire (CQ) in the usual way and attaches the corresponding source code. The magic happens after the PMC gives their approval: the Eclipse Genie process identifies every Type A third party content CQs that has been approved by the PMC, runs ScanCode on the source code attachments, and then attaches the report directly to the CQ.

If a single license is identified for all files in the third party content, and that license is on our white list (see below), then the CQ is automatically marked license_certified, its license information is updated, and the CQ is marked resolved. If multiple licenses, blacklisted licenses, or otherwise problematic licenses are detected (i.e. anything other a single white listed license), then the CQ is sent to the Eclipse IP Team for further investigation.

When you see something like the following on your CQ, you’re good-to-go.

Screenshot from 2017-07-18 15-19-46

At this point, the content can be used in project builds and included in milestone builds. Once all project CQs are either marked as license_certified (type A) or approved (type B), the project can do an official release.

I’m really curious to see what sort of hit rate we get on automatic license certification, vs. how many requests will have to be reviewed by the Eclipse IP Team. My hope is that we will get to a point where we have an 80% automatic approval rate; but we don’t have enough data to make a call yet.

Our current implementation has the following licenses in the white list:

  • Apache License 2.0
  • Apache License 1.0
  • Apache License 1.1
  • BSD 2 Clause
  • BSD 3 Clause
  • BSD 4 Clause
  • Eclipse Public License 1.0
  • Eclipse Distribution License 1.0
  • MIT License
  • ISC License
  • NTP License
  • OpenSSL License
  • Public Domain
  • SIL OPEN FONT LICENSE
  • W3C Software and Notice License
  • zlib license

We need to find a home for this. This list will grow.

We’re using Bug 496959 to track our work to update our processes and documentation regarding the Eclipse IP Process.

Advertisements
Posted in Eclipse 101, Intellectual Property, Uncategorized | Leave a comment

Devoxx4Kids Ottawa June 2017

We all had such a great time at the Devoxx4Kids session in San Jose this past March that we’ve decided to try running a session here in Ottawa.

The goals and mission of Devoxx4Kids is to:

  1. Teach children Computer Programming while having fun and introduce them to concepts of robotics, electronics and generally being creative with these kind of devices.
  2. Inspire not only children but also the classical education system, so they too can start including computer science in their curriculum.
  3. Demystify programming for girls and introduce them to computer science in order to improve gender equality in that field.

The full manifesto is on the Devoxx4Kids website. There’s also all sorts of information about the programme, including links to the workshops.

20170319_130428

Don’t let this picture fool you. Plenty of young women attended the session in San Jose, but we were so caught up in the fun that we didn’t take all that many pictures…

For this first attempt, we’re going to keep it simple and run only two workshops (they ran eight in four parallel streams in San Jose). Since we’re new at this, we’re going to stick to coding workshops with a plan to branch out and maybe try some of the hardware workshops in a future session (these workshops require that we acquire some supplies and equipment that we don’t have readily at hand).

For this first run, the good people at Carleton University have offered up some space. Registration will open at 9:00 am on Saturday, June 3, 2017; we’ll be in room 5345 of the Herzberg Physics building.

5345 Herzberg Physics
Carleton University
1125 Colonel By Dr
Ottawa, ON K1S 5B6

The target age range for attendees is between ten and fourteen years of age (close counts). We’ll post more information, including how to register, on our event page. Registration includes lunch. We’re charging a modest fee of $30 to cover our expenses. Attendees will need to bring their own laptop computer to complete the exercises (we may be able to bring a few spares).

If you’re interested in helping to mentor the session, please send us a note at emo@eclipse.org. We’ll get the mentors together in late May to go through the exercises and make sure that everybody is ready to hit the ground running.

The first workshop will focus on a simple game written in Javascript and HTML using a game engine called Phaser. Participants are shown some basic JavaScript expressions and are then invited to use their new knowledge to modify the game. The beautiful thing about this exercise is that it requires virtually no set-up: the code is all self-contained, any text editor (including Notepad) can be used for modifications, and it all runs in a browser. Further, it can be run successfully without requiring an Internet connection.

The second workshop is concerned with Minecraft Modding using Forge for Minecraft and Eclipse IDE as the development environment. This workshop has a few more moving parts than the first and so will require a bit more effort to set up and most certainly does require a stable Internet connection to at least assemble the initial development environment via a Gradle build. There’s certainly a lot of opportunities in this workshop to explain all sorts of interesting concepts without getting bogged down in too many details (which will be good if we end up having attendees with prior experience).

Screenshot from 2017-05-12 15-36-17

We’ll send out setup instructions a week or so ahead of the session; we can hit the ground running faster if everybody has the software that we’re going to need already downloaded.

Posted in Announcements, Devoxx4Kids, Other | Leave a comment

Service Releases of Third Party Content in the Eclipse IP Due Diligence Process

Some time ago, the Eclipse Foundation’s Board of Directors passed the following resolution.

RESOLVED, that previously approved dependencies of Eclipse projects can be
reviewed and approved by the EMO as follows:
a) Service releases (e.g. x.y., bug fixes, security fixes) will require no review.
b) Minor revisions (e.g. x..) will require a reduced review by the EMO.
c) Major revisions (e.g. ..) will require a full review by the EMO.

At the time the resolution was passed, it was decided that project teams would still be required to register the use of service releases for third party content with the Eclipse Intellectual Property (IP) team (i.e. create a Contribution Questionnaire); it was decided that the resolution really only impacted how the IP Team would process the content: at least in theory, the IP Team would rubber stamp requests for service releases. I say in theory, because the IP Team would still eyeball these requests and sometimes invest some modest effort to ensure that significant new IP hadn’t been introduced.

We’ve reconsidered this position.

Project teams are not required to engage with the Eclipse IP Team to review service releases for third party content, provided that the service release is based on a previously-approved release. This assumes that the providers of the third party content regard service releases in the manner described by Semantic Versioning (i.e. a patch of backwards-compatible bug fixes) and that the service release does not include significant new IP.  Of course, if a project team isn’t sure, they can always just submit the content for review anyway.

We envision a scenario where project teams will take advantage of this for milestone builds, but then get full approval for final versions of their third party content. This isn’t required, but it is an option that’s available for project teams that feel that it’s necessary.

For major and minor releases, the manner in which the the project team interacts with the Eclipse IP Team does not change. Note that it is implied that the reduced review for minor releases follows an approved full review for a corresponding major release and that any service release follows an approved major or minor release.

For more information, to add your thoughts and concerns, or to track our progress with regard to updating our documentation, add yourself in copy to Bug 512465.

Posted in Intellectual Property | Leave a comment

Run Eclipse IDE on One Version of Java, but Target Another

The Eclipse IDE for Java™ Developers (and the other Java developer variants) is itself a Java application that’s used to build Java applications. That relationship can be a bit weird to wrap your brain around.

Written almost entirely in Java, the Eclipse IDE requires a Java Runtime Environment (JRE) to run. A JRE provides just the runtime platform: it doesn’t include the source code and Javadoc for any of the base Java libraries, or any of the development tools that are included in the Java Development Kit (JDK). An Eclipse IDE runs just fine on a JRE.

If you’re building Java applications, however, you really need to have access to a JDK. By default, an Eclipse IDE will configure itself to build applications against the JRE that it was launched on. If that JRE is part of a JDK, then you’ll get access to all the goodies that you need to get useful content assist, documentation, debugging support, etc. If the runtime platform is just a JRE, then a lot of that valuable goodness will be missing (but compiling still works because the Java development tools include the Eclipse Compiler for Java)

Here’s where it gets a bit weird. You can run an Eclipse IDE on a JRE from one version of Java and build applications that target one or more different versions of Java. You can, for example, run your Eclipse IDE on Java 8, but use it to build applications based on basically any earlier version of Java. You can select the default Java version for your workspace in the preferences (on the Java > Compiler page), or individually in the properties for each Java Project (preferred).

JDK Compliance settings in the Preferences (Java > Compiler)

In order to actually build applications on a different version of Java, you need to connect your Eclipse IDE with the corresponding JDK. To do this, first install the JDK, and then tell the Eclipse IDE where to find it via the Java > Installed JREs page in the workspace preferences. With additional JDKs installed, you can configure individual projects to use specific versions of the compiler and runtime.

All of this is a long way of saying that you can configure your Eclipse IDE, Oxygen Edition milestone build to run on a Java 9 JRE, (download a JDK from the JDK 9 Early Access site) but use it to build applications that target earlier versions of Java (i.e. keep doing your day job). Even doing just this and providing feedback will be very helpful to the Eclipse projects involved in the Eclipse IDE. If you actually want to build Java 9 applications, you’ll need to install the Java 9 Support (BETA) for Oxygen from the Eclipse Marketplace and provide feedback to the team.

Note that the JDT product in the Eclipse Foundation’s Bugzilla instance is specifically for reporting bugs that are directly related to the Java development tools. Use the EGit product to report issues with the Git integration, the m2e product to report issues with the Maven integration, the Buildship product to report issues with the Gradle integration, the Platform product for issues with regard to the basic IDE framework, the Web Tools product for issues with enterprise Java and web development, or the EPP (Eclipse Packaging Project) product if you’re not sure.

Many committers from the various Eclipse projects that contribute to the Eclipse IDE, including at least a couple of committers from the Java development tools project in particular, will be at Devoxx US. If you want to learn more about Java 9 support in the Eclipse IDE, you’ll be able to find them at the Eclipse Foundation’s booth (or we’ll be able to help you find them).

I’ll also be doing a demonstration of Test First Java Development using the Eclipse IDE at the booth. Come by and see if I can keep myself within the twenty minute limit…

If you want to learn more about the great features available in the Eclipse IDE, follow @EclipseJavaIDE on Twitter (follow me while you’re at it).

devoxx_black_transparent400

Posted in Community, Conferences, Eclipse 101, Java, Screenshots | Leave a comment

Eclipse IDE, Oxygen Edition; and Java 9

Update: Note that as of October 11/2017, Java 9 is 100% supported “out of the box” by Eclipse IDE, Oxygen Edition; Java 9 can be used to run your Eclipse IDE, Oxygen Edition, and can be used to build Java 9 applications without additional configuration. Download or update today.

I’ll admit that I dropped the ball for a while, but I’m back to testing the Eclipse IDE with Java 9. Java 9 includes some pretty major changes, especially in the way that it restricts access to internal code, and so there’s a real risk that Java-based applications are going to break. Just confirming that the Eclipse IDE runs on Java 9 is a critical task (even before we talk about leveraging actual Java 9 features).

Today, I downloaded the Java 9 Early Access Build 158 (March 1/2017) and reconfigured my working installation of the Eclipse IDE for RCP and RAP Developers, Oxygen Edition (M5) to use it. I’ve added a few extra features to my installation, including the PHP Development Tools, Data Tools SQL Designer, Mylyn WikiText, TM Terminal, and a handy regular expression helper/tester that I found in the Eclipse Marketplace.

Configuration is relatively straight forward. The nice thing about the early access builds is that you can just decompress them into a directory on your system without installing them (which will have a impact on your system’s configuration). The downside is that you have to configure your Eclipse IDE to find it. Actually, this isn’t really much of a downside in my opinion: I like to have some degree of control over what Java  runtime I’m using to run the Eclipse IDE rather than just grabbing the system default.

I decompressed the Java build and updated my eclipse.ini file to point point to it:

...
-vm
/home/apps/jigsaw-jdk-9-ea+158_linux-x64_bin/jdk-9/bin/
-vmargs
--add-modules=ALL-SYSTEM
...

Note that you need to add the --add-modules option to tell Java 9 to load some modules that it wouldn’t normally load by default. I’m hopeful that we’ll have an update to the launcher to automatically add this option in the M6 builds (see Bug 493761); with this launching the Eclipse IDE will just work if you happen to have a Java 9 JVM installed as your system default.

As a sanity check, I opened the About Eclipse dialog and reviewed the Configuration in the Installation Details to confirm that I’m actually running on Java 9. The next step is to actually start writing some code that leverages Java 9 features; for that I’ll need to install the Java 9 Support for Oxygen from the Eclipse Marketplace.

I’m looking forward to learning a lot more about Java 9 at Devoxx US, including these sessions:

See you at Devoxx!

 


Update (2017/09/25): I’ve made a small change to the eclipse.ini configuration to match what needs to be done for the release version of Java 9, and have updated the title and reference to the Eclipse Marketplace record.


devoxx_black_transparent400

Posted in Conferences, Java | 5 Comments

Language Server Protocol Talks at Eclipse Converge and Devoxx US

I’m particularly interested in learning more about Language Server Protocol at the combined Eclipse Converge and Devoxx US conferences at the end of March. We have a handful of projects doing work on the topic, and the opportunity to connect directly with the developers doing the work is too good of an opportunity to miss.

I’m hoping to fit the following talks into my schedule.

Language Server Protocol Explained

Sven Efftinge

The Language Server Protocol (LSP) introduced by Microsoft’s VSCode team has been a hot topic recently. In a nutshell it is an effort to unify how editors communicate with advanced language tooling.

In this session we want to give you an overview of what the LSP is, why it is so important and how you could leverage it. We will also explain what it does and what it doesn’t do, discuss some misconceptions and show some cool demos based on a Java implementation of the protocol.

A sneak peek into the Spring Boot language server

Martin Lippert

Based in the language server protocol and the architectural ideas behind it, big chunks of the Spring Boot tooling for Eclipse are currently re-implemented. This talk provides a sneak peek into the implementation and discusses the early experiences using this approach. We dive into the details and challenges how we implemented tooling for Spring Boot property files (pure properties + yaml), support for Cloud Foundry manifest files, and how we extracted and refactored existing Spring IDE code to run inside of a language server.

Building the JDT Language Server and how can you build one from your own feature

Gorkem Ercan

Eclipse JDT LS (Language Server) project aims to develop a Java Language Server that will make JDT features available to any client that supports the Language Server Protocol. This talk covers the changes, challenges and lessons learned while building the JDT LS. It will also provide insights to using and developing language servers and some tips to converting existing features. There will also be demos of JDT LS features on different editors and their comparision with the Eclipse JDT to give a better understanding of the possibilities and problems of language servers.

Roll your own Development Environment with Eclipse Orion

Bogdan Gheorghe

Eclipse Orion is a cloud IDE that supports JavaScript development out of the box.  Recently, support for LSP (Language Server Protocol) has been added to the mix.  Come and learn how to create three different development environments using Docker, Orion and LSP (Language Sever Protocol).

Introduction to Eclipse Che

Stévan Le Meur and Florent Benoit

Eclipse Che introduces a new kind of workspace that is composed of projects and runtimes. This approach improves agile workflow and allows fast bootstrapping of developers. Eclipse Che can run locally or in the cloud which allow to scale the resources on-demand and benefit from high performances and resources.

In this session, we will explain how to setup a workspace cloud in Eclipse Che, how to create the environments using Docker, configure the tools that you need and register a set of commands to be executed in that workspace. We will show you how you can benefits from the workspace portability to easily share it onto another Che or to the cloud.

This session will also cover how Eclipse Che is providing support and intellisense for all the languages and explain the work did on the Language Server Protocol. The Language Server Protocol is a communication protocol between a tool and a Language Server than run all code analysis and operations

Eclipse Converge devoxx_black_transparent400

Posted in Uncategorized | Leave a comment

Prerequisite Dependencies

All third party content must be taken through the Eclipse Foundation’s Intellectual Property (IP) Due Diligence Process before being used by an open source project hosted by the Eclipse Foundation. This includes all third party content that is incorporated into project code, included in builds, or otherwise required by the project code to provide functionality.

The Eclipse Foundation’s Guidelines for the Review of Third Party Dependencies defines various classifications of third party content and how project teams should engage in the IP process. The classification of third party content most commonly used is the so-called prerequisite dependency: a prerequisite dependency is third party content that the project either needs to distribute along with the project code, or otherwise have present in order to function.

Third party content is likely a prerequisite dependency if:

  • project code accesses APIs;
  • project code includes an import statement for a package from a library;
  • project code uses reflection or other means to reference a library’s APIs and implementation;
  • the Java/OSGi manifest for a project bundles makes a direct reference; or
  • project code invokes a command line tool to access the functionality.

This list is not intended to be exhaustive, but rather to provide common examples. Fundamentally, if project code makes some kind of use of third-party content, then that content is likely a prerequisite. The guidelines provide for so-called works with and exempt prerequisite dependencies, but I’ll save that discussion for a future post.

Project committers can engage in the IP Due Diligence process by creating a contribution questionnaire (CQ) in the Eclipse Foundation’s IPZilla system.

The term contribution questionnaire is a bit misleading when it comes to third party content. Third party content is not a really a contribution to the project, but since the requirements for tracking project code and third party content is very similar, the same mechanism is used for both.

The workflow looks a little something like this:

prereq

screenshot-from-2017-02-08-13-25-01

Committer Tools

There’s an entry point that project committers can use to Create a Contribution Questionnaire in the Committer Tools block that is on every project’s information page (see the image on the right).

In the first step, the committer creates the CQ record with essential information about the third party content and then attaches the source code for that content (this currently happens in two separate steps). The corresponding Project Management Committee (PMC) must sign-off on the CQ before any further processing occurs.

As a general rule, a CQ should be created for each separate library. The process permits for some flexibility with regard to the definition of library. Content that has source in a single source repository that is distributed as multiple JAR files can very likely be regarded as a single library. There has also been cases where the IP Team has accepted a single CQ to do license certification for a collection of many different JavaScript sources. If you’re not sure, check with the IP Team.

License scanning software will be engaged for third party content that’s to be reviewed for license certification (type A). As we roll out this new type of IP due diligence, the IP Team is engaging the tool manually, evaluating the output, and making a determination. Our plan is to automate the invocation of the tool and make a determination automatically where possible (e.g. in cases where the licenses are clearly indicated in the content) and have the IP Team investigate further when necessary.

For requests to perform the more thorough IP due diligence process (type B), the workflow is different. For content that qualifies for parallel IP processing, the IP Team will do a cursory review and—assuming that the content passes muster—grant check in, meaning that the content can be pushed into the project’s source code repository and included in builds. The full due diligence review includes verification of the content’s provenance and a scan for all kinds of other anomalies (more on this in future posts). This process is a lot work that can take quite a lot of time to complete, and very often requires interaction with the committer. When the IP team completes a full review with a successful outcome, they’ll mark the CQ as approved.

All third party content must be either license certified or approved before an project can issue an official release. Any release that includes license certified content must be marked as a type A. A type A release may include type B content; A type B release, however, cannot include type A content.

I’ve thrown out a few concepts in this post without providing explanations. I’ll try and fill in the gaps in future posts, which will all be grouped in the intellectual property category.

Posted in Intellectual Property | Leave a comment