Automatic License Certification By The Numbers

In 2016, we introduced the notion of license certification intellectual property (IP) due diligence (“Type A”) into the Eclipse IP Policy with a goal in mind to automate the certification process. At that time, we started a process of evaluating tools that could be used for automatic validation and eventually discovered Scancode.

In our testing, we found that Scancode does a very good job of discovering licenses (and copyrights, which we may take better advantage of later), with the added bonus of having a very simple-to-use command-line interface (CLI) that produces reports in a handful of handy formats (including JSON and SPDX) that are easily machine readable.

The Eclipse Webmaster integrated Scancode into the Eclipse Genie process. Genie jumps into action when it notices a “Type A” third party content review request (“Contribution Questionnaire”) with attached source code, and PMC approval. If the licenses discovered in the attached content are compatible with the project license, Genie automatically license certifies the content.

After running for a few months, it looks like we’re getting about a 50% hit rate. That is, about half of all third party content license certification requests are being automatically approved without needing to engage the Eclipse IP Team.

typa

Note that some of the older bumps in the graph come from requests that were retroactively designated as “Type A”, and that little bump in automatic approvals in March is from Webmaster testing.

I’ll admit that it’s a little suspect to me that the graphs line up so closely starting in July. I’m pretty confident in the query and have reviewed the data, so it’s accurate as far as I know.

It’ll be interesting to see how the pattern tracks over time.

Advertisements
Posted in Eclipse 101, EDP, Intellectual Property | Leave a comment

Legal Documentation Requirements for Eclipse Projects

Late last week, I pushed out an update to our documentation regarding the legal documentation requirements for Eclipse projects that Sharon Corbett and I have been working on over the past quarter. In the process, we moved the guidelines off of the main website and rolled them in to the Eclipse Project Handbook. Our primary goal in revising this documentation was to make it more generally applicable to all open source projects and bring us more in line with what the rest of the open source world does.

Update probably isn’t the right word. It’s more accurate to say that we’ve pushed out some general guidelines and that the old/existing documentation becomes instructions for how to apply those general guidelines in the case where your project builds Eclipse Platform Plug-ins and Features.  In this regard, the existing documentation was very much concerned with the technical aspects of how to provide legal documentation in a manner that they can be presented to the user via the “About Eclipse” dialog (there’s no particular value in changing how all that works).

That is, if you’re building Eclipse Platform Plug-ins and Features under the Eclipse Public License (EPL) 1.0, we’re not asking you to change anything. 

I’m going to repeat that. Don’t panic. If you’re building Eclipse Platform Plug-ins and Features, just keep doing what you’re doing. If you’re planning to update to the Eclipse Public License 2.0 (EPL-2.0), we’ll need to talk (more on this later).

The gist of it is that in the general case, project teams are required to include license and notices in both their project repositories and in the output form of their project code (generally the result of compiling/building, e.g. JAR files). For consistency across our own projects and the greater community, we’re recommending that these files be named LICENSE and NOTICE (with optional extensions). Variations based on technical limitations are acceptable. The license and notice files should be in plain text. The use of a favourite markup language is fine, but not required.

There’s a lot more information in the document, so I’ll avoid trying to repeat it here. One thing that you’ll notice is that we’ve tightened up the copyright header text a bit by removing the “All rights reserved”. There’s no need to change any headers that include this statement: it’s not necessarily wrong to include the statement, it just doesn’t really add any value.

You’ll notice that there’s an appendix in the document for projects that are building Eclipse Platform Plug-ins and Features under the EPL-2.0. There’s a few rough edges here that we still need to work out (there a link to an issue in Bugzilla if you want to contribute your thoughts). One thing that we discovered while working on these changes is that basically nobody in the open source community has anything like our Software User Agreement (SUA) which is effectively an End User License Agreement (EULA). Since a project that updates to the EPL-2.0 will have to update all of their legal documentation, we figured that we use the opportunity to drop the SUA and instead just use the license text in places that call for a license (AFAICT most third party plug-in providers do this).

To make getting the legal documentation right a little easier for project teams, I’ve started working on an experimental script that uses the information in the Eclipse Foundation database to generate at least some or all of the required documentation.

https://www.eclipse.org/projects/tools/about.php?id=%5BProjectId%5D

Take note that this script is experimental and may not be entirely correct (maybe consider using this as a template or a starting point; ultimately it’s up to the project teams to make sure that the notice file is correct). If you do notice something that seems wrong with the data (e.g. wrong license or trademark statement), send a quick note to emo@eclipse.org; if it looks like the rendering is wrong or that there is some technical issue, please open a bug against Community/Process.

EclipseCon Europe 2017

Posted in Eclipse 101, EDP, Intellectual Property | Leave a comment

Running Eclipse IDE on Java 9

Out of the box, Java 9 makes only a subset of the modules available to applications. If your application makes use of APIs from other modules, you need to configure the JVM to make those modules available. The Eclipse IDE its itself a Java application that uses APIs from modules outside of that default group, so a little extra configuration is needed to make it run on Java 9.

In your Eclipse IDE installation directory, you’ll find the eclipse.ini file that your Eclipse IDE uses for configuration information. Add --add-modules=ALL-SYSTEM after the -vmargs argument (basically anything that follows the -vmargs argument is passed to the VM at start up (note that most VMs will barf if you pass in arguments that they don’t know or expect; adding this option when running on a Java 8 JVM will result only in disappointment and frustration).

...
-vm
/path/to/jdk/bin
...
-vmargs
--add-modules=ALL-SYSTEM
...

There’s more information in the Eclipse wiki.

Note that we’re pushing out some updates to make it so that the modification is not required. But that will only apply on future versions of Eclipse IDE (e.g. the Oxygen point releases, Photon, and later). For all older versions of Eclipse IDE, you’ll need to make this configuration change.

If you want to learn more about Java 9 support in the Eclipse IDE, including the Java development tools (JDT) and other great features like Eclipse Web Tools, Eclipse Code Recommenders and Eclipse Maven Integration, etc., consider attending EclipseCon Europe in Ludwigsburg, Germany from October 24 – 26, 2017.

There’s an entire Java and JDT track that includes the following:

There will also be BoFs, along with all sorts of opportunities to network and learn.

EclipseCon Europe 2017

If you want to learn more about the great features available in the Eclipse IDE, follow @EclipseJavaIDE on Twitter (follow me while you’re at it).

Posted in Community, Conferences, Eclipse 101, Java | 1 Comment

Java 9 Support for Eclipse IDE, Oxygen Edition

Java 9 has been released into the world and support to make your Eclipse IDE ready to build, run, and debug Java 9 applications is available from the Eclipse Marketplace. You can install this either from within the Marketplace Client in the Eclipse IDE itself (Help > Eclipse Marketplace) or by dragging and dropping from the website.

Or, just drag and drop this install button onto your running Eclipse.

Drag to your running Eclipse* workspace. *Requires Eclipse Marketplace Client

Installing the Java 9 support will also add the JUnit 5 support as well!

The Java 9 support includes:

  • the ability to add JRE and JDK 9 as installed JRE;
  • support for JavaSE-9 execution environment;
  • the ability to create Java and Plug-in projects that use a JRE or JDK 9; and
  • the ability to compile modules that are part of a Java project

Checkout https://wiki.eclipse.org/Java9/Examples for some working examples of Java 9.

We’ll be including both Java 9 and JUnit 5 support in our package downloads and the installer soon!

Posted in Announcements, Java | Leave a comment

Eclipse Projects: Level Playing Field

For many open source organisations, open means the same thing as transparent: open as in open book. At the Eclipse Foundation, we regard being transparent as the practice of making sure that the community can see and understand what the project is doing; and being open as the act of giving up absolute control and welcoming the community to participate as an equal player on a level playing field (i.e. being open to participation by the others).

Screenshot from 2017-07-31 23-22-35

Not really a field, but this is the closest thing that I have to a picture of field-based sporting event. Ice is about as level as you can get.

At the Eclipse Foundation, we take the open part of open source very seriously. It’s codified in the Open Source Rules of Engagement found in the Eclipse Development Process.

Everybody needs to play by the same set of rules. A level playing field doesn’t necessarily mean that a project team needs to accept every single contribution that comes their way. Rather, it means that the project team needs to have a set of rules by which everybody participates; and these rules can’t include things like for whom the contributor works.

Contribution rules can require that contributions fall within the project’s scope and current release plan. The rules can require that all code contributions be accompanied by unit tests and documentation; or that contributions implement a solution for an issue that’s been discussed by the project team in their issue tracker. Some sort of quality bar is a reasonable part of any set of contribution rules.

For most open source projects, these contribution rules aren’t formally captured. However, most of the rules that I’ve listed so far collectively form a pretty reasonable default set of participation rules. A quality bar is (obviously) hard to quantify, but for many project teams it’s enough that any committer feels that the contribution should be accepted (some projects require that two committers sign off on a contribution before it can be accepted).

Note that it’s also perfectly reasonable for a project team to require that significant contributions come with a promise of continued investment in the form of the contributor becoming a member of the project team.

Regardless of the rules that define the level playing field for any particular project, any content destined for the project’s code base should have some public record of contribution. Otherwise, the project would be operating (at least in part) hidden from community involvement and so counter to the open source rules of engagement. That public record can take the form of a Gerrit review, GitHub pull request, or (if you’re in a pinch) an attachment on a Bugzilla or GitHub Issue record.

Regardless of how a contribution is presented, the contributor must be listed as the author in the Git commit record and must complete the Eclipse Contributor Agreement before any contribution can be accepted.

The best way to get involved with an open source project is connect with the project team. All Eclipse project repositories should have a contribution guide in the root of every Git repository with this contact information and more. You can also search for project information on the Eclipse Projects website.

Posted in Eclipse 101, EDP, Project Management | Leave a comment

Follow @EclipseJavaIDE

If you’re new to the Eclipse IDE you’re already a big fan, you’ll find something of value every day by following @EclipseJavaIDE.  Frankly, there’s so much good stuff coming out of this account, that it’s hard to pick any favourites. So, I’ve pulled out a couple of recent ones.

Menu entries exist to help you complete incomplete thoughts and—of course—CTRL+1 is always there for Quick Fix or Quick Assist; but sometimes you just don’t want to take your hands off the keyboard.

Automating tedious operations like generating fields, getters and setters, etc. is something that the Eclipse IDE does well. Converting anonymous classes into lambda expressions? Easy.

Follow @EclipseJavaIDE to get your daily Eclipse IDE tips and tricks.

Download the Eclipse IDE, Oxygen Edition today.

EclipseCon Europe 2017

Posted in Eclipse 101, Java | Leave a comment

License Certification (Mostly) Just Happens

The Eclipse Intellectual Property Policy defines two types of intellectual property (IP) due diligence for third party content. The so-called Type A Due Diligence is concerned exclusively with license certification; and Type B Due Diligence is concerned with license certification, provenance checking and a deep dive scan of the content for various anomalies.

Regarding the analysis of Type A third party content, the IP Policy makes this statement:

It will be the responsibility of the Eclipse Project to run and analyze the results of a scan tool provided by the EMO, using parameters provided by the EMO, to obtain the terms and conditions under which such Content would be distributed by the Eclipse Foundation, and ensure that such terms are consistent with the Project Licenses. The Eclipse Project will certify that the terms and conditions of its Non-Eclipse Content conform to the then-current licensing guidelines as provided by the EMO. No further approvals will be required from the EMO prior to the Eclipse Project placing the Non-Eclipse Content into the Repository.

In short, it is the project team’s responsibility to run a tool, analyse the results, and certify the content as being consistent with the various conditions. It took us a while to identify the scan tool.

The Eclipse IP Team has been using open source Fossology for a while. While Fossology is a very comprehensive tool that’s great for IP Analysts, our experience suggests that the learning curve is too steep for it to be used by committers. What we wanted to provide was a tool that could generate a simple report (both human and machine readable) containing a manifest and corresponding licensing.

We found what we needed in ScanCode, which is produced by some old friends at NexB. ScanCode has the ability to generate the manifest and summarize findings in a number of different formats, including HTML, JSON, and SPDX RDF and Tag/Value. Using ScanCode is pretty easy. In fact, it’s so easy that the Webmaster integrated its use into an Eclipse Genie script. So… for a project team to run the tool, all they really need to do is create a CQ. The rest just happens automatically.

license-certification-workflow

In cases where the third party content has a single white listed license, committers only need to create the CQ and then add the content to their builds. If Eclipse IP Team review is required, the committer may need to participate in that review.

To leverage this automatic scan, a project committer creates a Type A contribution questionnaire (CQ) in the usual way and attaches the corresponding source code. The magic happens after the PMC gives their approval: the Eclipse Genie process identifies every Type A third party content CQs that has been approved by the PMC, runs ScanCode on the source code attachments, and then attaches the report directly to the CQ.

If a single license is identified for all files in the third party content, and that license is on our white list (see below), then the CQ is automatically marked license_certified, its license information is updated, and the CQ is marked resolved. If multiple licenses, blacklisted licenses, or otherwise problematic licenses are detected (i.e. anything other a single white listed license), then the CQ is sent to the Eclipse IP Team for further investigation.

When you see something like the following on your CQ, you’re good-to-go.

Screenshot from 2017-07-18 15-19-46

At this point, the content can be used in project builds and included in milestone builds. Once all project CQs are either marked as license_certified (type A) or approved (type B), the project can do an official release.

I’m really curious to see what sort of hit rate we get on automatic license certification, vs. how many requests will have to be reviewed by the Eclipse IP Team. My hope is that we will get to a point where we have an 80% automatic approval rate; but we don’t have enough data to make a call yet.

Our current implementation has the following licenses in the white list:

  • Apache License 2.0
  • Apache License 1.0
  • Apache License 1.1
  • BSD 2 Clause
  • BSD 3 Clause
  • BSD 4 Clause
  • Eclipse Public License 1.0
  • Eclipse Distribution License 1.0
  • MIT License
  • ISC License
  • NTP License
  • OpenSSL License
  • Public Domain
  • SIL OPEN FONT LICENSE
  • W3C Software and Notice License
  • zlib license

We need to find a home for this. This list will grow.

We’re using Bug 496959 to track our work to update our processes and documentation regarding the Eclipse IP Process.

Posted in Eclipse 101, Intellectual Property, Uncategorized | Leave a comment