Dogfooding the Eclipse Dash License Tool

There’s background information about this post in my previous post. I’ve been using the Eclipse Dash License Tool on itself.

$ mvn dependency:list | grep -Poh "\S+:(system|provided|compile)$" | java -jar licenses.jar -
Querying Eclipse Foundation for license data for 7 items.
Found 6 items.
Querying ClearlyDefined for license data for 1 items.
Found 1 items.
Vetted license information was found for all content. No further investigation is required.
$ _

Note that in this example, I’ve removed the paths to try and reduce at least some of the clutter. I also tend to add a filter to sort the dependencies and remove duplicates (| sort | uniq), but that’s not required here so I’ve left it out.

The message that “[v]etted license information was found for all content”, means that the tool figures that all of my project’s dependencies have been fully vetted and that I’m good to go. I could, for example, create a release with this content and be fully aligned with the Eclipse Foundation’s Intellectual Property Policy.

The tool is, however, only as good as the information that it’s provided with. Checking only the Maven build completely misses the third party content that was introduced by Jonah’s helpful contribution that helps us obtain dependency information from a yarn.lock file.

$ cd yarn
$ node index.js | java -jar licenses.jar -
Querying Eclipse Foundation for license data for 1 items.
Found 0 items.
Querying ClearlyDefined for license data for 1 items.
Rejected: https://clearlydefined.io/definitions/npm/npmjs/@yarnpkg/lockfile/1.1.0
Found 0 items.
License information could not automatically verified for the following content:

npm/npmjs/@yarnpkg/lockfile/1.1.0 (null)

Please create contribution questionnaires for this content.

$ _

So… oops. Missed one.

Note that the updates to the IP Policy include a change that allows project teams to leverage third-party content (that they believe to be license compatible) in their project code during development. All content must be vetted by the IP due diligence process before it may be leveraged by any release. So the project in its current state is completely onside, but the license of that identified bit of content needs to be resolved before it can be declared as proper release as defined by the Eclipse Foundation Development Process.

This actually demonstrates why I opted to create the tool as CLI that takes a flat list of dependencies as input: we use all sorts of different technologies, and I wanted to focus the tool on providing license information for arbitrary lists of dependencies.

I’m sure that Denis will be able to rewrite my bash one-liner in seven keystrokes, but here’s how I’ve combined the two so that I can get complete picture with a “single” command:

$ { mvn dependency:list | grep -Poh "\S+:(system|provided|compile)$" ; cd yarn && node index.js; } | java -jar licenses.jar -
Querying Eclipse Foundation for license data for 8 items.
Found 6 items.
Querying ClearlyDefined for license data for 2 items.
Rejected: https://clearlydefined.io/definitions/npm/npmjs/@yarnpkg/lockfile/1.1.0
Found 1 items.
License information could not automatically verified for the following content:

npm/npmjs/@yarnpkg/lockfile/1.1.0 (null)

Please create contribution questionnaires for this content.
$ _

I have some work to do before I can release. I’ll need to engage with the Eclipse Foundation’s IP Team to have that one bit of content vetted.

As a side effect, the tool generates a DEPENDENCIES file. The dependency file lists all of the dependencies provided in the input in ClearlyDefined coordinates along with license information, whether or not the content is approved for use or is restricted (meaning that further investigation is required), and the authority that determined the status.

maven/mavencentral/org.glassfish/jakarta.json/1.1.6, EPL-2.0 OR GPL-2.0 WITH Classpath-exception-2.0, approved, emo_ip_team
maven/mavencentral/commons-codec/commons-codec/1.11, Apache-2.0, approved, CQ15971
maven/mavencentral/org.apache.httpcomponents/httpcore/4.4.13, Apache-2.0, approved, CQ18704
maven/mavencentral/commons-cli/commons-cli/1.4, Apache-2.0, approved, CQ13132
maven/mavencentral/org.apache.httpcomponents/httpclient/4.5.12, Apache-2.0, approved, CQ18703
maven/mavencentral/commons-logging/commons-logging/1.2, Apache-2.0, approved, CQ10162
maven/mavencentral/org.apache.commons/commons-csv/1.8, Apache-2.0, approved, clearlydefined
npm/npmjs/@yarnpkg/lockfile/1.1.0, unknown, restricted, none

Most of the content was vetted by the Eclipse Foundation’s IP Team (the entries marked “CQ*” have corresponding entries in IPZilla), one was found in ClearlyDefined, and one requires further investigation.

The tool produces good results. But, as I stated earlier, it’s only as good as the input that it’s provided with and it only does what it is designed to do (it doesn’t, for example, distinguish between prerequisite dependencies and dependencies of “works with” dependencies; more on this later). The output of the tool is obviously a little rough and could benefit from the use of a proper configurable logging framework. There’s a handful of other open issues for your consideration.

Posted in EDP, Intellectual Property | Leave a comment

Updates to the Eclipse IP Due Diligence Process

In October 2019, The Eclipse Foundation’s Board of Directors approved an update to the IP Policy that introduces several significant changes in our IP due diligence process. I’ve just pushed out an update to the Intellectual Property section in the Eclipse Foundation Project Handbook.

I’ll apologize in advance that the updates are still a little rough and require some refinements. Like the rest of the handbook, we continually revise and rework the content based on your feedback.

Here’s a quick summary of the most significant changes.

License certification only for third-party content. This change removes the requirement to perform deep copyright, provenance and scanning of anomalies for third-party content unless it is being modified and/or if there are special considerations regarding the content. Instead, the focus for third-party content is on license compatibility only, which had previously been referred to as Type A due diligence.

Leverage other sources of license information for third-party content. With this change to license certification only for third-party content, we are able to leverage existing sources of information license information. That is, the requirement that the Eclipse IP Team personally review every bit of third-party content has been removed and we can now leverage other trusted sources.

ClearlyDefined is a trusted source of license information. We currently have two trusted sources of license information: The Eclipse Foundation’s IPZilla and ClearlyDefined. The IPZilla database has been painstakingly built over most of the lifespan of the Eclipse Foundation; it contains a vast wealth of deeply vetted information about many versions of many third-party libraries. ClearlyDefined is an OSI project that combines automated harvesting of software repositories and curation by trusted members of the community to produce a massive database of license (and other) information about content.

Piggyback CQs are no longer required. CQs had previously been used for tracking both the vetting process and the use of third-party content. With the changes, we are no longer required track the use of third-party content using CQs, so piggyback CQs are no longer necessary.

Parallel IP is used in all cases. Previously, our so-called Parallel IP process, the means by which project teams could leverage content during development while the IP Team completed their due diligence review was available only to projects in the incubation phase and only for content with specific conditions. This is no longer the case: full vetting is now always applied in parallel in all cases.

CQs are not required for third-party content in all cases. In the case of third-party content due diligence, CQs are now only used to track the vetting process.

CQs are no longer required before third-party content is introduced. Previously, the IP Policy required that all third-party content must be vetted by the Eclipse IP Team before it can be used by an Eclipse Project. The IP Policy updates turn this around. Eclipse project teams may now introduce new third-party content during a development cycle without first checking with the IP Team. That is, a project team may commit build scripts, code references, etc. to third-party content to their source code repository without first creating a CQ to request IP Team review and approval of the third-party content. At least during the development period between releases, the onus is on the project team to—​with reasonable confidence—​ensure any third-party content that they introduce is license compatible with the project’s license. Before any content may be included in any formal release the project team must engage in the due diligence process to validate that the third-party content licenses are compatible with the project license.

History may be retained when an existing project moves to the Eclipse Foundation. We had previously required that the commit history for a project moving to the Eclipse Foundation be squashed and that the initial contribution be the very first commit in the repository. This is no longer the case; existing projects are now encouraged (but not required) to retain their commit history. The initial contribution must still be provided to the IP Team via CQ as a snapshot of the HEAD state of the existing repository (if any).

The due diligence process for project content is unchanged.

If you notice anything that looks particularly wrong or troubling, please either open a bug report, or send a note to EMO.

Posted in Eclipse 101, Intellectual Property | Leave a comment

Revising the Eclipse IP Due Diligence Process for Third Party Content

The Eclipse Foundation’s board of directors approved an update to the Eclipse Foundation’s Intellectual Property (IP) Policy in October 2019. With help from some of our open source project teams and the Eclipse Architecture Council, we’ve been defining, refining, and rolling out updates to our corresponding IP due diligence process and supporting services. A big part of this update is a change in the way that we will manage third party content.

In the context of the Eclipse IP Policy, “third party content” is content that is leveraged by the Eclipse open source project, but not otherwise produced or managed by an Eclipse open source project. We use “content” in a general sense, independent of any particular technology or means of delivery; libraries, components, individual files, etc. are all considered content (and, content that comes from a third party provider is “third party content). A library produced by, say, an Apache open source project, is considered to be third party content.

Previously, the IP Policy required that all third party content must be vetted by the Eclipse IP Team before it can be used by an Eclipse Project. The IP Policy updates turn this around. Eclipse project teams may now introduce new third party content during a development cycle without first checking with the IP Team. That is, a project team may commit build scripts, code references, etc. to third party content to their source code repository without first creating a contribution questionnaire (CQ) to request IP Team review and approval of the third party content. At least during the development period between releases, the onus is on the project team to–with reasonable confidence–ensure any third party content that they introduce is license compatible with the project’s license. Before any content may be included in any formal release the project team must validate that the third party content licenses are compatible with the project license.

The “license compatible” part is important. The IP Policy change eliminates the notion of different types of IP due diligence for third party content. Henceforth, all third party content is subject to license certification only.

With this change to license certification only for third party content, we are able to leverage existing sources of information license information. The short version is that we’re getting out of the business of scanning through every single bit of source code ourselves, and will instead leverage what we’ve already learned and trust sources of license information (and contribute to these other sources of information).

We currently have two trusted sources of license information: The Eclipse Foundation’s IPZilla and ClearlyDefined. The IPZilla database has been painstakingly built over most of the lifespan of the Eclipse Foundation; it contains a vast wealth of deeply vetted information about many versions of many third party libraries. ClearlyDefined is an OSI project that combines automated harvesting of software repositories and curation by trusted members of the community to produce a massive database of license (and other) information about content. The Eclipse Foundation’s IP Team has been working closely with the ClearlyDefined team, providing input into their processes and helping to curate their data.

Both of these sources of information can be leveraged relatively easily to find license information on a single unit of third party content. But many (perhaps most) projects depend on long lists of third party content and identifying license information for a long list of dependencies can be cumbersome. Fortunately, some APIs are available that make it possible to query these data sources in bulk and we have developed a prototype License Tool that committers may use to query those APIs.

Before we can talk about leveraging APIs or using tools, we need to be able to identify the units of third party content. This is pretty natural for developers who are familiar with technologies like Apache Maven which has well defined means of uniquely identifying content in a well-defined software repository. Maven coordinates–which combine groupid, artifactid, and version (often referred to as a “GAV”)–identify a very specific bit of content (especially when combined with a specific source, e.g. Maven Central). For example, org.junit.jupiter:junit-jupiter:5.5.2 unambiguously identifies content on Maven Central that is known to be under the Eclipse Public License version 2.0 (EPL-2.0). Other systems use different means of identifying content: node.js, for example, uses a pURL-like identifier for content in the NPM repository (e.g., @babel/generator@7.62).

To bridge the gap between the different means of identifying content (and fill in some of the blanks), we’ve adopted the ClearlyDefined project’s five part identifiers which includes the type of content, its software repository source, its namespace and name, and version. ClearlyDefined coordinates are roughly analogous to Maven coordinates, but–by including the type and source–provide a greater degree of precision. ClearlyDefined uses slashes to separate the parts of the identifier, so the Maven GAV org.junit.jupiter:junit-jupiter:5.5.2 maps to maven/mavencentral/org.junit.jupiter/junit-jupiter/5.5.2 and the NPM identifier @babel/generator@7.62 maps to npm/npmjs/@babel/generator/7.6.2.

The License Tool was created with a command line interface that takes a list of dependencies as input and generates output that identifies the content that needs further scrutiny (i.e., it lists all content for which license information cannot be found in one of the approved sources). As a side effect, it also generates a DEPENDENCIES file (example) that lists all dependencies, along with their resolved licenses and some other information (this file will be used to augment intellectual property logs; more on this later). We created it with a command line interface so that the initial version of the tool could be used on a variety of technologies; at some point, it is our hope that–with some community help–we’ll be able to create, for example, a Maven plugin as well.

What the License Tool does is relatively simple:

  • It first sends the content list to IPZilla; if an entry has been vetted by the IP Team and has been approved for use, then it goes into the “approved” pile. If an entry has been vetted by the IP Team and flagged as somehow problematic, it will be put into the “restricted” pile.
  • It then sends the content that has not been flagged as either approved or restricted to ClearlyDefined; if an entry is known to ClearlyDefined and has a score of at least 75 and all discovered licenses are on our approved licenses list, then it goes into the “approved” pile.
  • Everything else goes into the “restricted” pile.

The License Tool CLI lists all content that ended up in the “restricted” pile as requiring further scrutiny. It is this list of content that requires further scrutiny that the project team must engage on with the Eclipse IP Team.

For many build technologies, it is relatively easy to generate lists of dependencies directly (e.g., the Maven Dependencies plugin). We’ve been working with Maven-, Gradle-, NPM-, and Yarn- based systems: there are some specific examples in the prototype tool’s README (I’ll walk through some specific examples in future posts). Project teams that use technologies that do not provide easily parsed/converted dependency lists can manually generate a list and feed that to the tool. Sorting out how we make this work for more technologies is on the to-do list.

It’s important to keep in mind that committers are ultimately responsible to ensure that all intellectual property is accounted for properly (e.g., if you have third party content buried in product content, you’ll likely need to account for that manually). If your build technology does not completely or accurately report the list of dependencies, then the committers need to account for that. Furthermore, it’s important to keep in mind that the License Tool is intended to assist committers in determining whether or not further scrutiny is required for their project dependencies; while our early work with it shows that it produces good results, those results should not be blindly accepted as authoritative (the tool is only as good as the input which which it is provided). If something doesn’t seem right, please open an issue so that we can address it.


Edit (May 29/2020): we’ve revised the acceptance criteria for content listed on ClearlyDefined to “a score of at least 75 and all discovered licenses are on our approved licenses list”

Posted in EDP, Intellectual Property | Leave a comment

Becoming an Eclipse Committer

Committers are the developers who hold the keys for an open source project. They are the ones who get to push code directly to source code repositories, configure build servers, push the output of builds to the download server, and so on. They are the ones who make decisions that set the path of their open source project.

Committers have another important role: turning contributors into committers. All committers should plan to spend some significant amount of their time interacting with contributors to whip their contributions into shape, working community forums to court contribution to the project, and otherwise evangelize their project.

The Eclipse Foundation considers having a growing list of committers from a diverse array of organizations and interests a important health metric for open source projects.

At the Eclipse Foundation, committer status is granted on a project-by-project basis. A developer who holds committer status on one Eclipse open source project, does not automatically hold any status on other Eclipse open source projects.

There are three ways to become a committer on an Eclipse Foundation open source project:

  • Get elected as a committer by the existing committers;
  • Be listed as an initial committer on a new project proposal; or
  • Be appointed as a participant representative for a specification project.

The most common way for somebody to join an Eclipse Foundation open source project as a committer is to be elected to an active project. The process starts with a developer establishing a record of making high quality contributions to the project (typically, but not necessarily, in the form of source code patches or code in some form). After an individual has, through contribution, demonstrated that they understand how the open source project works, and that they understand the Eclipse Development Process and are prepared to implement the Eclipse IP Policy, an existing committer will invite them to join the team and initiate an election. Committer elections start with a nomination by an existing committer that includes a statement of merit that usually takes the form of a list of the various contributions that the individual has made to the project. What constitutes a sufficient demonstration of merit varies by project team and team.

The second way to become a committer on an Eclipse open source project is to be listed as an initial committer on a new project proposal (naturally, this only makes sense if the project is still in the proposal phase). Since all new open source projects need to be bootstrapped with actual committers, the new open project creation process serves as the committer election. The community review period provides an opportunity for potential contributors to demonstrate merit and put their names forward as initial committers; it’s left to the discretion of the people who propose (proposers) the project to determine whether or not they’ll add a potential committer to project proposal (there’s no means by which anybody can force their way into committer status on a new Eclipse open source project without consent of the proposer). The creation review serves as the actual election, giving the Eclipse Foundation membership a final opportunity to challenge aspects of the new project proposal, including the initial committers.

It’s acceptable (perhaps not common, but certainly not uncommon) for the initial committers on a new Eclipse open source project to all work for the same employer. Again, the Eclipse Foundation expects that the project team will actively court contributors and committers from other organizations and interests as the project moves from the initial incubation phase into the mature phase.

Third way to become a committer is a special case: get appointed as a participant representative for a specification project operating under the purview of an Eclipse working group. A company that participates in a working group (at a particular membership level) that does not already have a committer representing their interests on a specification project may appoint one. The intent behind the notion of a participant representative is to ensure continuity of the flow of intellectual property grants when an participating organization is found to be without representation. Following their appointment, participant representatives become indistinguishable from other committers on a specification project. As such, following appointment, participant representative committers are subject to the same rules as any other committer: they do not hold any special status or formal role beyond that of committer once appointed, and they retain their committer status if they change employers.

Okay… so there’s actually four ways to become a committer. The Eclipse Development Process includes a provision that allows the project leadership–in the event that a project becomes dysfunctional (e.g., no active committers)–to add (or remove) committers to reinvigorate the project. This power is used rarely.

Your contributions are welcome.

Posted in EDP, Uncategorized | Leave a comment

Git Contribution Activity Charts for Eclipse Projects

Eclipse EGit: Git Integration for Eclipse

This post is brought to you today by the Eclipse EGit™ Project. Eclipse EGit is the Eclipse open source project that provides Git integration for the Eclipse Platform.

On Eclipse open source project pages (which we often refer to as the Project Management Interface or “PMI”), the “Who’s Involved” pages include some commit activity graphs.

The Contribution Activity chart shows the absolute number of commits made on project repositories over the last twelve months. Eclipse open source projects may have multiple Git repositories; this chart shows commit activity across all branches on all repositories owned by the project team.

The Individual Contribution Activity chart shows the commits attributed to individual developers over the last three months (across all repositories and branches). The identify of the individual committer comes directly from the author field in the Git commit. For project committers, we map author credentials to the information that committers provide in their Eclipse Foundation Account, so–for committers–the name shown here will match what’s in their account, not what’s on the commit record.

Note that any developer listed as an “Also-by” in the commit message will get credit for the commit (“Some Bodyelse”, from the example shown below, would share author credit for the commit). Because of the “Also-by” folks being counted as authors/contributors, some commits may be represented multiple times in the Individual Contribution Activity and Organization Contribution Activity charts (once each for each author).

If you click on a committer’s picture on the “Who’s Involved” page, you’ll see a chart of lifetime contributions to the project. If the chart is missing, that means that the person hasn’t authored any commits.

The Organization Contribution Activity chart shows relative contributions from Eclipse Foundation member companies. Commits made by all project committers affiliated with the company (across all project Git repositories and branches) are grouped together.

On this chart, “Other Committer” groups together contributions from committers who do not work for a member company, and “Contributor” refers to activity by contributors (i.e., developers who are not yet not committers).

Active Member Companies are those Eclipse Foundation member companies that have at least on committer that has made at least one commit on the project in the last three months (note that the order is random).

These charts take a very narrow view of “contribution”. There are many ways to contribute to an open source project. You can answer questions on forums or mailing lists, open and comment on issues, present at conferences, …

Posted in EDP, Project Management, Uncategorized | Leave a comment

Specification Project Committer Agreements

We identified a hole in our committer agreement process that excluded individuals with a certain employment status from participating in Eclipse Foundation open source specification projects operating under the Eclipse Foundation Specification Process (EFSP).

I’ll start by saying that you don’t need to be a committer to contribute to an open source specification project (in fact, notwithstanding the initial bootstrapping of a new open source project team, you cannot become a committer without first demonstrating merit as a contributor). If you just want to make a contribution in the form (for example) of a chunk of text in a specification document or a new method in an API, you can sign the Eclipse Contributor Agreement, create a pull request (the column in grey below) for review by a committer (the pull request can only be merged by a committer), and you’re off to the races!

After making a habit of contributing high quality contributions, an existing project committer may (I might argue should or must) nominate you to be a committer via election.

Specification Project Agreement Selection Flow

Elections generally run for a full week (they will end immediately when all existing project committers vote in the affirmative). Following a successful election, you will be put into our committer agreement workflow: the Eclipse Foundation’s system will send you, the newly elected committer, an email with the instructions you need to provide us with agreements we need to instate you as a committer (we refer to this as the “committer paperwork” process).

If you work for an Eclipse Foundation member company that is a participant in the working group, then we already have all of the agreements that we need and you’re good-to-go.

If you’re self employed, unemployed, or a student, we need you to sign the individual working group participation agreement (IWGPA). This agreement embeds the Individual Committer Agreement and Membership Agreement that are required to participate in a specification project. Once you’ve signed the IWGPA and have provided it to the EMO Records team, they’ll set you up as a committer and you’re good-to-go.

If you work for an Eclipse Foundation member company but that company is not a participant in the working group, or you work for a company that is not a member of the Eclipse Foundation at all, then you need to sign the individual working group participation agreement (IWGPA), and your employer needs to sign the Employer Consent Agreement for Eclipse Foundation Specification Projects (which we shorten to “Employer Consent Agreement” or just “Consent Agreement). This is the part highlighted in red in the above Specification Project Agreement Selection Flow chart.

Our committer provisioning process is automated. As a new committer, you will—following your successful election—be contacted by the EMO Records team by email to engage in our agreement workflow which guides you through signing those agreements that you need. These agreements are all available for your review on our Legal Resources page. The working group-specific agreements are on our Explore our Working Groups page.

Note that in order to get committer rights on a project hosted on GitHub, you need to set your GitHub Id in your Eclipse Foundation account.

The Employer Consent Agreement is the new bit. Due to the differences in the way that intellectual property flows are managed in a specification project (vs. a software project), we need this employer agreement. This agreement must be signed on behalf of the company, so—if you need one of these—you have to identify somebody within your organization with the authority to sign on behalf of the organisation (your manager will likely be able to help you with this).

So…

  • Step 1: Sign the ECA;
  • Step 2: Establish a pattern of contributing high-quality pull requests to an Eclipse open source specification project
  • Step 3: Get voted in to join the project by the existing committers;
  • Step 4: Complete the necessary working group participation agreements; and
  • Step 5: Take your place in history.

Posted in EDP, EFSP, Intellectual Property | Leave a comment

Organizing Your Eclipse Foundation Open Source Project Team

The Eclipse Foundation Development Process (AKA, the Eclipse Development Process or EDP) says nothing about how teams should organize. 

The EDP defines a committer role: committers are those developers who have the ability to make decisions for the project (e.g., push commits to project Git repositories and configure build servers). We often say that the committers are the ones with the real power: they’re the ones that hold all of the keys to all of the project resources. With great power comes great responsibility, so we have a well-defined process by which a contributor may demonstrate to the existing project committers that they are worthy of joining them as a peer.

The EDP also defines a project lead role: project leads are responsible for ensuring that the processes are being followed. So, for example, project leads need to ensure that committers are engaging in the IP due diligence process. But they are also responsible for ensuring that committers are “playing nice”. That is, they need to ensure that the level playing field and vendor neutrality bits in the EDP are being observed, but the EDP does not explicitly grant project leads any special powers to make technical decisions on behalf of the project team.

As the first link in the project leadership chain, one important power granted to project leads by the EDP is the ability to retire committers who are working against the interests of the team. This is essentially the nuclear option (per Merriam-Webster, “an extreme option regarded as a drastic step or last resort”) and it should be used with care.

Other than defining committers and project leads, the EDP says nothing about how teams organize. We expect that project teams sort this out for themselves.

While it is natural to have one or more committers become technical leaders in the project team, there is no formal technical lead role defined in the EDP. Likewise, while it is natural for certain members of a specification project team to become leaders in the specification process, there is no formal notion of specification lead (or “spec lead”) defined in the Eclipse Foundation Specification Process (EFSP). Any de facto technical or specification lead does not have any special authority beyond that which is granted to them by the other project committers. 

So the committers in a project team have some say over who their leaders are and what powers they grant to those leaders. While the role is not specifically defined for this sort of thing, the project lead role could be granted decision making powers. It is completely reasonable for a project team to decide, for example, that somebody in the project lead role must approve of all commits and anybody with that role can mitigate potential rogue actions by rolling back commits. One of the key benefits of organizing in this manner is that project lead is an elected position, so project committers have a build-in process for capturing the decision to grant those extra powers.

So how does all of this work?

For many open source projects, the committers follow the natural leaders, and that’s the end of it. But when a more formal relationship is desired, it must be arrived at by consensus of the project team (via public channels) and documented clearly so that everybody can understand how the project team works (making it very clear how a project team works is a critically important aspect of growing diversity in your project team). If the project team decides, for example, that all committers must contribute their updates as pull requests that may only be merged by a project lead, that must be documented (it’s fairly common for project teams to require that pull requests from one committer be merged by a different committer). When the role is formally defined, it’s also important to document how a committer ascends to into that role (to be clear, in the spirit of vendor neutrality, this criteria cannot be based on employment status).

Note that any rules that a project team decides to adopt don’t have to apply homogeneously across the entire project; a specification project team could decide, for example, that all contributions of text to a specification document must be approved by project lead before they can be merged, but that all committers can just merge their contributions to an API.

While I have your attention… I’d like to also highlight that the Eclipse Development Process says nothing about day-to-day development within a project. It is the project team’s responsibility to decide things like development methodology, merit criteria for new committers and project leads, etc.. 

Posted in EDP, EFSP | Tagged , , | Leave a comment

Yearly Release Reviews for Eclipse Projects

One of the key roles of he Eclipse Architecture Council is to maintain the Eclipse Development Process (EDP). Maintenance usually takes the form of an update every year or so. Updates to the EDP are approved by the Eclipse Board of Directors. The Architecture Council records issues that need to be addressed and tracks the work in the Community/Architecture component in the Eclipse Foundation’s Bugzilla instance.

The update to the EDP in late 2018 changed the way that we engage in release reviews. Project teams only need to engage in a release review for releases that occur more than one year after their last successful release review. For most project teams (i.e., active projects making regular releases), this means that they only need to engage in a release review once every year.

Project teams still need to create a release record, but do not have to engage with the Eclipse Management Organization (EMO) or their Project Management Committee (PMC), and don’t need to submit their IP Log for review. Project leads must still take care to ensure that the intellectual property included in all releases has been fully vetted by the IP due diligence process (in practical terms, this means that all CQs for content included in a release must be resolved, and the license compatibility of all third party content must be established before making the release official).

Eclipse open source project teams who aren’t sure whether or not they’re ready to release, can check with the EMO or can submit their IP Log for a sanity check.

Posted in EDP | Leave a comment

Eclipse Committer and Contributor Paperwork

The Eclipse Foundation has several agreements that we use to ensure that contributors understand the terms by which they make their contributions and, especially, to give them an opportunity to assert that they have the necessary rights to make those contributions under the terms of the corresponding project license.

The Eclipse Contributor Agreement (ECA) must be signed by anybody who wants to contribute to an Eclipse open source software or specification project. When a contributor has signed the ECA, a committer will be able to merge a contributor’s pull requests. Another way of looking at it is that a committer cannot merge a pull request when the contributor has not signed the ECA.

The Eclipse Individual Committer Agreement (ICA) is signed by folks who do not work for an Eclipse Foundation member company when they become a committer. For those developers who do work for an Eclipse Foundation member company, the company representative can complete an Eclipse Member Committer and Contributor Agreement (MCCA) which covers all of that member company’s employees.

Acquiring committer status starts with an election started by an Eclipse project team. Upon successful completion of that election, the nominee is prompted to provide the necessary paperwork. Completing that paperwork is the last step in the process, granting a developer the ability to commit (i.e., push their own content or merge pull requests from others). When a new committer signs the ICA (or their employer signs the MCCA) they also sign the ECA, and so gains the ability to contribute (but not directly push or merge) to any other Eclipse open source software or specification project.

When a developer is elected as a committer to another Eclipse project, they will not have to complete any additional paperwork. Committer status is specific to a particular Eclipse project; paperwork is not project specific.

Note that contributors retain ownership of their contributions; the ECA, for example, states in part (highlighting is mine):

This ECA, and the license(s) associated with the particular Eclipse Foundation projects You are contributing to, provides a license to Your Contributions to the Eclipse Foundation and downstream consumers, but You still own Your Contributions, and except for the licenses provided for in this ECA, You reserve all right, title and interest in Your Contributions. 

That is, the contributor owns their contributions, but grants a license to the Eclipse Foundation and downstream consumers to use them under the terms of the project license.

Posted in EDP, Intellectual Property | Leave a comment

Eclipse IP Policy: Reviewing Third Party Content

I’ll start this discussion with some background…

Under the original Eclipse Foundation Intellectual Property (IP) Policy, every bit of third party content needed to be thoroughly reviewed before it could be used by an Eclipse project. And the reviews were thorough: license scan, provenance check, scan for anomalies, … Reviews of third party content literally took days, weeks, and months. All of that review needed to be complete before the project team could commit any code that made any reference to that third party content.

As you might imagine, the time required to engage in all of that analysis was somewhat inconvenient for project teams, so we introduced the notion of Parallel IP. The idea behind the introduction of Parallel IP was that the IP Team could perform a cursory check of the content and grant checkin, thereby authorizing the project team to commit code into their repository that leverages the content, while the IP Team engaged in their more thorough review in parallel. The project team needed to wait until that thorough review was complete before engaging in a release. Initially, Parallel IP was available only to Eclipse projects in the Incubation Phase; it was later extended to Eclipse projects in the Mature Phase, but only for new versions of third party content that had already been reviewed and approved.

Parallel IP made the process better. Committers only had to wait for a day or two before they could leverage new third party content. It worked well for those Eclipse projects that engaged in annual releases, but as more and more Eclipse projects increased their release frequency, the time required to complete reviews of third party content (and the lead time required to engage with the IP due diligence process ahead of a release) became a problem (it’s likely more accurate to say that the existing problem became more acute).

To accommodate those projects that needed to move quickly, we introduced the notion of license check only IP due diligence and gave Eclipse project teams the ability decide what sort of IP due diligence they’d like to engage. We helpfully labeled the new type of IP due diligence as Type A (license check only) and the classic thorough IP due diligence as Type B (license check, provenance check, and anomolies scan). We introduced some automation that leveraged open source tools to scan and automatically approve third party content submitted for Type A review. Based on the rate of adoption, Type A was very successful.

It’s worth pointing out that our Type A, even though it is less thorough than our Type B IP due diligence, it is still far more than any other open source organization does. In fact, our Type A provides a far more thorough review of third party content than most organizations engage in.

While the introduction of Type A made the IP due diligence process flow faster for Eclipse project teams, it didn’t address the underlying problem: that the process required that every single bit of third party content must be reviewed before it can be used in any capacity.

The October 2019 updates that we introduced to the Eclipse Foundation’s IP Policy changes an important definition that lets us turn the process around. The definition of the term Distributed Content had previously forced us to implement a process that required the review of all third party content before any use or reference to it could be committed to an Eclipse project’s source code repository. By narrowing the definition of Distributed Content to refer only to content that is included in a release, Eclipse project teams may now push commits that reference third party content without first checking with the IP Team during a development cycle. It’s only when it comes time to release that we need to certify that the third party content included in and referenced by the release is license compatible.

This change shifts some of the onus onto the project team. Before pushing a commit that leverages third party content, a committer will need to (at least informally) check to see if the license on that content is compatible with the project license. We’re not expecting that project committers do any sort of deep analysis, only that they review the licensing terms on the content (if the third party content’s license in on the Eclipse Foundation’s Approved Licenses list, then you’re probably okay). If there’s any question, or a committer feels that shenanigans might be afoot, they can engage with the Eclipse Foundation’s IP Team for help. Primarily, committers need to provide some scrutiny to the license of the content that they adopt to avoid surprises when it comes time to certify compliance ahead of a release.

With no requirement to review content in advance, there is no requirement to engage with the Eclipse IP Team via contribution questionnaires (CQs) for every single piece of third party content. Instead, we can leverage the vast database of intellectual property metadata that we’ve assembled over the years to validate an entire dependency list as a unit (I posted about this a couple of weeks ago). For those readers who have been part of our community for a while, this means that there is no longer any requirement to create piggyback CQs. Further, we’re leveraging other sources of intellectual property metadata (e.g., ClearlyDefined), which means that there is generally no longer any requirement to create CQs of any kind. In practice, we will continue to use CQs to engage the Eclipse IP Team to research and vet content for which information is not already available, or to investigate content when we detect shenanigans (we will also continue to use CQs to track project code and third party content that includes cryptography).

Our intent is to make as many of the changes work as possible using the process and infrastructure that we currently have in place. In 2020, we will start researching and evaluating new tools; our hope is that we will be able to implement our updated process using existing open source tools. In the meantime, we have some tools that we’ve been using internally to validate license compliance of third party content and are working on making these tools available in a form they can be leveraged by committers (we’ve just started capturing requirements on, and will track progress using Bug 553016).

We’ve only just gotten approval for the policy changes and have only just started implementing process changes. We appreciate your patience and we work to make all of this happen. We’re tracking our progress on Bug 552967.

Posted in EDP, Intellectual Property | Leave a comment