Maybe it’s just me. But “Fetching children of Juno” made me laugh. Out loud (Sharon already thinks I’m nuts). Perhaps this is because I’ve been looking forward to reading the latest book in the “Percy Jackson” series, “Order The Heroes of Olympus: The Son of Neptune” with my son. Or perhaps it’s just a hint of latent sixth-grade Roman history knowledge. Or maybe I’m spending too much time working with the various projects at Eclipse that are participating in this year’s simultaneous release.

There are a great many children of Juno at Eclipse. At present, we have seventy children projects in the list. That’s seventy projects who are working together to produce an impressive stack of software.

How I Learned to Stop Worrying and Love the Build

With Hudson driving builds from the top; Git, Gerrit, Maven, and Tycho in the middle; and Mylyn controlling the pieces from the developer’s desktop, The Eclipse Foundation provides an impressive stack of technologies for building software. All this great technology combined with governance, intellectual property management, architectural guidance, and coordination via the simultaneous release, combine to deliver an Application Lifecycle Management (ALM) solution that is the envy of other open source projects and IT departments around the globe. In this session, we’ll present all the pieces and show how they work together; we’ll talk about evolution of the our processes and infrastructure, and muse about how they’ll grow in the future.

It really should amaze me that we’ve evolved a powerful common build infrastructure at Eclipse. But it doesn’t. The Eclipse community working in conjunction with our world-class IT team regularly does amazing things. It’s easy to get lost in it all and take it for granted.

The Eclipse Common Build Infrastructure is something that we’ve collectively been evolving for years. In this talk, you’ll learn about the current state of the technology and the processes that drive it. You can use what you learn to plan your own ALM stack and strategy, or implement your eclipse.org project build.

I hear the speakers are pretty good. That Beaton guy’s a little funny looking, though.

Vendor neutrality is a tricky thing to balance. Organizations that contribute to open source deserve some recognition, but if you cross the line from recognition to vendor bias, you may end up turning away potential contributors and their contributions.

Splashing a few logos on a project page is a great way to provide recognition. Committer email addresses can very often also provide some amount of recognition. The trick is to stay on the right side of the line that separates “interesting; company X contributes a lot to this project” and “company X obviously owns and runs this project”. It can be very difficult to manage perception.

Vendor neutrality is a requirement that’s baked right into the Eclipse Bylaws:

The Eclipse technology is a vendor-neutral, open development
platform supplying frameworks and exemplary, extensible tools (the “Eclipse Platform”).

It’s something that we take very seriously. It’s not something, however, for which we have very many hard-and-fast rules. Managing vendor neutrality tends to be a subjective matter that is highly-dependent on perception.

For example, it’s generally acceptable for a project to include a link or two on their download page to software that is hosted elsewhere. Some projects do this to provide more complete packages that contain the project code along with value-add software. There’s nothing wrong with doing this, provided that:

• the project code is accessible directly from the vendor-neutral eclipse.org download server;
• the eclipse.org-based downloads are the prominent choice; and
• there is a means for other organizations provide similar links to their project-related downloads.

Similarly, providing links to services related to a project can be helpful for the adopter community. Again, there’s nothing wrong with doing this, provided that the playing field is level. If a project chooses to provide links to services, it must be done in a vendor-neutral manner: other providers of similar project-related services must have a means of having their links included. It’s often enough to just document a reasonable set of conditions that must be met for another organization to list their services. Or you can just outsource the problem completely and include a link to services listed in the Eclipse Marketplace.

The Eclipse wiki provides further guidance concerning links to non-eclipse.org content.

Vendor neutrality is a big and important part of project diversity. If you are an Eclipse project, you have to care about project diversity. By extension, you have to care about vendor neutrality.

Eclipse is free and open source software that is distributed royalty-free under the terms of the Eclipse Software User Agreement. Please let me know if we can be of further assistance.

Now, the interesting thing for me, is the number of different product names people use to describe the thing that they want to buy. Here are a few of my favourites (in no particular order).

• Eclipse Foundation Eclipse Galileo SR1 3.5.1 EN 01
• IDE License JAVA programming: Open Source
• The “Eclipse” Software
• “For a Better World” (an Eclipse Publication)
• LTO Ultrium 5 1500 GB Data Cartridges
• Eclipse Indigo (3.7) Packages for Mac OS
• Eclipse + plug-in
• ECLIPSE IDE for Java EE Developers 1.2.0.20090621-0820 epp.package.jee for Win7
• ECLIPSE Practice Management Software/Medical Billing Software
• SCA dev lab Software and SCA tools

There’s a couple in there that seem to come out of nowhere, obviously from folks who have us confused with somebody else. Curiously, I get almost no email about soccer clubs or vampires.

Open – Eclipse is open to all; Eclipse provides the same opportunity to all. Everyone participates with the same rules; there are no rules to exclude any potential contributors which include, of course, direct competitors in the marketplace.
Transparent – Project discussions, minutes, deliberations, project plans, plans for new features, and other artifacts are open, public, and easily accessible.
Meritocracy – Eclipse is a meritocracy. The more you contribute the more responsibility you will earn. Leadership roles in Eclipse are also merit-based and earned by peer acclaim.

In more concise terms, transparency is about inviting participation; openness is about actually accepting it; and meritocracy is a means of limiting participation to those individuals who have demonstrated the desire and means to actually participate.

Transparency is one of those things that I believe most people understand in principle. Do everything in public: bug reports are public, along with all discussion; mailing lists are public; team meetings are public and minutes are captured (and disseminated). By operating transparently, the community around the project can gain insight into the direction that the project is moving and adjust their plans accordingly.

In practice, however, transparency is difficult to motivate in the absence of openness. What is the value to the community of discussing every little detail of an implementation in public? Does anybody really care? The fact of the matter is that a lot of people really don’t care. Most users of Eclipse are blissfully unaware that we even have bug tracking software and mailing lists. But some people do care, and transparency is a great way to hook those people who do are and get them to participate.

A lot of open source projects understand transparency. A lot, however, don’t understand openness. They’re not the same thing. To be “open” means that a project is “open” to participation. More than that, an “open” invites and actively courts participation. Participation in an open source project takes many forms. It starts by creating bug reports and providing patches, tests, and other bits of code. Overtime, contribution increases, and–eventually–some contributors become full-blown members of the project. Courting the community for contributors should be one of the first-class goals of every open source project.

But openness isn’t just about getting more help to implement your evil plans for world domination. It’s also about allowing participants to change your evil plans for world domination. Openness is about being open to new ideas, even–as the EDP states–if those new ideas come from your direct competitors in the marketplace. A truly open project actively courts diversity. Having different interests working together is generally good for the overall health of an open source project and the community that forms around it.

For the purposes of this graph, I have defined a project as active in any month if it had at least one commit in that month.

I’ve observed a couple of interesting things. First, total number of projects (top line) seems to increase at a pretty regular rate. There are two significant drops in the total number of projects that align with the retirement of projects in the DSDP Restructuring of 2010 and the great Technology Subproject Culls of 2010 and 2011.

The other thing that I’ve observed is that the number of active projects is relatively stable. I’m not sure exactly what to make of this information. It’s worth noting that the number of active committers (using the same definition of “active” is also stable through the same time period).

The coincidental drop offs in December 2011 are also interesting: one might conclude that we must have killed off some active projects. That’s not the case. None of the nine Technology subprojects that were terminated and archived in that period had a single commit in all of 2011. 186 Eclipse projects made at least one commit in 2011. Of the 186 Eclipse projects that made commits in 2011, 58 did not make commits in December. 20 Eclipse projects that made commits in November 2011 did not have any commits in December. There were 5,598 total commits in November, but only 4,691 in December. I have to conclude that it has something to do with Santa Claus. Or Pirates.

Andrew Robinson from Red Hat started things off with a demonstration of some Eclipse-based tools used to drive Koji, the software that builds RPM packages for the Fedora project.

Next up was Brian de Alwis, a committer on the Eclipse Platform UI project, who started with a short presentation describing CSS support in Eclipse 4.x and then wrapped things up with a demonstration of a new “CSS Spy” that should soon appear in the e4 incubator project.

Marcelo Paternostro, a long time committer and contributor on various Eclipse projects, and fellow martial artist, then stepped up to demonstrate the Oracle Enterprise Pack for Eclipse. He went quickly through that demonstration and ended up spending a lot more time discussing and demonstrating the Sapphire project. Sapphire is very cool; if you’re building SWT-based user interfaces, you really need to spend some time with this project.

Igor Feodorenko gave a demonstration of the m2e project’s Resolved Dependencies view. This view helps you review the full list of dependencies pulled in by a Maven build, and–most interestingly–makes it easy to drill into the dependency graph and sort out why a particular library is included. Igor wrapped up with a quick demonstration of their Onboarding software that’s recently been made open source on GitHub; this software is intended to hasten the process of bringing a new developer onto a project by making it easy provision a complete environment.

Peter Krogh of the EclipseLink project introduced multi-tenancy support by the project. As Peter described, with multi-tenancy, a single schema can be used to represent multiple virtual schemas.

I wrapped up the evening with a brilliant demonstration of the Orion project. There was a lot of interest and feedback. Perhaps the most interesting discussion (at least for me) was around whether or not we intended to move the Eclipse IDE into the browser. Multi-faceted questions like this are a bit of challenge. I used the opportunity to discuss how “he” is a part of “we”, and discussed how the community and committers–not the Eclipse Foundation–makes those sorts of decisions. Then I spent a few minutes musing over some ideas the collective “we” has had bouncing around. I’ll save a more thorough discussion of this for another time (or perhaps for the Orion blog).

The evening was capped off with a trivia contest (with questions provided by Kim Moir) for a beautiful gray Eclipse-branded golf shirt (as recently seen on the runways of Paris), drinks and snacks in the bar across the road, and the coolest cake I’ve ever seen (and I’ve watched 4½ minutes of Cake Boss).

The cake was yummy.

Thanks to Andrew for organizing, and Red Hat for co-sponsoring along with the Eclipse Foundation, a great event!

Moving forward, this is going to change.

I opened up Bug 363524 to track a feature I intend to add to the new project management infrastructure. With this feature, I intend to connect a project’s release records directly to the corresponding review, and plan. In the process, I think we’ll manage to reduce the overall burden on projects, and make the presentation of this information far more consistent.

In short, the release records–which today consist only of a date, name, and a small number of URLs–will be expanded to include more information. Information that can be used to generate the release review documentation and a project plan will be included directly in the release record. Rather than sending me a document in some arbitrary format via email and waiting for me upload and link the document, project members can themselves create and post the review materials directly. Rather than entering the same information multiple times, in different places, and formats, it’s all done in one place.

I envision projects entering basic information about their upcoming releases very early in the release cycle. Perhaps even entering multiple future releases. During the release cycle, the basic information will evolve, much like a plan is supposed to evolves today. By the end of the release cycle, while preparing those final bits, project members finalize the documentation and click a “submit for review” button. From there, the PMC reviews the provided information and either sends it back with comments, or approves it. From there it goes public on RSS, Twitter, Blogs, and where-ever else we choose to publish.

It’s also a pretty natural fit to have community comments about a review actually connected to the review itself, rather than directed into a sea of other posts on the project forum. Though we could probably do both if that makes sense…

Anyway, your comments on Bug 363524 are most welcome.

The decision was made based on several factors:

• The Eclipse Foundation’s IT infrastructure is PHP-based; our webmaster team are very effective at managing PHP-based solutions. We have several products running on Drupal today, including Eclipse Live and Eclipse Marketplace.
• First-class support for MySQL.
• We also have PHP and Drupal skills in-house: we have two full-time Drupal-savvy web developers and several other Eclipse Foundation staffers who are well-versed in PHP development.
• The Drupal platform provides a lot of out-of-the-box functionality, including rich text editing, revision management, and a very dynamic query framework.
• The Drupal community and eco-system are vast and diverse. There are hundreds of plug-ins available to handle things like authentication through LDAP, Twitter integration, workflows, and more
• Good development tools are available from the PHP Development Tools project.

We’re still in the early stages of development, so there’s not much to show. If you want to have a quick look, take a peek at projects.eclipse.org/.

Our initial work is concerned with representing projects. You’ll see useful things like a breadcrumb trail and a list of related projects in the left-nav (inspired by the automatically-generated project summary/info pages). If you log in, you’ll be able to edit the project information directly. Over the next couple of days, we’re working on implementing an LDAP-based authentication mechanism, and will then sort out authorization (so that only project committers can change project information).

The system is pre-populated with information taken from several sources including project metadata, descriptions provided via project websites, and other things like scope extracted from the project proposal (where possible). Expect updates on an ad hoc basis as new features are added. Note that we may occasionally have to rebuild our database, so any changes you make may be lost.

At this point, I do intend to open source our implementation, but not until we get a little further along in our implementation, and nail down a few software engineering issues. I’m not sure exactly how this will manifest. I’m pretty sure that it won’t be an Eclipse project due in part to licensing issues. That, and there are many aspects of being an Eclipse project that don’t apply here. More on that later…

We’re being a little ambitious with this new effort. We need to do more than what the current portal does. And we need to make it natural and easy to use. Further, I’ve decided to expand the target audience for the new implementation. It will be more than just a ‘developer portal’, it’s a reimagining of the entire Eclipse Project Management Infrastructure that provides a little something for everybody, including consumers of Eclipse technology.

As a first step, we’ve implemented a representation of projects. For consumers, this will manifest as something very much like the new-and-improved project summary pages (which I’ve been referring to as “project info pages”) for learning about projects and the relationships between projects. For authenticated committers, the page is a little different, featuring an “edit” button which will allow in-place editing of the various fields.

I’ll be showing off our modest beginnings at EclipseCon Europe 2011.

Depending on the type of user, and roles within a specific project, additional menus will appear to provide committer/lead elections, creation of releases, etc. The intent is to make it a one-stop shop for everything a community member needs to do.

In that vein, we’ll be rolling the project proposal process into the new infrastructure. A project proposal is a document that’s maintained by the system, taken through a well-defined process, and eventually turned into a project. Likewise, reviews will be documents managed by the system. One of my bigger frustrations is that too much of our information is buried in documents that are not easy to mine for information; by keeping this information in the system, we’ll be able to do clever things like reuse information from one review to the next.

I envision a simplification of the release review process. Instead of making releases and their corresponding review separate things, I intend to join them. A project member can create a “release” record for their project in the system. A release keeps track of basic information like the name/number, date, and type of release. Then we’ll tack on some other obvious information like a description of the release (what’s included), a link to a new and noteworthy document, screenshot images, a Bugzilla URL, link to the IP Log, etc. With this kind of information, the release record itself can serve as the review documentation and be taken through a workflow that doesn’t depend on checklists and email exchanges.

By connecting screenshots directly to a release record, we may be able to finally address Bug 214277 (by attaching screenshots to a specific release, we can generate a master screenshots page that is more-or-less guaranteed to be up-to-date). We can go one step further: if we capture the location and description of downloads, URL for update sites, and Eclipse Marketplace Client metadata, we can use that information to automatically generate a download page for each release.

I think is important is that we give projects the option to keep doing what they’re doing. I think the new project info pages (and the corresponding implementation in the new system) can serve well as a default project website. This will certainly make it much easier for new projects to get started. But projects that want to do something a little more custom will still be able to do so. The same holds for the automatically-generated downloads pages that are still currently in the dreaming stage.

It’s still very much early days on this effort and we’re not quite ready yet to show our results. While Nathan works furiously to show some early results, I’ve been working my way through the bug backlog against Portal. Over the next few days and weeks, I’ll be entering some new bugs to track the development of the various aspects of the new infrastructure (which we’ll use to solicit your direct input). Then, using the the early work as a guide, we should be able to get a good sense for development velocity and establish an actual plan and schedule. Stay tuned.

See you at EclipseCon Europe!