Student Application Period Starts on Monday April 22/2013

Students, start your engines…

The Google Summer of Code (GSoC) student application process started earlier today, April 22/2013.

You must apply to participate in the programme through Google’s Summer of Code 2013 website.

We have provided an application template that should be available to you through Google’s site. Please use that template as a starting point, but feel free to add whatever kind of information you believe will help us better understand your proposal and your ability to complete the tasks you describe.

Please note that you need to describe a specific set of tasks that you intend to complete during the summer term. Be specific. This isn’t a job application. Do not create a proposal that proclaims nothing more than “I’d love to work with the Eclipse community”. Such proposals are a waste of our mentors’ time; to avoid wasting our mentors’ time, I will just summarily reject any such proposals.

Your proposal needs to present a good project idea that adds value to the Eclipse community. You also need to demonstrate to us that you will be able to accomplish the tasks you describe. The onus is on you, the student, to convince the mentors that your proposal has merit, and is worth the considerable time and effort it will take to mentor you.

There will be time to update and revise your proposal after you submit it. Please do not submit an empty placeholder proposal. Do try with your initial submission to describe your complete project idea. We will let you know if you need to provide more detail. We will help you sort out how to identify an appropriate Eclipse project for your work.

If you have any questions, please ask on our soc-dev mailing list.

I look forward to reading your proposals.

Eclipse committers: let me know if you are able to help us review proposals and maybe mentor a student.

Eclipse at GitHub

Almost all Eclipse Git repositories are mirrored at GitHub. The mirrors were initially set up two years ago by the nice folks at GitHub with relatively little input from us. I provided a simple script–which lists all of our repositories in very simple CSV format–that was used to create the original set of mirrors and is occasionally used to update the mirrors. That is, every once-in-a-while, I ask the good people at GitHub to run their script which invokes my script to create any newly-added repositories to the list of mirrors. It’s a service that GitHub provides; it’s not supported by the Eclipse Foundation’s Webmaster team.

The script that generates the list of Git repositories at eclipse.org uses information gathered by Dash. Eclipse committers don’t have to do anything to make this happen: if you create a Git repository, it gets mirrored (eventually) on GitHub. It will also be automatically mirrored on Google Source.

We do draw attention to the mirrors. If a mirror exists for a Git repository, the project information pages will provide a link. Similar links are provided for Google Source mirrors.

There are some problems with the mirrors.

Mirroring repositories was something that GitHub set up in the early days, but it isn’t a first class service and the mirrors aren’t fully supported. The repositories themselves seem to stay up-to-date (within a few minutes), but the corresponding web sites on GitHub tend to be days and weeks behind (it’s not clear when/how they update). Unfortunately, this leaves an impression that the project is inactive. This is a pretty big problem because we see the GitHub mirrors as an opportunity to encourage additional contribution; with out-of-date web pages, I fear that the opposite is happening.

GitHub has no current plans to fix this problem. This presents us with an opportunity to rethink how GitHub mirrors are done.

The preferred means of making this happen, as far as I know, is to create a repository on GitHub and then push the contents of the eclipse.org repository into it. Unfortunately, there is currently no automated means of doing this. As far as I know, GitHub provides no mechanism to schedule a regular pull (correct me if I’m wrong), which means that it would become our responsibility to push to the GitHub clone.

This is where I’m going to need a little help. I’m pretty sure that it’s possible to create a git hook that can be installed into the git.eclipse.org Git repository that will automatically push to a remote repository on commit. Perhaps it makes sense to have the hook push to all remotes known to the repository, not just GitHub. Having this sort of hook would give projects the power to decide what they want to do; they would have control over whether or not the repository is automatically replicated and where.

Please add your comments on Bug 402183.

No seriously: We love Git. Seriously.

The cornerstone, I think, of a good April Fools Day joke is plausibility. The more reasonable or possible the topic, the more likely it is that you’ll fool somebody. I’m not sure what it says about the Eclipse Foundation that so many people really thought that we’d made a decision to move away from Git, but I am pretty proud that I caught a least a few people with today’s announcement.

In the aftermath, though, I should set a few things straight.

CVS is gone. Well, mostly gone. It is true that we’ve granted a open-ended stay of execution for Orbit on CVS, but all other CVS use has been completed turned off.

We’re very serious about Git. We are also very serious about Gerrit. This combination of technology provides a powerful means of attracting contribution that just can’t be touched by CVS or SVN.

Our entire committer base seems very happy with Git. The comments about confusing workflows, non-fast-forward merges, and missing commits are actually based in reality. But these are just growing pains that a few of our committers and projects have worked through. Real complaints are few and those that we do get are addressed pretty quickly. A great deal of credit goes to the EGit project for providing tools, as well as leadership and assistance in our migration to Git.

The chart is totally bogus. The Dash data tells a very different story:

This chart shows the number of commits that have different values in the “Author” and “Committer” fields. This chart does not provide an entirely accurate quantification of contribution (it excludes all contributions received via Gerrit, for example), but I do believe that the trend it displays is correct. I do have the necessary data to generate a more accurate chart, but–since Dash wasn’t designed with Git in mind–I don’t currently have it in a form that I can easily query. When this changes, I’ll publish a more accurate chart.

SVN support will end. Over the past 18 months, we have said “no” to projects requesting new SVN repositories. SVN support at Eclipse is effectively deprecated. We have not yet set an end-of-life date for SVN, but one will be set soon. Projects currently using SVN should start thinking about migration to Git.

The Great Git Experiment

While many projects have claimed success with Git, a great many projects and developers continue to struggle with its adoption. In the fall of 2012, several projects chose to terminate rather than migrate their CVS repositories to Git. We have infrequent reports of commits being erased and general confusion regarding the workflow. While EGit does a good job of smoothing out the rough corners, it’s still the case that just writing code to a Git repository takes too many steps. I hear many complaints about committers forgetting to push their commits or having them lost in non-fast-forward merges.

This undercurrent of dissatisfaction came to a head last week at EclipseCon. I was cornered by several committers who were quite angry at having been forced to adopt Git and demanded that I allow them to roll back to CVS. The general consensus was that while at face value, Git appears to be better suited for encouraging contribution, the confusing nature of its workflow was actually having an opposite effect.

Data collected by Dash confirms it: many projects are actually losing committers and contributors because of Git. Non-committer contributions to projects has steadily dropped over the last twelve months. Projects that switched to Git early in the migration process did on average experience a short spike in contributions in the last few months of 2011, but in the general case, the contribution curve drops almost linearly from that period.

Faced with the backlash and drop in contribution, we have no choice. After careful consideration, and extensive discussions with the Eclipse Webmaster, we have decided that we have no choice but to migrate back to CVS. Given resource constraints–the webmaster team cannot support three version control systems–support for Git has been deprecated effective today. The Git server will be shutdown on December 21/2013.

Nerdvana is a Data Centre

Earlier today, we started a pretty bold move from one data centre to a brand-spanking new one across the city. I say bold, because we had to take ourselves completely offline while we disconnected more than thirty bits of hardware, shipped it across the city, and then reassembled it all.

I didn’t take very many shots from old data center; just this one of Webmaster Matt behind a bunch of cables.

Denis coordinated the whole thing from an impressively comprehensive plan assembled over weeks of careful analysis, multiple mental walk-throughs, and–no doubt–many sleepless nights. Here’s a shot of Denis piecing some of our systems together in the sparkly-clean new data centre.

Chris was a big help. In this shot, Chris is installing the the back half of a rail in for one of the servers. I was able to take the picture because I did all the front screws which are way easier and less time consuming than the ones in the back.

I managed to take this next shot in a rare moment of inactivity from Mike (seriously, the guy’s a machine) who had stepped back for a minute to give Denis some room to connect a few systems into the switch. We actually managed a very high level of parallel activity despite the tight space. I credit Denis’ plan and the well-laid out documents and diagrams which allowed us to get done what he needed without bombarding him with questions.

By early afternoon, it was all coming together nicely. I think it was at this point that Denis said something to the effect of “our backends are up”. This made me giggle. I guess that deep down we never get any older than twelve.

The data centre is still relatively new and so there’s still some empty space (which was good because being able to spread the servers around a bit made things a lot easier). I was tempted to use the space to demonstrate some of my sweet Karate moves. But everybody was busy doing actual work and wouldn’t have been able to fully enjoy the experience.

Our other Chris–the guy who writes the cheques–joined in as well. In this shot, our two Chrises are assembling various bits of hardware while Denis wrestles the systems to life. As as aside, I quite like the portable workstands we used; they’re easy to wheel around, they’re very stable, and they’re height-adjustable to boot. I need to get one of these for home.

By late afternoon, we’d installed every single rail and shelf, and every bit of hardware was in its place. When I left, Mike and Chris were following Denis’ cabling plan while Denis and Matt bent the operating systems to their will. The speed and confidence that Matt and Denis demonstrate while elbows-deep in Linux configuration files is awesome. And humbling.

It was a great day. Frankly, I love playing with hardware. But I had to leave. Minor hockey needed me…

Denis reported late this evening that all was progressing according to plan. Many of our servers are up and running again and the rest should be taken care of tomorrow. I’m not 100% happy that the Foundation’s email server is back up, though…

Rolling out the new Project Management Infrastructure

I’ll admit that a really big part of me wanted to see who would notice.

Last week, I relatively quietly changed how we represent project metadata by switching parts of the Eclipse projects website and the corresponding elements in the developer portal over to the new Project Management Infrastructure (PMI). If you click on the “About this Project” link for any project, or click the “All Projects” link on the Eclipse Projects page, you’ll be taken over to the new stuff.

Projects are the central feature of the PMI. Here’s an example:

You’ll notice some structural similarities to the old “Project Info|Summary” pages.

The main content of the page shows various bits of information, including the licenses used by the project, releases, reviews, and various graphs. I’m particularly fond of the “Contribute to this project” section which provides a bunch of handy information for prospective contributors, including links to the project’s repositories, and Bugzilla coordinates. I’m planning to add in links to information regarding builds and such as well.

The left navigation area shows an overview of the project. In this case, the shows the list of simultaneous releases the project has participated in, along with some handy links to information. Among the links is one that will display a page of project developers, one that will display the full set of releases, and some convenience links to display the current and next releases. Each release gets its own page (but I’ll take more about that in a follow up post).

A couple of committers noticed this week that the “Manage project metadata” link disappeared from the developer portal. The edit functionality for the data projects collect has been moved into the PMI.

Any registered user can log into the PMI. If you’re a committer or project lead the a project, an “Edit” button (link) will appear. If you click that, an editor will open and you can change information about the project.

You may notice that we provide fields to collect a few more bits of data than we currently display. We don’t, for example, actually use the “Build URL” field yet; but that’s coming soon. You’ll also notice that some of the fields provide autocompletion help (the mailing list fields in particular).

Note that you can use wildcards when specifying source repositories. For example, specifying ‘/gitroot/cdt/*’ will result in all Git repositories found nested under /gitroot/cdt/ will appear in the “Contributing to this project:” section.

A project lead or committer can now directly update the description of the project. The description was taken from the value previously provided via a file containing some HTML in your project’s web directory; projects no longer need that file. You may also notice that a project’s scope is now explicit. The scope was taken from corresponding section in each project’s proposal (when linkage to the proposal could be determined). Both of these values can be edited. Take care when editing the scope that you don’t change the meaning. Changing the text is fine; changing the meaning of the scope requires community review.

The migration over to this new infrastructure will occur in stages. In this first stage, we’ve migrated projects, releases, reviews, and simultaneous releases. For committer management (electing and retiring committers) and creating CQs, you’ll need to keep using the developer portal.

I expect that a few things (including documentation) will break during the migration, but we’ll fix those breaks as they occur. When you notice that something isn’t working quite right, or it’s not clear how it’s supposed to work, let me know. Opening a bug against Community/Project Management & Portal is the preferred means of communicating the problem. If that doesn’t work for you, send a note to emo@eclipse.org.

Accepting Committers with Code

Most of the open source projects that I’m familiar with don’t tend to take significant contributions unless they come with development resources. Taking on significant new functionality can be pretty daunting for development teams that are already stretched.

But how do you on-board new developer (committers) while observing the open source rules of engagement (especially the part about meritocracy)?

Let me back up a bit. Before we even get to the point of wanting to bring on some new developers, we need to consider the contribution. Any decision to accept a significant contribution into an open source project should follow a certain amount of transparent due diligence. The actual amount varies based on many factors including the nature of the contribution, composition of the development team, source of the contribution, etc. The important point here is that some amount of due diligence must occur before a decision to accept a significant contribution of code is made.

During the due diligence, existing project developers get a feel for how well the contribution integrates with the existing code base, and the sort of quality that they can expect from the contribution’s developer(s). The due diligence process also provides a good opportunity for the contributors to learn about (i.e. experience) at least part of the Eclipse Development Process (EDP).

By the end of this process, a development team should have a good feel for whether or not a contribution’s developers are a good fit on the team. A development team may opt to accept the code, but not the developers; they may just decide that the contribution is not a good fit; or they may decide that the contribution’s developers are not a good fit and throw out the baby with the bathwater (ie. reject the contribution). Ultimately, it’s up to the project’s existing developers to decide what to do.

At Eclipse, we have a very well-defined process for electing developers (committers) onto a project. They must first be nominated and then all existing project developers have to vote in the resulting committer election.

The nomination can happen immediately following the point when the decision is made to accept the contribution and add contribution’s developers to the project. Since the due diligence process has occurred transparently, the contribution itself–along with the related conversation–can be cited as the merit for the nomination.

In parallel with the committer election, existing project developers can start the intellectual property (IP) due diligence process on the contribution.

Here’s what the process looks like for an Eclipse project:

1. A contributor provides a code contribution (either through Gerrit or Bugzilla);
2. Existing project committers review and transparently discuss the contribution (using a public forum like the project mailing list or Bugzilla);
3. Existing project committers decide to accept the contribution and the contributors as new committers;
4. An existing project committer nominates the contributor(s) for committer status on the project (citing the contribution as the nomination criteria);
5. An existing project committer opens a contribution questionnaire (CQ) and attaches the contribution for review by the IP team;
6. The election runs for at most one week (or until all existing project committers vote);
7. The contribution works through the IP due diligence process

For projects that can leverage the parallel IP process (i.e. incubating projects), the IP due diligence process will generally grant a project permission to “check in” the code in about the same amount of time it takes to run the election. For projects that cannot leverage the parallel IP process, it’s a pretty good bet that the committer elections will conclude well in advance of the code contribution being cleared for “check in”.