Kepler by the Numbers

We originally had 72 projects sign up to participate in the Eclipse Juno simultaneous release on June 27/2012. By release day, that number had dropped to 71 with the removal of EMF Query 2 (which has since been terminated). If you scrutinize the table, you may notice that there are 72 entries; last year, the Eclipse project contributed two different releases (3.8 and 4.2). My count correctly includes these both as a single project.

This year, as we enter into the final stretch for Kepler, the number is again 72. This time around, however, I’m quite confident that the number will stay at 72.

The composition this year is a little different. Most of the projects from last year’s Juno release have stayed on as part of Kepler. But a handful of projects have decided to drop out of the simultaneous release. Virgo, Jetty, and the Runtime Packaging Project (RTP) have all decided not to participate. The Xtend project merged into Xtext, so it too drops off the list (the bits from Xtend are still part of the simultaneous release as part of Xtext’s contribution).

While Jetty is not formally participating in the simultaneous release, some of the Jetty project’s bits are in the Kepler software repository: they have been pulled in as dependency for one or more projects that are participating in Kepler. For similar reasons, there are bits from the Gemini JPA project in the Kepler software repository. If you find this a bit confusing, you’re in good company.

Five projects joined: EMF Diff/Merge, Sphinx, Stardust, Hudson, and Maven Integration for Web Tools Platform all join the Kepler Simultaneous Release.

Some of the projects include subprojects. The Mylyn project contribution to Kepler, for example, is listed as a single entry, but includes Builds, Commons, Context, Docs, Reviews, Tasks, Versions, Model Focused Tools (MFT), and R4E. The Eclipse Project, Web Tools, and Data Tools each also contains a handful of subprojects; and the Parallel Tools Platform’s (PTP) release include Photran. All together, there are 114 (updated) total projects formally contributing something to the release.

Some of the projects don’t actually have code repositories of their own. The Eclipse, Web Tools, Data Tools, and Mylyn top-level projects, for example, do not have their own code. The java development tools (JDT) and Eclipse Platform projects are also, I believe, container projects that do not have their own code repositories (the code is all owned by subprojects). Of the 114 (updated) projects contributing to Kepler, 107 (updated) have their own code repositories. Eight Kepler projects are using Subversion source code repositories. The remaining 99 (updated) use Git (~93% updated).

A total of 428 (updated) committers from 54 different organizations each provided at least one of the 48K (updated) commits that have been made against ~200K files in Kepler project source code repositories in the past year (since June 27/2012). These numbers don’t include the non-participating dependency projects discussed above (e.g. Jetty, and Gemini JPA).

There are 4,786 OSGi bundles and 915 features in the Kepler software repository (as of June 7/2013),

Finally, we estimate that ~58 million lines of code contributed to the the Kepler repository.

Compare this against the Juno numbers.

June 12/2013: Note that I’ve updated some of the numbers. I had neglected to include the Web Tools subprojects in my queries. All updates are marked directly in the document.

The Great Kepler Release Review, Part 1

The Kepler edition of the Eclipse simultaneous release goes live on June 26/2013. It’s an exciting and busy time for Eclipse project committers as they put the finishing touches on their bits, and prepare documentation for the release.

The number of projects increases by one from last year’s Juno release. A few projects decided to leave the simultaneous release this year and several others stepped up to take their place; joining the simultaneous release are EMF Diff/Merge, Sphinx, Stardust, Hudson, and Maven Integration for Web Tools Platform.

In addition to the official Kepler page, there’s a some additional information on the projects’ Kepler page (including, for example, links to the project release information; you can also step through the links to see past simultaneous releases).

Over the next week, wrapping up on June 12th, we’ll be running the first round of release reviews for Kepler projects. A few projects needed a little more time, and so we’ll continue our tradition of running a second round of release reviews that will wrap up the following week. The projects participating in the first round are listed on the projects landing page (I’m also working on a new page that provides a more comprehensive summary of upcoming reviews, but it’s not quite ready to take over for the projects landing page).

Note that there are a couple of extra reviews listed that are not part of Kepler. EMF Client Platform is engaging in a 1.0 Release/Graduation review, and we’ve initiated the creation review for the Vert.x project‘s move to Eclipse.

Some of the projects involved with Kepler are contributing previous or service releases, and so are not represented in the release reviews (in the Eclipse Development Process, service/bug-fix-only releases do not require a review).

For those projects that are participating in this round of reviews, their review documentation is posted for your consideration. If you have questions regarding the contents of the review documentation, or features and issues regarding the current release, pose them on the corresponding project’s communication channel.

Student Application Period Starts on Monday April 22/2013

Students, start your engines…

The Google Summer of Code (GSoC) student application process started earlier today, April 22/2013.

You must apply to participate in the programme through Google’s Summer of Code 2013 website.

We have provided an application template that should be available to you through Google’s site. Please use that template as a starting point, but feel free to add whatever kind of information you believe will help us better understand your proposal and your ability to complete the tasks you describe.

Please note that you need to describe a specific set of tasks that you intend to complete during the summer term. Be specific. This isn’t a job application. Do not create a proposal that proclaims nothing more than “I’d love to work with the Eclipse community”. Such proposals are a waste of our mentors’ time; to avoid wasting our mentors’ time, I will just summarily reject any such proposals.

Your proposal needs to present a good project idea that adds value to the Eclipse community. You also need to demonstrate to us that you will be able to accomplish the tasks you describe. The onus is on you, the student, to convince the mentors that your proposal has merit, and is worth the considerable time and effort it will take to mentor you.

There will be time to update and revise your proposal after you submit it. Please do not submit an empty placeholder proposal. Do try with your initial submission to describe your complete project idea. We will let you know if you need to provide more detail. We will help you sort out how to identify an appropriate Eclipse project for your work.

If you have any questions, please ask on our soc-dev mailing list.

I look forward to reading your proposals.

Eclipse committers: let me know if you are able to help us review proposals and maybe mentor a student.

Eclipse at GitHub

Almost all Eclipse Git repositories are mirrored at GitHub. The mirrors were initially set up two years ago by the nice folks at GitHub with relatively little input from us. I provided a simple script–which lists all of our repositories in very simple CSV format–that was used to create the original set of mirrors and is occasionally used to update the mirrors. That is, every once-in-a-while, I ask the good people at GitHub to run their script which invokes my script to create any newly-added repositories to the list of mirrors. It’s a service that GitHub provides; it’s not supported by the Eclipse Foundation’s Webmaster team.

The script that generates the list of Git repositories at eclipse.org uses information gathered by Dash. Eclipse committers don’t have to do anything to make this happen: if you create a Git repository, it gets mirrored (eventually) on GitHub. It will also be automatically mirrored on Google Source.

We do draw attention to the mirrors. If a mirror exists for a Git repository, the project information pages will provide a link. Similar links are provided for Google Source mirrors.

There are some problems with the mirrors.

Mirroring repositories was something that GitHub set up in the early days, but it isn’t a first class service and the mirrors aren’t fully supported. The repositories themselves seem to stay up-to-date (within a few minutes), but the corresponding web sites on GitHub tend to be days and weeks behind (it’s not clear when/how they update). Unfortunately, this leaves an impression that the project is inactive. This is a pretty big problem because we see the GitHub mirrors as an opportunity to encourage additional contribution; with out-of-date web pages, I fear that the opposite is happening.

GitHub has no current plans to fix this problem. This presents us with an opportunity to rethink how GitHub mirrors are done.

The preferred means of making this happen, as far as I know, is to create a repository on GitHub and then push the contents of the eclipse.org repository into it. Unfortunately, there is currently no automated means of doing this. As far as I know, GitHub provides no mechanism to schedule a regular pull (correct me if I’m wrong), which means that it would become our responsibility to push to the GitHub clone.

This is where I’m going to need a little help. I’m pretty sure that it’s possible to create a git hook that can be installed into the git.eclipse.org Git repository that will automatically push to a remote repository on commit. Perhaps it makes sense to have the hook push to all remotes known to the repository, not just GitHub. Having this sort of hook would give projects the power to decide what they want to do; they would have control over whether or not the repository is automatically replicated and where.

Please add your comments on Bug 402183.

No seriously: We love Git. Seriously.

The cornerstone, I think, of a good April Fools Day joke is plausibility. The more reasonable or possible the topic, the more likely it is that you’ll fool somebody. I’m not sure what it says about the Eclipse Foundation that so many people really thought that we’d made a decision to move away from Git, but I am pretty proud that I caught a least a few people with today’s announcement.

In the aftermath, though, I should set a few things straight.

CVS is gone. Well, mostly gone. It is true that we’ve granted a open-ended stay of execution for Orbit on CVS, but all other CVS use has been completed turned off.

We’re very serious about Git. We are also very serious about Gerrit. This combination of technology provides a powerful means of attracting contribution that just can’t be touched by CVS or SVN.

Our entire committer base seems very happy with Git. The comments about confusing workflows, non-fast-forward merges, and missing commits are actually based in reality. But these are just growing pains that a few of our committers and projects have worked through. Real complaints are few and those that we do get are addressed pretty quickly. A great deal of credit goes to the EGit project for providing tools, as well as leadership and assistance in our migration to Git.

The chart is totally bogus. The Dash data tells a very different story:

This chart shows the number of commits that have different values in the “Author” and “Committer” fields. This chart does not provide an entirely accurate quantification of contribution (it excludes all contributions received via Gerrit, for example), but I do believe that the trend it displays is correct. I do have the necessary data to generate a more accurate chart, but–since Dash wasn’t designed with Git in mind–I don’t currently have it in a form that I can easily query. When this changes, I’ll publish a more accurate chart.

SVN support will end. Over the past 18 months, we have said “no” to projects requesting new SVN repositories. SVN support at Eclipse is effectively deprecated. We have not yet set an end-of-life date for SVN, but one will be set soon. Projects currently using SVN should start thinking about migration to Git.

The Great Git Experiment

While many projects have claimed success with Git, a great many projects and developers continue to struggle with its adoption. In the fall of 2012, several projects chose to terminate rather than migrate their CVS repositories to Git. We have infrequent reports of commits being erased and general confusion regarding the workflow. While EGit does a good job of smoothing out the rough corners, it’s still the case that just writing code to a Git repository takes too many steps. I hear many complaints about committers forgetting to push their commits or having them lost in non-fast-forward merges.

This undercurrent of dissatisfaction came to a head last week at EclipseCon. I was cornered by several committers who were quite angry at having been forced to adopt Git and demanded that I allow them to roll back to CVS. The general consensus was that while at face value, Git appears to be better suited for encouraging contribution, the confusing nature of its workflow was actually having an opposite effect.

Data collected by Dash confirms it: many projects are actually losing committers and contributors because of Git. Non-committer contributions to projects has steadily dropped over the last twelve months. Projects that switched to Git early in the migration process did on average experience a short spike in contributions in the last few months of 2011, but in the general case, the contribution curve drops almost linearly from that period.

Faced with the backlash and drop in contribution, we have no choice. After careful consideration, and extensive discussions with the Eclipse Webmaster, we have decided that we have no choice but to migrate back to CVS. Given resource constraints–the webmaster team cannot support three version control systems–support for Git has been deprecated effective today. The Git server will be shutdown on December 21/2013.

Nerdvana is a Data Centre

Earlier today, we started a pretty bold move from one data centre to a brand-spanking new one across the city. I say bold, because we had to take ourselves completely offline while we disconnected more than thirty bits of hardware, shipped it across the city, and then reassembled it all.

I didn’t take very many shots from old data center; just this one of Webmaster Matt behind a bunch of cables.

Denis coordinated the whole thing from an impressively comprehensive plan assembled over weeks of careful analysis, multiple mental walk-throughs, and–no doubt–many sleepless nights. Here’s a shot of Denis piecing some of our systems together in the sparkly-clean new data centre.

Chris was a big help. In this shot, Chris is installing the the back half of a rail in for one of the servers. I was able to take the picture because I did all the front screws which are way easier and less time consuming than the ones in the back.

I managed to take this next shot in a rare moment of inactivity from Mike (seriously, the guy’s a machine) who had stepped back for a minute to give Denis some room to connect a few systems into the switch. We actually managed a very high level of parallel activity despite the tight space. I credit Denis’ plan and the well-laid out documents and diagrams which allowed us to get done what he needed without bombarding him with questions.

By early afternoon, it was all coming together nicely. I think it was at this point that Denis said something to the effect of “our backends are up”. This made me giggle. I guess that deep down we never get any older than twelve.

The data centre is still relatively new and so there’s still some empty space (which was good because being able to spread the servers around a bit made things a lot easier). I was tempted to use the space to demonstrate some of my sweet Karate moves. But everybody was busy doing actual work and wouldn’t have been able to fully enjoy the experience.

Our other Chris–the guy who writes the cheques–joined in as well. In this shot, our two Chrises are assembling various bits of hardware while Denis wrestles the systems to life. As as aside, I quite like the portable workstands we used; they’re easy to wheel around, they’re very stable, and they’re height-adjustable to boot. I need to get one of these for home.

By late afternoon, we’d installed every single rail and shelf, and every bit of hardware was in its place. When I left, Mike and Chris were following Denis’ cabling plan while Denis and Matt bent the operating systems to their will. The speed and confidence that Matt and Denis demonstrate while elbows-deep in Linux configuration files is awesome. And humbling.

It was a great day. Frankly, I love playing with hardware. But I had to leave. Minor hockey needed me…

Denis reported late this evening that all was progressing according to plan. Many of our servers are up and running again and the rest should be taken care of tomorrow. I’m not 100% happy that the Foundation’s email server is back up, though…