Rubber Stamp Processes

I continually think about ways that we can improve the Eclipse Development Process and the various bits of infrastructure and process that exist to support it.

The whole review process has always been troubling for me because it’s really turned into a bit of a rubber stamp process.

Early on, reviews were something that were attended. A project undergoing a review was required to assemble presentation materials that would be delivered on a conference call with interested parties that included project members, relevant PMCs, adopters and other members of the community. It didn’t take too long for us to determine that these calls were pretty pointless: attendance was low, and–while there was sometimes some interesting related discussion–direct discussion of the review itself tended to be one-sided (i.e. the project representative did all of the talking). So the call became optional in favour of mailing-list/forum based discussion. Eventually, we just dropped the idea of the call altogether.

Today, we’ve changed the notion of a review event into a review period. Previously, review material had to be posted a full week in advance of the (potential) call so that that participants could be prepared. We turned this week of materials-review time into the review itself. During this review period, the community is encouraged to comment and question the review. At the end of the review period, the EMO makes a decision on whether or not the review is successful.

This is where the rubber stamp part comes in.

By this point, the IP team has–if necessary–reviewed the IP Log for the project and has determined that everything is okay; the project’s PMC has approved the event; and the EMO has confirmed that the review materials contain the required information. From a process point-of-view, everything is pretty much in order before the start of the review period (there are exceptions, but this is the general rule).

At the end of the review period, the review is deemed successful. In my tenure, I have never failed a review (with the possible exception of the TPTP Termination review, which I don’t believe ever actually got scheduled). So I find myself asking: what’s the point of the review period?

Reviews are undertaken primarily to inform the community of key events in the lifecycle of a project and give them opportunity to react. Adopters, for example, need to know about releases so that they can take advantage of new features, or react to API changes. But the review itself comes very late in the development cycle; any adopter who is not involved with a project far in advance of the release review, isn’t going to get very much advantage from that week long review period that precedes the actual release by a couple of days. No, an adopter who needs to keep abreast of developments in a project–much like the project committers themselves–is going to need to be involved very early in the development cycle. This is why project plans, and ongoing communication are important (probably in the opposite order).

Frankly, I’m not sure if reviews are useful or interesting for the user community; users probably get more value from a “new and noteworthy” document than they would from a release review document.

So what is the next step in the evolution of our review process? I have some thoughts about the various types of review that I intend to discuss in the coming days and weeks. In the meantime, if there’s something that you feel you can contribute to the discussion, I’d love to hear from you.

This entry was posted in EDP. Bookmark the permalink.

3 Responses to Rubber Stamp Processes

  1. Miles Parker says:

    Wayne,

    What about facilitating and encouraging some kind of voluntary non-binding peer code-review process. This might work in a similar way to academic peer review, where folks from other projects could on a rotating basis be assigned during say a week long period to take a look at the reviewed project’s code base and make comments and recommendations. These would be public so there would an incentive for projects to make an effort to respond but they wouldn’t be binding so that projects wouldn’t need to feel defensive about receiving constructive feedback. I think that this would go a long way toward pointing everyone in the same direction WRT to project and code quality without making things too process heavy.

    cheers,

    Miles

  2. Konstantin Komissarchik says:

    I have a suggestion regarding re-working termination review. The problem with the way that project termination is currently specified is that it requires the folks that have already distanced themselves from the project (due to whatever causes) to take the time to write a termination proposal document. That’s hardly going to be on top of people’s priority list, so projects that ought to be terminated are sitting around in a zombie state.

    I propose establishing an automatic metric of project liveliness. An automated system monitors this metric for all projects. When this metric falls beneath a certain threshold, an automated notice is generated on project’s dev list informing the project of this fact. The project at that point is given X days to schedule a continuation review (including putting together continuation review documentation that details why the project should be allowed to continue). If the project fails to request a continuation, it is automatically archived at the end of specified period. No termination review. Just a notice of the fact to the overall community.

  3. thomas says:

    I’ve also always considered reviews as neither interesting nor useful to the community and wouldn’t mind at all if that part of the process was simply removed. A new release should be accompanied by a N&N document. That’s what adopters of a release will look for.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s