Quantitative Community Involvement Metrics

This area is for tracking community involvement by several different quantitative metrics. This is what we want to track:

Mailing List Posting Metrics

Number of messages posted to OSAF lists by community members in a week, and the number of unique posters. Each are broken down into a total, volunteers (non-OSAF staff), and OSAF staff.

Date	Messages			Unique Posters			PyLucene
Week ending	total	vols	OSAF	total	vols	OSAF

Not Found

The requested URL /~bear/metrics/countMessages.txt was not found on this server.

---

Apache/2.2.3 (Debian) Server at kumu.osafoundation.org Port 80

This shows how many people are subscribed to the following OSAF mailing lists on the given day.

Date	Announce	Design subscr	Dev subscr	General	Process	PyLucene

Not Found

The requested URL /~bear/metrics/countListSubscriptions.txt was not found on this server.

---

Apache/2.2.3 (Debian) Server at kumu.osafoundation.org Port 80

Wiki accesses to meeting minutes and status for week ending

This table shows the weekly Wiki hits, excluding hits from OSAF's Howard Street office.

• Total wiki hits
• Staff meeting minutes ("Staff")
• Management meeting minutes ("Mgmt")
• Comprehensive status reports ("Status")
• Newsletters ("Newsletters")
• Design working group meetings ("Design")
• Apps working group meetings ("Apps")
• Repository working group meetings ("Repository")
• Community working group meetings ("Comm")
• The number of accounts on the given day
• The number of RSS requests in a week

Week ending	wiki hits	Staff	Mgmt	Status	Newsletters	Design	Apps	Repos	Comm	Wiki accounts	RSS gets

Not Found

The requested URL /~bear/metrics/countWikiPageHits.txt was not found on this server.

---

Apache/2.2.3 (Debian) Server at kumu.osafoundation.org Port 80

Web hits in the week ending

Weekly Web hits, excluding hits from OSAF's Howard Street offices (from http://stats.osafoundation.org):

Week ending	total web hits	page hits	downloads

Not Found

The requested URL /~bear/metrics/countWebHits.txt was not found on this server.

---

Apache/2.2.3 (Debian) Server at kumu.osafoundation.org Port 80

Note: the third column shows all downloads, including 206s (partial GETs), 304s (no modification), and HEADs (excluding accesses from Howard Street).

Note that for the 9 May and 16 May 2004 lines, we believe that roughly 20,000 hits are due to a change in the build system. The builds started being run much more frequently, and the Tinderbox page refreshes itself automatically. On 17 May, Markie modified the Tinderbox status page so that it does not pull the OSAF logo nor the css file from hula, so we thought the number of hits should've dropped significantly. We don't yet understand why they didn't.

On around July 1, we moved the Web server, and it took a while to get back to normal. Hence the missing data. Also, the log parsing tool needed to be rewritten because of a change in practice.

Downloads

This table shows the number of possibly successful Chandler downloads in a given week (excluding downloads from the Howard Street office):

	Release:	Milestones				older releases				0.4 release				0.5 release				0.6 release
Date	Total	Total	Win	Linux	Mac	Total	Win	Linux	Mac	Total	Win	Linux	Mac	Total	Win	Linux	Mac	Total	Win	Linux	Mac

Not Found

The requested URL /~bear/metrics/countDownloads.txt was not found on this server.

---

Apache/2.2.3 (Debian) Server at kumu.osafoundation.org Port 80

This table shows only the downloads that had a chance of retrieving the entire package (i.e. were GETs with a status code of 200).

Bugzilla

This table shows the number of bugs filed in a week, both by OSAF employees and others in the community ("vols").

	Bugs Opened			Bugs Closed			Patches			BZ accounts	% regressed
Week ending	total	vols	OSAF	total	vols	OSAF	total	vols	OSAF	(cumulative)

Not Found

The requested URL /~bear/metrics/countBugzilla.txt was not found on this server.

---

Apache/2.2.3 (Debian) Server at kumu.osafoundation.org Port 80

CVS in week ending

This shows how much CVS activity there was in a week. We don't yet have a way to count who is OSAF and who is not..

	files			lines
Week ending	# added	# removed	# changed	added	removed	net change	comments

Not Found

The requested URL /~bear/metrics/countChanges.txt was not found on this server.

---

Apache/2.2.3 (Debian) Server at kumu.osafoundation.org Port 80

IRC Office Hours

This table shows the number of people ("talkers") or present but not participating ("lurkers") in an IRC Office Hour. "OH lines" gives the number of lines in the transcript of the hour, and the topic gives what the Office Hour was about (and a link to the transcript).

Date	vol talkers	OSAF talkers	vol lurkers	OSAF lurkers	total vols	total OSAF	OH lines	topic

Not Found

The requested URL /~bear/metrics/countIRCs.txt was not found on this server.

---

Apache/2.2.3 (Debian) Server at kumu.osafoundation.org Port 80

Future Metrics

Here are some metrics that we want to track but don't yet:

• Number of patches per week submitted by the community
• code reviews done by community
• (Not a community metric per se, but useful) Bug regressions, i.e. how many bugs introduced when fixing other bugs
• Lines of code in Chandler
• number of hits to the RSS feed of the wiki
• StarterProjects projects posted / active / participants / completed
• attendance at public events
• Media exposure: number of references / citations / articles / blogs / etc. referring to Chandler and OSAF from the community and general press, e.g.:
  • news links (number of articles from news.google.com mentioning "open source applications foundation")
  • web links (Google of "link:www.osafoundation.org")
  • web references (Google of "osaf chandler -site:osafoundation.org" for the "last 3 months")
  • blog mentions1 (Google of "open source applications foundation blog")
  • blog mentions2 (Google of "osafoundation.org ext:xml" and ext:rdf and ext:rss)
  • blog mentions3 (Feedster search of osafoundation.org)

Date	News	Links	references	Blog1	Blog2	Blog3
18 Mar 2004	1	795	3,570	7480	1121	353

If you have an idea for a good metric that we aren't tracking, please send email to ducky at osafoundation.org. ("Number of active volunteers" is something we'd like to track, but it isn't clear how we would count it.)

-- DuckySherwood - 13 Jan 2004

---

Comments

This looks like a great idea. I like what the community group is doing here to try to track some quantitative metrics of success. I hope the design group and the other groups will end up following the lead of the community group.

After you have some raw data for different metrics, it seems like the next step would be to try to figure out how to use these metrics to keep track of how successful OSAF's community outreach effort is. Towards that end, it might be good to start thinking about what the metrics might look like for a successful effort. In my opinion, it seems like the main goals of the community effort are (1) to help get Chandler built as quickly as possible, (2) to help make sure Chandler is a quality product, and (3) to help speed the adoption of Chandler. I think that "community involvement" should not be a goal in an of itself, unless it helps achieve one of the 3 main goals.

So for goal #1, one of the metrics of success might be "number of lines of code contributed by volunteers". Or, more accurately, the metric should be "number of lines of contributed code which end up included in 1.0", and that would be a positive metric. And actually, there might also be a metric for "number of lines of contributed code which end up not included in 1.0", and that would be a negative metric. The long term goal would be to try to increase the values of positive metrics and decrease the values of negative metrics.

• BrianDouglasSkinner - 26 Jan 2004

---

Heikki explained how to get the bugs in a week:

1. 1. Go to Bugzilla query page: http://bugzilla.osafoundation.org/query.cgi
   2. Select product (Chandler)
   3. Clear Status (none selected)
   4. In the Email and numbering section, uncheck bug owner, check reporter, select "doesn't match regexp" from the drop down list and write "osafoundation.org" (without quotes) in the edit box
   5. In Bug Changes section select [bug creation] field, and write the dates from-to below
   • Sample: All bugs filed on Chandler by non-OSAF people between 2004-01-01 and 2004-01-28.

---

I don't think it is as straight forward as Brian suggested (positive/negative metrics). If an extension will not ship with 1.0, but is usable and downloadable from somewhere with 1.0, I think it is a positive thing (think of all the Mozilla extensions).

Also, I could easily imagine someone getting involved with Chandler on something that will not make it into 1.0 but would make it into 2.0. In the long run that is clearly positive, but short term distracting (if we did not support it in the short term, the long term gain would probably never happen).

Still, having said that, OSAF's number one priority is shipping 1.0 in good time.

-- HeikkiToivonen - 28 Jan 2004

---

Just FYI -- you might want to look at this TWiki ChartPlugin^?.

• BrianDouglasSkinner - 26 Jan 2004

---

Release download numbers as of xx/xx/xxxx:

Release	Windows	Linux	Mac OS X	Total
0.3 Release	xxx	xxx	xxx	xxx

We might also want to consider debug versions and source tarballs.

-- HeikkiToivonen - 02 Mar 2004

( Done. -- DuckySherwood)

---

To get a count of patches in bugzilla try this:

  select profiles.login_name AS email, count(a.bug_id) AS bugcount FROM profiles, attachments a 
  WHERE a.submitter_id=profiles.userid and a.ispatch = 1 GROUP BY email ORDER BY bugcount;
  

-- MikeT - 11 Feb 2004

---

I think Brian is on the right track, but I would redefine the main goals; I think goal #1 in particular is too narrow. I would rewrite it as something like "Making Chandler a long-term succesful offering" and then define success in terms of quality, adoption, functionality, and the other things OSAF discussed in its vision documents. Chandler 1.0 is only the beginning. We know that 1.0 won't be aversion aimed at institutions. We also know that 1.0 might be a version for early individual adopters but not necessarily the mass consumer market. So defining community success in terms of 1.0 seems too narrow, since the success of the Chandler project itself won't be evaluated this way. Mitchell

---

I just realized Bonsai can also provide some metrics since the tree was last opened (currently that is a LONG period for us):

• http://bonsai.osafoundation.org/toplevel.cgi
• http://bonsai.osafoundation.org/countcheckins.cgi

-- HeikkiToivonen - 31 Mar 2004

---