Fishing in the Bay

Most human activity has changed drastically over our lifetimes. And the rate of change is increasing – see for instance the next generations user interface for computers. You would hope academics would live at the forefront of change. But the double refereed hard copy journal system has not changed for a century. I think it is a bad system. In fact, I wonder why we need journals at all.

It is important at the outset to identify journal publication as it currently stands as serving two distinct purposes. One is to disseminate information amongst the research community. The other is to keep a record of research output for career advancement in order to assist the HR departments of universities in promotions and appointment. I will focus on the first one, since I think that is the main game. But I will come back to the second one at the end.

My idea is that draft papers in mathematical statistics are uploaded to a website – let’s call it the Mathematical Statistics Research Network. Submission format is standardized to some extent and also involves filling out a form of keywords, selecting from a list of say 50 subject categories and you input the MSRN code of cited papers. There is a modest fee for uploads to cover the cost of the site but there is no fee for downloads. Only registered users can use the site. Registration is free. The rest follows very easily indeed, especially for anyone who is familiar with using E-Bay.

The first point to note is that electronic papers will, all other things held constant, be better than hardcopy. Why? Because there are fewer media imposed limits on the author. Papers could include colour graphics, data links and even video. Most importantly, the author is not writing to a word limits and can use exactly as much space as is required to explain the ideas. The second point is that there is not a two year delay from first draft to “publication”. Of course, premature publication has its risks. Don’t worry. I will get to that.

The purpose of the site is to give you, the user, free access to good and recent research in your field of interest. When you log-on you select from a list of the say 50 subject categories – for instance exact inference & medical statistics and you see a list of papers in the intersection of these categories. The key to making the system work well is the provision of a powerful and personalisable sorting algorithm. There are various characteristics of the paper that need to be measured to inform the sorting. Some are easy to measure, some are not. One easy one is the date of upload, so you might initially order papers by how recent they are. Another might be a keyword, such as “intention to treat” which would promote these papers to the head of the list, but still time ordered.

But the main problem that this system would face is the same one that journals face. How do you identify the good papers from the bad? First, you can measure how often a paper has been downloaded. If a paper is currently hot, then I might want to see it. But download count is not the same as quality. How do you measure quality? This is where we learn from Ebay. Registered users can review and rate papers, on say a 1-10 scale. The reviews are not anonymous. And each rating links to a document giving reasons for the rating. Reviews can be withdrawn or edited by the reviewer. In fact, after reading dissenting opinions there may well emerge a consensus view. So for each paper, I can see two more fields – number of downloads and average rating. As a user, I can click on the average rating and see a list of who rated the paper and find out why. And reading these possibly conflicting reviews will help me in digesting the main issues in the paper.

What about capricious or malicious reviews? E-bay again. Reviewers can be rated. This might partly come through registered users rating individual reviews, so if a reviewer offers reviews that are consistently rated lowly, then the reviewer is rated lowly. This defines a weight that feeds into the average rating measure applied to papers. So as a user, when you look at the list of reviews you can order them by reliability. Of course, you can set weights for reviewers yourself. For instance, you might give automatic high weight to any review by Robert Kohn in the area of Bayesian model selection. And you might set zero weights to those who you have seen write stupid reviews in the past or people who you just have a low opinion of.

At a certain point, the author can choose to withdraw the paper or to upload a final version, taking into account the comments of the reviewers. At this point the paper becomes anchored and cannot be changed. The final paper also attracts reviews and rating. Those who want to publish in a hardcopy journal could still do so if they wanted to – though I am not sure how long journals publication two years hence would remain attractive. If the journal system can survive the market then so be it. It must have a use. Moreover, those journals who did remain would find it easy to referee submissions – just look at MSRN and see the on-line reviews. This will make life for the referees of these journals much easier.

Lastly, what about career advancement? The current system of promotion or appointment involves saying how many papers N you have published, how many X are in top journals, and recommendations from three referees. Only the N and the X would change. Instead you would have a list of your papers ranked perhaps by downloads and average ratings. Your referees and the institutions considering you could easily log-in to MSRN and look at the reviews and look at other rankings. Citation information would be easier to collect automatically within the system (though it is available now), including various variants of half life measures that take into account the fact that older papers have more citations than newer papers.

To set up such a system in the first place would be pretty expensive. But it can be done. Look for instance at the Social Science Research Network who have made a start in this direction. Another interesting site is Berkeley electronic press where my department is about to outsource its entire research webpage. Neither of these sites are currently implementing all the idea above, nor does uploading your research to these sites currently “count” either for promotion or even informally amongst your peers. But it seems to me that it is only a matter of time.

I expect this post might start a vigorous conversation. To start you all off, perhaps you could structure your comments around this question:

What cost/benefits does traditional double-blind review have over the system I am proposing?

You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.