vendredi 11 janvier 2013

A new reviewing process

I totally agree with the remarks of Pierre.

I would like to present a new reviewing process that I will propose for CPAIOR-14

CP conferences are very special:
  • publishing a paper in a conference is as hard (harder than?) as in a CP journal.
  • a lot of members of the program committee (PC) publish papers in the conference
The first point gives too much importance to the conference publication. Reviewers are fighting each others. Some of them are killing papers because they thing that it can increase their chance to have their own paper accepted.
The second point is not really good and causes some problems. In my University, my CV has been reviewed by a Professor who is not in computer science. He/she write in the review: "we cannot count papers that are published in the conference in which the author is in the program committee". Therefore, he/she kills a huge number of my papers! I was upset! But is he/she wrong? I am not sure... When a best paper is given to a paper with an author who is in PC, I think we make a big mistake: the conference will not be well considered outside.

Since publishing in the conference is hard (in order to have low acceptation rate as we like so much), people are complaining a lot about the reviews they receive. The problem is that there are right most of the time. We cannot have ultra selective conferences, if the chairmans are not doing their job very well, for instance by reading all the reviews. Unfortunately, this is not possible because there are too many submitted papers.
Thus, a lot of things have been invented like:
  • the authors may answer to the reviewer
  • a senior PC is created
I don't really understand the first idea, because the reviewer has not to answer to the answer and it does not mean that a reviewer cannot write whatever he wants. It could just give more work to the chairmans because there are more things to read.

The second point is more interesting but it does not consider a big flaw in the general process:

Any Professor at the university knows that two correctors will give different evaluations for the same homework. He knows that it is really difficult to have several correctors without having this issue and that we need to define very precisely how the homeworks have to be evaluated if we expect the same final score.
For conferences, we have a lot a reviewers and, as if by magic, there is no problem to order, to compare and to discuss about a slight difference of scores. We like so much to use scores even if we know that it is so much difficult to evaluate in the same way the same work by two different persons! In order to help the chairman I introduced at CPAIOR-04 the notion of "confidence in the review", by thinking that it could help the chairman to classify the paper. However, it does not change the problem.

In addition, it is very dangerous to use the scores today, because people use them to kill some other papers and because the chairmen do not read all the reviews.

For instance, I recently received a review saying (I cite): "the proposed method is simple and elegant", then the reviewer says that the experiments are not sufficient to convince him, because he would like to see the use of the method on super computers. Roughtly, this is the review. Then it gave me :"strong reject". This is not normal. The chairman should complain about the reviewer. This is not possible to find a new method "simple and elegant", consistent for the classical benchmarks and complain about the absence hypothetical benchmarking and give a "strong reject". I have no problem with the rejection of the paper. I have a problem with the strong rejection. Since we use score this is important.

So, even by creating senior committee we have not the guarantee that the members will read all the papers and the reviews. In addition we still have the issue of the quality of the evaluation.
Thus, wan can we do?

I propose to apply the two following rules:
  1. We accept a paper when at least one reviewer wants to accept it. In this case, the reviewer has to select a box "I want to see this paper accepted". We have just to ensure that the paper is not a republication of another work. The author will have to integrate the negative comments for improving his paper, but there are not part of the decision process. Honestly, if at least one person is interesting, this is positive!
  2. When a reviewer reject a paper, he has to destroy it. This rule comes from Alain Colmerauer. If you keep a paper in your office it means that it is interesting, so you should accept it.
People are going to tell me: "your first rule is not reasonnable because friends are going to accept papers of other friends". Maybe, but I have three answers: a) it already happens. I can tell you that some papers are almost always reviewed by the same group of people working on the same topic.
b) some people are also trying to group together as reviewers in order to kill some papers. With my method, this part will be almost useless.
At least, if need we can propose another thing:
the name of the reviewers having accepted a paper is made public.

With this rule I guess that the selected papers by reviewer will change...

7 commentaires:

  1. A workshop has been organized on the topic:
    Plenty of ideas for renewing reviewing there.
    Now for a next controversial post: what about the recruitment process ? Sarcasm.

  2. What did you think of this year CPAIOR process ?

  3. Naren,
    I didn't understand anything (I was in the PC).
    I had to vote on some other papers that I didn't read. Then they do something with the score and decide the accepted papers.
    As author, I received a lot of scores: several strong accepts and several strong rejects.

    Instead of inventing a lot of things and manage it, there is another possibility: the chairman READ all the reviews. I will be better than inventing a kind of bureaucratic stuff.

    We should really think about papers. What should we accept only non controversial papers? Especially for a conference, I think this is not a good idea.

  4. I like your idea a lot: accept a paper when at least one reviewer wants to accept it crearly.
    More than the name of the reviewer who accept it, I'd also like to see is the review of the person who wants to accept. This should decrease further the risk of accepting friend's papers.
    Papers are published to be read by other scientists and reviews is a very valuable information for a reader (everybody likes it for books, why not for papers?).
    I also believe that if reviews are of high quality it can also be beneficial for the reputation of the conference it self to publish it (the positives one).
    We could even ask to the authors of the paper and the reviews themselves if they agree.
    What is dangerous is giving the name of someone making a negative review. A colleague of mine did it by signing one of his review. I found it very honest and courageous but the authors where not happy at all :-(

    Another idea for the next controversial post: pro and cons of double blind reviews.

  5. If CP conferences are too selective then the fix is to make them less selective, not to change PCs IMHO. The easiest way to make them less selective is to accept more papers.

    The CP community will need to make a choice: does it want to behave like other scientific communities, where conferences are quite easy to get in, and where journals are highly selective, or does it want to treat its major conferences like a journal?