After papers are submitted on July 8, the bidding process will begin. I will send out an email to let you know exactly when you can start. Bidding will continue through July 13. Careful bidding by the PC is crucial to obtaining a good paper assignment. Thanks in advance for doing your part!
Every PC and ERC member must enter their bids by the end of July 13.
Bidding will be carried out in HotCRP. Each PC and ERC member should enter a bid for every paper. A bid (called a review preference in HotCRP) is a combination of two things:
Positive numbers in your review preference mean you have greater than average desire to review the paper. Negative numbers mean you have a less than average desire to review the paper. A score of −100 means you think you have a conflict. Examples:
There are probably a number of ways to game this preference system. I'd prefer it if you didn't try. For example, if you assign 20 papers a 3 and every other paper a -3, you probably won't get exactly those 20 and I won't know which ones you really want or don't want. I'd like everyone to be excited about the stack of papers they receive to review but naturally, I will have to balance that against the need to ensure papers have proper expertise assigned to them.
When bidding, you won't have read the entire paper (though, of course, you are free to look at any paper in depth when bidding), so you are only estimating your expertise. If, when you review a paper, you find you have made a mistake in your estimate, that is just fine, and is bound to happen from time to time. If you find yourself downgrading your expertise from an X, we might find a paper suddenly lacking expert reviewers. In such a case, feel free to alert the PC chair. I'll see what I can do.
Entering bids in HotCRP: There is more than one way you can enter bids in to HotCRP. One way to begin is to go the reviewer preferences page. There, you will see a list that shows all submitted papers. You may enter your preferences in the text boxes here. Alternatively, you may flip through the paper pages (use keys k and j to flip forwards and backwards through the paper pages efficiently). If you go through the papers in numeric order, flip a coin first to decide whether you will go through them back to front or front to back. You may also upload preferences from a text file; see the “Download” and “Upload” links below the paper list on your review preferences page.
Guardian reviews due Aug 15
ERC reviews due Sept 1
75% of each PC member's reviews due Sept 1
All PC reviews due Sept 5
Online paper discussion Sept 10-25
The programming languages community in general, and POPL especially, has an excellent reviewing culture. Our reviews are thoughtful, deep, clear, constructive and unbiased. Not all academic fields are this way, by any means --- we are pretty lucky. As 2015 POPL reviewers, it is our obligation to do our best to maintain POPL's very high reviewing standards. If each individual reviewer does a great job reviewing for POPL this year, we'll put pressure on each other and on next year's reviewers to do just as good a job. And our entire field will benefit for it. Thanks so much for your hard work!
When writing a POPL review, remember that it serves two primary purposes:
One commentary on writing reviews for systems conferences can be found here. I do not agree with everything in that article, but it is worth a read.
Rating Paper Acceptance: Each review will give a score to a paper indicating whether or not the reviewer believes the paper should be accepted or not and what their expertise on the topic of the paper is. We will use these scores as rough guides for filtering papers, but they are no substitute for detailed, textual commentary on the strengths and weaknesses of a paper.
We will summarize our judgement about paper acceptance using the A-B-C-D "Identify the Champion" scoring scheme:
Notice that an "A" does not mean that a reviewer thinks that a paper is perfect or award-winning. Rather, it means that the reviewer believes strongly that the paper should be included in the POPL 2015 program. That is quite a different thing. A flawed paper will recieve an "A" when a reviewer believes the paper, despite its flaws, contains some great ideas that the community will benefit from hearing about.
Likewise, a vote of "D" does not mean a paper is worthless --- it merely means that the reviewer believes strongly that a paper should not be included in the POPL 2015 conference program. In my experience, there are almost no "worthless" submissions to POPL. POPL submitters are highly selective and almost all POPL submissions contain good ideas. Nevertheless, many papers will receive scores of "D" from some reviewers.It is ok to reevaluate your reviews and your scores! The goal of the PC is to try to make the right final decisions about papers. PC members do not have to "stick" to their original reviews or scores, especially when new information comes to light. For example, if you review your first couple of papers and then enter your reviews right away (your PC chair will thank you for doing so!), you may find that after reading a few more papers, you are better calibrated and would like to move one of your previous scores up or down. It is just fine to do so. I hope this encourages reviewers to enter their reviews in to HotCRP early as opposed to hanging on to them while they deliberate about other papers.
A Thought Experiment: Suppose you encounter a paper with a great new idea, but a significant flaw. What do you do? Do you support the paper because of its great new idea? Or do you reject the paper because of the flaw?
Many people find it more difficult to support such a paper than to reject it. It often seems to take more courage to support the paper that has some flaws than to reject the paper that has a great core idea. Keep this in mind and search out the positives when reviewing your papers. Clearly state and explain the positive aspects of each paper you review in addition to the negative ones. A paper does not have to be perfect to be included in POPL (few papers are). Of course, we do not want to accept papers in which the main theorem is irreparably broken --- some flaws are too severe to overlook.
Younger reviewers: It is folklore that younger researchers tend to be harsher critics of their peers than older researchers. I do not know if this is true or not, but this year POPL will have many younger reviewers. In my opinion, the young researchers in our community are wicked smart, energetic, hard-working, dedicated and in the thick of the most exciting research our field has ever seen. And I think they have the talent and judgement to identify and appreciate the positive elements in the research they review. Prove that I'm right! :-)
Rating Reviewer Expertise: We will be using the standard X-Y-Z scores for reviewer expertise. These scores have the following meaning:
Each POPL submission will be assigned a guardian. The role of guardian reviewers is:
Overall, the guardians act as a double-checking mechanism for the PC chair with the advantage that each guardian has 6-10 papers to worry about whereas the PC chair will probably have 220+ papers to worry about.
Every guardian review must completed by August 15.
Even though guardian reviews must be completed by August 15, feel free to impress me by completing your guardian reviews by the end of July! The earlier we get those reviews done, the easier it will be to deal with papers we find do not have sufficient expertise.
PC and ERC members should do their own reviews, not delegate them to someone else. If doing so is problematic for some papers, e.g., you don't feel completely qualified, then consider the following options. First, submit a review for your paper that is as careful as possible, outlining areas where you think your knowledge is lacking. Assuming we have sufficient expert reviews, that could be the end of it: non-expert reviews are valuable too, since conference attendees are by-and-large not experts for any given paper. Second, if you feel like the gaps in your knowledge are substantial, submit a first cut review, and then work with the PC chair to solicit an external review. This is easy: after submitting your review the paper is unblinded, so you at least know not to solicit the authors! You will also know other reviewers of the paper that have already been solicited. If none of these expert reviewers is acceptable to you, just check with the PC Chair that the person you do wish to solicit is not conflicted with the authors. In addition, the PC chair will attempt to balance the load on external reviewers. Keep in mind that while we would like the PC to make as informed a decision as possible about each submitted paper, each additional review we solicit places a burden on the community.
If you are a guardian for a paper and feel you need to solicit an outsider review, please discuss with the PC Chair as soon as possible. We will undoubtedly come across submissions for which we do not have sufficient expertise on the committee. When this happens, some PC members will become non-expert guardians. In some cases, this might be quite clear from the outset. Please work with the PC Chair to come up with external reviewers who can help us obtain the necessary expertise to make an informed judgement on the paper in question.
Most of the above was repeated on the community FAQ here.