27 May 2016

The Plan for KDD

There are quite a few changes this year in KDD and we are excited about them. We wanted to explain the rationale behind them.

Tracks, Talks, Posters and Videos

Based on feedback from the past few years we learned that a large number of tracks with competing talks was not always desirable, since many attendees missed out on many great papers. On the other hand, reducing the number of tracks would mean accepting fewer papers, something we did not want to do. We also wanted to make sure that it’s possible to learn about the entire content of the conference. Moreover, we wanted to make it easy to learn about KDD even when not attending the conference. So many goals, so little time, hence what to do?

  • We decided to halve the number of talks, to keep the number of parallel sessions small. The decision as to whether to award a talk was based on excitement of the referees and also based on suitability of the content as a talk. I.e. a demo is probably easier to share than a detailed technical proof.
  • Every paper, regardless of whether it’s a talk or not gets a slot as a poster. This way there’s plenty of time to discuss technical details with the authors at the poster booth.
  • At the same time, we felt that it was necessary to give authors a chance to give a talk regardless, just not in person at the main conference. The way to fix this was to allow and encourage authors to upload a 30 minute video of their paper on YouTube. The link will be shared and will allow everyone to see the talk, even before the conference. In a way, this is better than giving everyone a talk, since it would not have been possible to attend multiple talks given simultaneously anyway.
  • Finally, the talks given at the main conference will be shared online, too.

No Authors Feedback

To get it out of the way - authors feedback is great and it is super useful, when it works. Hence, why did we decide against it? The sad reality that we all experienced as program chairs and SPC members is that referees very rarely take authors feedback into account to revise their opinion. This is due to a number of reasons. First off, it is extra work and sometimes reviewers are too busy to look at things again. Second, humans often find it difficult to revise their opinions, once they’ve settled on them. These two things conspire against effective authors’ feedback. Hence we decided not to waste authors’ time on this.

  • That said, there was discussion between referees regarding the quality of the papers. This discussion was moderated by the SPC members and the latter tried hard to ensure that the papers were actually discussed. Mostly this was the case, but not always.

Accepting Papers

Accepting papers is nontrivial. After all, there are reviewer biases, and, at the margins, preferences that span fields, ethnic groups, and institutions. This is inevitable since reviewers are human beings. We tried to make the best of it as follows:

  • Papers above a certain score were accepted by default, papers below a certain score were rejected by default. This left a grey zone of around ⅓ of all submissions.
  • We gave special considerations to papers that had at least a single strong accept or a single strong reject to understand why the referees felt this way.
  • The SPC members gave a thorough analysis of all of their batch of papers and made suggestions as to whether to accept a paper. This helped with cross-referee calibration and to homogenize reviews within a given field.
  • The PC skim-read through approximately the top 200 papers (that’s 2,000 pages), i.e. well below the cutoff line, to decide whether their opinion matched that of the SPC members. This was the case in all but three cases where the SPC overruled the PC.

In other words, there were at least three stages of consideration - reviews, SPC and then PC for a paper to pass through. No such process is error-free. Not all authors will agree with their reviews. That said, a misunderstanding by multiple referees might point to deficiencies in writing and in how the idea is being communicated. We encourage the authors to use this feedback to write a great re-submission for a conference or journal later this year.