Author | Jiang Baoshang
Editor | Chen Caixian
In the early morning of November 11, the ICLR 2021 preliminary review results were announced on the official website, and it has officially entered the Rebuttal stage. According to the statistical analysis of Criteo AI Lab machine learning research scientist Sergey Ivanov, it is concluded that if the acceptance rate of the paper is 20%, the average score of the paper must reach 6 points or more to be accepted. Another conclusion of
is: the average score of all papers is 5.16 (median is 5.25). This means that authors of papers with lower than average scores should prepare rebuttal materials. Another category of
authors has received clear answers. A netizen on reddit said that his paper had a score of 3 because of the use of commercial software, which increased the entry barrier for reinforcement learning.
1
was rejected for raising the AI threshold?
According to the official website of OpenReview, the paper is labeled 2137, and its main contribution is a data set that can be used for deep data-driven reinforcement learning. Specifically,
designed a benchmark for offline reinforcement learning. The design basis is the key attribute of the real data set. In addition, the author also released benchmark tasks and data sets, conducted a comprehensive evaluation of existing algorithms and evaluation protocols, and provided an open source code base.
At present, this article has 4 reviewers and a field chairperson for scoring, and got two 6 points, one 3 points and one 2 points. Among them, 2 points and 3 points represent "strong rejection".
As mentioned earlier, the reason why a reviewer gave 3 points is: MuJoCo is used, which is a commercial software package. The reviewer said: "Half of the six tasks (Maze2D, AntMaze, Gym-mujoco) rely heavily on the MuJoCo simulator, which is a commercial software and is not free even academically. In addition, the cost of a personal MuJoCo license It’s $500 per year. Therefore, I’m worried that most researchers will not be able to access MuJoCo.”
In addition, the review said: “In view of the potential high influence of MuJoCo, this article will indeed greatly promote the use of MuJoCo and make RL more privileged (meaning Yes, there are barriers to use). If the accessibility (easy-availability) problem is solved, such as using PyBullet as the engine, I would be happy to increase my score.
Regarding this review opinion, a reader named Rasool Fakoor holds Objections: 1. Agree with the reviewer’s statement about “licensing”, but this is not a reason for rejecting the paper. The review ignored the true contribution of the paper. 2. The review requires PyBullet to build a benchmark, but this may take several months 3. This kind of reason for refusal will only undermine the confidence of researchers to invest in establishing such benchmarks, which obviously has a negative impact on the field. After discussion,
has also changed the attitude of the 6-point judges. He said: When I was writing my review, I didn’t realize that this benchmark test requires commercial software (Mujoco). I totally agree with the view that "this is very unfavorable for standardized benchmark tests." Z2z
In addition, the 2 points of review did not focus. On the topic of "commercial software", his reason is: Although the data set provided by the author is very useful for offline reinforcement learning researchers, there is no progress in new ideas. In general, the main part of this work The contribution is: collecting offline data somewhere, reducing the time required for other researchers to do so. Therefore, apart from labeling the data, the author does not seem to do any important work. So, I don’t think this The work should be published in a "high-end conference."
Finally, the domain chairperson also expressed his opinion: the issue of license (accessibility) is understandable, but don't make a big fuss about this, the most important thing is to consider the latest article. Where is the contribution of the author. Just like the evaluation given by the first reviewer (giving 2 points), we will consider all factors to make a decision.
In other words, according to the recommendations of the 2-point judges, this article is probably It’s cold.
And on reddit, most netizens are sympathetic: it’s a fool to equate the repeatability of science with the repeatability of anyone.stupid!
2
"Tucao Conference"
This ICLR 2021 has a total of 3013 papers submitted, of which 856 papers were resubmitted after NeurIPS 2020 Rejection.
recalled the review of ICLR 2020 last year, which can be described as complaining and controversial.
For example, after an ICLR 2020 paper received a full score evaluation, the other two reviewers gave two consecutive 1-point evaluations, and the other three reviewers gave 6-6-6. However, the regional chairperson made comments that did not apply to his paper.
In addition, last year, Professor Zhou Zhihua of Nanjing University revealed that 47% of ICLR 2020 reviewers have never published a paper in this field.
Later, Professor Zhou pointed out: The open review is only effective when the participants are experts at a certain level, otherwise it is easier to be misled. Academic judgment cannot be "equal". The insights and judgments of ordinary practitioners and high-level experts are not the same. It is precisely because there are high-level experts to check, but now it is impossible... ..
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-