AI (AI) is obvious to all. Today, AI has reached or exceeded the level of humans in playing Go, playing StarCraft and other games, generating art works, and even optimizing the architecture of microchip . Next, will AI surpass humans in programming?
In fact, automated programming is no longer a new topic. Today there are already some powerful large-scale language models that show their potential in code generation in simple programming tasks such as web design. However, there is still a long way to go to let AI solve more complex and obscure problems through programming.
Recently developed from DeepMind, a system called AlphaCode, can generate code through a language model based on converter, and can create novel solutions for programming problems that require deep reasoning. AlphaCode ranks in the top 54.3% of 10 contest-level programming competitions with more than 5,000 human participants. That is to say, it has reached the average human level.
Recently, the relevant paper was published in Science, and became the cover paper of the current issue [1].
Figure | AlphaCode Related Papers (Source: Science)
makes AI programming have many challenges, especially even for the same problem, the solutions that can be provided may look completely different, and how to make AI judge some correct or incorrect programs is a difficult challenge.
Previously, there were some AI systems that could complete simple programming tasks, such as Codex, which can complete short narrative summary and other tasks. However, when faced with complex programming problems, these AI systems appear to be short of limits. In particular, how to make AI program, using general programming languages such as C++ or Python to generate the entire program to solve long natural language description tasks, has not made much progress in this regard.
To solve this problem, DeepMind sets the following three key components for AlphaCode to achieve a level that can compete with humans in code generation tasks: First, select a wide and concise competitive programming dataset for training and evaluation; Second, adopt a large-scale architecture with efficient sampling capabilities; Third, reduce the exploration space through large-scale model sampling, and filter a group of submissions according to program behavior.
AlphaCode The data set selected during training is a competitive programming data set. This dataset is also a representative set of these complex programming problems. AI not only needs to understand complex natural language descriptions, reason about problems that have not been seen before, but also master a wide range of algorithms and data structures, and be able to accurately implement solutions across hundreds of lines. Furthermore, since hundreds of thousands of programmers around the world are widely involved every year, competitive programming problem sets can better ensure that the best solution shortcuts are found for problems, thus providing a better discriminant benchmark.
Figure | Overview of the AlphaCode system (Source: arXiv)
As shown in the figure above, it is an overview of the AlphaCode system. Simply put, the system work is mainly divided into the following four processes:
First, the system will pre-train the converter-based language model on the GitHub code set with standard language modeling goals. In this way, the model can reasonably locate the problem in human encoding space, thereby greatly reducing the scope of problem search.
Then, the system takes GOLD as the training target and fine-tunes the model on the competitive programming dataset. This can further narrow the search space and can compensate for a small amount of competitive programming data through pre-training.
Next, the system will generate a large number of samples for each problem.
Finally, the system filters these samples and obtains a small group of candidate samples with no more than 10 submissions. The selected samples are then subjected to hidden test evaluations by using sample testing and clustering, etc., and then the samples are selected based on the feedback from the program.
It can be seen that there is a unique setting in AlphaCode, which is to filter after large-scale sampling, which greatly improves the problem-solving rate and can promote faster and more efficient sampling.
To evaluate the performance of AlphaCode, researchers evaluated it on the famous programming competitions Codeforces and CodeContests platforms.
Figure | AlphaCode system ranking in 10 Codeforces (lower the better) (Source: arXiv)
After evaluation, the AlphaCode system ranking in 10 Codeforces competitions can reach the top 54.3%, which is already at the level of an average human programmer.
In summary, the development of the artificial intelligence-assisted coding platform will significantly increase the productivity of programmers . During the same period, the Science commented: "It represents a substantial step forward of the machine learning model. This model can solve some challenging problems by synthesizing computer program ." It is also reported that the author of the paper told the media that although AlphaCode is an important step from 0% to 30%, there is still a lot of work to be done, and the next step will continue to improve AlphaCode.
Reference
1.Yujia Li et al. Science 378, 6624, 1092-1097 (2022).DOI: 10.1126/science.abq1158
