compiled from: https://cacm.acm.org/magazines/2017/10/221326-a-large-scale-study-of-programming-languages-and-code-quality-in-GitHub/fulltext?imm_mid=0f7103cmp=em-prog-na-na-newsltr_20171007
Author: Acm
Translator: qhwdw
What is the impact of programming language on software quality? This issue has become a subject that has caused a lot of debate for a long time. In this study, we collected a large amount of data from GitHub (728 projects, 63 million lines of source code, 29,000 authors, 1.5 million submissions, 17 programming languages) to try to provide some empirical evidence on this problem. This relatively large sample size allows us to use a mixed method, combining multiple visual regression models and text analysis to study the impact of language characteristics, such as in software quality, static and dynamic types and types that allow confusion and non-confusion. By conducting triangulation studies from different methods (LCTT translation note: a method of measurement research) and controlling confusing factors, such as team size, project size and project history, our report shows that language design does have a great impact (in many aspects), but the impact of language is very limited in terms of software quality. The most obvious thing seems to be that types that do not allow confusion are slightly better than types that allow confusion, and, in functional languages, static types are also better than dynamic types. It is worth noting that most of these slight effects caused by language design are dominated by process factors, such as project size, team size, and number of submissions. However, we need to remind readers that even these inconspicuous slight effects are caused by other intangible process factors, such as preference for certain function types, and static languages that do not allow type confusion.
1 Preface
During the discussion of whether a given programming language is the "right tool for this job", multiple debates followed. Although some debates have shown religious fanaticism, most people agree that the choice of programming language can have an impact on the encoding process and the resulting results.
People who advocate strong static types tend to think that static methods can capture defects early; they believe that a little prevention is better than a lot of corrections. Dynamic type advocates argue that conservative static type checking is a waste of developer resources anyway, and it is best to rely on strong dynamic type checking to capture error types. However, most of these debates are "paper talk" and are supported only by "legendary" evidence.
These "legends" may not be unreasonable; taking into account the large number of other factors that affect the results of software engineering, obtaining such empirical evidence support is a challenging task, such as code quality, language characteristics, and application areas. For example, software quality, considering that it has a large number of well-known influencing factors such as code quantity, 6 team size, 2 and age/proficiency. 9
controlled experiments are one way to test language selection in the face of such a discouraging confusion effect, however, such studies often introduce a kind of confusion of their own, that is, limiting the scope due to cost. In this kind of research, the complete task must be limited and cannot simulate development in the real world. Here are several recent studies of such undergraduates using, or, through an experimental factor, to compare static or dynamic types of languages. 7 , 12 , 15
Luckily, now we can study these problems based on a large number of real-world software projects. GitHub contains a large number of projects in multiple languages and varies greatly in size, age, and number of developers. Each project's repository provides a detailed record of contribution history, project size, authorship identity, and defect repair. We then use a variety of tools to study the impact of language characteristics on defect occurrence.The best description of our research method should be "mixed approach", or triangulation; 5 We use text analysis, clustering and visualization to confirm and support the results of quantitative regression studies. This experience-based approach helps us understand the specific impact of programming languages on software quality, because they are used informally by developers.
2 Method
Our method is a typical large-scale observation and research method in software engineering. We first used a large number of automated methods to collect data from several data sources. The data were then filtered and cleaned using a pre-constructed statistical analysis model. The choice of filters is driven by a series of factors including the nature of the problem we studied, the data quality and the data that we believe is best suited for this statistical analysis study. In particular, GitHub contains a lot of projects written by a large number of programming languages. For this study, we spent a lot of effort focusing on collecting data on popular projects written in most mainstream programming languages. We chose the appropriate method to evaluate the influencing factors on the counting data.
2.1 Data Collection
We chose the top 19 programming languages on GitHub. CSS, Shell scripts, and Vim scripts were excluded because they are not common programming languages. We include TypeScript 5, which is a superset of JavaScript. Then, for each of the studied programming languages, we searched for the top 50 projects with it as the main programming language. We analyzed 17 different languages in total, with a total of 850 projects.
Our programming language and project data is extracted from GitHub Archive, a database that records all active public GitHub projects. It records 18 different GitHub events, including new commits, fork events, PRs (pull requests), developer information and issue tracking for all open source GitHub projects on an hourly basis. Upload the packaged data to the interactive data analysis interface provided by Google BigQuery.
Identification Programming Language Ranking List
We summarize items based on their main programming language classification. We then selected most of the projects for further analysis, as shown in Table 1. A project may use multiple programming languages; it is difficult to determine it as a single programming language. The information saved by Github Archive is collected from GitHub Linguist, which uses the extension of the source file in the project repository to determine what the project's publishing language is. The most used programming language in the source file was identified as the main programming language for this project.
Table 1 Top three projects for each programming language
Search for popular projects
For each selected programming language, we first select the project based on the main programming language used by the project, and then rank the popularity of the project according to the number of relevant stars for each project. The number of stars indicates how many people actively expressed interest in the project and it is an appropriate indicator of popularity. Therefore, the top three projects in C are linux, git, and php-src; for C++, they are node-webkit, phantomjs, and mongo; for Java, they are storm, elasticsearch, and ActionBarSherlock. We selected 50 projects for each programming language.
To ensure that each project has a long enough development history, we eliminated less than 28 submitted projects (28 is the number of first quartile values for the candidate). This way we have 728 projects left. Table 1 shows the first three projects for each programming language.
Retrieve project evolution history
For each of the 728 projects, we downloaded their non-merge submissions, submission records, author data, author names using git. We calculate the code changes and the number of modified files per commit from the number of lines added and deleted for each file. We search for the programming language used by the programming language represented by the extension of the file modified in each commit (a commit may have multiple programming language tags). For each commit, we calculate its commit age by subtracting the date of the first commit of the item by subtracting its commit date.We also calculate other project-related statistics, including the maximum submission age of the project and the total number of developers, for the control variables used for our regression analysis model, which is discussed in Section 3. We identify bug fix submissions by searching for keywords related to errors in the submission record, such as error, bug, fix, issue, mistake, incorrect, fault, defect, flaw. This is similar to previous studies. 18
Table 2 summarizes our dataset. Because a project may use multiple programming languages, the second column of the table shows the total number of projects using a certain programming language. We further excluded those programming languages in the project that had fewer than 20 submissions. Because 20 is the first quartile value of the total number of submissions for each project in each programming language. For example, we found more than 20 submissions in C. This ensures that each "programming language-project" pair is sufficiently active.
Table 2 Research topics
In summary, we studied 728 projects developed in 17 programming languages over the past 18 years. In total, 29,000 different developers, 1.57 million commits, and 564,625 bug fix commits were included.
2.2 Language Classification
We define language categories based on several programming language characteristics that affect language quality, 7, 8, 12, as shown in Table 3.
Programming Paradigm means whether the project is written in command, script, or functional language. In the following part of this article, we use the terms "procedural" and "scripting" to represent command methods and scripting methods respectively.
Table 3. Different types of language classification
type checking (Type Checking) represents static or dynamic types. In statically typed languages, type checking is performed at compile time, and the variable name is bound to a value and a type. In addition, expressions (including variables) are classified according to the type of values they may produce at runtime. In dynamic typed languages, type checking occurs at runtime. Therefore, in a dynamically typed language, it may appear in the same program and a variable name may be bound to objects of different types.
Implicit Type Conversion allows an operand of type T1 to be accessed as another different type T2 without explicit type conversion. Such implicit type conversion can cause type confusion in some cases, especially when it represents an explicit operand of T1 type, and then use it as another different T2 type. Because not all implicit type conversions will immediately have problems, we show our definition through examples that implicit type conversion may occur in all programming languages that we identified. For example, in programming languages like Perl, JavaScript, CoffeeScript, adding a character and a number is allowed (for example, "5" + 2 results in "52"). But in Php, the same operation, the result is 7. Operations like this are not allowed in some programming languages, such as Java and Python, because they do not allow implicit conversions. In strong data types C and C++, the results of this operation are unpredictable. For example, int x; float y; y=3.5; x=y; is legal C code, and the result for x and y is different. Which value depends on the meaning, which may cause problems later. a In Objective-Ch, the data type id is a general object pointer that can be used for any data type object, regardless of the classification. b General data types like this offer great flexibility, which can lead to implicit type conversions and also have unpredictable results. c Therefore, we classify programming languages based on whether its compiler allows or does not allow implicit type conversion as described above; programming languages that do not allow implicit type conversion will explicitly detect type obfuscation and report type mismatch errors.
does not allow implicit type conversion programming languages, using a type judgment algorithm, such as Hindley 10 and Milner, 17, or using a dynamic type checker at runtime, which can judge the results of static types in a compiler (such as Java). In contrast, a type of confusion may occur silently because, it may be because it is not detected or reported. Either way, allowing implicit type conversions can also end up with errors that are difficult to determine the cause. For simplicity, we will use implicit to represent programming languages that allow implicit type conversion, and not implicit type conversion. We will use explicit to represent them.
Memory Classification (Memory Class) indicates whether developers are required to manage memory. Although Objective-C follows a hybrid pattern, we still treat it in a non-management classification because, we observe a lot of memory errors in its code base, which will be discussed in RQ4 in Section 3.
Please note that we use this method to classify and study programming languages because this method is used informally by a large number of developers in a "real world". For example, TypeScript is intentionally classified into a static programming language classification, which does not allow implicit type conversion. However, in practice, we noticed that developers often use any type (with 50% of variables, and across TypeScript – projects used in our dataset) which is a general union type, and, therefore, in practice, TypeScript allows dynamic, implicit type conversion. To reduce confusion, we exclude TypeScript from our programming language classification and related models (see Tables 3 and 7).
2.3 Identify project fields
We use an automatic plus manual hybrid technology to classify the research projects into different fields based on the characteristics and functions of the programming language. On GitHub, projects use project descriptions and README files to describe their characteristics. We use a document topic generation model (Latent Dirichlet Allocation, abbreviated as: LDA) 3 to analyze these texts. Provide a set of documents to it, and the LDA will generate different keywords and then identify possible topics. For each document, LDA also estimates the probability of documents assigned to each topic.
We detected 30 different fields (in other words, topics) and evaluated the probability that each item belongs to each field. Because these areas of automatic detection include keywords for several specific projects, such as Facebook, it is difficult to define its underlying common functions. To assign each field a meaningful name, we manually checked 30 domain identification keywords for the domain that were not related to the project name. We manually renamed all 30 areas of automatic detection and found most of the projects in the following six areas: applications, databases, code analysis, middleware, libraries, and frameworks. We also found some projects that did not meet any of the above areas, so we generally labeled this area as others. Subsequently, another member of our study group examined and confirmed the way this project domain classification was classified. Table 4 summarizes the domain results identified by this process.
Table 4 Domain features
2.4 bug classification
Although when fixing software bugs, developers often leave original important information about this bug in the submission log; for example, why a bug occurs and how to fix a bug. We use a lot of information to classify bugs, similar to Tan's et al. 13, 24
First of all, we classify based on the cause (Cause) and impact (Impact) of the bug. _ Cause _ Further decompose into irrelevant error subclasses: algorithmic, concurrency, memory, ordinary programming errors, and unknown. The impact of bugs is also divided into four unrelated subclasses: security, performance, failure, and other unknown classes. Therefore, each bug fix commit also contains the types of causes and impact. Table 5 shows each bug classification described.This category is performed in two stages:
Table 5 bug classification and description in the entire dataset
(1) keyword search We randomly selected 10% of bug fix information and used a keyword-based search technology to automatically classify them as possible bug types. We use this annotation for both types (cause and impact). We selected a limited set of keywords and idioms, as shown in Table 5. Limited keywords and idioms like this can help us reduce false positives.
(2) Supervised classification We use the annotated bug fix log in the previous steps as training data to supervise the classification technology, correct the remaining bug fix information through test data. We first convert each bug fix information into a bag of words (LCTT translation note: bag-of-words, an information retrieval model). Then, delete words that only appear once in all bug fix messages. This reduces the keywords for specific projects. We also use standard natural language processing techniques to solve this problem. Ultimately, we use Support Vector Machine (LCTT translation note: Support Vector Machine, abbreviated as SVM, in the field of machine learning, to classify test data.
To accurately evaluate the bug classifier, we manually annotated 180 randomly selected bug fixes, distributed evenly across all categories. We then compare the results of manually annotated datasets in the automatic classifier. After final processing, the accuracy shown is acceptable, with the lowest accuracy in performance, 70%, and the highest accuracy in concurrency error, 100%, with an average of 84%. Run again, the accuracy from low to high is 69% to 91%, and the average accuracy is 84%.
The results of our bug classification are shown in Table 5. The causes of most defects are related to ordinary programming errors. This result is not surprising, because a large number of programming errors are involved in this category, such as type errors, input errors, writing errors, etc. Our technology cannot reclassify bug fix information that accounts for 1.4% of any (cause or impact) classification; we classify it as unknown.
2.5 Statistical Methods
We used a regression model to model the number of defective submissions in other factors related to the software project. All models use negative binomial regression (NBR) (LCTT translation note: a regression analysis model) to model project attribute counts, such as the number of submissions. NBR is a generalized linear model for responsive modeling of non-negative integers. 4
In our model, we control several factors that may affect the final result for each project’s programming language. Therefore, in our regression analysis, each (language/project) pair is a row and can be considered as a sample from a popular open source project. We convert objects based on variable counts to keep the variable stable and improve the applicability of the model. 4 We validate them by testing comparisons of non-nested models using AIC and Vuong.
to check those excessive multicollinearities (LCTT translation note: Multicollinearity refers to the fact that the model estimation is distorted or difficult to estimate accurately due to the precise or high correlation between explanatory variables in the linear regression model.) It is not a problem. We use a conservative maximum value of 5 in all models to calculate the variance of the expansion factor of each dependent variable. 4 We removed the high leverage points by visually examining the residual and leverage graphs of each model to find the separation and maximum values of the Cook distance (LCTT translation note: a statistical term used to diagnose the presence of abnormal data in regression analysis).
We use effects, or differences, to encode them into our research to improve the performance of programming language regression coefficients. 4 Weighted Effect Code allows us to compare the effects of each programming language with all programming languages, while compensating for the inhomogeneity of programming languages used across projects.23 To test the relationship between two variable factors, we used an independent chi-square test (LCTT translation note: Chi-square, a statistical hypothesis test method). 14 After confirming a dependency, we use Cramer's V, which is the phi(φ) coefficient of normal data equivalent to r × c, to create an effect data.
3 Results
We started with a simple and clear problem, which solved some core problems that people firmly believed in, namely:
problem 1: Are some programming languages more prone to defects than other languages?
We used a regression analysis model to compare the impact of each programming language on the average number of defects in all programming languages and its impact on defect repair submissions (see Table 6).
Table 6. Some languages have fewer defects than other languages
. We include some variables as control factors that clearly affect the response. Project age (age) is also included because the older the project generates a larger number of defect fixes. The number of commits will also have a slight impact on project response. In addition, the number of developers (devs) working on the project and the original size (size) of the project will increase as the project is active.
The size and symbol of the estimated coefficient in the above model (LCTT translation note: refers to "+" or "-") is related to the predictor of the result. The initial four variables are control variables, and we are not interested in the effect of these variables on the final result, just saying that they are all positive and meaningful. Language variables are indicator variables, which are the variable factors for each project, which compare each programming language with the weighted averages for the programming languages for all projects. Programming language coefficients can be roughly divided into three categories. The first category is that those coefficients that are statistically insignificant and cannot be distinguished from 0 during the modeling process. These programming languages perform similarly to the mean, or they may also have greater variance. The remaining coefficients are very obvious, either positive or negative. For those positive coefficients, we guess that it may be related to the large number of defect fixes in this programming language. These languages include C, C++, Objective-C, Php, and Python. All programming languages with a negative coefficient, such as Clojure, Haskell, Ruby, and Scala, suggest that the defect fix commits of these languages may be less than the average.
It should be noted that although there is a clear connection between programming languages and defects from a statistical perspective, don’t overestimate the impact of programming languages on defects, because this impact effect is very small. The results of the anomaly analysis show that this effect is less than 1% of the total anomaly.
ut1.jpg
We can understand the coefficient of the model in this way, which represents a predictor whose unit changes in a predictor reflects a logarithmic change in the expected response while all other predictors remain unchanged; in other words, for a coefficient βi, a unit change in βi produces an expected change in the eβi response. For variable factors, this expected change is compared with the average of all programming languages. Therefore, if a specific project developed in a programming language with an average value has four defect commits for a certain number of commits, then if you choose to develop in C++, it means we expect that there should be an additional (LCTT translation note: 1 more relative to the average value of 4) defect commits, because e0.18 × 4 = 4.79. For the same project, if you choose to use Haskell for development, it means we expect one less defect submission (LCTT translation note: supra, 4 relative to the average). Because, e−0.26 × 4 = 3.08. The accuracy of the prediction depends on the remaining other factors remain the same, and all of them are a very challenging proposition except for those trivial items. All observational studies face similar limitations; we will address these matters in detail in Section 5.
Conclusion 1: Some programming languages have higher defect correlation than other programming languages, but the impact is very small.
In the rest of this article, we will elaborate on the basic conclusions. By considering different types of applications, defects, and programming languages, we can further understand the relationship between programming languages and defect tendencies.
software bugs usually fall into two broad categories:
- domain-specific bugs: project-specific features and does not depend on the underlying programming language.
- Normal bug: Most ordinary bugs are born and have nothing to do with project functions, such as input errors, concurrent errors, etc.
Therefore, in a project, the application domain and programming language interaction may affect the number of defects, the conclusion is considered reasonable. Because some programming languages are considered to be more prominent in some tasks than others, for example, C for low-level (lower) work, or Java for user applications, an inappropriate choice for programming languages may bring more flaws. To study this situation, we will ideally ignore domain-specific bugs, because ordinary bugs rely more on the characteristics of programming languages. However, because a domain-specific bug can also occur in a normal programming error, it is difficult to distinguish between the two. One possible workaround is to study programming languages while controlling the field. From a statistical point of view, while, using 17 programming languages across 7 domains, understanding large numbers of terms will be a huge challenge in a given sample size.
In view of this situation, we first consider testing the dependencies between the domain and programming language usage in a project, using a Chi-square test independently. Of the 119 units, it is 46, which means 39%, which is above the conservative value we set, which is too high. This number cannot exceed 20%, and should be less than 5.14 We include the full value here; d However, the value tested by Cramer is 0.191, which is low correlation, indicating that the correlation between any programming language and domain is very small, and that including domains in the regression model does not produce meaningful results. One option to solve this situation is to remove programming languages, or mix fields, but our existing data is not fully selected. Or, we mix programming languages; this choice leads to a related but slightly different problem.
Question 2: Which programming language features are related to defects?
We aggregate them by programming language category rather than considering individual programming languages, as described in Section 2.2, and then analyze the relationship with defects. Overall, each of these properties divides the programming language into errors often discussed in the context, user debate-driven, or by previous work topics. Therefore, individual properties are highly correlated, and we created six model factors that synthesize all individual factors into our study. We then model the effect of six different factors on the number of defects while controlling for the same basic covariates in the model we used in Section 1 of Question (LCTT) (LCTT) Note: Covariates refer to independent variables that cannot be artificially manipulated in the experiment).
Regarding the programming language used (in the previous Table 6), we use the average response across all language classes to compare programming language classes. This model is expressed in Table 7. It is obvious that the Script-Dynamic-Explicit-Managed class has the smallest coefficient of magnitude. This coefficient is trivial. In other words, the Z-test of this coefficient cannot be distinguished from 0. Given the order of standard errors, we can assume that the behavior of programming languages in this category is very close to the average of all programming language behaviors. We can prove this by using Proc-Static-Implicit-Unmanaged as the base level and for processing, or using the base level to falsely encode and compare each language class. In this case, Script-Dynamic-Explicit-Managed is significantly different from p = 0.00044.Note that although we chose different encoding methods in this way, affecting the coefficients and Z values, this method is the same as all other aspects. When we change the encoding, we adjust the coefficients to reflect the comparison we want to generate. 4 Comparing other classes of programming languages with the overall average, the Proc-Static-Implicit-Unmanaged programming language is more likely to cause defects. This means that implicit type conversion or managing memory will lead to more flaws than other process programming languages.
Table 7. Functional languages have lower correlation with defects than other languages, while procedural languages are greater or close to the average.
In script-class programming languages, we observe a relationship similar to that between programming languages that allow and do not allow implicit type conversion, which provides some evidence that implicit type conversion (compared to explicit type conversion) is the reason for this difference, not memory management. Given the correlation between various factors, we cannot draw this conclusion. However, when they are compared with the mean, as a group, programming languages that do not allow implicit type conversion have a lower tendency to have errors, while programming languages that have a higher tendency to have errors have a relatively higher chance of errors. The difference between static and dynamic types is also obvious in functional programming languages.
Functional language as a group shows a very obvious difference from the average. The coefficients of statically typed languages are much smaller, but functional language classes have the same standard errors. The tendency of functional static programming languages to have less errors than functional dynamic programming languages is strong evidence, though, Z-checking is merely to test whether the coefficients can be distinguished from 0. To strengthen this inference, we recoded the model above using processing encoding and observed that the error tendency of the Functional-Static-Explicit-Managed programming language class is significantly smaller than p = 0.034 for the Functional-Dynamic-Explicit-Managed programming language class.
ut2.jpg
Like programming languages and defects, the impact of the relationship between programming language classes and defects is very small. This difference in interpreted programming languages is also similar, although small, and this difference in interpreted programming languages is less than 1%.
We are now back to the application field. Does the application field affect each other with language classes? How to choose? For example, a functional programming language has certain advantages for specific fields? As above, a chi-square test is done for the relationship between these factors and the project domain, with a value of 99.05, df = 30, p = 2.622e–09, and we reject the meaningless assumption that the V of Cramer produces a value of 0.133, indicating a weak association. Therefore, although there are some correlations between the domain and the programming language, there is only a very weak correlation between the application domain and the programming language class here.
Conclusion 2: There is a small but obvious relationship between programming language classes and defects. Compared with procedural or scripted languages, functional languages have fewer defects. The conclusion of
is somewhat unsatisfactory because we do not have strong evidence to prove the correlation between programming languages or language classes and application fields in a project. An alternative is to ignore the total number of projects and defects based on all programming languages and application areas and view the same data. Because, in this way, no longer produces irrelevant samples, we do not try to analyze them from a statistical perspective, but use a descriptive, visual-based method.
We define Defect Proneness as the ratio of bug fix commits to total commits per language per domain. Figure 1 uses a heat map to show the interaction between the application domain and the programming language, with the tendency to represent defects from light to dark. We examined which programming language factors influenced the defect repair submission of projects written across multiple languages. It leads to the following research questions:
Figure 1. Interaction between programming language defect tendencies and application fields.For a given field (bottom of column), each grid in the heat map represents a programming language's flaw tendency (row header). The "whole" column represents a programming language's tendency to be based on all domains. A grid marked with a white crosshair represents a null value, in other words, a commit that does not match in that grid.
Question 3: Does the programming language’s error tendency depend on the application field?
To answer this question, we first filter out items considered anomalies in our regression model with high leverage points. This method is necessary here, although this is a non-statistical approach, and some relationships may affect visualization. For example, we find a simple project, Google v8, a JavaScript project, responsible for all errors in the middleware. This was a surprise for us, as JavaScript is not usually used in middleware. This model has been repeated continuously in other applications, so the defects of the projects we filtered out are below 10% and above 90%. This result is in Figure 1.
We see only a small difference in this heat map, as seen in question 1, this result only represents the inherent misleading tendency of programming languages. To verify this inference, we measured the flaw tendency of programming languages to each application area and to all application areas. For all domains except the database, the correlation is positive and the p-value makes sense (0.01). Therefore, with regard to defect tendencies, the language sorting in each field is basically the same as the language sorting in all fields.
ut3.jpg
Conclusion 3: There is no overall connection between the application field and the tendency to deficiencies in programming languages.
We demonstrate that different languages have a large number of flaws, and that this relationship is not only related to a specific language, but also applies to general language classes; however, we find that project types cannot coordinate this relationship to a certain extent. Now that we change our attention to reaction classification, I want to understand what the connection is between programming languages and specific kinds of defects, and how this relationship is compared with the more common relationship we observe. We divide the defects into different categories, as described in Table 5, and then ask the following question:
question 4: What is the relationship between programming languages and bug classification?
We used a method similar to that used in question 3 to understand the relationship between programming languages and bug classifications. First, we examine the relationship between bug classification and programming language classes. A heatmap (Figure 2) shows the total number of defects on top of programming language classes and bug types. To understand the interaction between bug classification and language, we use an NBR regression model for each category. For each model, we used the same control factors as in question 1, and the language encoded with weighted effects to predict defect repair submissions.
Figure 2. The relationship between bug category and programming language class. Each grid represents the percentage of all bug fix commits per language class (row header) per bug category (bottom of column). This value is normalized by columns. The results of
and the ANOVA value of the programming language are shown in Table 8. The overall exceptions of each model are very small, and for specific defect types, the proportions displayed by language are similar in the order of magnitude in most categories. We explain this relationship as the impact of programming languages on specific bug categories is greater than the overall impact. Although our conclusion summarizes all categories, in the following section, we further study the bug categories with a large number of bugs reflected in Table 5.
Table 8. Although the impact of programming languages on defects varies by defect category, programming languages have greater impact on specific categories than general categories.
Programming Error Normal programming errors account for about 88.53% of all bug fix submissions and are available in all programming language classes. Therefore, the regression analysis gives a conclusion similar to that in question 1 (see Table 6).All programming languages can cause such programming errors, such as processing errors, definition errors, input errors, etc.
Memory Error Memory Errors account for 5.44% of all bug fix commits. Thermal diagram Figure 2 demonstrates a very close connection between the Proc-Static-Implicit-Unmanaged class and memory errors. A memory bug occurred in programming languages that do not manage memory, which is expected. Table 8 also demonstrates this, for example, C, C++, and Objective-C raise many memory errors. Java raises more memory errors in languages that manage memory, although it is less than programming languages that do not manage memory. Although Java does memory recycle by itself, it is not surprising that it has memory leaks, because object references often prevent memory recycle. 11 In our data, of all memory errors in Java, 28.89% are caused by memory leaks. In terms of quantity, memory defects in programming languages have a much greater impact than other categories.
Concurrent Errors In the total bug fix commits, the repair commits related to concurrent errors account for 1.99%. Thermal graph shows that Proc-Static-Implicit-Unmanaged is the main error type. Among this error, C and C++ account for 19.15% and 7.89% respectively, and they are distributed across various projects.
ut4.jpg
Programming languages that belong to the Static-Strong-Managed language class have been proven to be in the dark area in the heat map, and ordinary static languages produce more concurrent errors than other languages. In dynamic languages, Erlang only has more tendencies to concurrent errors, which may be related to the many concurrent applications developed using this language. Similarly, the negative coefficients in Table 8 indicate that projects written in dynamic programming languages such as Ruby and `Php have fewer concurrent errors. Note that some languages, such as JavaScript, CoffeeScript, and TypeScript, do not support concurrency, although Php has limited concurrency support (depending on its implementation). These languages introduce false 0s into our data, so the coefficients of the concurrency model of these languages in Table 8 cannot be explained like other languages. Because of these false 0s, the average of all languages in this model is very small, which may affect the magnitude of the coefficients, so they give w.r.t. an average, however, this does not affect their relative relationship, because we only focus on their relative relationship.
Text analysis of high-frequency words in bug-fixed messages shows that most concurrency errors occur in a conditional contention, deadlock, or incorrect synchronization, as shown in the table above. Iterating through all languages, conditional contention is the reason why concurrent errors occur the most, for example, 92% in Go. The improvement in conditional contention errors in Go may be due to the use of a contention detection tool to help developers locate contention. Synchronization errors are mainly related to Message Passing Interface (MPI) or Shared Memory Operations (SHM). Erlang and Go use MPI e for inter-thread communication, which is why there are no SHM-related errors in both languages, such as shared locks, mutex locks, etc. In contrast, projects written in languages with earlier SHM technology for inter-thread communication may have lock-related errors.
security and other conflict errors Among all bug fix commits, commits related to conflict (Impact) errors account for about 7.33%. Among them, Erlang, C++, and Python security-related errors are higher than the average (Table 8). There are fewer security errors related to the Clojure project (Figure 2). We can also see from the thermal diagram that static languages are generally more prone to failures and performance errors, followed by the Functional-Dynamic-Explicit-Managed language, such as Erlang. Analysis of abnormal results shows that programming languages are closely related to conflict failure. Although security errors are weakly correlated in this category, the differences in interpreted languages are still relatively large compared to residuals.
Conclusion 4: Defect types are strongly related to programming languages; some defect types such as memory errors and concurrency errors also depend on early languages (technology used). For a specific category, programming languages cause more errors than overall.
4. Related work
The work done before comparing programming languages is divided into the following three categories:
(1) Controlled experiment
For a given task, developers are monitored when programming in different languages. Research and compare results, such as development results and code quality. Hanenberg 7 developed a parser that monitored 48 programmers for 27 hours to compare static and dynamic types. He found that there was no significant difference in code quality between the two, but dynamically-typed languages took less development time. Their research was conducted in a lab using undergraduate students, setting up custom languages and IDEs. Our research is exactly the opposite, and is a practical study of popular software applications. While we are able to control confounders only by using regression models (and post hockey) control, our advantage is that the sample size is large and more realistic and widely used software. We found that under the same conditions, statically typed languages have less error tendency than dynamically typed languages, and that languages that do not allow implicit type conversion are better than languages that allow implicit type conversion, which has a very small impact on the results. This is reasonable because the sample size is very large, so this very small effect can be seen in this study.
Harrison et al. 8 compared C++ with SML, one is a procedural programming language and the other is a functional programming language. There was no significant difference in the total number of errors, but SML has a higher defect density than C++. SML is not reflected in our data, but it is believed that functional programming languages have fewer defects than procedural programming languages. Another key task is to compare development work across different languages. 12 , 20 However, they do not analyze programming language flaw tendencies.
(2) Investigation
Meyerovich and Rabkin 16 investigated developers' views on programming languages to study why some programming languages are more popular than others. Their report notes that non-programming language factors have a huge impact: previous programming language skills, available open source tools, and existing old-fashioned systems. Our research also demonstrates that available external tools also affect software quality; for example, concurrent bugs in Go (please check out the contents of issue 4).
(3) Mining of software repositories
Bhattacharya and Neamtiu 1 studied four projects developed with C and C++, and found that components developed in C++ are generally more reliable than those developed in C. We found that C and C++ have higher error tendencies than the average for all programming languages. However, for some bug types, like concurrent errors, C has a higher flaw tendency than C++ (see Issue 4 in Section 3).
5. Valid Risk
We believe that the conclusions of our report are almost risky. First, when it comes to identifying bug fix commits, the keywords we rely on are keywords that developers often use to represent bug fixes. Our choices were carefully considered. In an ongoing development process, we capture the problems that developers have been facing, rather than the bugs they reported. However, this option has the risk of overestimating. We categorize domains to explain the tendency to defects, and other members of our study group have verified classification. In addition, we spend our energy on classifying bug fix submissions, which may also be at risk of being contaminated by the keywords that were originally selected. Each project is also different in the description of the commit log. To mitigate these risks, we evaluated our categories using manual annotations, as described in Section 2.4.
We judge that the programming language to which the file belongs is based on the file's extension. If files written in different programming languages use the extension of the common programming language file we studied, it is also prone to error tendencies in this case. To reduce this error, we manually verified our language classification using a random sample file set.
Based on the situation shown in our dataset, we made some assumptions based on the main uses of programming language properties in Section 2.2.For example, we divide Objective-C into non-managed memory types, rather than mixed types. Similarly, we annotate Scala as a functional programming language and C# as a procedural programming language, although both support them in terms of design choices. 19 , 21 In this research work, we did not separate object-oriented programming languages (OOPs) from procedural programming languages because, there is no clear difference between them, and the main difference is in the programming type. We classify C++ into categories that allow implicit type conversion because some types of memory areas can be processed differently by using pointer operations, 22 and we noticed that most C++ compilers can detect type errors at compile time.
Finally, we associate the defect fix commits to programming language properties, which can reflect the style of the report or other developers' attributes. Available external tools or libraries can also affect the number of bugs in a related programming language.
6. Summary
We conducted a large-scale study of programming languages and usage because it involves software quality. The data we use on Github has high complexity and differentiation in multiple dimensions. Our sample size allows us to conduct a mixed method study on the effects of programming languages and the interactions between programming languages, application areas and defect types while controlling for some confounding factors. Research data shows that functional languages are better than procedural languages; languages that do not allow implicit type conversion are better than languages that allow implicit type conversion; languages that are statically typed are better than languages that are dynamically typed; and languages that manage memory are better than non-managed languages. Furthermore, the flaw tendency of programming languages has no connection with the software application field. In addition, each programming language is more associated with a specific bug category than with all bugs.
On the other hand, even large-scale data sets, they become small and incomplete after being segmented by multiple methods at the same time. Therefore, as more and more variables depend on it, it is difficult to answer the question of how big the impact a variable has, especially when the variables interact. Therefore, we cannot quantify the specific effects of programming languages in use. Other methods, such as investigations, may be helpful. We will address these challenges in future work.
Acknowledgements
This material was completed under the authorization and support of the National Science Foundation (NSF) and the Office of Scientific Research (AFOSR). Authorization number 1445079, 1247280, 1414172, 1446683, FA955-11-1-0246.
compiled from: https://cacm.acm.org/magazines/2017/10/221326-a-large-scale-study-of-programming-languages-and-code-quality-in-GitHub/fulltext?imm_mid=0f7103cmp=em-prog-na-na-newsltr_20171007
Author: Acm
Translator: qhwdw
What is the impact of programming language on software quality? This issue has become a subject that has caused a lot of debate for a long time. In this study, we collected a large amount of data from GitHub (728 projects, 63 million lines of source code, 29,000 authors, 1.5 million submissions, 17 programming languages) to try to provide some empirical evidence on this problem. This relatively large sample size allows us to use a mixed method, combining multiple visual regression models and text analysis to study the impact of language characteristics, such as in software quality, static and dynamic types and types that allow confusion and non-confusion. By conducting triangulation studies from different methods (LCTT translation note: a method of measurement research) and controlling confusing factors, such as team size, project size and project history, our report shows that language design does have a great impact (in many aspects), but the impact of language is very limited in terms of software quality. The most obvious thing seems to be that types that do not allow confusion are slightly better than types that allow confusion, and, in functional languages, static types are also better than dynamic types. It is worth noting that most of these slight effects caused by language design are dominated by process factors, such as project size, team size, and number of submissions. However, we need to remind readers that even these inconspicuous slight effects are caused by other intangible process factors, such as preference for certain function types, and static languages that do not allow type confusion.
1 Preface
During the discussion of whether a given programming language is the "right tool for this job", multiple debates followed. Although some debates have shown religious fanaticism, most people agree that the choice of programming language can have an impact on the encoding process and the resulting results.
People who advocate strong static types tend to think that static methods can capture defects early; they believe that a little prevention is better than a lot of corrections. Dynamic type advocates argue that conservative static type checking is a waste of developer resources anyway, and it is best to rely on strong dynamic type checking to capture error types. However, most of these debates are "paper talk" and are supported only by "legendary" evidence.
These "legends" may not be unreasonable; taking into account the large number of other factors that affect the results of software engineering, obtaining such empirical evidence support is a challenging task, such as code quality, language characteristics, and application areas. For example, software quality, considering that it has a large number of well-known influencing factors such as code quantity, 6 team size, 2 and age/proficiency. 9
controlled experiments are one way to test language selection in the face of such a discouraging confusion effect, however, such studies often introduce a kind of confusion of their own, that is, limiting the scope due to cost. In this kind of research, the complete task must be limited and cannot simulate development in the real world. Here are several recent studies of such undergraduates using, or, through an experimental factor, to compare static or dynamic types of languages. 7 , 12 , 15
Luckily, now we can study these problems based on a large number of real-world software projects. GitHub contains a large number of projects in multiple languages and varies greatly in size, age, and number of developers. Each project's repository provides a detailed record of contribution history, project size, authorship identity, and defect repair. We then use a variety of tools to study the impact of language characteristics on defect occurrence.The best description of our research method should be "mixed approach", or triangulation; 5 We use text analysis, clustering and visualization to confirm and support the results of quantitative regression studies. This experience-based approach helps us understand the specific impact of programming languages on software quality, because they are used informally by developers.
2 Method
Our method is a typical large-scale observation and research method in software engineering. We first used a large number of automated methods to collect data from several data sources. The data were then filtered and cleaned using a pre-constructed statistical analysis model. The choice of filters is driven by a series of factors including the nature of the problem we studied, the data quality and the data that we believe is best suited for this statistical analysis study. In particular, GitHub contains a lot of projects written by a large number of programming languages. For this study, we spent a lot of effort focusing on collecting data on popular projects written in most mainstream programming languages. We chose the appropriate method to evaluate the influencing factors on the counting data.
2.1 Data Collection
We chose the top 19 programming languages on GitHub. CSS, Shell scripts, and Vim scripts were excluded because they are not common programming languages. We include TypeScript 5, which is a superset of JavaScript. Then, for each of the studied programming languages, we searched for the top 50 projects with it as the main programming language. We analyzed 17 different languages in total, with a total of 850 projects.
Our programming language and project data is extracted from GitHub Archive, a database that records all active public GitHub projects. It records 18 different GitHub events, including new commits, fork events, PRs (pull requests), developer information and issue tracking for all open source GitHub projects on an hourly basis. Upload the packaged data to the interactive data analysis interface provided by Google BigQuery.
Identification Programming Language Ranking List
We summarize items based on their main programming language classification. We then selected most of the projects for further analysis, as shown in Table 1. A project may use multiple programming languages; it is difficult to determine it as a single programming language. The information saved by Github Archive is collected from GitHub Linguist, which uses the extension of the source file in the project repository to determine what the project's publishing language is. The most used programming language in the source file was identified as the main programming language for this project.
Table 1 Top three projects for each programming language
Search for popular projects
For each selected programming language, we first select the project based on the main programming language used by the project, and then rank the popularity of the project according to the number of relevant stars for each project. The number of stars indicates how many people actively expressed interest in the project and it is an appropriate indicator of popularity. Therefore, the top three projects in C are linux, git, and php-src; for C++, they are node-webkit, phantomjs, and mongo; for Java, they are storm, elasticsearch, and ActionBarSherlock. We selected 50 projects for each programming language.
To ensure that each project has a long enough development history, we eliminated less than 28 submitted projects (28 is the number of first quartile values for the candidate). This way we have 728 projects left. Table 1 shows the first three projects for each programming language.
Retrieve project evolution history
For each of the 728 projects, we downloaded their non-merge submissions, submission records, author data, author names using git. We calculate the code changes and the number of modified files per commit from the number of lines added and deleted for each file. We search for the programming language used by the programming language represented by the extension of the file modified in each commit (a commit may have multiple programming language tags). For each commit, we calculate its commit age by subtracting the date of the first commit of the item by subtracting its commit date.We also calculate other project-related statistics, including the maximum submission age of the project and the total number of developers, for the control variables used for our regression analysis model, which is discussed in Section 3. We identify bug fix submissions by searching for keywords related to errors in the submission record, such as error, bug, fix, issue, mistake, incorrect, fault, defect, flaw. This is similar to previous studies. 18
Table 2 summarizes our dataset. Because a project may use multiple programming languages, the second column of the table shows the total number of projects using a certain programming language. We further excluded those programming languages in the project that had fewer than 20 submissions. Because 20 is the first quartile value of the total number of submissions for each project in each programming language. For example, we found more than 20 submissions in C. This ensures that each "programming language-project" pair is sufficiently active.
Table 2 Research topics
In summary, we studied 728 projects developed in 17 programming languages over the past 18 years. In total, 29,000 different developers, 1.57 million commits, and 564,625 bug fix commits were included.
2.2 Language Classification
We define language categories based on several programming language characteristics that affect language quality, 7, 8, 12, as shown in Table 3.
Programming Paradigm means whether the project is written in command, script, or functional language. In the following part of this article, we use the terms "procedural" and "scripting" to represent command methods and scripting methods respectively.
Table 3. Different types of language classification
type checking (Type Checking) represents static or dynamic types. In statically typed languages, type checking is performed at compile time, and the variable name is bound to a value and a type. In addition, expressions (including variables) are classified according to the type of values they may produce at runtime. In dynamic typed languages, type checking occurs at runtime. Therefore, in a dynamically typed language, it may appear in the same program and a variable name may be bound to objects of different types.
Implicit Type Conversion allows an operand of type T1 to be accessed as another different type T2 without explicit type conversion. Such implicit type conversion can cause type confusion in some cases, especially when it represents an explicit operand of T1 type, and then use it as another different T2 type. Because not all implicit type conversions will immediately have problems, we show our definition through examples that implicit type conversion may occur in all programming languages that we identified. For example, in programming languages like Perl, JavaScript, CoffeeScript, adding a character and a number is allowed (for example, "5" + 2 results in "52"). But in Php, the same operation, the result is 7. Operations like this are not allowed in some programming languages, such as Java and Python, because they do not allow implicit conversions. In strong data types C and C++, the results of this operation are unpredictable. For example, int x; float y; y=3.5; x=y; is legal C code, and the result for x and y is different. Which value depends on the meaning, which may cause problems later. a In Objective-Ch, the data type id is a general object pointer that can be used for any data type object, regardless of the classification. b General data types like this offer great flexibility, which can lead to implicit type conversions and also have unpredictable results. c Therefore, we classify programming languages based on whether its compiler allows or does not allow implicit type conversion as described above; programming languages that do not allow implicit type conversion will explicitly detect type obfuscation and report type mismatch errors.
does not allow implicit type conversion programming languages, using a type judgment algorithm, such as Hindley 10 and Milner, 17, or using a dynamic type checker at runtime, which can judge the results of static types in a compiler (such as Java). In contrast, a type of confusion may occur silently because, it may be because it is not detected or reported. Either way, allowing implicit type conversions can also end up with errors that are difficult to determine the cause. For simplicity, we will use implicit to represent programming languages that allow implicit type conversion, and not implicit type conversion. We will use explicit to represent them.
Memory Classification (Memory Class) indicates whether developers are required to manage memory. Although Objective-C follows a hybrid pattern, we still treat it in a non-management classification because, we observe a lot of memory errors in its code base, which will be discussed in RQ4 in Section 3.
Please note that we use this method to classify and study programming languages because this method is used informally by a large number of developers in a "real world". For example, TypeScript is intentionally classified into a static programming language classification, which does not allow implicit type conversion. However, in practice, we noticed that developers often use any type (with 50% of variables, and across TypeScript – projects used in our dataset) which is a general union type, and, therefore, in practice, TypeScript allows dynamic, implicit type conversion. To reduce confusion, we exclude TypeScript from our programming language classification and related models (see Tables 3 and 7).
2.3 Identify project fields
We use an automatic plus manual hybrid technology to classify the research projects into different fields based on the characteristics and functions of the programming language. On GitHub, projects use project descriptions and README files to describe their characteristics. We use a document topic generation model (Latent Dirichlet Allocation, abbreviated as: LDA) 3 to analyze these texts. Provide a set of documents to it, and the LDA will generate different keywords and then identify possible topics. For each document, LDA also estimates the probability of documents assigned to each topic.
We detected 30 different fields (in other words, topics) and evaluated the probability that each item belongs to each field. Because these areas of automatic detection include keywords for several specific projects, such as Facebook, it is difficult to define its underlying common functions. To assign each field a meaningful name, we manually checked 30 domain identification keywords for the domain that were not related to the project name. We manually renamed all 30 areas of automatic detection and found most of the projects in the following six areas: applications, databases, code analysis, middleware, libraries, and frameworks. We also found some projects that did not meet any of the above areas, so we generally labeled this area as others. Subsequently, another member of our study group examined and confirmed the way this project domain classification was classified. Table 4 summarizes the domain results identified by this process.
Table 4 Domain features
2.4 bug classification
Although when fixing software bugs, developers often leave original important information about this bug in the submission log; for example, why a bug occurs and how to fix a bug. We use a lot of information to classify bugs, similar to Tan's et al. 13, 24
First of all, we classify based on the cause (Cause) and impact (Impact) of the bug. _ Cause _ Further decompose into irrelevant error subclasses: algorithmic, concurrency, memory, ordinary programming errors, and unknown. The impact of bugs is also divided into four unrelated subclasses: security, performance, failure, and other unknown classes. Therefore, each bug fix commit also contains the types of causes and impact. Table 5 shows each bug classification described.This category is performed in two stages:
Table 5 bug classification and description in the entire dataset
(1) keyword search We randomly selected 10% of bug fix information and used a keyword-based search technology to automatically classify them as possible bug types. We use this annotation for both types (cause and impact). We selected a limited set of keywords and idioms, as shown in Table 5. Limited keywords and idioms like this can help us reduce false positives.
(2) Supervised classification We use the annotated bug fix log in the previous steps as training data to supervise the classification technology, correct the remaining bug fix information through test data. We first convert each bug fix information into a bag of words (LCTT translation note: bag-of-words, an information retrieval model). Then, delete words that only appear once in all bug fix messages. This reduces the keywords for specific projects. We also use standard natural language processing techniques to solve this problem. Ultimately, we use Support Vector Machine (LCTT translation note: Support Vector Machine, abbreviated as SVM, in the field of machine learning, to classify test data.
To accurately evaluate the bug classifier, we manually annotated 180 randomly selected bug fixes, distributed evenly across all categories. We then compare the results of manually annotated datasets in the automatic classifier. After final processing, the accuracy shown is acceptable, with the lowest accuracy in performance, 70%, and the highest accuracy in concurrency error, 100%, with an average of 84%. Run again, the accuracy from low to high is 69% to 91%, and the average accuracy is 84%.
The results of our bug classification are shown in Table 5. The causes of most defects are related to ordinary programming errors. This result is not surprising, because a large number of programming errors are involved in this category, such as type errors, input errors, writing errors, etc. Our technology cannot reclassify bug fix information that accounts for 1.4% of any (cause or impact) classification; we classify it as unknown.
2.5 Statistical Methods
We used a regression model to model the number of defective submissions in other factors related to the software project. All models use negative binomial regression (NBR) (LCTT translation note: a regression analysis model) to model project attribute counts, such as the number of submissions. NBR is a generalized linear model for responsive modeling of non-negative integers. 4
In our model, we control several factors that may affect the final result for each project’s programming language. Therefore, in our regression analysis, each (language/project) pair is a row and can be considered as a sample from a popular open source project. We convert objects based on variable counts to keep the variable stable and improve the applicability of the model. 4 We validate them by testing comparisons of non-nested models using AIC and Vuong.
to check those excessive multicollinearities (LCTT translation note: Multicollinearity refers to the fact that the model estimation is distorted or difficult to estimate accurately due to the precise or high correlation between explanatory variables in the linear regression model.) It is not a problem. We use a conservative maximum value of 5 in all models to calculate the variance of the expansion factor of each dependent variable. 4 We removed the high leverage points by visually examining the residual and leverage graphs of each model to find the separation and maximum values of the Cook distance (LCTT translation note: a statistical term used to diagnose the presence of abnormal data in regression analysis).
We use effects, or differences, to encode them into our research to improve the performance of programming language regression coefficients. 4 Weighted Effect Code allows us to compare the effects of each programming language with all programming languages, while compensating for the inhomogeneity of programming languages used across projects.23 To test the relationship between two variable factors, we used an independent chi-square test (LCTT translation note: Chi-square, a statistical hypothesis test method). 14 After confirming a dependency, we use Cramer's V, which is the phi(φ) coefficient of normal data equivalent to r × c, to create an effect data.
3 Results
We started with a simple and clear problem, which solved some core problems that people firmly believed in, namely:
problem 1: Are some programming languages more prone to defects than other languages?
We used a regression analysis model to compare the impact of each programming language on the average number of defects in all programming languages and its impact on defect repair submissions (see Table 6).
Table 6. Some languages have fewer defects than other languages
. We include some variables as control factors that clearly affect the response. Project age (age) is also included because the older the project generates a larger number of defect fixes. The number of commits will also have a slight impact on project response. In addition, the number of developers (devs) working on the project and the original size (size) of the project will increase as the project is active.
The size and symbol of the estimated coefficient in the above model (LCTT translation note: refers to "+" or "-") is related to the predictor of the result. The initial four variables are control variables, and we are not interested in the effect of these variables on the final result, just saying that they are all positive and meaningful. Language variables are indicator variables, which are the variable factors for each project, which compare each programming language with the weighted averages for the programming languages for all projects. Programming language coefficients can be roughly divided into three categories. The first category is that those coefficients that are statistically insignificant and cannot be distinguished from 0 during the modeling process. These programming languages perform similarly to the mean, or they may also have greater variance. The remaining coefficients are very obvious, either positive or negative. For those positive coefficients, we guess that it may be related to the large number of defect fixes in this programming language. These languages include C, C++, Objective-C, Php, and Python. All programming languages with a negative coefficient, such as Clojure, Haskell, Ruby, and Scala, suggest that the defect fix commits of these languages may be less than the average.
It should be noted that although there is a clear connection between programming languages and defects from a statistical perspective, don’t overestimate the impact of programming languages on defects, because this impact effect is very small. The results of the anomaly analysis show that this effect is less than 1% of the total anomaly.
ut1.jpg
We can understand the coefficient of the model in this way, which represents a predictor whose unit changes in a predictor reflects a logarithmic change in the expected response while all other predictors remain unchanged; in other words, for a coefficient βi, a unit change in βi produces an expected change in the eβi response. For variable factors, this expected change is compared with the average of all programming languages. Therefore, if a specific project developed in a programming language with an average value has four defect commits for a certain number of commits, then if you choose to develop in C++, it means we expect that there should be an additional (LCTT translation note: 1 more relative to the average value of 4) defect commits, because e0.18 × 4 = 4.79. For the same project, if you choose to use Haskell for development, it means we expect one less defect submission (LCTT translation note: supra, 4 relative to the average). Because, e−0.26 × 4 = 3.08. The accuracy of the prediction depends on the remaining other factors remain the same, and all of them are a very challenging proposition except for those trivial items. All observational studies face similar limitations; we will address these matters in detail in Section 5.
Conclusion 1: Some programming languages have higher defect correlation than other programming languages, but the impact is very small.
In the rest of this article, we will elaborate on the basic conclusions. By considering different types of applications, defects, and programming languages, we can further understand the relationship between programming languages and defect tendencies.
software bugs usually fall into two broad categories:
- domain-specific bugs: project-specific features and does not depend on the underlying programming language.
- Normal bug: Most ordinary bugs are born and have nothing to do with project functions, such as input errors, concurrent errors, etc.
Therefore, in a project, the application domain and programming language interaction may affect the number of defects, the conclusion is considered reasonable. Because some programming languages are considered to be more prominent in some tasks than others, for example, C for low-level (lower) work, or Java for user applications, an inappropriate choice for programming languages may bring more flaws. To study this situation, we will ideally ignore domain-specific bugs, because ordinary bugs rely more on the characteristics of programming languages. However, because a domain-specific bug can also occur in a normal programming error, it is difficult to distinguish between the two. One possible workaround is to study programming languages while controlling the field. From a statistical point of view, while, using 17 programming languages across 7 domains, understanding large numbers of terms will be a huge challenge in a given sample size.
In view of this situation, we first consider testing the dependencies between the domain and programming language usage in a project, using a Chi-square test independently. Of the 119 units, it is 46, which means 39%, which is above the conservative value we set, which is too high. This number cannot exceed 20%, and should be less than 5.14 We include the full value here; d However, the value tested by Cramer is 0.191, which is low correlation, indicating that the correlation between any programming language and domain is very small, and that including domains in the regression model does not produce meaningful results. One option to solve this situation is to remove programming languages, or mix fields, but our existing data is not fully selected. Or, we mix programming languages; this choice leads to a related but slightly different problem.
Question 2: Which programming language features are related to defects?
We aggregate them by programming language category rather than considering individual programming languages, as described in Section 2.2, and then analyze the relationship with defects. Overall, each of these properties divides the programming language into errors often discussed in the context, user debate-driven, or by previous work topics. Therefore, individual properties are highly correlated, and we created six model factors that synthesize all individual factors into our study. We then model the effect of six different factors on the number of defects while controlling for the same basic covariates in the model we used in Section 1 of Question (LCTT) (LCTT) Note: Covariates refer to independent variables that cannot be artificially manipulated in the experiment).
Regarding the programming language used (in the previous Table 6), we use the average response across all language classes to compare programming language classes. This model is expressed in Table 7. It is obvious that the Script-Dynamic-Explicit-Managed class has the smallest coefficient of magnitude. This coefficient is trivial. In other words, the Z-test of this coefficient cannot be distinguished from 0. Given the order of standard errors, we can assume that the behavior of programming languages in this category is very close to the average of all programming language behaviors. We can prove this by using Proc-Static-Implicit-Unmanaged as the base level and for processing, or using the base level to falsely encode and compare each language class. In this case, Script-Dynamic-Explicit-Managed is significantly different from p = 0.00044.Note that although we chose different encoding methods in this way, affecting the coefficients and Z values, this method is the same as all other aspects. When we change the encoding, we adjust the coefficients to reflect the comparison we want to generate. 4 Comparing other classes of programming languages with the overall average, the Proc-Static-Implicit-Unmanaged programming language is more likely to cause defects. This means that implicit type conversion or managing memory will lead to more flaws than other process programming languages.
Table 7. Functional languages have lower correlation with defects than other languages, while procedural languages are greater or close to the average.
In script-class programming languages, we observe a relationship similar to that between programming languages that allow and do not allow implicit type conversion, which provides some evidence that implicit type conversion (compared to explicit type conversion) is the reason for this difference, not memory management. Given the correlation between various factors, we cannot draw this conclusion. However, when they are compared with the mean, as a group, programming languages that do not allow implicit type conversion have a lower tendency to have errors, while programming languages that have a higher tendency to have errors have a relatively higher chance of errors. The difference between static and dynamic types is also obvious in functional programming languages.
Functional language as a group shows a very obvious difference from the average. The coefficients of statically typed languages are much smaller, but functional language classes have the same standard errors. The tendency of functional static programming languages to have less errors than functional dynamic programming languages is strong evidence, though, Z-checking is merely to test whether the coefficients can be distinguished from 0. To strengthen this inference, we recoded the model above using processing encoding and observed that the error tendency of the Functional-Static-Explicit-Managed programming language class is significantly smaller than p = 0.034 for the Functional-Dynamic-Explicit-Managed programming language class.
ut2.jpg
Like programming languages and defects, the impact of the relationship between programming language classes and defects is very small. This difference in interpreted programming languages is also similar, although small, and this difference in interpreted programming languages is less than 1%.
We are now back to the application field. Does the application field affect each other with language classes? How to choose? For example, a functional programming language has certain advantages for specific fields? As above, a chi-square test is done for the relationship between these factors and the project domain, with a value of 99.05, df = 30, p = 2.622e–09, and we reject the meaningless assumption that the V of Cramer produces a value of 0.133, indicating a weak association. Therefore, although there are some correlations between the domain and the programming language, there is only a very weak correlation between the application domain and the programming language class here.
Conclusion 2: There is a small but obvious relationship between programming language classes and defects. Compared with procedural or scripted languages, functional languages have fewer defects. The conclusion of
is somewhat unsatisfactory because we do not have strong evidence to prove the correlation between programming languages or language classes and application fields in a project. An alternative is to ignore the total number of projects and defects based on all programming languages and application areas and view the same data. Because, in this way, no longer produces irrelevant samples, we do not try to analyze them from a statistical perspective, but use a descriptive, visual-based method.
We define Defect Proneness as the ratio of bug fix commits to total commits per language per domain. Figure 1 uses a heat map to show the interaction between the application domain and the programming language, with the tendency to represent defects from light to dark. We examined which programming language factors influenced the defect repair submission of projects written across multiple languages. It leads to the following research questions:
Figure 1. Interaction between programming language defect tendencies and application fields.For a given field (bottom of column), each grid in the heat map represents a programming language's flaw tendency (row header). The "whole" column represents a programming language's tendency to be based on all domains. A grid marked with a white crosshair represents a null value, in other words, a commit that does not match in that grid.
Question 3: Does the programming language’s error tendency depend on the application field?
To answer this question, we first filter out items considered anomalies in our regression model with high leverage points. This method is necessary here, although this is a non-statistical approach, and some relationships may affect visualization. For example, we find a simple project, Google v8, a JavaScript project, responsible for all errors in the middleware. This was a surprise for us, as JavaScript is not usually used in middleware. This model has been repeated continuously in other applications, so the defects of the projects we filtered out are below 10% and above 90%. This result is in Figure 1.
We see only a small difference in this heat map, as seen in question 1, this result only represents the inherent misleading tendency of programming languages. To verify this inference, we measured the flaw tendency of programming languages to each application area and to all application areas. For all domains except the database, the correlation is positive and the p-value makes sense (0.01). Therefore, with regard to defect tendencies, the language sorting in each field is basically the same as the language sorting in all fields.
ut3.jpg
Conclusion 3: There is no overall connection between the application field and the tendency to deficiencies in programming languages.
We demonstrate that different languages have a large number of flaws, and that this relationship is not only related to a specific language, but also applies to general language classes; however, we find that project types cannot coordinate this relationship to a certain extent. Now that we change our attention to reaction classification, I want to understand what the connection is between programming languages and specific kinds of defects, and how this relationship is compared with the more common relationship we observe. We divide the defects into different categories, as described in Table 5, and then ask the following question:
question 4: What is the relationship between programming languages and bug classification?
We used a method similar to that used in question 3 to understand the relationship between programming languages and bug classifications. First, we examine the relationship between bug classification and programming language classes. A heatmap (Figure 2) shows the total number of defects on top of programming language classes and bug types. To understand the interaction between bug classification and language, we use an NBR regression model for each category. For each model, we used the same control factors as in question 1, and the language encoded with weighted effects to predict defect repair submissions.
Figure 2. The relationship between bug category and programming language class. Each grid represents the percentage of all bug fix commits per language class (row header) per bug category (bottom of column). This value is normalized by columns. The results of
and the ANOVA value of the programming language are shown in Table 8. The overall exceptions of each model are very small, and for specific defect types, the proportions displayed by language are similar in the order of magnitude in most categories. We explain this relationship as the impact of programming languages on specific bug categories is greater than the overall impact. Although our conclusion summarizes all categories, in the following section, we further study the bug categories with a large number of bugs reflected in Table 5.
Table 8. Although the impact of programming languages on defects varies by defect category, programming languages have greater impact on specific categories than general categories.
Programming Error Normal programming errors account for about 88.53% of all bug fix submissions and are available in all programming language classes. Therefore, the regression analysis gives a conclusion similar to that in question 1 (see Table 6).All programming languages can cause such programming errors, such as processing errors, definition errors, input errors, etc.
Memory Error Memory Errors account for 5.44% of all bug fix commits. Thermal diagram Figure 2 demonstrates a very close connection between the Proc-Static-Implicit-Unmanaged class and memory errors. A memory bug occurred in programming languages that do not manage memory, which is expected. Table 8 also demonstrates this, for example, C, C++, and Objective-C raise many memory errors. Java raises more memory errors in languages that manage memory, although it is less than programming languages that do not manage memory. Although Java does memory recycle by itself, it is not surprising that it has memory leaks, because object references often prevent memory recycle. 11 In our data, of all memory errors in Java, 28.89% are caused by memory leaks. In terms of quantity, memory defects in programming languages have a much greater impact than other categories.
Concurrent Errors In the total bug fix commits, the repair commits related to concurrent errors account for 1.99%. Thermal graph shows that Proc-Static-Implicit-Unmanaged is the main error type. Among this error, C and C++ account for 19.15% and 7.89% respectively, and they are distributed across various projects.
ut4.jpg
Programming languages that belong to the Static-Strong-Managed language class have been proven to be in the dark area in the heat map, and ordinary static languages produce more concurrent errors than other languages. In dynamic languages, Erlang only has more tendencies to concurrent errors, which may be related to the many concurrent applications developed using this language. Similarly, the negative coefficients in Table 8 indicate that projects written in dynamic programming languages such as Ruby and `Php have fewer concurrent errors. Note that some languages, such as JavaScript, CoffeeScript, and TypeScript, do not support concurrency, although Php has limited concurrency support (depending on its implementation). These languages introduce false 0s into our data, so the coefficients of the concurrency model of these languages in Table 8 cannot be explained like other languages. Because of these false 0s, the average of all languages in this model is very small, which may affect the magnitude of the coefficients, so they give w.r.t. an average, however, this does not affect their relative relationship, because we only focus on their relative relationship.
Text analysis of high-frequency words in bug-fixed messages shows that most concurrency errors occur in a conditional contention, deadlock, or incorrect synchronization, as shown in the table above. Iterating through all languages, conditional contention is the reason why concurrent errors occur the most, for example, 92% in Go. The improvement in conditional contention errors in Go may be due to the use of a contention detection tool to help developers locate contention. Synchronization errors are mainly related to Message Passing Interface (MPI) or Shared Memory Operations (SHM). Erlang and Go use MPI e for inter-thread communication, which is why there are no SHM-related errors in both languages, such as shared locks, mutex locks, etc. In contrast, projects written in languages with earlier SHM technology for inter-thread communication may have lock-related errors.
security and other conflict errors Among all bug fix commits, commits related to conflict (Impact) errors account for about 7.33%. Among them, Erlang, C++, and Python security-related errors are higher than the average (Table 8). There are fewer security errors related to the Clojure project (Figure 2). We can also see from the thermal diagram that static languages are generally more prone to failures and performance errors, followed by the Functional-Dynamic-Explicit-Managed language, such as Erlang. Analysis of abnormal results shows that programming languages are closely related to conflict failure. Although security errors are weakly correlated in this category, the differences in interpreted languages are still relatively large compared to residuals.
Conclusion 4: Defect types are strongly related to programming languages; some defect types such as memory errors and concurrency errors also depend on early languages (technology used). For a specific category, programming languages cause more errors than overall.
4. Related work
The work done before comparing programming languages is divided into the following three categories:
(1) Controlled experiment
For a given task, developers are monitored when programming in different languages. Research and compare results, such as development results and code quality. Hanenberg 7 developed a parser that monitored 48 programmers for 27 hours to compare static and dynamic types. He found that there was no significant difference in code quality between the two, but dynamically-typed languages took less development time. Their research was conducted in a lab using undergraduate students, setting up custom languages and IDEs. Our research is exactly the opposite, and is a practical study of popular software applications. While we are able to control confounders only by using regression models (and post hockey) control, our advantage is that the sample size is large and more realistic and widely used software. We found that under the same conditions, statically typed languages have less error tendency than dynamically typed languages, and that languages that do not allow implicit type conversion are better than languages that allow implicit type conversion, which has a very small impact on the results. This is reasonable because the sample size is very large, so this very small effect can be seen in this study.
Harrison et al. 8 compared C++ with SML, one is a procedural programming language and the other is a functional programming language. There was no significant difference in the total number of errors, but SML has a higher defect density than C++. SML is not reflected in our data, but it is believed that functional programming languages have fewer defects than procedural programming languages. Another key task is to compare development work across different languages. 12 , 20 However, they do not analyze programming language flaw tendencies.
(2) Investigation
Meyerovich and Rabkin 16 investigated developers' views on programming languages to study why some programming languages are more popular than others. Their report notes that non-programming language factors have a huge impact: previous programming language skills, available open source tools, and existing old-fashioned systems. Our research also demonstrates that available external tools also affect software quality; for example, concurrent bugs in Go (please check out the contents of issue 4).
(3) Mining of software repositories
Bhattacharya and Neamtiu 1 studied four projects developed with C and C++, and found that components developed in C++ are generally more reliable than those developed in C. We found that C and C++ have higher error tendencies than the average for all programming languages. However, for some bug types, like concurrent errors, C has a higher flaw tendency than C++ (see Issue 4 in Section 3).
5. Valid Risk
We believe that the conclusions of our report are almost risky. First, when it comes to identifying bug fix commits, the keywords we rely on are keywords that developers often use to represent bug fixes. Our choices were carefully considered. In an ongoing development process, we capture the problems that developers have been facing, rather than the bugs they reported. However, this option has the risk of overestimating. We categorize domains to explain the tendency to defects, and other members of our study group have verified classification. In addition, we spend our energy on classifying bug fix submissions, which may also be at risk of being contaminated by the keywords that were originally selected. Each project is also different in the description of the commit log. To mitigate these risks, we evaluated our categories using manual annotations, as described in Section 2.4.
We judge that the programming language to which the file belongs is based on the file's extension. If files written in different programming languages use the extension of the common programming language file we studied, it is also prone to error tendencies in this case. To reduce this error, we manually verified our language classification using a random sample file set.
Based on the situation shown in our dataset, we made some assumptions based on the main uses of programming language properties in Section 2.2.For example, we divide Objective-C into non-managed memory types, rather than mixed types. Similarly, we annotate Scala as a functional programming language and C# as a procedural programming language, although both support them in terms of design choices. 19 , 21 In this research work, we did not separate object-oriented programming languages (OOPs) from procedural programming languages because, there is no clear difference between them, and the main difference is in the programming type. We classify C++ into categories that allow implicit type conversion because some types of memory areas can be processed differently by using pointer operations, 22 and we noticed that most C++ compilers can detect type errors at compile time.
Finally, we associate the defect fix commits to programming language properties, which can reflect the style of the report or other developers' attributes. Available external tools or libraries can also affect the number of bugs in a related programming language.
6. Summary
We conducted a large-scale study of programming languages and usage because it involves software quality. The data we use on Github has high complexity and differentiation in multiple dimensions. Our sample size allows us to conduct a mixed method study on the effects of programming languages and the interactions between programming languages, application areas and defect types while controlling for some confounding factors. Research data shows that functional languages are better than procedural languages; languages that do not allow implicit type conversion are better than languages that allow implicit type conversion; languages that are statically typed are better than languages that are dynamically typed; and languages that manage memory are better than non-managed languages. Furthermore, the flaw tendency of programming languages has no connection with the software application field. In addition, each programming language is more associated with a specific bug category than with all bugs.
On the other hand, even large-scale data sets, they become small and incomplete after being segmented by multiple methods at the same time. Therefore, as more and more variables depend on it, it is difficult to answer the question of how big the impact a variable has, especially when the variables interact. Therefore, we cannot quantify the specific effects of programming languages in use. Other methods, such as investigations, may be helpful. We will address these challenges in future work.
Acknowledgements
This material was completed under the authorization and support of the National Science Foundation (NSF) and the Office of Scientific Research (AFOSR). Authorization number 1445079, 1247280, 1414172, 1446683, FA955-11-1-0246.
References
- Bhattacharya, P., Neamtiu, I. Assessing programming language impact on development and maintenance: A study on C and C++. In Proceedings of the 33rd International Conference on Software Engineering, ICSE'11 (New York, NY USA, 2011). ACM, 171–180.
- Bird, C., Nagappan, N., Murphy, B., Gall, H., Devanbu, P. Don't touch my code! Examined the effects of ownership on software quality. In Proceedings of the 19th ACM SIGSOFT Symposium and the 13th European Conference on Foundations of Software Engineering (2011). ACM, 4–14.
- Blei, D.M. Probabilistic topic models. Commun. ACM 55 , 4 (2012), 77–84.
- Cohen, J. Applied Multiple Regression/Correlation Analysis for the Behavioral Sciences.Lawrence Erlbaum, 2003.
- Easterbrook, S., Singer, J., Storey, M.-A., Damian, D. Selecting empirical methods for software engineering research. In Guide to Advanced Empirical Software Engineering (2008). Springer, 285–311.
- El Emam, K., Benlarbi, S., Goel, N., Rai, S.N. The confounding effect of class size on the validity of object-oriented metrics. IEEE Trans. Softw. Eng. 27 , 7 (2001), 630–650.
- Hanenberg, S. An experiment about static and dynamic type systems: Doubts about the positive impact of static type systems on development time. In Proceedings of the ACM International Conference on Object Oriented Programming Systems Languages and Applications, OOPSLA'10 (New York, NY, USA, 2010). ACM, 22–35.
- Harrison, R., Smaraweera, L., Dobie, M., Lewis, P. Comparing programming paradigms: An evaluation of functional and object-oriented programs. Softw. Eng. J. 11 , 4 (1996), 247–254.
- Harter, D.E., Krishnan, M.S., Slaughter, S.A. Effects of process maturity on quality, cycle time, and effort in software product development. Manage. Sci. 46 4 (2000), 451–466.
- Hindley, R. The principal type-scheme of an object in combinational logic. Trans. Am. Math. Soc. (1969), 29–60.
- Jump, M., McKinley, K.S. Cork: Dynamic memory leak detection for garbage-collected languages. In ACM SIGPLAN Notices , Volume 42 (2007). ACM, 31–38.
- Kleinschmager, S., Hanenberg, S., Robbes, R., Tanter, É., Stefik, A. Do static type systems improve the maintenance ability of software systems? An empirical study. In 2012 IEEE 20th International Conference on Program Comprehension (ICPC) (2012). IEEE, 153–162.
- Li, Z., Tan, L., Wang, X., Lu, S., Zhou, Y., Zhai, C. Have things changed now? An empirical study of bug characteristics in modern open source software. In ASID'06: Proceedings of the 1st Workshop on Architectural and System Support for Improving Software Dependability (October 2006).
- Marques De Sá, J.P. Applied Statistics Using SPSS, Statistica and Matlab , 2003.
- Mayer, C., Hanenberg, S., Robbes, R., Tanter, É., Stefik, A. An empirical study of the influence of static type systems on the usability of undocumented software. In ACM SIGPLAN Notices , Volume 47 (2012). ACM, 683–702.
- Meyerovich, L.A., Rabkin, A.S. Empirical analysis of programming language adoption. In Proceedings of the 2013 ACM SIGPLAN International Conference on Object Oriented Programming Systems Languages Applications (2013). ACM, 1–18.
- Milner, R. A theory of type polymorphism in programming. J. Comput. Syst. Sci. 17 , 3 (1978), 348–375.
- Mockus, A., Votta, L.G. Identifying reasons for software changes using historic databases. In ICSM'00. Proceedings of the International Conference on Software Maintenance (2000). IEEE Computer Society, 120.
- Odersky, M., Spoon, L., Venners, B. Programming in Scala. Artima Inc, 2008.
- Pankratius, V., Schmidt, F., Garretón, G. Combining functional and important programming for multicore software: An empirical study evaluating scala and java. In Proceedings of the 2012 International Conference on Software Engineering (2012). IEEE Press, 123–133.
- Pankratius, T., Skeet, J. Real World Functional Programming: With Examples in F# and C#.Manning Publications Co., 2009.
- Pierce, B.C. Types and Programming Languages. MIT Press, 2002.
- Posnett, D., Bird, C., Dévanbu, P. An empirical study on the influence of pattern roles on change-proneness. Emp. Softw. Eng. 16 , 3 (2011), 396–423.
- Tan, L., Liu, C., Li, Z., Wang, X., Zhou, Y., Zhai, C. Bug characteristics in open source software. Emp. Software. Eng. (2013).
Author
Baishakhi Ray ( [email protected] ), Department of Computer Science, University of Virginia, Charlottesville, VA.
Daryl Posnett ( [email protected] ), Department of Computer Science, University of California, Davis, CA.
Premkumar Devanbu ( [email protected] ), Department of Computer Science, University of California, Davis, CA.
Vladimir Filkov ( [email protected] ), Department of Computer Science, University of California, Davis, CA.
Footnotes
- a. Wikipedia's article on type conversion, https://en.wikipedia.org/wiki/Type_conversion , has more examples of unintended behavior in C.
- b. This Apple developer article describes the usage of "id" http://tinyurl.com/jkl7cby .
- c. Some examples can be found here http://dobegin.com/objc-id-type/ and here http://tinyurl.com/hxv8kvg .
- d. Chi-squared value of 243.6 with 96 df. and p = 8.394e–15
- e. MPI does not require locking of shared resources.
via: https://cacm.acm.org/magazines/2017/10/221326-a-large-scale-study-of-programming-languages-and-code-quality-in-github/fulltext?imm_mid=0f7103cmp=em-prog-na-na-newsltr_20171007
Author: Baishakhi Ray, Daryl Posnett, Premkumar Devanbu, Vladimir Filkov Translator: qhwdw Proofreading: wxy
This article was originally compiled by LCTT, Linux China Honored launch
