Author: Zhou Jihai, Zhou Yifan, Ma Songsong, Tao Fen, Yang Weiqiang, Cheng Shengcong, Chen Yaping Source: Huazhang Technology Wikipedia, DevOps (a combination of Development and Operations) refers to a culture, movement or practice that attaches importance to communication and co

Author: Zhou Jihai Zhou Yifan Ma Songsong Tao Fen Yang Weiqiang Cheng Shengcong Chen Yaping

Source: Huazhang Technology

On Wikipedia, DevOps (a combination of Development and Operations) refers to a culture, movement or practice that attaches importance to communication and cooperation between software developers (Dev) and IT operation and maintenance technicians (Ops). By automating the process of software delivery and architecture changes, building, testing and publishing software can be faster, more frequently and more reliable.

Now, it is difficult to define what DevOps is even for those who have learned about DevOps and even already use this development model. Some views believe that DevOps is different from the traditional waterfall model, based on the agile model, and expands agile ideas and practices from development to operation and maintenance (there are also radical views that it is completely different from these two R&D models), and is a new way of thinking and action.

01 A brief history of DevOps development

At the Agile Conference held in 2008, Patrick Debois and Andrew Clay Shafer proposed to discuss the topic of "agile infrastructure". At the second year of Agility, two Flickr employees gave a speech titled “10+ Deploys per Day: Dev and Ops Cooperation at Flickr” (“10 Deployments per Day”), which can be seen as pioneering what we now call DevOps concept.

After it inspired Patrick to hold the first DevOpsDays in Ghent, Belgium in October of the same year, which represented the beginning of DevOps promotion.

Since then, DevOps has spread around the world with DevOpsDays and peaked in 2019—80 cities around the world held DevOpsDays in 2019. From October 28 to 30, 2019, DevOps event organizers and promoters from countries and cities around the world gathered in Gent, Belgium to hold the 10th anniversary celebration of DevOpsDays.

Compared with European and American countries, DevOpsDays entered China relatively late . On March 18, 2017, Beijing held the first China DevOpsDays, and in the next few years, DevOpsDays summits were held in Shanghai and Shenzhen respectively.

Due to the impact of DevOpsDays on China's DevOps industry, on July 22, 2018, the China DevOps community was established and rapidly developed to 18 cities across the country. DevOps continues to promote DevOps in various cities through small activities such as localized DevOps Meetup.

In addition to DevOpsDays, another landmark event was that Alanna Brown drafted the first edition of the annual "DevOps Status Report" in 2012. Since 2012, this annual report has been used by DevOps industry and practitioners as a reference for understanding the current situation of DevOps and a weather vane for the development direction of DevOps.

In addition, in 2013, Gene Kim published the novel "Phoenix Project", which vividly introduced readers to the collision of ideas during the transformation process, as well as various problems and related solutions by describing a company that is undergoing DevOps transformation.

In 2016, Gene Kim, together with Jez Humble, Patrick Debois and John Wills, jointly published the most authoritative classic work in the DevOps industry: "The DevOps Practice Guide". Figure 1-1 shows the development history of DevOps over 10 years.

▲Figure 1-1 History of DevOps Development

02 DevOps concept

DevOps goal is to improve the overall R&D efficiency, carry out more convenient, faster and more reliable delivery, thereby improving product competitive advantages. DevOps blurs the boundaries between positions and roles such as development, testing, operation and maintenance in the previous R&D model, strengthens their collaboration , and even encourages the reorganization of each role from the traditional expert team's organizational structure into a full-function team to strengthen collaboration (as shown in Figure 1-2).

▲Figure 1-2 DevOps organizational structure transformation

At the technical level, developers can obtain higher performance through pipelines, a series of automation mechanisms, mature and scalable infrastructure (such as the cloud), etc., so as to turn code into products more frequently and quickly, and obtain continuous feedback and verification from this speed to obtain higher reliability.

In order to achieve the goal of DevOps: to deliver more conveniently and more frequently more reliable , in addition to thinking patterns and culture, DevOps also needs some technology and tools to support it.It is also thanks to the development and maturity of some infrastructure and tools that more and more companies can practice DevOps.

From the current industry best practices, DevOps technology and concepts mainly include several key elements: Continuous Integration (CI), Continuous Delivery (CD), Microservice (Microservice), Automated Test , Infrastructure as Code, Infrastructure as Code, implies technologies and concepts such as virtualization, containers, automatic orchestration, configuration as code, monitoring and logging (Monitoring and Logging), etc. The industry has formed a series of tools and solutions around DevOps.

Finally, through the change of cultural awareness and the use of automation tools, the value that DevOps can bring is also obvious, including faster R&D delivery speed, faster product innovation and trial speed; effectively managing larger systems and providing more reliable quality; from a cultural perspective, deepening collaboration between R&D roles. Not only the Internet industry, but also many traditional industries such as finance, retail, and manufacturing are also trying DevOps.

In the traditional mode as shown in Figure 1-3, security review is required after the entire R&D process (requirement, development and testing) is completed and before it goes online to ensure the security of the application. Therefore, simply put, the delivery cycle of the entire software development is the R&D time plus the duration of security evaluation.

▲Figure 1-3 From traditional mode to DevOps mode

In DevOps mode, we have improved R&D efficiency through automation, agile development, team collaboration, microservice design and other DevOps concepts and technical means. The duration of the R&D stage is shortened, which also reduces the duration of the entire delivery cycle and improves delivery speed and efficiency.

However, since the traditional DevOps model does not consider security, the security review time before going online has not changed. It can be clearly seen from Figure 1-3 that with the mature DevOps, the bottleneck in the team's continued improvement of R&D efficiency is no longer in the R&D stage, but in the security review stage before going online.

So, how to further improve R&D efficiency and improve delivery efficiency in DevOps mode? In addition, from a security perspective, traditional application security modes (such as SDL) in waterfall mode can no longer keep up with the faster delivery speed in DevOps mode, so it is necessary to explore a new application security mode methodology suitable for continuous iteration and rapid delivery.

03 From DevOps to DevSecOps

In the previous section, we used a diagram to briefly describe the transition from the traditional R&D model to the DevOps model. However, traditional DevOps mainly considers speed and quality, but does not consider information security. Therefore, when DevOps is relatively mature, information security has become a bottleneck for the continued improvement of R&D efficiency. The ultimate goal of

DevSecOps is to shorten the duration of the security review phase by moving the security left to the development and testing team, thereby further shortening the delivery cycle (as shown in Figure 1-4). And it can detect and fix security vulnerabilities at an earlier stage, thereby reducing the rework cost of discovering security vulnerabilities before going online.

▲Figure 1-4 From DevOps mode to DevSecOps mode

DevSecOps is a concept proposed by Gartner in 2012. Its original term is DevOpsSec. After the 2017 RSA Summit, DevSecOps began to become a hot topic in the world. DevSecOps continues the concept of DevOps, and its design and execution are still under the framework of Agile.

DevSecOps' goal is to embed security into various processes of DevOps (demand, architecture, development, testing, etc.), thus realizes the left shift of security, let everyone be responsible for security, , transform security from passive to active, and ultimately allow the team to develop better quality products faster and safer.

is called security left shift, in practice, to make the team responsible for the content they develop. By moving security and other tasks (such as testing security) left from the security review stage before deployment to an earlier stage, we can discover and solve security problems earlier and faster, rather than waiting until the deployment is deployed a few days later, or sending out a penetration test report a few months later.The emergence of

DevSecOps is not accidental, it is an inevitable product of the continuous delivery and evolution of software. Under this new software delivery model, security behavior will be scattered at all stages of software delivery, and the responsibility for security will also fall on participants at all stages, rather than the main responsibility will fall on the security team. DevSecOps can provide many benefits to R&D efficiency, mainly reflected in the following three aspects (see Figure 1-5):

▲Figure 1-5 The benefits of DevSecOps compared to DevOps

1. Faster delivery

DevSecOps scans through automated security tools, shifting the security scanning work in some traditional modes in the last stage before going online without any sense, making the entire delivery cycle shorter and the delivery speed faster. For example, in Figure 1-5, due to the reduction of the security review phase duration (T7), the delivery cycle has changed from T1 in DevOps mode to "T1–T7" in DevSecOps mode.

2. Cost saving

DevSecOps Since security risks and vulnerabilities were discovered and corrected in the early stage of SDLC, avoids rework in the traditional model after security scans were conducted in the last stage before going online, thus saving costs from the process.

For example, in Figure 1-5, after reworking and repairing the security vulnerability before going online, the entire development, testing and security review process must be repeated again, so the additional cost is the manpower under T2 time.

In DevSecOps mode, since the security left moves to the development or testing stage, if a high-risk security vulnerability is discovered in the development stage, the additional manpower consumed is only the manpower under the development time T4, and what is saved is the manpower under the "T2–T4" time.

If the high-risk security vulnerability is discovered during the testing phase, the additional manpower consumed by rework is the manpower under "T4 + T5", so what is saved is the manpower under "T2–T4–T5".

3. Control risk

DevSecOps reduces the development team's dependence on security departments/teams. allows the development team to discover and correct some security risks and vulnerabilities through security left shift.

In addition, under the traditional model, security departments/teams often play the role of "police" to provide protection for the safety of the enterprise, so sometimes they prevent or delay the delivery of the development team from being launched due to security risks or risks. Based on this relationship, because everyone has different purposes, the relationship between the development team and the security team is often not so harmonious, and sometimes even conflicts arise.

However, the purpose of DevSecOps is to ultimately hold everyone responsible for security by shifting security left. Therefore, there will no longer be a security "police" role to oversee the development team, but the development team will be responsible for the safety of the products they develop.

Although DevSecOps is an inevitable result of the evolution of DevOps, still faces difficulties and challenges from technology, processes, people and culture in the process of implementing DevSecOps practice (as shown in Figure 1-6). Among them, the technical challenges mainly come from two aspects:

1. Since DevSecOps is a brand new concept, there are not many open source and commercial tools available on the market. Many of the existing DevSecOps tools in
2. are not mature (such as false positive rates, high professional requirements, etc.), so it also increases the difficulty of the DevSecOps tools in the promotion and use process.

▲Figure 1-6 Challenge to realize DevSecOps

Compared with the challenges from technology, the challenges in people and culture have a greater impact. For programmers, their main job is to write code, so many programmers may lack relevant security awareness and simply think that security is not their responsibility, but the responsibility of the security team.

A survey and report by Threat Stack, a US threat detection company, on more than 200 security, development and operation and maintenance professionals from large, medium and small enterprises in North America, shows that DevSecOps is still at the theoretical stage. The main reason for this situation is that information security knowledge and capabilities have not been popularized, and the second is that there is a lack of high-level support, and business leaders do not even encourage this.The report

pointed out that only 27% of the operation and maintenance teams and 18% of the development teams are equipped with security experts; more than 44% of the developers have not received any training in security coding; and 42% of the operation and maintenance personnel have not received training in basic security practices.

Therefore, even if some developers are aware of security, they may not have the ability to securely encode and fix security vulnerabilities, so relevant security training is required. However, information security is an independent discipline after all, and therefore increases the learning cost of programmers.

Finally, just like when DevOps first appeared, as a brand new concept, the concept of DevSecOps has not been popularized, so it often does not receive support from senior management. The report also pointed out that 52% of companies acknowledged that they would cut security measures to achieve business goals by the deadline. 68% of the CEOs surveyed do not allow business delivery to be slowed down due to security issues. From this report, it can be seen that without top-down support from management, the push of DevSecOps will be very slow and even stagnant.

DevSecOps also provides corresponding best practices (see Figure 1-7) to further promote it in enterprises. For example, at the technical level, DevSecOps best practice emphasizes automated information security, and even moves security scans further to the IDE stage to detect and fix problems earlier, thereby saving costs.

▲Figure 1-7 DevSecOps Best Practices

In addition, security indicators can also be used as quality access control to ensure the security of delivery. The human and cultural level emphasizes on the development of continuous training and the development of security awareness, as well as the definition of new roles such as DevSecOps "experts" in the DevSecOps leader and the development team. The process level emphasizes regular code reviews, red and blue confrontations, discover bottlenecks in the R&D process through DevSecOps measurements, and evaluate the effectiveness of DevSecOps improvements.

About the author: Zhou Jihai, Ph.D., Imperial College London, UK, has more than 10 years of experience in the transformation and implementation of DevOps/DevSecOps from many large international banks (Barclays, HSBC, etc.) and Tencent. From 2018 to 2021, DevOps and DevSecOps experience has been shared in more than 30 technology summits, forums and communities at home and abroad.

Zhou Yifan, senior information security analyst, registered information security engineer, and cloud security engineer in HSBC Technology China. Have many years of experience in the fintech and information security industry in China and Singapore. Currently, it is mainly engaged in the implementation and implementation of application system architecture security review, security risk assessment and DevSecOps.

Ma Songsong, Tencent security expert, has 13 years of work experience in the field of basic security. Currently, the team is mainly responsible for the construction and operation of the specifications, processes related to the R&D vulnerability control and the construction and operation of automation systems. Since 2019, we have been paying attention to and promoting the exploration and implementation of DevSecOps within Tencent.

Tao Fen, a master's degree in information security at Wuhan University, has worked for Baidu, a large domestic Internet company for more than ten years, and is responsible for the implementation of DevSecOps, data security capabilities and operational practices within the enterprise.

Yang Weiqiang, general manager of HSBC Technology China Information Security Department, has worked in the field of large-scale multinational financial information security for 15 years. His main research directions include security development, security architecture, and threat modeling.

This article is excerpted from "DevSecOps Practical Practice" and is published with the authorization of the publisher.

Further reading "DevSecOps Practical Practice"

Recommended words: This book helps readers understand and understand easily and immersively how to implement and practice DevSecOps within the enterprise. The implementation and implementation methods of DevSecOps based on various industry characteristics are the main technical points of this book.