| Zhidongxi Open Class
Instructor | Liu Longze Hongpu Information Chief AI Consultant
Reminder | Follow the Zhidongxi Open Class official account and reply to the keyword Intel 06 to get the courseware.
playback | https://appoSCMf8kb5033.h5.xeknow.com/st/5yIUK0QfI
Introduction:
htmlOn May 19, Liu Longze, chief AI consultant of Hongpu Information, gave a live explanation at the Zhidongxi Open Class, with the theme of "Using OpenVINO to Accelerate Industrial Defect Detection Efficiency", which is also the 6th Lecture of the Intel AI Top 100 Innovation Incentive Program Series Open Class.In this explanation, Teacher Liu Longze systematically explained from the current status and challenges of industrial defect detection application, the design of defect detection algorithm based on Faster R-CNN, the photovoltaic EL intelligent identification system, how to use OpenVINO to accelerate the deployment of practical applications, and the industrial vision detection quality management system.
This article is a collection of pictures and texts for this special lecture session:
Text:
Information Chief AI Consultant Liu Longze, I am honored to share today's topic with you in Zhidongxi Open Class. The topic shared today is "Using OpenVINO to Accelerate Industrial Defect Detection Efficiency", which is mainly divided into the following 4 parts:
, Current Application Status and Challenges in Industrial Defect Detection Current , Design of Photovoltaic Module Defect Detection Algorithm Based on Faster R-CNN 3, Using OpenVINO to Accelerate Defect Detection Efficiency of Photovoltaic EL Intelligent Identification System Defect Detection Efficiency
4, Quality Management System Based on Industrial Vision Detection
Current Application Status and Challenges in Industrial Defect Detection
Industrial Defect Detection Field has developed for a long time. Since the manufacturing industry and quality inspection, industrial defect detection is needed. In the past, industrial defect detection was needed in many scenarios, including lithium batteries, OLED panels, tires, chips, power station components, etc. These are classic scenarios encountered when exploring the industrial field. When was first used to detect industrial defects, it used more traditional algorithms. These algorithms were mainly based on feature point extraction or Opencv technology. The biggest feature is that the algorithm is not stable enough, which is mainly reflected in the following aspects:
is the complexity of the scene, and the low accuracy is that the scene is too complex, with many types of products, or the product forms very many, which will cause a relatively low accuracy rate; secondly, in other scenarios, such as pedestrian detection or face recognition, the effect is generally relatively constant, but in the industrial field, due to the high accuracy requirements, the effect attenuation will occur. The effect attenuation is mainly caused by equipment changes or some subtle adjustments of the product; finally, the industry is iterating quickly and the quality standards change frequently. Therefore, the complexity of the scenario is one of the biggest challenges facing industrial inspection in the past.
Faster R-CNN photovoltaic module defect detection algorithm design
Let us help our solution engineer, Link, come to share with you.
Hello everyone, I am Lin Kexiang, a solution engineer from Hongpu Information. I am very honored to introduce to you a quality management product that Hongpu is in a leading position in the photovoltaic industry in this open class: the EL intelligent detection system of photovoltaic modules, and how to use OpenVINO to design new intelligent detection products.
Everyone will definitely be curious why intelligent detection is related to quality management? What is the intelligent detection system for photovoltaic modules?
First, look at the pictures above, which are some applications of photovoltaic cells in Northwest power stations and home scenarios. In fact, batteries only have one function in these scenarios: generating electricity, and every user hopes that these batteries can supply them with stable power for a long time. To keep batteries supplying them efficiently for a long time, it is necessary to troubleshoot some problems with the battery as soon as possible in the production process. From the picture above, we can see that there are many small squares arranged on each battery module. These squares are solar cells. The solar cells may be a thing with thickness in the picture, but in fact, the thickness of each cell is usually only 2 to 3 hairs. Therefore, during the production process, it is difficult to ensure that the battery cells are not damaged.
In addition, detecting appearance damage is not the difficulty of the project. The difficulty of the project lies in some invisible defects hidden in the battery cells, such as the slight cracks of some batteries and the failure of the battery cells, and the problem that some batteries may not be welded in place, which will cause the power generation efficiency of photovoltaic modules to decrease and the life of the photovoltaic modules to shorten. In response to these situations, the photovoltaic industry usually uses infrared cameras to shoot and find these defects, and this detection method of finding defects is called EL detection.
Traditional EL detection is usually done by people, but since each component may have dozens to hundreds of pictures, and manual detection has certain uncertainty, the missed detection rate of manual detection in this link is relatively high. When we learned about the pain points of the photovoltaic industry under EL detection, we began to try to use visual detection methods instead of manual labor, and recently we also involved a new product based on OpenVINO.
encountered many problems in the early development process of the product. At the beginning, I tried to use traditional image algorithms to capture defects, but I encountered several problems not long after I started: The first problem of
is that the defect patterns of photovoltaic modules are very different. Compared with other image problems, defects have fewer commonalities, and these defects are not easy to gather on the same category, resulting in the fact that when using machine learning to extract these features later, it is found that overfitting or underfitting will occur during the learning process; the second problem is that the judgment conditions of defects are relatively complex. In general object detection, the characteristics of objects are relatively obvious, and the objects to be detected are obviously different from the surrounding things, including the background. However, under EL components, defects may be embedded in the background, and defects are difficult to detect, resulting in missed detection. There are also a large number of scratches and dirt similar to defects in the background. These situations may also be detected, making it possible to use other information other than the image to help suppress the detection rate of the entire detection; the third problem is that the scale of the defect is different. In the actual research process, it may be found that the shadow surface of some batteries on the component diagram may be several millimeters, but some may be particularly large or close to 1/2 of the battery cells, which poses a great challenge to traditional image algorithms.
Combining the above problems, we try to use the Faster R-CNN method of deep learning to train the object detection model. Faster R-CNN proposed a RPN candidate box algorithm. The proposed algorithm greatly improves the target detection speed of the overall model and also enables the detection efficiency of the model to meet the needs of our customers. How to complete object detection through Faster R-CNN? First, we need to input a picture of the photovoltaic component, and then this picture enters the convolutional neural network for feature extraction. We will first use RPN to generate a bunch of anchor boxes, then crop and filter these anchor boxes, and then judge whether the anchor is a defect through softmax. At the same time, the region proposal network of another branch will correct these anchor boxes to form a more accurate positioning defect box, and then map the defect box to another convolutional feature map of the CNN, and then use the RoI pooling layer, so that each RoI feature map uses softmax loss and smooth l1 loss to jointly train the classification probability and border regression. After applying Faster R-CNN,
found that the overall detection level of the model has been greatly improved, and then through tag engineers and a deep understanding of EL components in the photovoltaic industry. We iterated several models later. To this day, this model has reached a level where the missed detection rate is less than 5% of ten thousand and the passing detection rate is less than 1% of ten thousand. Below we will use a video to show the operation process of our entire system.
After the detection station opens the system, the system will obtain the EL picture from the detection station, then transmit the picture to the computer room, infer it by the algorithm server, and return the inference result to the picture, displaying the defect location and defect type, and at the same time, a signal of whether the component is qualified. If it fails, it will be transmitted to the return repair station for remediation. The return repairer can also assist in revising the defect based on the results determined by the system.Of course, if our products want to achieve quality management, it is not just the above operations. Quality management will be focused on in Part 4. After the launch of this product,
has also been recognized by many customers, but there are still some customers who will make such a sound. Your products are really good, and they can also reduce people and increase production capacity for us. But this batch of servers is a bit expensive, and a server costs tens of thousands. Such a high hardware cost will make them feel that the benefits on this product are a little less, so I hope that other methods can be used to reduce the corresponding costs. After hearing the feedback from these customers, we began to think about whether we could make an inference through other methods and hardware. After that, we cooperated with Intel and found that Intel supports CPU inference, so we made a new product based on the CPU server.
Use OpenVINO to accelerate the defect detection efficiency of photovoltaic EL intelligent identification system
Briefly introduce what OpenVINO is? OpenVINO is a tool suite developed by Intel based on its own hardware platform. It can speed up the development of high-performance computer vision and deep learning vision applications, while supporting hardware accelerators from various Intel platforms and performing deep learning on these hardware accelerators. OpenVINO also supports windows systems, Linux systems, Python and C++ languages. After a certain research on OpenVINO, we found that OpenVINO has the following characteristics:
, OpenVINO focuses on the field of computer vision, and we are now using computer vision-related technologies such as Opencv and deep learning, so OpenVINO is very consistent with our products in the technical field; , OpenVINO has various capabilities for product algorithm model deployment, it has three built-in API deep learning deployment toolkits, as well as the general learning reasoning toolkits, and the optimization capabilities of Opencv and Openvx, and there is an optimization processing in the convolution layer and the full connection layer, and these optimization processing can help the model to reason well; 3, OpenVINO supports more than 151 neural network structures, which is the same as the model capabilities of the algorithm server. Because we want to see the effect of different models on the same thing. If OpenVINO does not support so many network structures, it means that OpenVINO can only be used to accelerate a small number of network structures, and once a network model that is not effective or that OpenVINO does not support, it cannot be migrated accordingly. However, OpenVINO supports enough network structures, which gives us greater freedom in technical choices.
4 and OpenVINO can quickly deploy applications and solutions that simulate human vision. This is also based on previous points. The most core feature of OpenVINO is that its hardware cost is very low and its inference efficiency is particularly high. Why? Because it can make inference based on CPU, among past customers, the CPUs on their devices actually have inference capabilities. Even if the CPU does not have inference capabilities, it can also configure a CPU device with better performance on the edge to perform operations. Moreover, we have calculated the overall hardware cost. The CPU can save at least 30% of the hardware cost compared to the GPU server, and the inferred efficiency also meets the needs of customers. Therefore, based on the low-cost features of OpenVINO, we can use it to let the CPU do GPU things without much loss of efficiency, which is why we are very happy to use OpenVINO.
just introduced that OpenVINO integrates three new APIs. These three new APIs can well assist our products, and most of this product are completed based on a deployment toolkit of deep learning. So first give a brief introduction to the deep learning toolkit and how it serves our model.
This toolkit mainly includes two parts: the model optimizer and the inference engine. The model optimizer is written in Python or C++. Its working principle is to optimize models trained under Tensorflow, Pytorch and Caffe 2 through the model optimizer, and then convert the optimization results into an intermediate representation file, that is, the IR file. The IR file contains many optimized network topology structures, as well as optimized model parameters and model variables.The reasoning engine will read the IR file, and then select the corresponding hardware plug-in according to the selected target platform, and then download the IR file to the target platform for execution. Currently, the plug-ins supported by OpenVINO include CPU plug-ins, GPU plug-ins, and FPGA plug-ins. After verifying the complete model, we will download the inference engine and the intermediate IR file together, or integrate it into our photovoltaic component EL intelligent detection system for deployment.
After learning so much about OpenVINO, I found that OpenVINO is very suitable for our products because it not only contains the optimization function of Opencv, but also optimizes the models trained in Tensorflow, Pytorch and Caffe2 in the past, and can significantly reduce the hardware cost of the product through CPU inference. These advantages are very consistent with our products, so we carried out a development work based on OpenVINO.
First, the algorithm team conducted a survey on OpenVINO and confirmed that it is in line with the EL intelligent detection system. Then, based on the characteristics of OpenVINO, it designed its product framework. We will connect each GPU server to the EL detection machine, and then uniformly place all GPU servers in the workshop room, because these servers have relatively high requirements for the workshop environment. Then the client will collect the pictures and send them to the computer room to make inferences, and then the inference results will be returned to the front. There is also a very special hub in the computer room. What is its function? First, we can schedule our computing power to ensure that the computing power of each GPU server is balanced and that each inference is stable; second, we can store the inference results in the database, and our database can be connected with some earlier systems in the photovoltaic industry, and store these detection images in the database. We can also perform some data mining based on these detection results.
We can dig out the current production line production conditions and the defect conditions of this batch of products, and then apply for a visual report based on these and hand it over to quality personnel to better assist in completing quality management. The entire process not only saves time in statistical samples, but also saves time in analyzing samples.
What changes has OpenVINO made in the original architecture? In fact, most of them have not changed. The most important change is the CPU supported by OpenVINO. We put it on the edge side, so what are the benefits of putting it on the edge side? This means that it can complete hardware deployment on site in the workshop, the distance to the machine is relatively close, and the network transmission time is relatively small, which improves the stability of inference.
After designing the product framework, we conducted a series of tests in the laboratory. At the beginning, we took more EL pictures from the production line, and then put the models trained under the Tensorflow framework into the model optimizer for optimization, and used the optimized model to locate defects. Throughout the experiment, it was found that the inference time was slightly slower than that of the Tensorflow model.
But overall, it meets customer requirements and can also accurately frame the location of defects. Then we started to conduct the test on site. The entire test results showed that after OpenVINO accelerated inference, the inference speed and inference effect of the product were slightly worse than the original model, but the overall effect was OK. In short, using OpenVINO to accelerate the EL detection system can not only help us on the technical side, but also reduce costs by nearly 30%. At the same time, the deployment conditions on the edge will be more convenient than those of GPU servers. Since the network transmission time is smaller, the stability of its inferred time is also greatly guaranteed. If you want to develop similar computer vision products in the future, I believe OpenVINO will definitely be a good choice.
. My part of the explanation is here. Next, Mr. Liu will bring you a quality management system based on industrial vision inspection.
quality management system based on industrial vision detection
First of all, from a horizontal perspective, if OpenVINO is combined with deep learning, the entire system used in the scene of lithium batteries and photovoltaic power stations, I believe it will have a good effect, especially like a power station, where all its detection occurs in difficult-to-control locations. Unlike production lines, people can debug next to it, but component power stations usually appear on the roof, so we can directly deploy it on the edge side through a cost-controllable edge calculator, and then do some back-passing through 5G base stations or wireless facilities, which will be more advantageous than traditional solutions.
In terms of quality management, if a manufacturing practitioner wants to see whether deep learning can bring significant improvements to the detection site, the answer is yes. First of all, the object of detection can be abstracted. In the photovoltaic industry, if you look at the photovoltaic EL intelligent detection system to detect between the welding site and the lamination site, after passing such a system, compared with the absence of a system, you can see that the overall distribution of yield rates will indeed be optimized. But this is not the whole of quality management, because if the inspection is just completed, this matter has not been completed yet. Why? Although it can save labor costs and improve detection accuracy, it does not make any substantial money.
Looking at the above structure, you can see that the location of the inspection before the layer has changed. In fact, there are other inspections before the welding site. Between the series welding site and the welding site, a new monitoring point can be added to call series inspection. And between the incoming material port and the string welding, we can add an incoming material to check. The current situation of the products tested in each link is actually similar, and the difference is not particularly large. However, after a round of inspection, we will find that during each round of incoming material inspection, the defects at the inlet have been picked out. If this link is not available, all the defects will continue to flow downward and accumulate until the last layer before the inspection link. However, if all the defects can be picked out in the incoming inspection ring, this part of the cost can be saved and thus help customers make money.
If you don’t look at it from the perspective of money, since each problem is captured, the yield rate will definitely be improved to a greater extent. There is a special name in industry called zero defect management. Can zero defect management be done in the past? It can be done, but the cost of zero defect management will be very high. Why? Because a testing site means buying new equipment and hiring new people, these things are new costs. However, it will become very simple if AI algorithms are used. If you expand your capabilities horizontally, in fact, AI algorithms can keep doing well at this site today and at another site tomorrow. This is the application of zero-defect management in the photovoltaic industry. Going further, there is another new thing in all these links: data.
What is the relationship between artificial intelligence and data? Because artificial intelligence is the easiest to generate high-quality data, this data has high value-addedness. What does it mean? In the past, when I was doing big data applications, I often mentioned a problem that the data was particularly "dirty". The "dirty" of this data did not necessarily mean that there were various things that made me unable to input into the system. Instead, the dirty data directly caused the availability of all data to be very low. When the dirty data enters the system, the system does not know what it is useful because it has been reviewed or just collected. Such problems were commonly encountered in the applications of industrial big data in the past.
Now, in the photovoltaic scene, you can see that the data is compared between different links through AI. If you find any defects that occur from the incoming material to the middle of the string welding, this is a number. Next, you find that the string welding to the middle of the front layer is another number. Then these numbers can be used to identify the productivity of each site, thereby discovering more problems.
After establishing the factory, I came to the battery factory again. You can look at the two battery cells in the picture above. What are the differences? The piece on the left is a bad scratch caused during equipment processing, while the piece on the right is a bad result caused by improper manual operation. The difference is very subtle and only a very experienced master can see it.
So what does this mean? If there are many bad things like the left, it proves that there are problems with the equipment that may have bad things. If the proportion of bad things on the right increases, then it is inferred that it is a problem for the people at the site. You can further find the equipment department or production squad leader. If it is a bad thing on the equipment, you can find directions to improve. Similarly, for the production squad leader, if it is a bad thing caused by humans, you can go back and solve this problem. So this is a quality management system.
Now there is an AI detection system and model. Can AI detection system be used to generate some quality reports? These quality reports can trace back to some causes of defects and further connect with the model. In this way, when you know what the cause of the bad is, you can find a way to create a better model. What should you do? We have a training platform that can hand over all training capabilities to customers. After developing some toolkits, we can quickly master AI capabilities. Finally, they use the entire closed loop to create high-quality models and continuously iterate. This is the quality management system of the entire AI.
So, if the manufacturing industry implements quality inspection, you can refer to this path. Why talk about paths? Because we found that all paths are not only a quality management, but also more predictive maintenance, intelligent yield optimization, etc., the applications we do are actually very easy to replicate, and this has a lot to do with OpenVINO.
OpenVINO is not just a platform that can help save costs and improve efficiency, but also a channel platform. What does it mean? When they first came into contact with Intel, they proposed a very important concept to us that Intel will mass-produce the standard version of public areas, integrate a series of algorithms, some inference frameworks, or some platform capabilities of training models, which can be integrated into the public areas.
As a supplier, just open our license in a public area and customers can try it or even buy it. We can put these things on the customer's site and guide them to use them, so they can use the public areas to bring their own products to customers and create more value for them. I think this is a more core capability of OpenVINO, which means that it is not only a technical platform, but also a channel ecological platform.
