11 November 15th, the 2022 Snapdragon Summit was held in Hawaii. When the official introduced the second generation of Snapdragon 8 mobile platform, he specifically mentioned the cognitive ISP (Cognitive-ISP). In an era when the public is increasingly demanding on mobile phone cameras, if you want to discuss the imaging technology of smartphones, ISP is a topic that cannot be bypassed.
ISP Leverage the advancement of smartphone imaging technology
So, what is cognitive ISP? According to the official introduction, cognitive ISP is a set of technologies to improve mobile phone imaging performance. With the first cognitive ISP, the second generation Snapdragon 8 realizes automatic enhancement of photos and videos through real-time semantic segmentation. The AI neural network allows the camera to perceive faces, facial features, hair, clothes and sky in the situation, and independently optimize, so that every detail can be customized and professional image tuning.
In layman's terms, before this, smartphones could not fully recognize the details of objects appearing in the camera, and the optimization of faces was also an overall optimization. Now, with the help of AI technology, smartphones can accurately identify and optimize each part individually, greatly improving the user's image experience.
"Dual-camera era", HDR recording and slow-motion videos are all related to the progress of ISP
In fact, before the release of cognitive ISPs, the ISPs integrated by Snapdragon mobile platforms have led to mobile images for many years. It can be said that every leap in smartphone imaging technology is accompanied by the breakthrough of Snapdragon Mobile Platform in ISP technology. Nowadays, with the help of AI, smartphones are quickly approaching the performance of professional-grade cameras.
In fact, the Snapdragon platform has integrated "dual ISP" a long time ago. Since then, high-pixel, 2PD focus, 4K video/HDR video shooting has gradually been implemented. In 2016, the Snapdragon 835 platform was equipped with Spectra 180 ISP, which supports up to two 16-megapixel cameras, and can greatly improve the speed of photography and image processing. It was the strongest ISP capability in the industry at that time, and mobile phones also entered the "dual camera era".
Only then can mobile photography take its current prototype.
In December 2017, the Snapdragon 845 mobile platform released at the Snapdragon Technology Summit integrated the new Spectra 280 ISP. At that time, the Snapdragon mobile platform integrating Spectra ISP supported 10-bit color depths in more than 1 billion tones, and realized motion compensation , time filtering (MCTF) and electronic image stabilization (EIS).
Today, those video features that have become popular, such as slow-motion video, HDR recording and high-speed performance capture, are all possible because of Spectra ISP. This generation of Spectra ISP also integrates a depth sensing system based on parallax . Its working principle is very similar to that of the human eye, and it can judge the relative distance of an object from a dual-lens perspective.
These technologies have greatly improved the smartphone image experience.
332018 released CV-ISP, and the era of optimization according to "frame" has arrived
In 2018, ISP technology ushered in a major upgrade. Snapdragon 855 brings the new Spectra 380 ISP, the world's first computer vision (CV) ISP, making a huge leap in computing photography and camera AI.
Spectra 380 eliminates bottlenecks in the image processing pipeline and performs machine vision processing directly on the ISP itself, thus saving up to 4 times the power consumption compared to previous generations of products, and greatly improves AI processing capabilities through cooperation with the AI engine.
For ordinary users, its experience improvement is huge.The Spectra 380 adds support for full HDR10+ video recording, making it possible to record videos with a larger dynamic range and a wider color space to take advantage of the capabilities of modern phones and TVs. Additionally, it will be able to handle higher resolution imaging, with its dual 14-bit CV-ISP pipeline supporting 22MP 30Hz concurrent capture of dual cameras, and 48MP 30Hz capture of a single image sensor.
This generation of ISPs allows smartphones to have computer vision (CV) capabilities, and the optimization of photos and videos can be specific to each frame. integrates a large number of computer vision capabilities that support hardware acceleration, and combines it with the Qualcomm AI engine, bringing another breakthrough in mobile phone imaging capabilities.
Although ISP technology has raised smartphone imaging technology to a new level, this generation of products still cannot replace human role in some aspects. For example, when taking night scene photos, ISP can take multiple photos at once and then synthesize them quickly, and the optimization of the photos is overall and comprehensive. However, in the real world, photographers edit photos are modified according to different parts of the picture, such as the sky, face, eyes, a tree, or even a leaf.
This time, the cognitive ISP released in 2022 finally solved this problem with the help of more powerful AI technology - to optimize each part separately.
Cognitive ISP: Let AI "refine" the image
Snapdragon 8's three-core 18-bit ISP can process three 36-megapixel cameras at the same time 30fps data, support up to 200 million pixels for a single camera, and also support a 108MP sensor with zero shutter lag.
combined with the real-time semantic segmentation technology of cognitive ISP, the smartphone image experience will be pushed to a new level. Although smartphones do not have super large CMOS like SLR cameras, the use of AI technology allows smartphones to take professional-grade images.
So, what is real-time semantic segmentation?
ISP Separate up to eight layers in the scene using the AI neural network inside the Hexagon processor, and then process each of these layers individually to better fine-tune the volume image quality. This allows the image processing pipeline to independently apply different levels of noise reduction, sharpening and color correction to various layers of the scene, such as personal faces, facial features, hair, clothing, sky, etc., and even remove glare from glasses. The entire process above
above occurs in real time. so-called "real-time" means that it happens instantly during the shooting process, and the completed picture and preview effect are consistent. Such complex calculations can be processed in a very short time, which also demonstrates the extremely high computing efficiency of cognitive ISPs.
Qualcomm said that cognitive ISP helps to enable Snapdragon Sight Snapdragon imaging technology to open a new era of professional quality camera experience, providing breakthrough, customized, professional-level quality enhancement technology.
So, how does cognitive ISP do it?
Vice President of Product Management, Judd Heape explained this in the example. In video mode, the second-generation Snapdragon 8 optimizes each layer with the separate tuning parameter , , all of which are done with the upgraded Hexagon processor in the new AI engine.
The second generation of Snapdragon 8 has also made deeper adjustments to the new image sensor. Samsung ISOCELL HP3 is the first 200 million pixel image sensor optimized for the platform, providing professional quality photos and videos.
revealed that it was establishing a partnership with Sony when Qualcomm released its first generation Snapdragon 8 mobile platform in December 2021. Now a year later, Qualcomm shares its latest labor results with Sony.The second generation Snapdragon 8 supports new quadruple exposure technology, which will make the most of Sony's new quadruple exposure digital overlap (QDOL) sensor, which allows it to gain greater dynamic range. QDOL4 technology enables videos and photos to present details in the darkest and brightest areas on the same screen.
Some users will have doubts. Will the ISP's such a large computing volume make the second-generation Snapdragon 8 a power-consuming power user?
However, with the help of AI, the AI energy efficiency and overall energy efficiency of Snapdragon 8 have been improved. First, the AI processors used to handle situational awareness tasks are added to two, and the recognition efficiency and computing speed are doubled. Second, the Hexagon processor adopts a larger tensor accelerator, bringing up to 4.35 times the AI performance improvement in some scenarios; it supports INT4 AI precision format, and can achieve a 60% energy efficiency improvement in continuous AI inference.
Overall, the CPU performance of the second generation Snapdragon 8 has been improved by 35%, and the energy efficiency has been improved by 40% . Moreover, a physical connection called "Hexagon Direct Connect" has been added between the Hexagon processor and the ISP, which means that the ISP can directly call the chip's AI capabilities, which is to provide more powerful image AI computing support. Graphic computing can be processed within the ISP, greatly improving the computing efficiency of AI in images.
Looking forward to a new experience in smartphone imaging
The first dual ISP brings mobile phones into the "dual photography era"; the first CV-ISP gives mobile phones a "computer vision" function... Every generation of ISPs on the Snapdragon Mobile Platform has promoted the continuous progress of mobile phone imaging functions.
Nowadays, ISP has become the standard configuration for smartphones. After the release of the second-generation Snapdragon 8 mobile platform, many OEM manufacturers and brands around the world, including Asus ROG, Honor , iQOO, Motorola, Nubia , OnePlus, OPPO, Red Magic , Sharp , Sony, vivo, Xiaomi, Redmi, Xingji Times/Meizu and ZTE , cognition ISP is expected to bring a brand new imaging experience to many smartphones.
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-