Shenzhen Wangchen Technology Company has the leading AI vision algorithm team and real-time rendering engine . For ten years, it has been focusing on the research and development of sports games and its related AI vision technology, and has accumulated a huge database of player roles. Not only are there 20,000 professional players registered in FIFA accurate to centimeters of player body size, bone length, BMI, body fat and muscle ratio, but also thousands of top players with high precision 3D models, and high-precision models of almost all major stadiums around the world.
Based on these high-quality structured data, Arena4D can render the captured results in a real-time rendering engine with nearly photo-level real quality, using completely free lens effects and mirror-based methods to achieve an unprecedented shocking experience.
Arena4D Space tracking and identification and data processing
faces computing power challenges
- Players and balls move fast, have large posture changes, and have a large appearance similarity between clothes, making it difficult to use the industry-wide MOT algorithm for high-quality tracking and posture recognition.
- Even if it is a 4k image, the players and balls are too small in the image, and it is difficult to track.
- Multiple HD cameras each frame of the image needs to be uploaded to the graphics card for real-time transcoding, noise reduction and other pre-processing, and the data throughput is relatively large.
- is based on the neural network computing pipeline, which requires real-time tracking, recognition, posture estimation and noise reduction calculations of multiple perspectives and multiple athletes.
- In multiple AI model cascade computing pipelines, data processing and copying between each AI model takes up a lot of time.
NVIDIA AI computing platform provides computing power support for Arena4D
to realize the full process GPU acceleration
Based on the above challenges, Wangchen Technology chose to use NVIDIA AI computing platform to provide support, which brought huge support and improvement to R&D and project implementation after use.
. Generate huge amounts of synthetic data based on GalaSports sports game rendering and accelerate training on NVIDIA data center GPU. First, train the basic network on synthetic data with GT, and then finetune on the real data marked manually.
and Arena4D use NVIDIA TensorRT inference acceleration engine . TensorRT first quantizes the neural network, then optimizes and merges operators, and finally passes Batch to improve the inference processing speed by 4-12 times compared to the original one. It can efficiently process 4k high-definition data of multiple cameras, extract the player's 3D Pose and appearance characteristics in real time and perform matching calculations.
. Copy the original data running based on CPU, image format conversion, image crop/resize and other time-consuming operation pipelines, using CUDA to fully switch to GPU processing, which is accelerated by 50~100 times compared to the CPU implemented solutions.
4. In view of the characteristics of fast athletes in sports scenes, large movement amplitude and small size in images, the network structure and operators are improved to make them suitable for sparse changes in the stadium and track scenes with small characters; the MOT algorithm is improved, and the 2D tracking results are finetune through the fusion network of multiple views to achieve multi-view tracking in 3D space.
Figure 1: Overall process
Picture source and belongings: Galasports Wangchen Technology
5. For the calculation delay problem of neural network pipelines, the model structure is first optimized based on the use scenarios of sports games and camera perspectives. According to the camera position and field scale of different sports types, a recognition network specifically for specific games is designed, which greatly reduces the complexity of the network; based on the posture of continuous frames, the player characteristics such as bone length and BMI are calculated, and the recognition accuracy is matched in the database to improve the recognition accuracy; in response to the IObound problem of multi-camera from memory to video memory, the parallelization of memory copying and data processing is achieved using CUDA multi-stream technology, which reduces the parallelization of memory copying and data processing is reduced. overhead, the copy and transcoding of 4k camera data is reduced from 50ms to 30ms; then the network is quantized using quantization tools to accelerate fp16/int8 (QAT), and finally NVIDIA TensorRT is used to compile for NVIDIA data center GPUs to achieve optimal performance on NVIDIA data center GPUs.
Figure 2: Computational delay solution ideas
Image source and belongings: Galasports Wangchen Technology
6. The entire pipeline is compiled and runs on the NVIDIA data center GPU through TensorRT. Each video signal is calculated in real time by one GPU, and finally the multiple calculation results are noise-reduced and fused in the CPU. Finally, taking the football field scene as an example, the tracking target is 1 football + 22 players + 3 coaches' positions and bones. On one NVIDIA data center GPU device, we achieved an average speed of 50ms/frame, and on two NVIDIA data center GPU devices, the average speed of 30ms/frame is achieved, and the entire pipeline is 18 times higher than the prototype.
uses NVIDIA TensorRT inference acceleration engine and NVIDIA data center computing card . The entire pipeline of Arena4D can run in real time, further improving the user experience and laying a good foundation for expanding other businesses in the direction of AI sports, and even adding XR scenarios.
NVIDIA helps Wangchen Technology
to create a new generation of real-time AI sports full-scene multi-dimensional reconstruction engine
Arena4D Through NVIDIA TensorRT and other technologies, it realizes multi-view tracking in 3D space, improves the accuracy of identifying player characteristics such as bone length and BMI, completes parallelization of memory copy and data processing, improves network structure and operators to make it suitable for sparse changes in the stadium, and ultimately realizes the full process GPU acceleration, and speed increases by more than 50~100 times.
AI The sports industry is one of the most popular fields of artificial intelligence application. Traditional sports companies have also begun to embrace technology and have introduced the artificial intelligence industry one after another. A technology enterprise integrating technology, talents and scenario advantages must actively explore and develop in the long term with an excellent partner. The GPU solutions provided by NVIDIA solve the technical problems of deep learning and machine learning, allowing Wangchen Technology to continuously innovate products in the field of AI sports, broaden AI application scenarios, and provide audiences and fans with a new digital viewing experience in the new era.
NVIDIA Startup Acceleration Plan
Wangchen Technology is a member company of NVIDIA Startup Acceleration Plan (NVIDIA Inception). NVIDIA's startup acceleration program is a free membership system that aims to cultivate outstanding startups that disrupt the industry landscape. The plan combines well-known venture capital institutions at home and abroad, entrepreneurship incubators, entrepreneurship accelerators, industry partners, technology and entrepreneurship media, etc. to create an entrepreneurship acceleration ecosystem. It can provide a series of services such as product discounts, technical support, market promotion, financing docking, and business recommendation to accelerate the development of startups.
Want to get NVIDIA startup acceleration ecosystem support? Scan the QR code below and fill out the intention application form in just one minute to get quick contact.
