Heart of the Machine Column
QQ Browser Search Technology Center, Tencent PCG ARC Lab
In recent years, the explosive growth of short video content on the Internet has put forward higher requirements for the ability of video-text retrieval (Video-Text Retrieval). In QQ browser search, videos usually contain multiple modal information such as title, cover image, video frame, audio, etc. Therefore, the video retrieval model introduces multi-modal features to characterize the correlation between the query and the video in multiple modalities, and perform comprehensive sorting to improve the overall satisfaction of the video search results. In addition, QQ Browser can also recommend search terms that the user may be interested in based on the video content the user is watching. The mining and sorting of recommended terms also require the model to effectively utilize the multi-modal information of the video.
In academia, there are currently two mainstream models for video retrieval:
A model based on CLIP image and text pre-training: the representative work is CLIP4Clip. The advantage is that it successfully transfers the knowledge of image and text pre-training to the video-text retrieval task. The disadvantage is the lack of utilization of other modal information in the video;
The model based on multi-modal information encoding, the representative work is MMT (Multi-Modal Transformer), the advantage is the use of transformer Jointly encode multiple modal information of the video (objects, actions, scenes, audio, etc.); the disadvantage is that multi-modal features (embeddings) with huge distribution differences are input into a black box, which not only fails to achieve effective fusion, but also increases the difficulty of model training.
Therefore, we propose a video retrieval model that not only learns image and text pre-training knowledge, but also effectively uses multi-modal tags to guide visual-text alignment: TABLE (TAgging Before aLignmEnt). The
TABLE model has achieved SOTA results on four public video-text retrieval data sets, and related research has been accepted by AAAI 2023. Advantages of
TABLE
video - Text retrieval is one of the very important tasks in the field of artificial intelligence . The key to the task lies in the alignment of visual information and text information in the same semantic space. In recent years, with the introduction of large-scale pre-training models, the performance of video retrieval models has been significantly improved. However, most current models (such as CLIP4Clip, CLIP2VIDEO, etc.) only use the visual modality for retrieval and recall, and the information of other video modalities is not effectively utilized. Therefore, the recall results are lacking in both relevance and diversity.
Some methods (such as MMT, MDMMT, etc.) try to use pre-trained models (experts) on multiple tasks to extract multi-modal embeddings, and then use the transformer structure to fuse the embeddings. However, the differences in embedding distributions extracted by different experts are huge, so it is difficult to complete the fusion of multi-modal information through a simple module. Furthermore, this black-box fusion method is difficult to explain, and it is impossible to know whether the final output really represents multi-modal information, or whether it just introduces some "learnable random inputs".
In response to the above problems, we propose a new video-text retrieval model guided by multi-modal tags: TABLE (TAgging Before aLignmEnt). The TABLE model first extracts information from each modality of the video and fuses it in the label domain. The fused multi-modal labels are ultimately used as anchors to promote visual-text semantic alignment.
We use a variety of pre-trained experts models to extract multi-modal information, including objects, people, scenes, actions and audio. Object and character information focus on the local or physical features of the video; scene information focuses on the overall and background features of the video; action information is an important feature that distinguishes the video from the image, and the timing information contained in it is often easily ignored; audio information provides some auxiliary information other than visual. In order to effectively and efficiently fuse information from different modalities,
uses label fields as a bridge for fusion. It has the following two main advantages:
embeddings generated by different experts are difficult to fuse with each other, and the label field method can generate unified and easy-to-interpret representations for different modal information.The
multi-modal label serves as an anchor, which can force the model to comprehensively learn the local features, global features, temporal features and other supplementary features of the video. Important segments (time) and important areas (space) of the video are highlighted under the influence of multi-modal tags, and the overall effect of video-text recall can be improved.
is shown in Figure 1. Through the object detector, you can get object labels such as "bowl, bottle" and the person label "woman"; through the image classifier, you can get the scene label "kitchen"; through the action detector, you can get the action label "cooking"; finally, through automatic speech recognition (Automatic Speech Recognition) and keyword extraction, you can get audio labels such as "marinating a chicken". The above tags provide rich information from multiple modalities of the video and serve as a bridge for interaction between vision and text, which can make alignment learning more accurate and efficient.
Figure 1: Multimodal information can be converted into tags to facilitate video-text alignment.
As shown in Figure 2, the TABLE model contains four encoders: visual encoder, label encoder, text encoder and cross-modal encoder. Among them, cross-modal coding jointly encodes the visual features of multiple frames and the label features of multi-modality, and the multi-modal labels serve as anchors for visual-text alignment. In addition, we introduce Video Text Matching Loss (VTM) and Masked Language Modeling (MLM) as additional supervision during model training. Compared with previous methods, the advantages of TABLE can be summarized as follows:
not only transfers the pre-trained knowledge of graphics and texts to video-text retrieval tasks, but also makes full use of the multi-modal information of videos, including objects, characters, scenes, actions, and audio;
multi-modal information is fused in the label domain, resulting in a unified representation, strong interpretability, and can be used as an anchor Promote the alignment of vision and text in semantic space;
jointly encodes multiple video frames and multi-modal tags, and introduces VTM and MLM tasks to enhance the finer-grained interaction between vision and text;
achieves SOTA results on four commonly used video retrieval data sets: MSR-VTT, MSVD, LSMDC and DiDeMo.
Figure 2: The overall framework of the TABLE model
model details
, multi-modal label mining
TABLE uses a variety of pre-trained models to extract multi-modal information from videos, as shown in Table 1. To reduce noise as much as possible, only high-confidence labels are retained in each model.
Table 1: Specific models used in tag mining
, visual and tag, text encoder
Visual encoder: Use the ViT model in CLIP for initialization. The video frame sequence is expressed as:
, then the output of the visual encoder can be expressed as:
, N represents the number of video frames.
tag, text encoder: Use the BERT model in the CLIP model for initialization. The transformer parts of the two encoders share parameters, but the linear projection layers are independent of each other. The output of the label and text encoder can be expressed as:
respectively. K and M represent the token length of the label and text respectively.
, cross-modal encoder
As shown in Figure 2, we built a cross-modal encoder guided by tags. The input to the encoder can be represented as:
, where
represents the output of the label encoder at [EOS] as an overall representation of the multi-modal label. The cross-modal encoder consists of a four-layer transformer structure, initialized with the first four layers of the CLIP text encoder. The cross-modal encoder deeply fuses visual information and label information. At the same time, due to the ordered input of video frames, the model can also learn the temporal information of the video. Among them, label information is used as a guide to filter out important video frames and visual areas from noisy visual features.The fusion output of the cross-modal encoder can be expressed as:
, and then we use the pooling layer and residual connection to obtain the overall representation of the cross-modal fusion:
,
is a learnable weight factor. The
text encoding end takes the features at [EOS] as the overall representation of the title,
. We define a video-text similarity function:
as a linear projection function. Finally, we can construct the loss function:
is a learnable temperature coefficient; B is the training batch size;
represents the contrast loss function from text to video and video to text respectively;
is the overall contrast loss.
, additional supervision
video text matching loss (VTM): use the output of the cross-modal encoder as the overall representation of the video to determine whether it matches the output of the text encoder. The overall representation of the video and the text features are simultaneously input into a joint encoder for further fusion. The joint encoder and the cross-modal encoder share parameters and are discarded during inference. Take the first feature
of the cross-modal encoder as the overall representation of the video, the text feature is
, and the predicted result is
. Then the loss function of VTM is:
t=0 represents a positive sample pair, t=1 represents a negative sample pair;
is a symbolic function, when t=1 The value is 1, otherwise it is 0;
is the sample logarithm. Difficult example mining is performed here, that is, negative samples with higher correlation scores have a greater probability of being sampled.
Masked Language Model Loss (MLM): Predicts occluded words in titles based on the overall representation of the video and the title context.
represents the occluded text,
represents the prediction result of the occluded word, then the loss function of MLM is:
is a symbolic function. When the occluded word of the i-th sample is v, the value is 1, otherwise it is 0; V is the vocabulary size; Q is the number of samples. The overall loss function of the
TABLE model:
experimental evaluation
, evaluation data set:
MSR-VTT, MSVD, LSMDC, DiDeMo.
. Evaluation indicators:
R@K: Recall at rank K, K=1, 5, 10, TOP-K The proportion of correct results included in the recall results;
MdR: Median Rank, the ranking of correct results median ;
MnR: Mean Rank, the average number of rankings of correct results.
. Performance comparison with SOTA
With the introduction of large-scale pre-training models such as CLIP, the performance of video retrieval models has also achieved major breakthroughs. Therefore, when comparing performance, we divide existing methods into CLIP-based and NO-CLIP methods. It can be seen that CLIP-based methods usually have better performance. For example, MMT’s R@1 on the text-to-video task is only 26.6, while CLIP4Clip can achieve 43.1.
Table 2: Performance comparison on the MSR-VTT data set, * indicates the use of the inference strategy proposed in CAMoE.
As shown in Table 2, the performance of the TABLE model on the MSR-VTT data set exceeds all methods and achieves SOTA results. After using the inference strategy, TABLE achieved an R@1 of 52.3 on the text-to-video task, significantly improving the effect of video retrieval.
Table 3: Performance comparison on the MSVD data set
As shown in Table 3, TABLE also achieves SOTA results on the MSVD data set. We believe that in training learning on small sample data sets, the guiding role of multi-modal labels is more important.
Table 4: Performance comparison on the LSMDC dataset
LSMDC The dataset contains the most videos, and each video only corresponds to one text, so the performance of most methods is poor. TABLE slightly improves the score on this dataset, as shown in Table IV.
Table 5: Performance comparison on DiDeMo dataset
On DiDeMo, consistent with other methods, we perform a video-paragraph retrieval task, that is, splicing all texts corresponding to a video into one long text. As shown in Table 5, TABLE has achieved a significant improvement of 5.3 percentage points in the R@1 indicator of text-to-video. The difficulty of this data set lies in the alignment of long videos and long texts. This is because long videos contain many meaningless fragments, from which meaningful frames and regions need to be selected, and key sentences and keywords also need to be found in long texts. In the TABLE model, the multimodal label is equivalent to an aligned anchor. The cross-modal encoder can highlight important video frames and spatial areas from complex visual information based on the multimodal label, thereby strengthening the alignment of the video and text, so the gain on this data set is obvious.
. Visual results analysis
Figure 3: Visual results of the TABLE model. The color bar below the frame sequence indicates the importance of each frame, which refers to the correlation with the multimodal label as a whole (a, b) or a specific word in the title (c, d), calculated by cross attention in the cross-modal encoder or joint encoder. Important spatial regions in each frame are highlighted to varying degrees, calculated by attention in the visual encoder.
In Figure 3, temporal attention and spatial attention are visualized simultaneously. In (a) and (b), multi-modal features are treated as a whole to calculate cross attention with visual features. As shown in the figure, cross attention pays more attention to video frames that are highly related to multi-modal labels. For example, in (a), the weights of frame 1 (related to "man" and "ballplayer") and frame 10 (related to "catching or throwing baseball" and "baseball glove") are the largest. In each frame, the model also focuses on some important spatial areas, such as the head, torso, baseball gloves , etc. Although the multi-modal label may contain some noise, such as "bird" in (a), since the multi-modal label guides visual attention as a whole, the model is more robust to these weak noises than and . In addition, we also found that for the task of video retrieval, action labels are more dominant in cross attention. For example, in (b), the model pays more attention to frames 9 and 12 that are highly related to "dancing ballet", while frames related to scenes and characters do not receive high weights.
Additionally, we visualize the cross attention in the joint encoder. The difference is that what is calculated here is the cross attention between visual features and specific word vectors in the title. For example, in (d), frames 10 and 11 depict close shots of "monkey" and are thus highlighted in temporal attention, while other frames depicting "people" or "swimming" are relatively suppressed. It can also be seen that the joint encoder can accurately model the fine-grained correlation between the video and individual text words, thanks to the two auxiliary tasks of VTM and MLM.
Since the joint encoder and the cross-modal encoder share parameters, the visualization results further prove the ability of the cross-modal encoder: it can filter out key frames and key areas from redundant visual features under the guidance of multi-modal tags, which is conducive to accurate video-text retrieval. The application of
TABLE in QQ browser
In order to meet the search needs of users when watching videos, QQ browser will display recommended words in the form of "related searches" at the bottom of the video. Click on the recommended words to learn more related information.
Figure 4: Application scenario example in QQ browser
Recommended words must not only be related to the video content, but also stimulate users’ click needs. In the sorting of relevance, it is necessary to consider the information of multiple modalities of the video, such as title, video screen, text, audio, etc. The TABLE model can combine multi-modal information and output a comprehensive score, representing the overall relevance of the video content and a single recommendation.Recommended words that meet the relevance conditions can be used as the final display result through some other strategies (such as CTR estimation, sensitive word suppression).
author team introduction
QQ Browser Search Technology Center team is the team responsible for search technology research and development of Tencent PCG information platform and service line. Relying on Tencent content ecology, it drives product innovation through user research to provide users with multi-faceted information needs such as graphics, information, novels, long and short videos, and services. In terms of algorithms, based on technologies such as natural language processing, deep learning, multi-modal understanding and generation, knowledge calculation and application, we build technical directions such as content understanding, correlation and sorting, multi-modal search, intelligent question and answer, multi-language translation, search recommendations, etc., to explore and apply the industry's advanced technologies to create a better user search experience; in terms of engineering, we build a mid-stage industrialized system for search technology and polish a high-performance, high-availability, low-cost tens of billions-level retrieval system to provide Tencent PCG The search scenarios of various content businesses provide basic search engine services. Currently, it supports multiple PCG product lines such as QQ Browser, Tencent Video, Tencent News, and Tencent Microvision .
Tencent PCG ARC Lab is the "scout" and "special forces" of Tencent PCG, standing on the front line of Tencent's exploration and challenge of cutting-edge technologies related to smart media. The affiliated Tencent PCG is a large-scale business group integrating social networking, traffic and content platforms, and its business needs cover almost all technologies in artificial intelligence and big data. In the past two years, ARC Lab has adhered to the purpose of doing top-notch (world-leading) ground-level (application-based) scientific research, and has attracted a group of outstanding talents and outstanding interns at home and abroad. It has published more than 50 papers at top international conferences, and many technical achievements have been implemented in practical applications.
