| |
| \justifying Referring expression grounding is a multimodal matching task involving language and vision, with the goal of locating the object in an image that is closest to the current referring expression(RE). The key to this task is not only to use the attribute of the subject in the text, but also to fully utilize the complex location information (absolute and relative location) in the image. Existing methods only encode location feature using information such as 5-dimensional coordinate and object area, which ignore some possible fine-grained clues, such as the overlap between two objects, which can be helpful in distinguishing. This paper proposes a general structure modeling approach based on mask information that is applicable to both absolute and relative location. By modeling at a fine-grained level, this paper achieves the use of the same structure for both types of location information, thereby improving modular training efficiency. Specifically, for any two objects in an image, the model extracts small-scale binary feature constructed by mask information, which correspond to the subject and object parts of the relationship, respectively. Then, it performs phrase-guided object attention on this feature and update the initial representation of the objects through multi-layer message passing to obtain cross-feature information. Conducting experiments on three of the most commonly used related datasets, results show that compared to previous methods, the model can improve the performance of modular-based referring expression grounding models in a generalizable manner, further achieving superior performance. |
|
| Keywords:Multimodality; Referring expression grounding; Location matching |
|
