Idan/fga-vqa

Open-ended VQA on factor graph attention

VQA v1, open-ended: the question and the image are attended jointly by the attention layer of Factor Graph Attention (CVPR 2019), and the answer is chosen from a 3000-answer vocabulary.

Code: github.com/idansc/fga.

Results

Trained on COCO train2014, evaluated on val2014, with 36 bottom-up region features. Scored with the official metric — answer normalization, and the average over the ten leave-one-annotator-out subsets.

objective	VQA accuracy
soft_ce (this checkpoint)	61.55
bce	60.71
ce (single label)	60.47

VQA is graded rather than single-label: ten annotators answer each question and an answer earns min(matches/3, 1). Supervising those scores rather than one "correct" id is worth about a point. The sigmoid-and-binary-cross-entropy form recommended by the 2017 challenge writeup also helps, but is 0.8 behind the softmax form here — with a 3000-way vocabulary read out by an argmax, keeping the answers competing suits the evaluation better than scoring them independently.

Usage

from fga.tasks.vqa import OpenEndedVQAModel

model = OpenEndedVQAModel.from_pretrained("Idan/fga-vqa")
out = model(question_input_ids=q, image_features=v)
answer_id = out.logits.argmax(-1)

Note on the multiple-choice model

The repository also contains HighOrderAttentionForVQA, the port of High-Order Attention Models for VQA (NeurIPS 2017), which adds the candidate answers as a third modality and a ternary factor over (region, word, answer) triples. It is kept for that factor and is not published here: it scores 61.4, below this model, despite being handed eighteen candidates to choose between. Something in it is wrong and has not been found.

Citation

@inproceedings{schwartz2019factor,
  title={Factor graph attention},
  author={Schwartz, Idan and Yu, Seunghak and Hazan, Tamir and Schwing, Alexander G},
  booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
  pages={2039--2048},
  year={2019}
}