(Graduate Research Assistant, Faculty of Engineering & Technology (FET) , Multimedia University, Melaka, Malaysia)(2) * Joseph Emerson Raja
(Assistant Professor, FACULTY OF ENGINEERING AND TECHNOLOGY (FET), Multimedia University, Malaysia)(3) Md. Jakir Hossein
(Associate Professor, FACULTY OF ENGINEERING AND TECHNOLOGY (FET), Multimedia University, Malaysia)*corresponding author
AbstractAutomatic Video Captions are increasingly important to use in applications in the accessibility, search, education, and content moderation areas. Generating accurate and context-aware captions is difficult for lengthy and complex videos. Recent surveys focus more on language models than on the vision side. This means that the contribution of visual feature extraction towards the performance of the model is not studied properly. Here we attempt to fill this gap by conducting reviews of video captioning literature from the vision viewpoint, comparing CNNs-based and transformer-based feature extraction, and detecting trends, pros and cons. Approximately 100 papers published between 2017 - 2025 were reviewed and analyzed based on base vision, dataset and evaluation metrics. The review classified the selected papers into CNN-based pipelines and transformer-based techniques. Moreover, it compared their rank measured by application popularity, dataset usage, and performance scores across popular datasets/benchmarks. The review shows that CNNs were dominated before 2021, but transformer-based approaches now lead due to their ability to capture long-range temporal dependencies and multimodal interactions. However, CNNs continue to perform well for short videos. As for Dataset usage, patterns show that ActivityNet Captions and YouCook2 are suited for long-form reasoning because of timely manner captioning, while MSR-VTT and MSVD are effective for short clips (one sentence caption). In terms of Special semantic quality metrics, CIDEr and METEOR are more semantically appropriate compared to n-gram quality metrics like BLEU. This review emphasizes transformer-based architecture, particularly multimodal models, which will be the future of video captioning. However, challenges remain in handling lengthy videos, adapting across domains and languages, and ensuring efficient deployment (small, optimized model). Addressing these gaps will add value by making video captioning systems more reliable, interpretable, and broadly usable in real-world contexts.
KeywordsVideo Captioning, Video feature extraction, Vision-Language Models
|
DOIhttps://doi.org/10.26555/ijain.v12i3.2280 |
Article metricsAbstract views : 211 | PDF views : 3 |
Cite |
Full Text Download
|
References
[1] S. Venugopalan, M. Rohrbach, J. Donahue, R. Mooney, T. Darrell, and K. Saenko, “Sequence to Sequence -- Video to Text,” in 2015 IEEE International Conference on Computer Vision (ICCV), IEEE, Dec. 2015, pp. 4534–4542. doi: 10.1109/ICCV.2015.515.
[2] A. Vaswani et al., “Attention is All you Need,” in Advances in Neural Information Processing Systems, 2017, p. 11. Accessed: Feb. 17, 2026. [Online]. Available: https://arxiv.org/abs/1706.03762
[3] A. Dosovitskiy et al., “An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale,” in ICLR 2021 - 9th International Conference on Learning Representations, International Conference on Learning Representations, ICLR, Jun. 2021, p. 22. Accessed: Feb. 17, 2026. [Online]. Available: http://arxiv.org/abs/2010.11929
[4] G. Bertasius, H. Wang, and L. Torresani, “Is Space-Time Attention All You Need for Video Understanding?,” in Proceedings of Machine Learning Research, Jun. 2021, pp. 813–824. [Online]. Available: http://arxiv.org/abs/2102.05095
[5] J. Li, D. Li, S. Savarese, and S. Hoi, “BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models,” Proc. Mach. Learn. Res., vol. 202, pp. 20351–20383, Jun. 2023, [Online]. Available: http://arxiv.org/abs/2301.12597
[6] J.-B. Alayrac et al., “Flamingo: A Visual Language Model for Few-Shot Learning,” in Advances in Neural Information Processing Systems 35, San Diego, California, USA: Neural Information Processing Systems Foundation, Inc. (NeurIPS), 2022, pp. 23716–23736. doi: 10.52202/068431-1723.
[7] B. Lin et al., “Video-LLaVA: Learning United Visual Representation by Alignment Before Projection,” in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Y. Al-Onaizan, M. Bansal, and Y.-N. Chen, Eds., Stroudsburg, PA, USA: Association for Computational Linguistics, Nov. 2024, pp. 5971–5984. doi: 10.18653/v1/2024.emnlp-main.342.
[8] S. Islam, A. Dash, A. Seum, A. H. Raj, T. Hossain, and F. M. Shah, “Exploring Video Captioning Techniques: A Comprehensive Survey on Deep Learning Methods,” SN Comput. Sci., vol. 2, no. 2, p. 120, Apr. 2021, doi: 10.1007/s42979-021-00487-x.
[9] R. Akhare and S. Shinde, “Query Focused Video Summarization: A Review,” in Artificial Intelligence, A. A. Sk, T. Turki, T. K. Ghosh, S. Joardar, and S. Barman, Eds., Cham: Springer Nature Switzerland, 2022, pp. 202–212. doi: 10.1007/978-3-031-22485-0_19.
[10] M. Liu, L. Nie, Y. Wang, M. Wang, and Y. Rui, “A Survey on Video Moment Localization,” ACM Comput. Surv., vol. 55, no. 9, pp. 1–37, Sep. 2023, doi: 10.1145/3556537.
[11] M. Abdar et al., “A Review of Deep Learning for Video Captioning,” IEEE Trans. Pattern Anal. Mach. Intell., pp. 1–20, 2024, doi: 10.1109/TPAMI.2024.3522295.
[12] J. Zhang, J. Huang, S. Jin, and S. Lu, “Vision-Language Models for Vision Tasks: A Survey,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 46, no. 8, pp. 5625–5644, Aug. 2024, doi: 10.1109/TPAMI.2024.3369699.
[13] I. Qasim, A. Horsch, and D. Prasad, “Dense Video Captioning: A Survey of Techniques, Datasets and Evaluation Protocols,” ACM Comput. Surv., vol. 57, no. 6, pp. 1–36, Jun. 2025, doi: 10.1145/3712059.
[14] Z. Liu and R. Song, “Survey of Dense Video Captioning: Techniques, Resources, and Future Perspectives,” Appl. Sci., vol. 15, no. 9, p. 4990, Apr. 2025, doi: 10.3390/app15094990.
[15] Z. Li, X. Wu, H. Du, F. Liu, H. Nghiem, and G. Shi, “A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challenges,” in 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), IEEE, Jun. 2025, pp. 1578–1597. doi: 10.1109/CVPRW67362.2025.00147.
[16] B. Günsel, A. M. Tekalp, and P. J. L. va. Beek, “Content-based access to video objects: Temporal Segmentation, visual summarization, and feature extraction,” Signal Processing, vol. 66, no. 2, pp. 261–280, Apr. 1998, doi: 10.1016/S0165-1684(98)00010-3.
[17] L.-C. Chiu, T.-S. Chang, J.-Y. Chen, and N. Y.-C. Chang, “Fast SIFT Design for Real-Time Visual Feature Extraction,” IEEE Trans. Image Process., vol. 22, no. 8, pp. 3158–3167, Aug. 2013, doi: 10.1109/TIP.2013.2259841.
[18] Yuan Xie, Li-Feng Liu, Cui-Hua Li, and Yan-Yun Qu, “Unifying visual saliency with HOG feature learning for traffic sign detection,” in 2009 IEEE Intelligent Vehicles Symposium, IEEE, Jun. 2009, pp. 24–29. doi: 10.1109/IVS.2009.5164247.
[19] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet classification with deep convolutional neural networks,” Commun. ACM, vol. 60, no. 6, pp. 84–90, May 2017, doi: 10.1145/3065386.
[20] K. Simonyan and A. Zisserman, “Very Deep Convolutional Networks for Large-Scale Image Recognition,” 3rd Int. Conf. Learn. Represent. ICLR 2015 - Conf. Track Proc., vol. 6, pp. 1–14, Apr. 2015, [Online]. Available: http://arxiv.org/abs/1409.1556
[21] K. He, X. Zhang, S. Ren, and J. Sun, “Deep Residual Learning for Image Recognition,” in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, Jun. 2016, pp. 770–778. doi: 10.1109/CVPR.2016.90.
[22] D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri, “Learning Spatiotemporal Features with 3D Convolutional Networks,” in 2015 IEEE International Conference on Computer Vision (ICCV), IEEE, Dec. 2015, pp. 4489–4497. doi: 10.1109/ICCV.2015.510.
[23] J. Carreira and A. Zisserman, “Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset,” in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, Jul. 2017, pp. 4724–4733. doi: 10.1109/CVPR.2017.502.
[24] C. Yan et al., “STAT: Spatial-Temporal Attention Mechanism for Video Captioning,” IEEE Trans. Multimed., vol. 22, no. 1, pp. 229–241, Jan. 2020, doi: 10.1109/TMM.2019.2924576.
[25] S. Ren, K. He, R. Girshick, and J. Sun, “Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 39, no. 6, pp. 1137–1149, Jun. 2017, doi: 10.1109/TPAMI.2016.2577031.
[26] Z. Liu et al., “Swin Transformer: Hierarchical Vision Transformer using Shifted Windows,” in 2021 IEEE/CVF International Conference on Computer Vision (ICCV), IEEE, Oct. 2021, pp. 9992–10002. doi: 10.1109/ICCV48922.2021.00986.
[27] H. Zhang, X. Li, and L. Bing, “Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Y. Feng and E. Lefever, Eds., Stroudsburg, PA, USA: Association for Computational Linguistics, Dec. 2023, pp. 543–553. doi: 10.18653/v1/2023.emnlp-demo.49.
[28] B. He et al., “MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understanding,” in 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, Jun. 2024, pp. 13504–13514. doi: 10.1109/CVPR52733.2024.01282.
[29] M. A. Rahim, A. S. M. Miah, H. S. Akash, J. Shin, M. I. Hossain, and M. N. Hossain, “An Advanced Deep Learning Based Three-Stream Hybrid Model for Dynamic Hand Gesture Recognition,” vol. 1, pp. 1–26, Aug. 2024, [Online]. Available: http://arxiv.org/abs/2408.08035
[30] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “BLEU,” in Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL ’02, in ACL ’02. Morristown, NJ, USA: Association for Computational Linguistics, 2001, p. 311. doi: 10.3115/1073083.1073135.
[31] S. Banerjee and A. Lavie, “METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments,” in Proceedings of the {ACL} Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, J. Goldstein, A. Lavie, C.-Y. Lin, and C. Voss, Eds., Ann Arbor, Michigan: Association for Computational Linguistics, Jun. 2005, pp. 65–72. [Online]. Available: https://aclanthology.org/W05-0909/
[32] R. Vedantam, C. L. Zitnick, and D. Parikh, “CIDEr: Consensus-based image description evaluation,” in 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, Jun. 2015, pp. 4566–4575. doi: 10.1109/CVPR.2015.7299087w.
[33] C.-Y. Lin, “ROUGE: A Package for Automatic Evaluation of Summaries,” 2004, pp. 74–81. [Online]. Available: https://aclanthology.org/W04-1013/
[34] R. Krishna, K. Hata, F. Ren, L. Fei-Fei, and J. C. Niebles, “Dense-Captioning Events in Videos,” in 2017 IEEE International Conference on Computer Vision (ICCV), IEEE, Oct. 2017, pp. 706–715. doi: 10.1109/ICCV.2017.83.
[35] L. Zhou, N. Louis, and J. J. Corso, “Weakly-Supervised Video Object Grounding from Text by Loss Weighting and Object Interaction,” Br. Mach. Vis. Conf. 2018, BMVC 2018, vol. 1, pp. 1–16, Jul. 2018, [Online]. Available: http://arxiv.org/abs/1805.02834
[36] J. Xu, T. Mei, T. Yao, and Y. Rui, “MSR-VTT: A Large Video Description Dataset for Bridging Video and Language,” in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, Jun. 2016, pp. 5288–5296. doi: 10.1109/CVPR.2016.571.
[37] D. Chen and W. Dolan, “Collecting Highly Parallel Data for Paraphrase Evaluation,” in Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, D. Lin, Y. Matsumoto, and R. Mihalcea, Eds., Portland, Oregon, USA: Association for Computational Linguistics, Jun. 2011, pp. 190–200. [Online]. Available: https://aclanthology.org/P11-1020/
[38] X. Wang, J. Wu, J. Chen, L. Li, Y.-F. Wang, and W. Y. Wang, “VaTeX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research,” in 2019 IEEE/CVF International Conference on Computer Vision (ICCV), IEEE, Oct. 2019, pp. 4580–4590. doi: 10.1109/ICCV.2019.00468.
[39] B. A. Plummer, L. Wang, C. M. Cervantes, J. C. Caicedo, J. Hockenmaier, and S. Lazebnik, “Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models,” in 2015 IEEE International Conference on Computer Vision (ICCV), 2015, pp. 2641–2649. doi: 10.1109/ICCV.2015.303.
[40] G. A. Sigurdsson, A. Gupta, C. Schmid, A. Farhadi, and K. Alahari, “Charades-Ego: A Large-Scale Dataset of Paired Third and First Person Videos,” no. 2, pp. 1–3, Apr. 2018, [Online]. Available: http://arxiv.org/abs/1804.09626
[41] H. Agrawal et al., “nocaps: novel object captioning at scale,” in 2019 IEEE/CVF International Conference on Computer Vision (ICCV), IEEE, Oct. 2019, pp. 8947–8956. doi: 10.1109/ICCV.2019.00904.
[42] K. Grauman et al., “Ego4D: Around the World in 3,600 Hours of Egocentric Video,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 47, no. 11, pp. 9468–9509, Nov. 2025, doi: 10.1109/TPAMI.2024.3381075.
[43] S. Pini, M. Cornia, F. Bolelli, L. Baraldi, and R. Cucchiara, “M-VAD names: a dataset for video captioning with naming,” Multimed. Tools Appl., vol. 78, no. 10, pp. 14007–14027, May 2019, doi: 10.1007/s11042-018-7040-z.
[44] V. Estevam, R. Laroca, H. Pedrini, and D. Menotti, “Dense video captioning using unsupervised semantic information,” J. Vis. Commun. Image Represent., vol. 107, p. 104385, Mar. 2025, doi: 10.1016/j.jvcir.2024.104385.
[45] E. Blanco-Fernández, C. Gutiérrez-Álvarez, N. Nasri, S. Maldonado-Bascón, and R. J. López-Sastre, “Live video captioning,” Multimed. Tools Appl., vol. 84, no. 35, pp. 44863–44895, May 2025, doi: 10.1007/s11042-025-20908-w.
[46] Z. Xie, Y. Yang, Y. Yu, J. Wang, Y. Jiang, and X. Wu, “Exploring Temporal Event Cues for Dense Video Captioning in Cyclic Co-Learning,” Proc. AAAI Conf. Artif. Intell., vol. 39, no. 8, pp. 8771–8779, Apr. 2025, doi: 10.1609/aaai.v39i8.32948.
[47] S. Chu, S. Seo, and B. Han, “Fine-Grained Captioning of Long Videos through Scene Graph Consolidation,” Proc. Mach. Learn. Res., vol. 267, pp. 10801–10817, Jul. 2025, [Online]. Available: http://arxiv.org/abs/2502.16427
[48] T. Nishimura, A. Hashimoto, Y. Ushiku, H. Kameko, and S. Mori, “Recipe Generation from Unsegmented Cooking Videos,” ACM Trans. Multimed. Comput. Commun. Appl., vol. 22, no. 9, pp. 1–25, Sep. 2026, doi: 10.1145/3649137.
[49] X. Zhou et al., “Streaming Dense Video Captioning,” in 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, Jun. 2024, pp. 18243–18252. doi: 10.1109/CVPR52733.2024.01727.
[50] A. Hanna-Asaad, D. Aspandi-Latif, and T. Zaharia, “MI-Cap: A Multi-Modal Interpretable Model for Video Captioning,” in 2025 International Conference on Content-Based Multimedia Indexing (CBMI), IEEE, Oct. 2025, pp. 1–8. doi: 10.1109/CBMI66578.2025.11339325.
[51] A. Nadeem, F. Sardari, R. Dawes, S. S. Husain, A. Hilton, and A. Mustafa, “NarrativeBridge: Enhancing Video Captioning with Causal-Temporal Narrative,” 13th Int. Conf. Learn. Represent. ICLR 2025, pp. 21619–21647, Feb. 2025, [Online]. Available: http://arxiv.org/abs/2406.06499
[52] M. Kim, H. B. Kim, J. Moon, J. Choi, and S. T. Kim, “Do You Remember? Dense Video Captioning with Cross-Modal Memory Retrieval,” in 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, Jun. 2024, pp. 13894–13904. doi: 10.1109/CVPR52733.2024.01318.
[53] J. Xu et al., “Retrieval-Augmented Egocentric Video Captioning,” in 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, Jun. 2024, pp. 13525–13536. doi: 10.1109/CVPR52733.2024.01284.
[54] W. Chai et al., “AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmark,” 13th Int. Conf. Learn. Represent. ICLR 2025, vol. 4, pp. 83337–83381, Apr. 2025, [Online]. Available: http://arxiv.org/abs/2410.03051
[55] A. Yang et al., “Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning,” in 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, Jun. 2023, pp. 10714–10726. doi: 10.1109/CVPR52729.2023.01032.
[56] K. Yamazaki, K. Vo, Q. S. Truong, B. Raj, and N. Le, “VLTinT: Visual-Linguistic Transformer-in-Transformer for Coherent Video Paragraph Captioning,” Proc. AAAI Conf. Artif. Intell., vol. 37, no. 3, pp. 3081–3090, Jun. 2023, doi: 10.1609/aaai.v37i3.25412.
[57] T. Xu, Y. Cui, X. He, and C. Liu, “A latent topic‐aware network for dense video captioning,” IET Comput. Vis., vol. 17, no. 7, pp. 795–803, Oct. 2023, doi: 10.1049/cvi2.12195.
[58] Y. Xie, J. Niu, Y. Zhang, and F. Ren, “Global-Shared Text Representation Based Multi-Stage Fusion Transformer Network for Multi-Modal Dense Video Captioning,” IEEE Trans. Multimed., vol. 26, pp. 3164–3179, 2024, doi: 10.1109/TMM.2023.3307972.
[59] Y. Wei, S. Yuan, M. Chen, and L. Wang, “Enhancing Multimodal Alignment with Momentum Augmentation for Dense Video Captioning,” in ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, Jun. 2023, pp. 1–5. doi: 10.1109/ICASSP49357.2023.10095659.
[60] Y. Wei et al., “MPP-net: Multi-perspective perception network for dense video captioning,” Neurocomputing, vol. 552, no. October, p. 126523, Oct. 2023, doi: 10.1016/j.neucom.2023.126523.
[61] T. Wang, J. Zhang, F. Zheng, W. Jiang, R. Cheng, and P. Luo, “Learning Grounded Vision-Language Representation for Versatile Understanding in Untrimmed Videos,” vol. 2, pp. 1–1, May 2023, [Online]. Available: http://arxiv.org/abs/2303.06378
[62] J. Scherer, D. Bhowmik, and A. Scherp, “Event and Entity Extraction from Generated Video Captions,” in Machine Learning and Knowledge Extraction, A. Holzinger, P. Kieseberg, F. Cabitza, A. Campagner, A. M. Tjoa, and E. Weippl, Eds., Cham: Springer Nature Switzerland, 2023, pp. 280–300. doi: 10.1007/978-3-031-40837-3_17.
[63] Y. Qian, Y. Mao, Z. Chen, C. Li, O. T. Bloh, and Q. Huang, “Dense video captioning based on local attention,” IET Image Process., vol. 17, no. 9, pp. 2673–2685, Jul. 2023, doi: 10.1049/ipr2.12819.
[64] L. H. Palivela, S. Swetha, M. Nithish Guhan, and M. Prasanna Venkatesh, “Dense Video Captioning Using Video-Audio Features and Topic Modeling Based on Caption,” in Proceedings of Fourth International Conference on Communication, Computing and Electronics Systems, V. Bindhu, J. M. R. S. Tavares, and C. Vuppalapati, Eds., Singapore: Springer Nature Singapore, 2023, pp. 515–525. doi: 10.1007/978-981-19-7753-4_40.
[65] P. Li, P. Zhang, T. Wang, and H. Xiao, “Time–frequency recurrent transformer with diversity constraint for dense video captioning,” Inf. Process. Manag., vol. 60, no. 2, p. 103204, Mar. 2023, doi: 10.1016/j.ipm.2022.103204.
[66] Y. Jo, S. Lee, A. S. Lee, H. Lee, H. Oh, and M. Seo, “Zero-Shot Dense Video Captioning by Jointly Optimizing Text and Moment,” vol. 1, pp. 1–19, Jul. 2023, [Online]. Available: http://arxiv.org/abs/2307.02682
[67] X. Huang, K.-H. Chan, W. Wu, H. Sheng, and W. Ke, “Fusion of Multi-Modal Features to Enhance Dense Video Caption,” Sensors, vol. 23, no. 12, p. 5565, Jun. 2023, doi: 10.3390/s23125565.
[68] S. Han, J. Liu, J. Zhang, P. Gong, X. Zhang, and H. He, “Lightweight dense video captioning with cross-modal attention and knowledge-enhanced unbiased scene graph,” Complex Intell. Syst., vol. 9, no. 5, pp. 4995–5012, Oct. 2023, doi: 10.1007/s40747-023-00998-5.
[69] W. Choi, J. Chen, and J. Yoon, “Step by Step: A Gradual Approach for Dense Video Captioning,” IEEE Access, vol. 11, pp. 51949–51959, 2023, doi: 10.1109/ACCESS.2023.3279816.
[70] W. Chen, J. Niu, and X. Liu, “MRCap: Multi-modal and Multi-level Relationship-based Dense Video Captioning,” in 2023 IEEE International Conference on Multimedia and Expo (ICME), IEEE, Jul. 2023, pp. 2615–2620. doi: 10.1109/ICME55011.2023.00445.
[71] Z. Yan, Y. Chen, J. Song, and J. Zhu, “Multimodal feature fusion based on object relation for video captioning,” CAAI Trans. Intell. Technol., vol. 8, no. 1, pp. 247–259, Mar. 2023, doi: 10.1049/cit2.12071.
[72] W. Zhu, B. Pang, A. V. Thapliyal, W. Y. Wang, and R. Soricut, “End-to-end Dense Video Captioning as Sequence Generation,” Proc. - Int. Conf. Comput. Linguist. COLING, vol. 29, no. 1, pp. 5651–5665, Sep. 2022, [Online]. Available: http://arxiv.org/abs/2204.08121
[73] Q. Zhang, Y. Song, and Q. Jin, “Unifying Event Detection and Captioning as Sequence Generation via Pre-training,” in Computer Vision -- ECCV 2022, S. Avidan, G. Brostow, M. Cissé, G. M. Farinella, and T. Hassner, Eds., Cham: Springer Nature Switzerland, 2022, pp. 363–379. doi: 10.1007/978-3-031-20059-5_21.
[74] K. Yamazaki et al., “VLCAP: Vision-Language with Contrastive Learning for Coherent Video Paragraph Captioning,” in 2022 IEEE International Conference on Image Processing (ICIP), IEEE, Oct. 2022, pp. 3656–3661. doi: 10.1109/ICIP46576.2022.9897766.
[75] T. Wang, Z. Liu, F. Zheng, Z. Lu, R. Cheng, and P. Luo, “Semantic-Aware Pretraining for Dense Video Captioning,” vol. 1, pp. 2019–2021, Apr. 2022, [Online]. Available: http://arxiv.org/abs/2204.07449
[76] J. Prudviraj, M. I. Reddy, C. Vishnu, and C. K. Mohan, “AAP-MIT: Attentive Atrous Pyramid Network and Memory Incorporated Transformer for Multisentence Video Description,” IEEE Trans. Image Process., vol. 31, pp. 5559–5569, 2022, doi: 10.1109/TIP.2022.3195643.
[77] A. Mittal, S. Ghosal, and R. Bansal, “SAVCHOI: Detecting Suspicious Activities using Dense Video Captioning with Human Object Interactions,” vol. 2, pp. 1–14, Oct. 2022, [Online]. Available: http://arxiv.org/abs/2207.11838
[78] J. Madake, S. Bhatlawande, S. Purandare, S. Shilaskar, and Y. Nikhare, “Dense Video Captioning using BiLSTM Encoder,” in 2022 3rd International Conference for Emerging Technology (INCET), IEEE, May 2022, pp. 1–6. doi: 10.1109/INCET54531.2022.9824569.
[79] Y. Lu, Z. Zhang, Y. Chen, C. Yuan, B. Li, and W. Hu, “PIC 4th Challenge,” in Proceedings of the 4th on Person in Context Workshop, in PIC ’22. New York, NY, USA: ACM, Oct. 2022, pp. 17–21. doi: 10.1145/3552455.3555816.
[80] C. Liang, W. Wang, T. Zhou, and Y. Yang, “Visual Abductive Reasoning,” in 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, Jun. 2022, pp. 15544–15554. doi: 10.1109/CVPR52688.2022.01512.
[81] J. Jacob and V. . Devassia, “Dense Captioning of Videos using Feature Context Integrated Deep LSTM with Local Attention,” in 2022 Sixth International Conference on I-SMAC (IoT in Social, Mobile, Analytics and Cloud) (I-SMAC), IEEE, Nov. 2022, pp. 810–818. doi: 10.1109/I-SMAC55078.2022.9987416.
[82] Z. Guo, T.-J. J. Wang, and J. Laaksonen, “Post-Attention Modulator for Dense Video Captioning,” in 2022 26th International Conference on Pattern Recognition (ICPR), IEEE, Aug. 2022, pp. 1536–1542. doi: 10.1109/ICPR56361.2022.9956260.
[83] J. Dave and S. Padmavathi, “Hierarchical Language Modeling for Dense Video Captioning,” in Inventive Computation and Information Technologies, S. Smys, V. E. Balas, and R. Palanisamy, Eds., Singapore: Springer Nature Singapore, 2022, pp. 421–431. doi: 10.1007/978-981-16-6723-7_32.
[84] W. Choi, J. Chen, and J. Yoon, “Parallel Pathway Dense Video Captioning With Deformable Transformer,” IEEE Access, vol. 10, pp. 129899–129910, 2022, doi: 10.1109/ACCESS.2022.3228821.
[85] Z. Chang, D. Zhao, H. Chen, J. Li, and P. Liu, “Event-centric multi-modal fusion method for dense video captioning,” Neural Networks, vol. 146, pp. 120–129, Feb. 2022, doi: 10.1016/j.neunet.2021.11.017.
[86] N. Aafaq, A. Mian, N. Akhtar, W. Liu, and M. Shah, “Dense Video Captioning With Early Linguistic Information Fusion,” IEEE Trans. Multimed., vol. 25, pp. 2309–2322, 2023, doi: 10.1109/TMM.2022.3146005.
[87] H. Ye, G. Li, Y. Qi, S. Wang, Q. Huang, and M.-H. Yang, “Hierarchical Modular Network for Video Captioning,” in 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, Jun. 2022, pp. 17918–17927. doi: 10.1109/CVPR52688.2022.01741.
[88] Z. Liu, X. Wu, and Y. Yu, “Multi-Task Video Captioning with a Stepwise Multimodal Encoder,” Electronics, vol. 11, no. 17, p. 2639, Aug. 2022, doi: 10.3390/electronics11172639.
[89] [Z. Yu and N. Han, “Accelerated masked transformer for dense video captioning,” Neurocomputing, vol. 445, pp. 72–80, Jul. 2021, doi: 10.1016/j.neucom.2021.03.026.
[90] M. Yu, H. Zheng, and Z. Liu, “Dense Video Captioning with Hierarchical Attention-Based Encoder-Decoder Networks,” in 2021 International Joint Conference on Neural Networks (IJCNN), IEEE, Jul. 2021, pp. 1–8. doi: 10.1109/IJCNN52387.2021.9534164.
[91] X. Yan, Z. Fei, S. Wang, Q. Huang, and Q. Tian, “DVCFlow: Modeling Information Flow Towards Human-like Video Captioning,” vol. 1, pp. 1–10, Nov. 2021, [Online]. Available: http://arxiv.org/abs/2111.10146
[92] B. Wu, G. Niu, J. Yu, X. Xiao, J. Zhang, and H. Wu, “Weakly Supervised Dense Video Captioning via Jointly Usage of Knowledge Distillation and Cross-modal Matching,” in Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, Z.-H. Zhou, Ed., California: International Joint Conferences on Artificial Intelligence Organization, Aug. 2021, pp. 1157–1164. doi: 10.24963/ijcai.2021/160.
[93] T. Wang, R. Zhang, Z. Lu, F. Zheng, R. Cheng, and P. Luo, “End-to-End Dense Video Captioning with Parallel Decoding,” in 2021 IEEE/CVF International Conference on Computer Vision (ICCV), IEEE, Oct. 2021, pp. 6827–6837. doi: 10.1109/ICCV48922.2021.00677.
[94] C.-H. Lu and G.-Y. Fan, “Environment-Aware Dense Video Captioning for IoT-Enabled Edge Cameras,” IEEE Internet Things J., vol. 9, no. 6, pp. 4554–4564, Mar. 2022, doi: 10.1109/JIOT.2021.3104289.
[95] S. Lee and I. Kim, “DVC‐Net: A deep neural network model for dense video captioning,” IET Comput. Vis., vol. 15, no. 1, pp. 12–23, Feb. 2021, doi: 10.1049/cvi2.12013.
[96] C. S. Kanani, S. Saha, and P. Bhattacharyya, “Global Object Proposals for Improving Multi-Sentence Video Descriptions,” in 2021 International Joint Conference on Neural Networks (IJCNN), IEEE, Jul. 2021, pp. 1–7. doi: 10.1109/IJCNN52387.2021.9533883.
[97] X. Dang, G. Wang, K. Xiong, and Z. Qin, “Dense Video Captioning for Incomplete Videos,” in Artificial Neural Networks and Machine Learning -- ICANN 2021, I. Farkaš, P. Masulli, S. Otte, and S. Wermter, Eds., Cham: Springer International Publishing, 2021, pp. 665–676. doi: 10.1007/978-3-030-86383-8_53.
[98] S. Chen and Y.-G. Jiang, “Towards Bridging Event Captioner and Sentence Localizer for Weakly Supervised Dense Event Captioning,” in 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, Jun. 2021, pp. 8421–8431. doi: 10.1109/CVPR46437.2021.00832.
[99] E. Boran, A. Erdem, N. Ikizler-Cinbis, E. Erdem, P. Madhyastha, and L. Specia, “Leveraging auxiliary image descriptions for dense video captioning,” Pattern Recognit. Lett., vol. 146, no. June, pp. 70–76, Jun. 2021, doi: 10.1016/j.patrec.2021.02.009.
[100] N. Aafaq, A. Mian, W. Liu, N. Akhtar, and M. Shah, “Cross-Domain Modality Fusion for Dense Video Captioning,” IEEE Trans. Artif. Intell., vol. 3, no. 5, pp. 763–777, Oct. 2022, doi: 10.1109/TAI.2021.3134190.
[101] P. S et al., “Exploration of Visual Features and their weighted-additive fusion for Video Captioning,” vol. 1, pp. 1–6, Jan. 2021, [Online]. Available: http://arxiv.org/abs/2101.05806
[102] H. Ryu, S. Kang, H. Kang, and C. D. Yoo, “Semantic Grouping Network for Video Captioning,” Proc. AAAI Conf. Artif. Intell., vol. 35, no. 3, pp. 2514–2522, May 2021, doi: 10.1609/aaai.v35i3.16353.
[103] M. Zhu, “Video Captioning in Compressed Video,” in 2021 6th International Conference on Image, Vision and Computing (ICIVC), IEEE, Jul. 2021, pp. 336–341. doi: 10.1109/ICIVC52351.2021.9526927.
[104] M. Tang, Z. Wang, Z. LIU, F. Rao, D. Li, and X. Li, “CLIP4Caption: CLIP for Video Caption,” in Proceedings of the 29th ACM International Conference on Multimedia, in MM ’21. New York, NY, USA: ACM, Oct. 2021, pp. 4858–4862. doi: 10.1145/3474085.3479207.
[105] M. M. Rahman, T. Abedin, K. S. S. Prottoy, A. Moshruba, and F. H. Siddiqui, “Video captioning with stacked attention and semantic hard pull,” PeerJ Comput. Sci., vol. 7, p. e664, Aug. 2021, doi: 10.7717/peerj-cs.664.
[106] Z. Zhang, D. Xu, W. Ouyang, and L. Zhou, “Dense Video Captioning Using Graph-Based Sentence Summarization,” IEEE Trans. Multimed., vol. 23, pp. 1799–1810, 2021, doi: 10.1109/TMM.2020.3003592.
[107] T. Wang, H. Zheng, M. Yu, Q. Tian, and H. Hu, “Event-Centric Hierarchical Representation for Dense Video Captioning,” IEEE Trans. Circuits Syst. Video Technol., vol. 31, no. 5, pp. 1890–1900, May 2021, doi: 10.1109/TCSVT.2020.3014606.
[108] T. Wang, H. Zheng, and M. Yu, “Dense-Captioning Events in Videos: SYSU Submission to ActivityNet Challenge 2020,” vol. 2, pp. 3–6, Aug. 2020, [Online]. Available: http://arxiv.org/abs/2006.11693
[109] M. Suin and A. N. Rajagopalan, “An Efficient Framework for Dense Video Captioning,” Proc. AAAI Conf. Artif. Intell., vol. 34, no. 07, pp. 12039–12046, Apr. 2020, doi: 10.1609/aaai.v34i07.6881.
[110] Y. Song, S. Chen, Y. Zhao, and Q. Jin, “Team RUC_AIM3 Technical Report at Activitynet 2020 Task 2: Exploring Sequential Events Detection for Dense VidTeam RUC_AIM3 Technical Report at Activitynet 2020 Task 2: Exploring Sequential Events Detection for Dense Video Captioningeo Captioning,” vol. 1, pp. 1–5, Jun. 2020, [Online]. Available: http://arxiv.org/abs/2006.07896
[111] V. lashin and E. Rahtu, “Multi-modal Dense Video Captioning,” in 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), IEEE, Jun. 2020, pp. 4117–4126. doi: 10.1109/CVPRW50498.2020.00487.
[112] V. Iashin and E. Rahtu, “A Better Use of Audio-Visual Cues: Dense Video Captioning with Bi-modal Transformer,” Proc. Br. Mach. Vis. Conf. 2020, pp. 1–16, Aug. 2020, doi: 10.5244/C.34.29.
[113] G. Huang, B. Pang, Z. Zhu, C. Rivera, and R. Soricut, “Multimodal Pretraining for Dense Video Captioning,” in Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, K.-F. Wong, K. Knight, and H. Wu, Eds., Stroudsburg, PA, USA: Association for Computational Linguistics, Dec. 2020, pp. 470–490. doi: 10.18653/v1/2020.aacl-main.48.
[114] B. Korbar, F. Petroni, R. Girdhar, and L. Torresani, “Video Understanding as Machine Translation,” vol. 2, Sep. 2020, [Online]. Available: https://arxiv.org/abs/2006.07203
[115] Z. Fang, T. Gokhale, P. Banerjee, C. Baral, and Y. Yang, “Video2Commonsense: Generating Commonsense Descriptions to Enrich Video Captioning,” in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), B. Webber, T. Cohn, Y. He, and Y. Liu, Eds., Stroudsburg, PA, USA: Association for Computational Linguistics, Nov. 2020, pp. 840–860. doi: 10.18653/v1/2020.emnlp-main.61.
[116] J. Hou, Y. Jia, X. Wu, and Y. Qi, “Video Captioning Using Weak Annotation,” vol. 1, pp. 1–10, Sep. 2020, [Online]. Available: http://arxiv.org/abs/2009.01067
[117] Z. Zhang, D. Xu, W. Ouyang, and C. Tan, “Show, Tell and Summarize: Dense Video Captioning Using Visual Cue Aided Sentence Summarization,” IEEE Trans. Circuits Syst. Video Technol., vol. 30, no. 9, pp. 3130–3139, Sep. 2020, doi: 10.1109/TCSVT.2019.2936526.
[118] S. Yang, L. Li, S. Wang, D. Meng, Q. Huang, and Q. Tian, “Structured Stochastic Recurrent Network for Linguistic Video Prediction,” in Proceedings of the 27th ACM International Conference on Multimedia, in MM ’19. New York, NY, USA: ACM, Oct. 2019, pp. 21–29. doi: 10.1145/3343031.3350859.
[119] H. Xu, B. Li, V. Ramanishka, L. Sigal, and K. Saenko, “Joint Event Detection and Description in Continuous Video Streams,” in 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), IEEE, Jan. 2019, pp. 396–405. doi: 10.1109/WACV.2019.00048.
[120] T. Rahman, B. Xu, and L. Sigal, “Watch, Listen and Tell: Multi-Modal Weakly Supervised Dense Event Captioning,” in 2019 IEEE/CVF International Conference on Computer Vision (ICCV), IEEE, Oct. 2019, pp. 8907–8916. doi: 10.1109/ICCV.2019.00900.
[121] J. Mun, L. Yang, Z. Ren, N. Xu, and B. Han, “Streamlined Dense Video Captioning,” in 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, Jun. 2019, pp. 6581–6590. doi: 10.1109/CVPR.2019.00675.
[122] S. Chen et al., “Activitynet 2019 Task 3: Exploring Contexts for Dense Captioning Events in Videos,” vol. 1, pp. 1–6, Jul. 2019, [Online]. Available: http://arxiv.org/abs/1907.05092
[123] C. Sun, A. Myers, C. Vondrick, K. Murphy, and C. Schmid, “VideoBERT: A Joint Model for Video and Language Representation Learning,” in 2019 IEEE/CVF International Conference on Computer Vision (ICCV), IEEE, Oct. 2019, pp. 7463–7472. doi: 10.1109/ICCV.2019.00756.
[124] L. Li and B. Gong, “End-to-End Video Captioning With Multitask Reinforcement Learning,” in 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), IEEE, Jan. 2019, pp. 339–348. doi: 10.1109/WACV.2019.00042.
[125] J. Zhang and Y. Peng, “Object-Aware Aggregation With Bidirectional Temporal Graph for Video Captioning,” in 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, Jun. 2019, pp. 8319–8328. doi: 10.1109/CVPR.2019.00852.
[126] S. Olivastri, G. Singh, and F. Cuzzolin, “End-to-End Video Captioning,” in 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), IEEE, Oct. 2019, pp. 1474–1482. doi: 10.1109/ICCVW.2019.00185.
[127] L. Zhou, Y. Zhou, J. J. Corso, R. Socher, and C. Xiong, “End-to-End Dense Video Captioning with Masked Transformer,” in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, Jun. 2018, pp. 8739–8748. doi: 10.1109/CVPR.2018.00911.
[128] Y. Xiong, B. Dai, and D. Lin, “Move Forward and Tell: A Progressive Generator of Video Descriptions,” in Computer Vision -- ECCV 2018, V. Ferrari, M. Hebert, C. Sminchisescu, and Y. Weiss, Eds., Cham: Springer International Publishing, 2018, pp. 489–505. doi: 10.1007/978-3-030-01252-6_29.
[129] J. Wang, W. Jiang, L. Ma, W. Liu, and Y. Xu, “Bidirectional Attentive Fusion with Context Gating for Dense Video Captioning,” in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, Jun. 2018, pp. 7190–7198. doi: 10.1109/CVPR.2018.00751.
[130] Y. Li, T. Yao, Y. Pan, H. Chao, and T. Mei, “Jointly Localizing and Describing Events for Dense Video Captioning,” in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, Jun. 2018, pp. 7492–7500. doi: 10.1109/CVPR.2018.00782.
[131] X. Duan, W. Huang, C. Gan, J. Wang, W. Zhu, and J. Huang, “Weakly Supervised Dense Event Captioning in Videos,” Adv. Neural Inf. Process. Syst., vol. 2018-Decem, no. Nips, pp. 3059–3069, Dec. 2018, [Online]. Available: http://arxiv.org/abs/1812.03849
[132] S. Chen, Y. Song, Y. Zhao, J. Qiu, Q. Jin, and A. Hauptmann, “RUC+CMU: System Report for Dense Captioning Events in Videos,” vol. 1, pp. 1–4, Jun. 2018, [Online]. Available: http://arxiv.org/abs/1806.08854
[133] B. Wang, L. Ma, W. Zhang, and W. Liu, “Reconstruction Network for Video Captioning,” in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, Jun. 2018, pp. 7622–7631. doi: 10.1109/CVPR.2018.00795.
[134] B. Zhao, X. Li, and X. Lu, “Video Captioning with Tube Features,” in Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, California: International Joint Conferences on Artificial Intelligence Organization, Jul. 2018, pp. 1177–1183. doi: 10.24963/ijcai.2018/164.
[135] X. Wang, W. Chen, J. Wu, Y.-F. Wang, and W. Y. Wang, “Video Captioning via Hierarchical Reinforcement Learning,” in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, Jun. 2018, pp. 4213–4222. doi: 10.1109/CVPR.2018.00443.
[136] Sujin Lee and Incheol Kim, “Video Captioning with Visual and Semantic Features,” J. Inf. Process. Syst., vol. 14, no. 6, pp. 1318–1330, 2018, [Online]. Available: https://jips-k.org/digital-library/2018/14/6/1318
[137] R. Pasunuru and M. Bansal, “Reinforced Video Captioning with Entailment Rewards,” in Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, M. Palmer, R. Hwa, and S. Riedel, Eds., Stroudsburg, PA, USA: Association for Computational Linguistics, Sep. 2017, pp. 979–985. doi: 10.18653/v1/D17-1103.
[138] Y. Liu, X. Li, and Z. Shi, “Video Captioning with Listwise Supervision,” Proc. AAAI Conf. Artif. Intell., vol. 31, no. 1, pp. 4197–4203, Feb. 2017, doi: 10.1609/aaai.v31i1.11239.
[139] Z. Shen et al., “Weakly Supervised Dense Video Captioning,” in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, Jul. 2017, pp. 5159–5167. doi: 10.1109/CVPR.2017.548.
[140] Y. Pan, T. Yao, H. Li, and T. Mei, “Video Captioning with Transferred Semantic Attributes,” in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, Jul. 2017, pp. 984–992. doi: 10.1109/CVPR.2017.111.
[141] H. S. Akash, M. A. Rahim, A. S. M. Miah, H.-S. Lee, S.-W. Jang, and J. Shin, “Two-Stream Modality-Based Deep Learning Approach for Enhanced Two-Person Human Interaction Recognition in Videos,” Sensors, vol. 24, no. 21, p. 7077, Nov. 2024, doi: 10.3390/s24217077.
[142] A. Radford et al., “Learning Transferable Visual Models From Natural Language Supervision,” Proc. Mach. Learn. Res., vol. 139, pp. 8748–8763, Feb. 2021, [Online]. Available: http://arxiv.org/abs/2103.00020
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
___________________________________________________________
International Journal of Advances in Intelligent Informatics
ISSN 2442-6571 (print) | 2548-3161 (online)
Organized by UAD and ASCEE Computer Society
Published by Universitas Ahmad Dahlan
W: http://ijain.org
E: info@ijain.org (paper handling issues)
andri.pranolo.id@ieee.org (publication issues)
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0