Training and Finetuning Multimodal Embedding & Reranker Models with Sentence Transformers

This post demonstrates how to train and finetune multimodal embedding and reranker models using Sentence Transformers on custom domain data.

• •Multimodal models handle text, images, audio, and video inputs simultaneously, requiring specialized training approaches beyond text-only methods
• •Finetuning on domain-specific data significantly improves performance: Visual Document Retrieval example improved NDCG@10 from 0.888 to 0.947 and outperformed larger models
• •Key training components include the model, dataset, loss function (CachedMultipleNegativesRankingLoss, MatryoshkaLoss), training arguments, evaluator, and trainer
• •Dataset format must match the loss function requirements, supporting multimodal inputs: text strings, PIL images, file paths, URLs, audio and video arrays
• •