Scaling Semantic Insight: GPU-Accelerated Multimodal Feature Extraction for FrameNet Expansion

Introduction

As we move into an era where communication goes far beyond just words, the study of multimodal genres where meaning arises from a blend of images, text, and context has become a core challenge in computational linguistics and Goal is to computationally model the semantic representation of multimodal genres by extending the FrameNet framework to capture the complex interplay between communicative modes.

Why Extend FrameNet to Multimodal Data?

The FrameNet Legacy

FrameNet, rooted in frame semantics, has long provided a structured mapping between words and the conceptual “frames” they evoke. But classic FrameNet is built around text. It doesn’t capture meaning that comes from visual cues like body posture, spatial layouts, or symbolic imagery.

The Multimodal Imperative

Modern genres—think news stories, ads, memes, political posters—draw meaning from the synergy between images and text. Understanding these requires a model that can integrate not just lexical frames, but also visual entities, spatial relationships, emotional cues, and scene dynamics. That’s where our work comes in.

To support this, the FrameNet Brasil team and Red Hen Lab curated a dataset of 13,000+ news images, laying the groundwork for visual semantic parsing.

What We Needed: Structured Multimodal Outputs

To enrich our multimodal FrameNet model, we needed to extract from each image:

• entities_relationships: Who and what is in the image, and how are they interacting?
• scene_description_en: A rich narrative of the visual context in English.
• scene_description_pt: The Portuguese version for multilingual alignment.
• event_description: A high-level semantic summary of what’s happening.
• objects_list: A detailed list of objects in the scene.

This required more than just detection or classification—it demanded a deep semantic analysis of complex visual scenes, detailed enough to support frame-based reasoning.

The Model: Gemma-3B with Quantization

We used Gemma-3B, a March 2025 release known for its multimodal reasoning, and deployed it in its 4-bit quantized form. This let us reduce memory usage without losing semantic depth.

Quantization advantages:

• Memory savings: 4x smaller model size.
• Faster inference: Critical when processing thousands of high-res images.
• Scalability: Fit more models or bigger batches per GPU.

Scaling the Pipeline: Parallel Processing with PyTorch and Multiprocessing

Motivation

Processing 13,000+ images sequentially or on a single GPU would have taken weeks. Instead, we used three NVIDIA L40S GPUs in parallel on Case HPC, dramatically reducing the time required.

Reference Code Highlights

Here’s a code Snippet of our parallel image analysis pipeline:

num_gpus = torch.cuda.device_count()
if num_gpus == 0:
    raise RuntimeError("No GPUs available.")
print(f"Detected {num_gpus} GPU(s).")

tasks = [(img, i % num_gpus) for i, img in enumerate(image_paths)]

csv_file = 'structured_image_analysis_results.csv'
total_images = len(tasks)

with Pool(processes=num_gpus, initializer=init_worker) as pool:
    with open(csv_file, 'w', newline='', encoding='utf-8') as file:
        writer = csv.writer(file)
        writer.writerow([
            "Image Path", "GPU", "Entities & Relationships",
            "Scene Description (English)", "Scene Description (Portuguese)",
            "Event Description", "Objects List"
        ])
        for result in tqdm(pool.imap_unordered(process_image, tasks), total=total_images, desc="Processing images"):
            if result['status'] == 'success':
                writer.writerow([
                    result['image_path'],
                    result['gpu'],
                    result['entities_relationships'],
                    result['scene_description_en'],
                    result['scene_description_pt'],
                    result['event_description'],
                    result['objects_list']
                ])
            else:
                print(f"Failed to process {result['image_path']} on GPU {result['gpu']}")

Check the full code here: Github

How it works:

• Automatically detects Number of GPU’s and Images are distributed using round-robin GPU assignment.
• Each image is processed with a custom prompt to extract structured fields.
• Results are cleaned, split into English and Portuguese, and exported to CSV.

Performance: With 3 GPUs, we processed 13,500+ images in under ~60 hours, averaging about ~40 seconds per image (depending on complexity).

Prompt Engineering for Semantic Rigor

The quality of the model’s output depended heavily on prompt design. We crafted a multi-section prompt that instructed the model to:

• Tag entities like [person], [clothing], [object], etc.
• Generate scene descriptions in both English and Portuguese, preserving tags.
• Offer multiple plausible event hypotheses when ambiguity existed.

This approach made it possible for our pipeline to automatically parse and index semantic content, supporting future training of multimodal parsers and knowledge graphs.

Semantic Outputs: A Real Example

Let’s break down what our pipeline extracted from this soccer celebration image—and why each part matters.

Why this output is remarkable:

Our model doesn’t just list objects. It understands roles (who is celebrating, who is observing), captures relationships, and generates rich, multilingual scene descriptions. This is far beyond basic object detection—it’s structured, context-aware, and ready for advanced research.

Technical Takeaways

• Parallelism is essential for real-world multimodal workloads.
• Quantized LLMs like Gemma-3B make advanced processing possible on limited GPU resources.
• Prompt design and output parsing are as important as the model itself.
• The CSV output format made it easy to integrate with downstream NLP and annotation tools.

Conclusion

Our work proves that extracting structured meaning from images at scale is not just possible, it’s efficient and practical. With quantized models, smart prompts, and parallel GPU processing, we’ve built a pipeline that makes semantic multimodal annotation a reality for real research.

Acknowledgements

My sincere thanks to:

• Prof. Marcelo Viridiano for his clear direction and insights into feature extraction
• Prof. Mark Turner for his outstanding guidance on frame semantics
• Prof. Tiago Torrent for his expertise and support with Portuguese translation

Their mentorship and collaboration have been invaluable throughout this project.