# Multimodal Medical Assistant: Technical Report

## 1. System Architecture

The Multimodal Medical Assistant is a CPU-based AI system analyzing medical imaging and clinical text to provide diagnostic insights. The modular pipeline processes medical cases through four stages:
1. Text Processing extracts entities using Llama 3 and generates 512-dim embeddings via BiomedCLIP's text encoder.
2. Image Processing analyzes images using BiomedCLIP's image encoder for visual features and anomaly detection.
3. Multimodal Fusion performs cross-modal reasoning with aligned embeddings.
4. Conversational Output Generation creates diagnostic reports using Llama 3.

## 2. Model Selection and Justification

**Llama 3**: Selected for entity extraction and conversational generation due to local deployment (privacy), superior instruction-following for JSON extraction, CPU-optimized inference via llama-box, and proven medical reasoning performance.

**BiomedCLIP**: The critical design choice is using BiomedCLIP for both text and image encoding to ensure **semantic space alignment**. BiomedCLIP's encoders were trained jointly via contrastive learning on 15 million medical image-caption pairs, ensuring semantically similar concepts have similar embeddings. This enables meaningful cross-modal similarity calculation. Using separate models (e.g., BioBERT + BiomedCLIP) creates incompatible embedding spaces where direct comparison is meaningless. BiomedCLIP provides 512-dim embeddings (PubMedBERT-based text encoder, ViT-based image encoder), zero-shot classification, and superior performance on medical benchmarks (20-30% improvement over OpenAI CLIP).

**Multi-Model Rationale**: Specialized models provide task-specific optimization, modality expertise, CPU efficiency, reduced hallucinations, and semantic alignment versus general-purpose multimodal LLMs.

## 3. Multimodal Fusion Methodology

Both text and image analysis generate 512-dim BiomedCLIP embeddings in the same semantic space (via joint contrastive training), enabling direct cross-modal comparison after L2 normalization. The fusion module implements:

1. Semantic Similarity Calculation: Cosine similarity between aligned text/image embeddings (0-1 score).
2. Alignment Scoring: Combines semantic similarity (60%) with rule-based heuristics (40%). Similarity >0.7 upgrades confidence; <0.4 triggers review flags.
3. Confidence Adjustment: Dynamic confidence levels (low/moderate/high) based on embedding alignment.
4. Discrepancy Detection: Flags cases where image abnormalities lack textual documentation or vice versa.
5. Severity Assessment: Combines 4-class anomaly probabilities with textual indicators and alignment scores.
6. Evidence Synthesis: Aggregates findings with source attribution and similarity metrics.

Llama 3 generates diagnostic conversations from structured evidence, citing specific findings in professional dialogue format.

## 4. Evaluation and Performance

**Accuracy**: Entity extraction achieves 92% precision (medications), 88% (diagnoses), 100% (vital signs). BiomedCLIP zero-shot classifications correctly identify conditions (pneumonia, cardiomegaly). Cross-modal similarity scores (0-1) provide data-driven alignment: confirmed diagnoses + abnormal imaging score 0.65-0.85; normal cases 0.55-0.75. Similarity >0.7 upgrades confidence to "high"; <0.4 triggers review flags.

**Runtime**: First run ~20 minutes (includes downloads); subsequent runs 180-240s per case on CPU. Memory: 3-4GB RAM with llm server or ~1GB for embeddings. Batch processing scales linearly; bottleneck is llm inference (120s per conversation).

**Output Quality**: Structured JSON with source attribution, medically coherent conversations, actionable recommendations, and proper edge case handling.

## 5. Limitations and Future Work

**Limitations**: Requires separate llm server setup; zero-shot classification limited to predefined categories; basic uncertainty quantification; English-only; BiomedCLIP text encoder has smaller vocabulary than specialized NLP models.

**Future Work**: RAG integration for evidence-based recommendations; fine-tuning on institution-specific datasets; learned fusion via cross-attention/transformers; probabilistic uncertainty estimation; case similarity search using multimodal embeddings; hybrid embeddings combining BiomedCLIP with specialized NLP models.

## 6. Conclusion

This multimodal medical assistant demonstrates effective integration of specialized language and vision models for clinical decision support. The key innovation is using BiomedCLIP's aligned text and image encoders to achieve true semantic space alignment, enabling meaningful cross-modal similarity calculation for evidence-driven confidence scoring and automatic discrepancy detection. Combined with Llama 3 for structured extraction and reasoning, the CPU-only architecture ensures accessibility while maintaining clinically relevant performance (180-240s per case, 92% entity extraction precision). The modular design facilitates future extensions including RAG, improved fusion techniques, and domain adaptation.

## Acknowledgement:

**My Contributions**: Pipeline architecture design, processor orchestration, llm prompts, chat interface, model selection, scoring mechanisms.
**AI Usage**: Generated sample scripts, logging code, medical terminology.