DICE: Disentangled Instance-Class knowlEdge Prompt Tuning via SAE for Vision-Language Models

CLIP, known for its strong zero-shot generalization, has received significant attention in prompt tuning for its ability to adapt effectively to new tasks in few-shot settings. Recent studies have employed descriptions generated by Large Language Models (LLMs) as predefined prompts to obtain class-level semantics. However, because these predefined descriptions are not directly grounded in visual information, they often misalign with an image's visual semantics. They tend to generate generic class-level descriptions rather than instance-specific ones. As a result, LLM-based prompts are structurally biased toward class-level semantics, making it difficult to capture and reflect instance-level cues, which in turn limits generalization in few-shot settings where both levels of knowledge are crucial. To address this limitation, we propose Disentangled Instance-Class knowlEdge (DICE) prompt tuning, a framework that restructures LLM-derived class-level priors to capture instance-level semantics. Specifically, DICE decomposes LLM-derived priors into instance-level components using a Sparse Autoencoder (SAE), which selects instance-specific concept vectors. These concept vectors are then fused with class embeddings to form enriched representations. This synergy preserves class-level semantic coherence while capturing instance-level details, improving generalization to unseen classes. Our approach achieves competitive performance across 11 few-shot recognition benchmarks, while additionally offering a plug-and-play solution that enhances interpretability through SAE.

DICE builds a unified context prompt from two complementary branches. The Class Context Module projects cached LLM embeddings into the CLIP space to capture general class-level semantics. The Instance Context Module passes both the projected class embedding and the image feature through a shared Sparse Autoencoder (SAE), computes a co-activation score over the sparse concept dictionary, and selects the top-k concept vectors aligned with both the image and its class. The two branches are fused and passed through a lightweight meta-network to produce prompts that reflect both general class knowledge and instance-specific detail.

Contributions

• Concept Dictionary via Sparse Autoencoders. We show that an SAE can serve as a learnable concept dictionary for vision–language representations, enabling explicit concept selection and improving interpretability.
• Class–Instance Semantic Integration. We introduce a unified prompt formulation that integrates class-level semantic priors with instance-specific visual concepts for robust few-shot adaptation.
• From Handcrafted Descriptions to Concept-based Prompts. We replace predefined class descriptions with prompts built from disentangled concept units that adapt dynamically to each input instance.

Each SAE latent is matched to its most strongly activated WordNet concept. Meaningful and diverse concepts emerge (e.g., freckled for skin texture, mozzarella for food), and even images within the same class emphasize distinct but related concepts — evidence that the SAE captures interpretable, instance-level variation.