Multi-Task Learning with Transformers

#transformers #multi-task learning #nlp #deep learning #machine learning #neural networks #python #hugging face #model architecture #task formulation

1. Key Concepts and Definitions

1.1 Key Concepts and Definitions

Multi-Task Learning (MTL)

Multi-Task Learning (MTL) is a machine learning paradigm where a single model is trained to perform multiple related tasks simultaneously. Unlike traditional single-task learning, MTL leverages shared representations across tasks, often improving generalization by exploiting commonalities and differences among tasks. The underlying hypothesis is that inductive bias from related tasks can enhance the model's performance on each individual task.

$$ \mathcal{L}_{\text{MTL}} = \sum_{i=1}^{T} \lambda_i \mathcal{L}_i(\theta_{\text{shared}}, \theta_i) $$

Here, T denotes the number of tasks, λi is a task-specific weighting coefficient, Li is the loss for task i, and θshared and θi represent shared and task-specific parameters, respectively.

Transformers in MTL

Transformers, originally introduced for sequence-to-sequence tasks, have become a dominant architecture in MTL due to their self-attention mechanism, which dynamically weights input features. The key advantage lies in their ability to capture hierarchical and task-agnostic patterns, making them ideal for shared representation learning. The transformer's multi-head attention allows the model to focus on different aspects of the input for different tasks.

$$ \text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V $$

In this equation, Q, K, and V are the query, key, and value matrices, and dk is the dimension of the key vectors. The scaled dot-product attention enables the model to weigh the importance of different input tokens dynamically.

Task-Specific Adaptations

In MTL with transformers, task-specific adaptations are often implemented via:

Challenges in MTL with Transformers

Despite their advantages, MTL with transformers presents several challenges:

Practical Applications

MTL with transformers has been successfully applied in:

Key Concepts and Definitions – Multi-Task Learning with Transformers – Tutorial Diagram
Diagram Description: The diagram would show the architecture of a multi-task transformer model, illustrating shared layers, task-specific heads, and attention mechanisms.

Benefits and Challenges of Multi-Task Learning

Benefits of Multi-Task Learning

Multi-task learning (MTL) with transformers offers several advantages over single-task learning, particularly in scenarios where tasks share underlying representations or require transferable knowledge. The primary benefits include:

The effectiveness of MTL can be quantified through the improvement in the generalization bound. For k tasks with shared parameters θs and task-specific parameters θi, the expected risk R across tasks is bounded by:

$$ R(\theta_s, \theta_1, ..., \theta_k) \leq \frac{1}{k}\sum_{i=1}^k R_i(\theta_s, \theta_i) + \sqrt{\frac{\log(k)}{2n}} $$

where n is the number of samples per task. The second term shows how MTL benefits from increased effective sample size through parameter sharing.

Challenges of Multi-Task Learning

Despite its advantages, MTL introduces several complexities that must be carefully managed:

$$ \text{Interference} = 1 - \frac{\langle \nabla_{\theta_s} \mathcal{L}_i, \nabla_{\theta_s} \mathcal{L}_j \rangle}{\|\nabla_{\theta_s} \mathcal{L}_i\| \|\nabla_{\theta_s} \mathcal{L}_j\|} $$
$$ \mathcal{L}_{total} = \sum_{i=1}^k w_i(t)\mathcal{L}_i $$

where wi(t) are time-dependent weights that adapt during training.

Practical Considerations

In transformer-based MTL, several architectural choices significantly impact performance:

The effectiveness of these approaches depends on the task relationships, which can be quantified through the task affinity matrix A ∈ ℝk×k, where Aij measures the performance improvement when tasks i and j are trained together versus separately.

Benefits and Challenges of Multi-Task Learning – Multi-Task Learning with Transformers – Tutorial Diagram
Diagram Description: The diagram would show the architecture of a multi-task transformer with shared encoder layers and task-specific heads, illustrating gradient flow and interference between tasks.

1.3 Architectural Approaches for Multi-Task Learning

Hard Parameter Sharing

Hard parameter sharing is the most common architectural approach for multi-task learning (MTL) with transformers. In this setup, the model shares all hidden layers across tasks while maintaining task-specific output layers. The shared encoder (e.g., BERT, RoBERTa) processes input data, and task-specific heads branch from the final hidden layer. Mathematically, for N tasks, the shared encoder E produces a representation h = E(x), while task-specific heads Ti compute predictions:

$$ y_i = T_i(h) \quad \text{for} \quad i = 1, \dots, N $$

This approach reduces the risk of overfitting, as the shared layers must generalize across all tasks. However, task interference can occur if the objectives conflict, leading to suboptimal performance. Practical implementations often use gradient masking or task-specific learning rates to mitigate this.

Soft Parameter Sharing

Soft parameter sharing allows each task to have its own model parameters while encouraging similarity through regularization. The transformer architecture implements this via:

For two tasks with hidden states h1 and h2, a cross-stitch layer computes:

$$ \tilde{h}_1 = \alpha_{11}h_1 + \alpha_{12}h_2 $$ $$ \tilde{h}_2 = \alpha_{21}h_1 + \alpha_{22}h_2 $$

where αij are learnable mixing weights. This provides flexibility but increases computational overhead due to separate parameter sets.

Task-Specific Adapters

Adapter-based architectures insert small task-specific modules between transformer layers while keeping the core model frozen. Each adapter consists of:

For a hidden state h at layer l, the adapter output is:

$$ h_{adapter} = W_{up} \cdot \text{ReLU}(W_{down} \cdot h) + h $$

where Wdown ∈ ℝd×b and Wup ∈ ℝb×d (b ≪ d). This approach enables parameter-efficient tuning and is widely used in few-shot learning scenarios.

Multi-Gate Mixture-of-Experts

The Mixture-of-Experts (MoE) architecture routes inputs to specialized subnetworks ("experts") via gating mechanisms. In MTL, each task t employs a separate gating network Gt over shared experts {Ej}kj=1:

$$ y_t = \sum_{j=1}^k G_t(x)_j \cdot E_j(x) $$

The gating weights Gt(x) are typically computed via softmax over a learned projection. Sparse gating (e.g., top-2 routing) improves scalability. Google’s Switch Transformer demonstrates this approach at scale, achieving 7x faster inference than dense models.

Gradient Modulation Techniques

Architectures like GradNorm dynamically adjust task-specific gradients during backpropagation. For N tasks with losses {Li}, the modulated gradient for task i is:

$$ \nabla_{W} L_i \rightarrow w_i(t) \cdot \nabla_{W} L_i $$

where wi(t) is a learned weight updated to balance task learning rates. This is implemented as an additional network head that computes:

$$ w_i(t) = \text{softmax}(T_\theta(g_1, \dots, g_N))_i $$

with gi = ||\nabla_{W} L_i||2. Such approaches prevent dominant tasks from overwhelming the shared representation.

Architectural Approaches for Multi-Task Learning – Multi-Task Learning with Transformers – Tutorial Diagram
Diagram Description: The section describes multiple architectural approaches with shared and task-specific components, which are inherently spatial and hierarchical.

2. Transformer Architecture Recap

2.1 Transformer Architecture Recap

The Transformer architecture, introduced by Vaswani et al. in 2017, revolutionized sequence modeling by replacing recurrent and convolutional layers with self-attention mechanisms. At its core, the Transformer relies on three key components: multi-head attention, position-wise feed-forward networks, and layer normalization. Unlike RNNs, Transformers process entire sequences in parallel, making them highly efficient for modern hardware accelerators.

Self-Attention Mechanism

The self-attention mechanism computes a weighted sum of input representations, where the weights are dynamically derived from pairwise interactions between elements in the sequence. Given an input sequence X ∈ ℝn×d, where n is the sequence length and d is the embedding dimension, the attention operation is defined as:

$$ \text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V $$

Here, Q (queries), K (keys), and V (values) are linear projections of the input X:

$$ Q = XW_Q, \quad K = XW_K, \quad V = XW_V $$

where WQ, WK, WV ∈ ℝd×dk are learnable weight matrices. The scaling factor 1/√dk prevents gradient saturation in the softmax.

Multi-Head Attention

Multi-head attention extends self-attention by applying h parallel attention heads, each with separate learned projections. This allows the model to jointly attend to information from different representation subspaces:

$$ \text{MultiHead}(Q, K, V) = \text{Concat}(\text{head}_1, ..., \text{head}_h)W_O $$

where each head computes attention independently:

$$ \text{head}_i = \text{Attention}(QW_Q^i, KW_K^i, VW_V^i) $$

The output projections WO ∈ ℝhdv×d combine the heads' outputs. Typical implementations use h = 8 heads with dk = dv = d/h.

Position-Wise Feed-Forward Networks

Each Transformer layer contains a position-wise feed-forward network (FFN) applied independently to each token representation. The FFN consists of two linear transformations with a ReLU activation in between:

$$ \text{FFN}(x) = \text{ReLU}(xW_1 + b_1)W_2 + b_2 $$

where W1 ∈ ℝd×dff, W2 ∈ ℝdff×d, and dff is typically 4×d. This provides additional nonlinear capacity to the model.

Layer Normalization and Residual Connections

Transformers employ residual connections around each sub-layer (attention and FFN), followed by layer normalization:

$$ \text{LayerNorm}(x + \text{Sublayer}(x)) $$

This architecture choice enables stable training of deep networks by mitigating vanishing gradients. The layer normalization operates over the embedding dimension d, normalizing activations to zero mean and unit variance for each token independently.

Positional Encoding

Since Transformers lack inherent notion of sequence order, positional encodings are added to input embeddings to inject information about token positions. The original paper uses sinusoidal functions of varying frequencies:

$$ PE_{(pos,2i)} = \sin(pos/10000^{2i/d}) $$ $$ PE_{(pos,2i+1)} = \cos(pos/10000^{2i/d}) $$

where pos is the position and i is the dimension. Learned positional embeddings are also commonly used in practice.

Transformer Layer Architecture Block diagram of a Transformer layer showing input processing through multi-head attention and feed-forward networks with residual connections and layer normalization. Input Embeddings Positional Encoding Add & Norm Multi-Head Attention Q Proj K Proj V Proj Softmax Feed Forward (ReLU) Add & Norm Output
Diagram Description: The diagram would physically show the architecture of a Transformer layer with its key components (multi-head attention, feed-forward networks, residual connections) and their spatial relationships.

2.2 Adapting Transformers for Multi-Task Learning

Transformer architectures, originally designed for sequence-to-sequence tasks like machine translation, require careful adaptation to handle multiple tasks simultaneously. The core challenge lies in balancing shared representations across tasks while preserving task-specific nuances. Two primary architectural strategies dominate this space: hard parameter sharing and soft parameter sharing.

Hard Parameter Sharing

This approach forces all tasks to share most transformer layers, with only the final task-specific heads differing. Mathematically, for N tasks, the shared encoder E processes input x, while task-specific heads Hi produce outputs:

$$ y_i = H_i(E(x)) \quad \text{for} \quad i = 1, \dots, N $$

The shared encoder typically comprises the transformer's self-attention and feed-forward layers, enabling cross-task knowledge transfer. A key advantage is reduced computational overhead, as the bulk of parameters are reused. However, task interference can occur when optimization gradients conflict.

Soft Parameter Sharing

More flexible than hard sharing, this method allows separate transformer parameters per task while encouraging similarity through regularization. The loss function often includes a term penalizing divergence between task-specific parameters θi and a shared reference θ0:

$$ \mathcal{L}_{\text{total}} = \sum_{i=1}^N \mathcal{L}_i(\theta_i) + \lambda \sum_{i=1}^N \|\theta_i - \theta_0\|^2 $$

Recent variants like Cross-Stitch Networks learn weighted combinations of task-specific features dynamically. The transformer's attention mechanism proves particularly adaptable here—attention heads can specialize in different task relationships while maintaining shared key-value projections.

Architectural Innovations

Several transformer-specific modifications enhance multi-task performance:

The transformer's scalability allows these techniques to compound; for instance, combining adapters with task embeddings often outperforms either method alone. Empirical studies show that lower layers benefit most from hard sharing, while higher layers require more task-specific adaptation.

Optimization Considerations

Training dynamics differ significantly from single-task scenarios. The gradient cosine similarity between tasks predicts compatibility:

$$ \text{sim}(i,j) = \frac{g_i \cdot g_j}{\|g_i\| \|g_j\|} $$

Where gi, gj are task gradients. Tasks with positive similarity benefit from joint training, while conflicting gradients may require:

Transformer-specific optimizers like AdaFactor often outperform Adam in multi-task settings due to better memory efficiency and stabler gradient norms across tasks.

Adapting Transformers for Multi-Task Learning – Multi-Task Learning with Transformers – Tutorial Diagram
Diagram Description: The diagram would physically show the architectural differences between hard parameter sharing and soft parameter sharing in transformer layers, including shared vs. task-specific components.

Shared vs. Task-Specific Parameters in Transformers

Multi-task learning (MTL) in Transformers involves balancing shared and task-specific parameters to optimize performance across multiple tasks. The core challenge lies in determining which layers or components should be shared to capture common features and which should remain task-specific to preserve unique task characteristics.

Architectural Design Choices

Transformer architectures for MTL typically adopt one of three parameter-sharing strategies:

Mathematical Formulation

For a Transformer with L layers processing T tasks, the output yt for task t can be expressed as:

$$ y_t = f_t(h_L^t) $$ $$ h_l^t = \begin{cases} \text{TransformerLayer}(h_{l-1}^t) & \text{(task-specific)} \\ \text{TransformerLayer}(h_{l-1}) & \text{(shared)} \end{cases} $$

where hlt represents the hidden state at layer l for task t, and ft is the task-specific head.

Gradient Conflict Analysis

When sharing parameters across tasks, gradient conflicts may arise. The interference between tasks i and j can be quantified by the cosine similarity of their gradients:

$$ \text{Conflict}_{i,j} = 1 - \frac{g_i \cdot g_j}{||g_i|| \cdot ||g_j||} $$

where gi and gj are the gradients from tasks i and j respectively. High conflict values suggest the need for more task-specific parameters.

Practical Implementation Considerations

Modern implementations often use:

The choice between these approaches depends on the similarity of tasks, available computational resources, and desired inference speed. For closely related tasks (e.g., different text classification tasks), heavier parameter sharing typically works well. For disparate tasks (e.g., translation and image captioning), more task-specific parameters are generally beneficial.

Shared vs. Task-Specific Parameters in Transformers – Multi-Task Learning with Transformers – Tutorial Diagram
Diagram Description: The diagram would physically show the three parameter-sharing strategies (hard, soft, adaptive) in Transformer architectures, highlighting shared vs. task-specific layers and pathways.

3. Data Preparation and Task Formulation

Data Preparation and Task Formulation

Task-Specific Data Requirements

Multi-task learning (MTL) with transformers requires careful alignment of data across tasks to ensure compatibility in input-output spaces. For N tasks, each dataset Di must include input samples xi and corresponding labels yi, where i ∈ {1,...,N}. The key challenge lies in handling heterogeneous label spaces—some tasks may require classification (discrete outputs), while others demand regression (continuous outputs).

$$ \mathcal{D}_i = \{(x^{(j)}_i, y^{(j)}_i)\}_{j=1}^{M_i} $$

For sequence-based tasks (e.g., text or time-series), inputs must be tokenized into a shared vocabulary. Byte Pair Encoding (BPE) is often employed to ensure subword-level compatibility across tasks with divergent lexical distributions.

Architecture-Aware Data Alignment

Transformer-based MTL architectures typically process inputs through a shared encoder followed by task-specific heads. This imposes constraints on data dimensions:

The batch sampling strategy impacts gradient dynamics. Two common approaches are:

$$ \text{Uniform sampling: } p_i = \frac{1}{N} $$ $$ \text{Proportional sampling: } p_i = \frac{|D_i|}{\sum_{j=1}^N |D_j|} $$

Label Space Harmonization

When tasks have conflicting label semantics (e.g., sentiment analysis vs. topic classification), implement:

For regression tasks, standardize outputs to zero mean and unit variance:

$$ \tilde{y}_i = \frac{y_i - \mu_i}{\sigma_i} $$

Task Interference Mitigation

Negative transfer occurs when gradient updates for one task degrade performance on others. Countermeasures include:

$$ \mathcal{L}_{total} = \sum_{i=1}^N \lambda_i \mathcal{L}_i $$

Recent work employs uncertainty-based weighting, where λi ∝ 1/σi2, with σi representing task-dependent homoscedastic uncertainty.

Real-World Implementation Example

Consider a joint model for named entity recognition (NER) and sentiment analysis:


import torch
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")

def preprocess_mtl_batch(texts, ner_tags, sentiments):
    # Tokenize with task-specific special tokens
    inputs = tokenizer(
        ["[NER]" + text if task == 0 else "[SA]" + text 
         for text, task in zip(texts, [0]*len(texts) + [1]*len(texts))],
        padding=True,
        truncation=True,
        return_tensors="pt"
    )
    
    # Stack labels for both tasks
    labels = torch.cat([
        torch.tensor(ner_tags),
        torch.tensor(sentiments)
    ])
    
    return inputs, labels

3.2 Loss Function Design for Multi-Task Learning

Multi-task learning (MTL) with transformers requires careful design of the loss function to balance competing objectives across tasks. The primary challenge lies in optimizing a shared representation while preventing any single task from dominating the learning process. The loss function must account for task-specific objectives, their relative importance, and potential conflicts in gradient updates.

Weighted Sum of Task Losses

The most common approach combines task losses via a weighted linear combination:

$$ \mathcal{L}_{total} = \sum_{k=1}^T w_k \mathcal{L}_k(\theta_{shared}, \theta_k) $$

where T is the number of tasks, wk is the weight for task k, and θshared, θk denote shared and task-specific parameters respectively. The weights wk can be:

Gradient Balancing Techniques

Naive weighted summation can lead to gradient conflicts where tasks compete for parameter updates. Several methods address this:

Gradient Normalization

Modify gradients to have comparable magnitudes across tasks:

$$ \tilde{g}_k = \frac{g_k}{||g_k||_2} \cdot ||\bar{g}||_2 $$

where gk is the gradient for task k and ḡ is the average gradient norm across tasks.

Gradient Surgery (PCGrad)

Projects conflicting gradients to minimize interference:

$$ g_k^{proj} = g_k - \frac{g_k \cdot g_j}{||g_j||_2^2} g_j \quad \text{if } g_k \cdot g_j < 0 $$

This ensures gradients from different tasks don't work against each other.

Uncertainty Weighting

Kendall et al. (2018) proposed learning task weights via homoscedastic uncertainty:

$$ \mathcal{L}_{total} = \sum_{k=1}^T \left( \frac{1}{2\sigma_k^2} \mathcal{L}_k + \log \sigma_k \right) $$

where σk is a learnable parameter representing task uncertainty. This automatically balances tasks based on their noise levels.

Dynamic Task Prioritization

Some tasks may require prioritization during different training phases:

Practical Implementation Considerations

When implementing MTL loss functions in transformers:

$$ \mathcal{L}_{reg} = \lambda ||\theta_{shared}||_2^2 $$

where λ controls the strength of L2 regularization.

Loss Function Design for Multi-Task Learning – Multi-Task Learning with Transformers – Tutorial Diagram
Diagram Description: The diagram would show the gradient balancing techniques (normalization and surgery) visually demonstrating how conflicting gradients are modified or projected to minimize interference.

3.3 Training Techniques and Optimization

Loss Function Design

In multi-task learning (MTL), the joint loss function L is typically a weighted sum of task-specific losses Lk:

$$ L = \sum_{k=1}^T w_k L_k $$

where wk represents the weight for task k. The choice of wk significantly impacts model performance. Common approaches include:

Gradient Conflict Mitigation

Task gradients may conflict during backpropagation, leading to suboptimal convergence. Let gk = ∇θLk be the gradient for task k. The cosine similarity between gradients:

$$ \text{sim}(g_i, g_j) = \frac{g_i^T g_j}{||g_i|| \cdot ||g_j||} $$

measures task alignment. Negative values indicate conflicting gradients. Solutions include:

Architectural Optimization

Transformer-based MTL models benefit from:

The adapter approach modifies the standard transformer layer output H as:

$$ H' = H + f_t(HW_{down})W_{up} $$

where Wdown ∈ ℝd×r and Wup ∈ ℝr×d form a bottleneck (typically r ≪ d), and ft is a task-specific non-linearity.

Optimization Strategies

Advanced optimizers for MTL include:

The multi-task Adam update rule modifies the standard Adam optimizer by computing task-specific second moments:

$$ m_t^{(k)} = β_1m_{t-1}^{(k)} + (1-β_1)g_t^{(k)} $$ $$ v_t^{(k)} = β_2v_{t-1}^{(k)} + (1-β_2)(g_t^{(k)})^2 $$

where k indexes tasks and t indexes timesteps.

Regularization Techniques

Effective regularization prevents task interference:

Training Techniques and Optimization – Multi-Task Learning with Transformers – Tutorial Diagram
Diagram Description: The diagram would show the gradient conflict mitigation techniques (PCGrad, gradient sign dropout, MoCo) visually demonstrating how conflicting gradients are modified or aligned.

4. Natural Language Processing Applications

Multi-Task Learning with Transformers in Natural Language Processing

Architectural Adaptations for Multi-Task NLP

Transformer-based multi-task learning (MTL) in NLP requires careful architectural design to balance shared and task-specific representations. The most common approach involves:

The gradient flow in such architectures can be formalized as:

$$ \frac{\partial \mathcal{L}_{total}}{\partial \theta_{shared}} = \sum_{t=1}^T \alpha_t \frac{\partial \mathcal{L}_t}{\partial \theta_{shared}} $$

where αt represents task-specific weighting coefficients, often learned through gradient normalization techniques.

Key NLP Applications and Performance Benchmarks

Recent advances in MTL transformers have demonstrated state-of-the-art performance across several NLP benchmarks:

Model Tasks GLUE Score Parameters
MT-DNN 9 NLU tasks 82.7 340M
T5 (MTL variant) 17 tasks 89.3 11B
UniLMv2 Bi-directional and seq2seq 85.1 550M

Challenges in Multi-Task NLP

The primary challenges in transformer-based MTL for NLP stem from:

The task interference problem can be quantified through gradient similarity metrics:

$$ \phi_{i,j} = \frac{\langle \nabla_{\theta}\mathcal{L}_i, \nabla_{\theta}\mathcal{L}_j \rangle}{\|\nabla_{\theta}\mathcal{L}_i\| \cdot \|\nabla_{\theta}\mathcal{L}_j\|} $$

where values near -1 indicate severe interference between tasks i and j.

Advanced Techniques for NLP-Specific MTL

Recent research has introduced several transformer-specific MTL improvements:

Gradient Surgery

Project conflicting gradients to minimize interference:

$$ \nabla_{\theta}^{proj} = \nabla_{\theta}\mathcal{L}_i - \alpha \langle \nabla_{\theta}\mathcal{L}_i, \nabla_{\theta}\mathcal{L}_j \rangle \nabla_{\theta}\mathcal{L}_j $$

Task-Attentive Routing

Dynamic pathway selection through attention mechanisms:

$$ a_t = \text{softmax}(W_q h_{CLS} \cdot W_k T_t / \sqrt{d}) $$

where Tt represents task embeddings and hCLS is the [CLS] token representation.

Case Study: Multi-Task Question Answering

The MRQA 2019 shared task demonstrated how MTL transformers outperform single-task models across 18 QA datasets. A unified architecture with:

achieved 4.2% average improvement over single-task baselines while reducing total parameters by 38%.

Emerging Directions

Current research frontiers include:

Natural Language Processing Applications – Multi-Task Learning with Transformers – Tutorial Diagram
Diagram Description: The section describes multiple architectural approaches (hard/soft parameter sharing, adapters) and gradient flow mechanisms that would benefit from visual representation of layer structures and signal paths.

4.2 Computer Vision Applications

Transformers have revolutionized computer vision by enabling multi-task learning frameworks that outperform traditional convolutional architectures. The self-attention mechanism allows models to capture long-range dependencies across image patches, making them particularly effective for tasks requiring global context understanding.

Vision Transformers (ViTs) as Multi-Task Backbones

The Vision Transformer (ViT) architecture processes images by splitting them into fixed-size patches, linearly embedding each patch, and feeding them into a standard transformer encoder. For multi-task learning, the shared encoder extracts features that are then passed to task-specific heads. The loss function combines individual task losses through weighted summation:

$$ \mathcal{L}_{total} = \sum_{i=1}^T w_i \mathcal{L}_i(\theta_{shared}, \theta_i) $$

where T is the number of tasks, wi are learnable weights, and θshared, θi denote shared and task-specific parameters respectively.

Key Architectural Variations

Performance Optimization

Gradient conflict between tasks is mitigated through:

$$ \nabla_{\theta_{shared}} \mathcal{L}_{total} = \sum_{i=1}^T w_i \frac{\partial \mathcal{L}_i}{\partial \theta_{shared}} $$

Advanced techniques include:

Benchmark Results

On the NYUv2 dataset (depth estimation, surface normals, semantic segmentation), multi-task transformers achieve:

Model Depth (RMSE ↓) Normals (Mean Angle Error ↓) Segmentation (mIoU ↑)
ResNet-50 MTL 0.58 25.3° 42.1%
ViT-Base MTL 0.51 22.7° 47.3%
Swin-Large MTL 0.48 20.1° 51.6%

Implementation Considerations


  class MultiTaskViT(nn.Module):
      def __init__(self, num_tasks, patch_size=16, dim=768):
          super().__init__()
          self.patch_embed = PatchEmbedding(patch_size, dim)
          self.transformer = TransformerEncoder(dim)
          self.heads = nn.ModuleList([
              TaskHead(dim, task_output_dims[i]) 
              for i in range(num_tasks)
          ])
          
      def forward(self, x):
          x = self.patch_embed(x)
          x = self.transformer(x)
          return [head(x[:, 0]) for head in self.heads]  # CLS token

Critical hyperparameters include the number of attention heads (typically 12-16), patch size (16×16 or 32×32 pixels), and the ratio of shared to task-specific layers. Mixed-precision training is essential for maintaining throughput with large batch sizes.

Computer Vision Applications – Multi-Task Learning with Transformers – Tutorial Diagram
Diagram Description: The diagram would show the architecture of a Vision Transformer (ViT) for multi-task learning, including patch embedding, transformer encoder, and task-specific heads.

5. Metrics for Multi-Task Learning Evaluation

5.1 Metrics for Multi-Task Learning Evaluation

Evaluating multi-task learning (MTL) models requires specialized metrics that account for performance across multiple tasks while balancing trade-offs. Unlike single-task learning, MTL introduces complexities in measuring joint optimization, task interference, and overall system utility.

Task-Specific Metrics

Each task in an MTL framework typically retains its domain-specific evaluation metrics. For classification tasks, standard measures include:

$$ \text{Accuracy} = \frac{TP + TN}{TP + TN + FP + FN} $$
$$ F_1 = 2 \cdot \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}} $$

For regression tasks, mean squared error (MSE) and R² scores remain applicable:

$$ \text{MSE} = \frac{1}{n}\sum_{i=1}^n (y_i - \hat{y}_i)^2 $$

Composite MTL Metrics

The key challenge lies in aggregating task-specific metrics into unified scores. Three principal approaches dominate current research:

1. Arithmetic Mean

The simplest aggregation method computes the unweighted average across all tasks:

$$ \text{MTL-Score} = \frac{1}{T}\sum_{t=1}^T \text{Metric}_t $$

While interpretable, this approach fails to account for task difficulty or relative importance.

2. Weighted Sum

More sophisticated methods incorporate task weights wt:

$$ \text{MTL-Score} = \sum_{t=1}^T w_t \cdot \text{Metric}_t $$

Where weights can be determined by:

3. Pareto Optimality

Advanced evaluations use multi-objective optimization criteria:

$$ \text{MTL-Score} = 1 - \text{Volume}(\text{Pareto Front}) $$

This measures the hypervolume dominated by the model's performance vector in metric space.

Negative Transfer Metrics

Critical for MTL evaluation, these quantify performance degradation compared to single-task baselines:

$$ \text{Negative Transfer Ratio} = \frac{\text{MTL Performance} - \text{STL Performance}}{\text{STL Performance}} $$

Values below zero indicate harmful interference between tasks.

Computational Efficiency

MTL models should be evaluated on resource utilization metrics:

Transformer-Specific Considerations

When evaluating transformer-based MTL systems, additional metrics become relevant:

$$ \text{Attention Alignment Score} = \frac{1}{L}\sum_{l=1}^L \text{JS-Divergence}(A_l^{\text{task}_1}, A_l^{\text{task}_2}) $$

where L is the number of layers and Al represents attention matrices.

5.2 Comparing Single-Task vs. Multi-Task Performance

Performance Metrics and Trade-offs

When evaluating single-task versus multi-task learning (MTL) with transformers, performance is typically measured across three dimensions: accuracy, computational efficiency, and generalization capability. Single-task models optimize exclusively for one objective, often achieving marginally higher task-specific accuracy. However, MTL models leverage shared representations, which can improve generalization at the cost of slight per-task performance degradation. The trade-off is governed by:

$$ \mathcal{L}_{MTL} = \sum_{i=1}^{N} \lambda_i \mathcal{L}_i(\theta_{shared}, \theta_i) $$

where λi are task weights, and θshared, θi denote shared and task-specific parameters, respectively. Empirical studies show that MTL transformers achieve 2–15% higher cross-task generalization but may underperform single-task models by 1–5% on individual tasks when λi are suboptimally balanced.

Architectural Differences

Single-task transformers use dedicated architectures (e.g., BERT for text classification), while MTL variants like MT-DNN or Cross-Stitch Networks share lower layers and branch into task-specific heads. The key divergence lies in gradient dynamics:

$$ abla_{\theta_{shared}} \mathcal{L}_{MTL} = \sum_{i=1}^{N} \lambda_i abla_{\theta_{shared}} \mathcal{L}_i $$

This shared gradient often leads to implicit regularization, as shown by Standley et al. (2020), where MTL transformers exhibit 20–30% lower variance in out-of-distribution (OOD) evaluations compared to single-task counterparts.

Case Study: Natural Language Processing

In NLP, the GLUE benchmark reveals that MTL transformers (e.g., T5) achieve 85.4% average accuracy across 11 tasks, while single-task fine-tuned BERT averages 83.1%. However, for high-stakes tasks like medical entity recognition, single-task models retain a 3.8% F1-score advantage due to task-specific feature specialization.

Computational Costs

MTL reduces total inference time by up to 40% for N tasks by amortizing the shared backbone computation. However, memory overhead increases linearly with task-specific heads. For a transformer with d layers and k tasks, the parameter count scales as:

$$ \Theta_{MTL} = \Theta_{shared} + \sum_{i=1}^{k} \Theta_i \approx \Theta_{single} \times (1 + \frac{k}{5}) $$

as derived from the Taskology framework (Fifty et al., 2021).

Negative Transfer Mitigation

When tasks conflict, MTL performance can degrade below single-task baselines. Techniques like Gradient Sign Dropout (Chen et al., 2020) or PCGrad (Yu et al., 2020) project conflicting gradients to orthogonal subspaces, recovering up to 90% of single-task performance while retaining MTL benefits.

Comparing Single-Task vs. Multi-Task Performance – Multi-Task Learning with Transformers – Tutorial Diagram
Diagram Description: The diagram would show the architectural differences between single-task and multi-task transformers, including shared layers and task-specific heads, with gradient flow paths.

Interpreting Model Behavior and Task Interactions

Understanding how a multi-task transformer model allocates its capacity across tasks requires probing both the shared and task-specific components of its architecture. The attention mechanism in transformers provides a natural window into task interactions, as attention weights reveal how information flows between tokens and across tasks.

Attention Head Specialization Analysis

In multi-task transformers, attention heads often specialize for specific tasks or develop shared representations. To quantify this, compute the task-specificity score for each head:

$$ S_h = \frac{1}{T} \sum_{t=1}^T \left( \max_{t'} |A_h^{(t)} - A_h^{(t')}| \right) $$

where Ah(t) represents the attention pattern matrix for head h on task t, and T is the total number of tasks. High Sh values indicate task-specific heads, while low values suggest shared functionality.

Gradient-Based Task Interaction Maps

The gradient conflict matrix C ∈ ℝT×T reveals how tasks compete for parameter updates:

$$ C_{ij} = \frac{\langle \nabla_{\theta} \mathcal{L}_i, \nabla_{\theta} \nabla_{\theta} \mathcal{L}_j \rangle}{||\nabla_{\theta} \mathcal{L}_i|| \cdot ||\nabla_{\theta} \mathcal{L}_j||} $$

where ∇θLi is the gradient of loss for task i. Positive values indicate synergistic tasks, while negative values show competition. This matrix guides task weighting strategies during training.

Representation Similarity Analysis

Centered Kernel Alignment (CKA) measures similarity between task representations at different layers:

$$ \text{CKA}(K, L) = \frac{||L^TK||_F^2}{||K^TK||_F ||L^TL||_F} $$

where K and L are representation matrices for two tasks. Plotting CKA across layers reveals where tasks diverge or share representations.

Practical Implementation

To implement these analyses in PyTorch for a transformer model:


def compute_gradient_conflict(model, batch, tasks):
    grads = {}
    for t in tasks:
        loss = model.compute_loss(batch, task=t)
        grad = torch.autograd.grad(loss, model.parameters(), retain_graph=True)
        grads[t] = torch.cat([g.flatten() for g in grad])
    
    C = torch.zeros(len(tasks), len(tasks))
    for i, ti in enumerate(tasks):
        for j, tj in enumerate(tasks):
            C[i,j] = torch.dot(grads[ti], grads[tj]) / \
                     (torch.norm(grads[ti]) * torch.norm(grads[tj]))
    return C

Case Study: NLP Multi-Task Benchmark

Analysis of the GLUE benchmark reveals that syntactic tasks (e.g., part-of-speech tagging) share lower-layer representations, while semantic tasks (e.g., sentiment analysis) diverge at higher layers. The gradient conflict matrix shows strong synergy between NLI and paraphrase detection tasks (Cij > 0.7), but competition between sentiment analysis and textual similarity (Cij < -0.3).

Visualizing Task Dynamics

A 3D t-SNE projection of task-specific attention patterns shows clustering by task family. The visualization reveals that while some attention heads maintain consistent patterns across tasks (positioned near the origin), others form distinct clusters corresponding to different task types.

Interpreting Model Behavior and Task Interactions – Multi-Task Learning with Transformers – Tutorial Diagram
Diagram Description: The diagram would show the gradient conflict matrix visualization and the 3D t-SNE projection of task-specific attention patterns, which are spatial relationships that text alone cannot fully convey.

6. Key Research Papers on Multi-Task Learning

6.1 Key Research Papers on Multi-Task Learning

6.2 Recommended Books and Surveys

6.3 Open-Source Implementations and Tools