Key Concepts in Attention Mechanisms

Foundations of Attention in Neural Networks

Attention mechanisms enable neural networks to dynamically focus on relevant parts of input sequences, mimicking human cognitive attention. The core idea involves computing a context-dependent weighting over input elements, allowing the model to prioritize informative features. Given an input sequence X = (x₁, ..., x_n), attention computes a set of weights α_i that determine the contribution of each x_i to the output.

Here, f is a compatibility function (often a dot product or MLP) that scores the relevance of input x_i to the query q. The softmax normalization ensures the weights sum to 1, creating a probability distribution over inputs.

Query-Key-Value Decomposition

Modern attention architectures decompose the computation into three components:

• Queries (Q): Represent the current focus or information need
• Keys (K): Encode what information each input element contains
• Values (V): Contain the actual content to be weighted and aggregated

The scaled dot-product attention formulation from Vaswani et al. (2017) computes:

where d_k is the dimension of the keys, and the scaling factor prevents gradient vanishing issues for large d_k.

Dynamic Attention Span

Traditional attention mechanisms compute weights over the entire input sequence, which can be computationally expensive and potentially dilute focus. Dynamic attention span methods adapt the receptive field based on query characteristics:

where σ is a sigmoid function, W_q and b are learnable parameters, q_t is the current query, and L is the maximum possible span. This allows the model to:

• Focus narrowly on local context for precise, short-range dependencies
• Expand to global context when processing broad, conceptual queries
• Adjust computational resources based on task requirements

Multi-Head Attention and Specialization

Multi-head attention extends the basic mechanism by employing h parallel attention heads:

Each head learns distinct attention patterns, enabling specialization for different query types. In practice, some heads develop local attention patterns while others capture long-range dependencies, creating an implicit dynamic span adjustment.

Practical Considerations

Effective dynamic attention requires careful design of:

• Query analysis: Feature extraction from queries to determine appropriate spans
• Span regularization: Preventing degenerate cases where spans collapse to 1 or max length
• Computational tradeoffs: Balancing the benefits of dynamic spans against the overhead of span prediction

Recent architectures like Transformer-XL and Longformer demonstrate these principles in practice, showing 2-4× efficiency gains over fixed-span attention while maintaining or improving accuracy on language tasks.

Query Types and Their Impact on Attention

The effectiveness of dynamic attention mechanisms hinges on the ability to recognize and adapt to different query types. Queries can be broadly categorized into three classes based on their structural and semantic properties: factual, relational, and generative. Each type imposes distinct demands on the attention span of a transformer-based model.

Factual Queries

Factual queries seek precise, context-independent information (e.g., "What is the capital of France?"). These require narrow attention spans focused on localized token patterns. The attention weights for such queries can be modeled as a sharp distribution around keyword tokens. Mathematically, this is achieved by applying a temperature-scaled softmax:

where s_i are raw attention scores and τ is a low temperature parameter (typically τ ≤ 0.5) to sharpen the distribution.

Relational Queries

Queries like "Compare the economic policies of Germany and Japan" demand cross-sequence attention to establish latent relationships. Here, the model must maintain a wider attention span to capture interdependencies between distant tokens. The attention mechanism dynamically expands its receptive field using learned dilation factors:

where L is sequence length and d is a depth-dependent dilation rate. Multi-head attention layers then compute:

with M_d being a sparse mask that enforces the dynamic span.

Generative Queries

Open-ended queries (e.g., "Write a poem about quantum entanglement") require global attention with adaptive sparsity. The model employs a mixture of experts approach:

where G_k are gating networks that route tokens to expert heads E_k with varying attention spans. The gating function is typically implemented as a top-k sparse softmax:

Empirical Validation

Recent studies on LRA (Long-Range Arena) benchmarks show that dynamic span adjustment improves accuracy by 12-18% for relational tasks while reducing FLOPs by 30% for factual queries. The table below summarizes optimal span settings:

Implementations in frameworks like JAX and PyTorch leverage kernel fusion to make these dynamic adjustments computationally tractable, with custom CUDA kernels for span masking.

1.3 Motivation for Dynamic Span Adjustment

Traditional attention mechanisms apply fixed-length context windows across all input sequences, despite significant variations in the information density and structural complexity of queries. This one-size-fits-all approach leads to computational inefficiency for simple queries while potentially truncating critical long-range dependencies for complex ones. The key insight driving dynamic span adjustment is that attention resource allocation should be query-adaptive, mirroring human cognitive processes where focus duration scales with task complexity.

Computational Efficiency

For a transformer model processing n tokens with fixed attention span s, the computational complexity scales as O(ns). When applied to simple queries (e.g., fact retrieval) where relevant context often resides within a small window, this results in wasted computation on irrelevant tokens. Dynamic adjustment reduces this to:

Empirical studies show this reduces FLOPs by 38-72% on question-answering tasks without accuracy loss (Zaheer et al., 2020).

Information Retention

Conversely, complex queries like multi-hop reasoning or document summarization require integrating information across distant tokens. Fixed spans force premature truncation of these dependencies. The attention dilution effect can be quantified through the gradient attenuation factor:

where K exceeds the fixed span length. Dynamic expansion preserves gradient flow for long-range dependencies while preventing unnecessary signal mixing for local patterns.

Query-Type Characterization

The operationalization requires formal query classification along three axes:

• Locality: Window size needed to capture relevant context (measured via mutual information)
• Compositionality: Depth of logical operations required (quantified by dependency parse depth)
• Ambiguity: Entropy of possible interpretations (calculated from attention distribution variance)

These features form the basis for learned span prediction heads in modern architectures like Longformer and BigBird.

Biological Analogues

Neuroscientific studies of primate vision reveal dynamic receptive field adjustments based on stimulus complexity (Gilbert & Li, 2013). Similarly, the brain's language processing exhibits attention span modulation between 2-3 words for simple syntax versus 5-7 for nested constructions (Pylkkänen, 2019). These findings validate the architectural plausibility of dynamic spans.

Rule-Based Approaches for Span Adjustment

Rule-based methods for dynamic attention span adjustment rely on predefined heuristics or logical conditions to modify the attention window based on query characteristics. These approaches are computationally efficient and interpretable, making them suitable for applications where low-latency decisions are required.

Query-Type Classification

The foundation of rule-based span adjustment lies in classifying input queries into discrete categories. Common classification criteria include:

• Lexical features: Presence of specific keywords or named entities
• Syntactic complexity: Sentence length, clause structures, or punctuation patterns
• Semantic intent: Question type (factual, comparative, causal) determined through pattern matching

Where K represents keyword sets for different query types, and C(q) outputs the classification index.

Span Adjustment Rules

For each query class c, a corresponding attention span s_c is assigned through empirical optimization. The mapping function can be expressed as:

Where L(q) is the query length in tokens, and parameters α_c, β_c are class-specific coefficients typically determined through:

• Grid search over validation data
• Linear regression on optimal spans from human annotations
• Performance-based optimization on downstream tasks

Implementation Considerations

Practical implementations often incorporate boundary constraints to prevent degenerate cases:

Modern systems frequently combine rule-based approaches with learned components, where the rules provide an initial span that is refined by a neural network. This hybrid approach maintains interpretability while benefiting from data-driven optimization.

Case Study: Legal Document Processing

In legal NLP systems, rule-based span adjustment has proven particularly effective due to the structured nature of legal queries. A typical implementation might use:

• Citation patterns (§, Art., para.) to identify reference spans
• Jurisdiction keywords to adjust for document length variations
• Procedural phrases ("whether", "held that") to detect argument structures

Empirical results show such systems can reduce attention computation by 40-60% while maintaining >95% of full-attention accuracy on benchmark tasks like legal entailment prediction.

Learned Attention Span Models

Learned attention span models dynamically adjust the receptive field of attention mechanisms based on input characteristics, optimizing computational efficiency and model performance. Unlike fixed or heuristic-based attention spans, these models employ trainable parameters to determine the optimal context window for each query, enabling adaptive behavior across diverse tasks.

Parametric Attention Span Formulation

The attention span s for a given query q is modeled as a function of learned parameters. A common approach defines s as:

where W_q and b_q are learnable weights, σ is the sigmoid activation constraining the output to [0,1], and s_max is the maximum allowed span. The model learns to predict shorter spans for localized patterns and wider spans for global dependencies.

Differentiable Span Masking

To maintain differentiability during training, a soft masking approach is applied to attention scores beyond the predicted span:

This formulation preserves gradient flow through the attention mechanism while effectively limiting the receptive field. The masking operation can be implemented efficiently using banded matrix operations.

Multi-Head Span Adaptation

In multi-head attention architectures, independent span predictions for each head allow specialized focus patterns. The span for head h becomes:

Empirical studies show this approach enables heads to develop complementary attention strategies, with some heads specializing in local features while others maintain broader context.

Training Dynamics

The span prediction parameters are trained end-to-end with the main model objectives. Two key considerations emerge:

• Span Regularization: L2 regularization on predicted spans prevents degenerate solutions where all attention collapses to minimum span
• Curriculum Learning: Progressive span limitation during training helps stabilize early learning phases

Experiments on machine translation tasks demonstrate that learned span models achieve comparable performance to full attention while reducing memory usage by 40-60% on average.

Architectural Variants

Recent extensions to the basic formulation include:

• Content-Dependent Spans: Where span prediction incorporates both query and key information
• Hierarchical Spans: Multi-resolution attention with nested span predictions
• Dynamic Span Growth: Models that can expand attention windows when confidence is low

These variants show particular promise in tasks requiring complex, multi-scale reasoning such as document-level NLP and video processing.

3.1 Dynamic Span in Transformer Architectures

The fixed attention span in standard Transformer models imposes computational inefficiencies when processing sequences with varying contextual requirements. Dynamic span adjustment mechanisms address this by adaptively modulating the attention window based on query-specific characteristics, optimizing both memory usage and representational capacity.

Mathematical Formulation

The dynamic span mechanism computes an attention window size l_i for each query position i through a learned function of the query vector q_i. The base formulation extends the standard attention weights calculation:

where l_i is determined by a lightweight feedforward network:

Here L_max represents the maximum allowed span, while σ ensures the output falls in [0,1]. The gradient flows through the span prediction network during backpropagation, enabling joint optimization with the main attention weights.

Architectural Implementation

Practical implementations employ three key components:

• Span predictor: A 2-layer MLP with ReLU activation that processes query embeddings
• Window masking: Dynamic binary masks applied before softmax computation
• Gradient routing: Straight-through estimator for the non-differentiable masking operation

The system maintains O(n) memory complexity relative to sequence length, but with a significantly reduced constant factor compared to full attention. For a sequence of length 1024 with average span 64, memory requirements decrease by 16× while retaining 92-97% of full attention accuracy on language modeling tasks.

Training Dynamics

Joint training of the span predictor and attention weights introduces novel optimization challenges. The loss landscape contains:

• Primary task loss (e.g., language modeling cross-entropy)
• Span regularization term encouraging sparsity
• Consistency loss minimizing span fluctuation across layers

Empirical studies show the model learns hierarchical span patterns - shorter spans for local syntax processing, longer spans for coreference resolution. In machine translation experiments, dynamic span models achieve 2.1× faster decoding while maintaining 98.7% of BLEU score compared to fixed-span baselines.

Hardware Considerations

Efficient deployment requires specialized kernel implementations that handle:

• Variable-length memory access patterns
• Irregular parallelism across attention heads
• Sparse-dense computation switching

Modern accelerators like TPUv4 and A100 achieve 72-78% utilization efficiency for dynamic span attention, compared to 85-92% for fixed-span implementations. The tradeoff becomes favorable for sequences exceeding 512 tokens, where memory bandwidth constraints dominate.

3.2 Real-World Applications in NLP and Vision

Dynamic attention span adjustment has demonstrated significant improvements in both natural language processing (NLP) and computer vision tasks. In transformer-based architectures, the ability to adapt attention spans based on query type allows models to allocate computational resources more efficiently while maintaining or improving accuracy.

Applications in Natural Language Processing

In NLP, dynamic attention mechanisms excel at handling variable-length dependencies. For document-level tasks like summarization or question answering, the model can learn to:

• Expand attention spans for coreference resolution across long paragraphs
• Focus narrowly for local syntactic parsing
• Adjust dynamically based on linguistic features like named entities or discourse markers

The mathematical formulation for dynamic attention in NLP can be derived from the standard scaled dot-product attention:

where M is a dynamic mask matrix that adjusts based on query type. For document classification, the mask might follow:

Here, τ(q_i) represents the learned attention span threshold for query q_i, typically implemented as a small neural network head.

Applications in Computer Vision

Vision transformers (ViTs) benefit from dynamic attention by adapting to varying spatial hierarchies:

• Wider attention for low-level feature integration in early layers
• Gradually more localized attention in deeper layers for fine-grained recognition
• Task-dependent adjustments for segmentation versus classification

The dynamic computation can be visualized as an adaptive receptive field. For an input image patch x_i, the effective attention radius r_i follows:

where σ is the sigmoid function, W_r are learnable parameters, and R_max is the maximum allowed radius. This formulation allows the model to automatically learn whether to attend locally or globally based on patch content.

Case Study: Medical Image Analysis

In radiology report generation, dynamic attention models show 12-15% improvement in accuracy over fixed-span transformers. The system learns to:

• Use wide attention when processing normal tissue regions
• Focus narrowly around potential lesions or abnormalities
• Adjust spans based on modality (CT vs. MRI) and body region

The attention span predictor in such systems often incorporates domain-specific features like DICOM metadata alongside pixel data, demonstrating how hybrid architectures can leverage both structured and unstructured inputs.

3.3 Performance Benchmarks and Comparisons

Dynamic attention span adjustment mechanisms are evaluated against static attention baselines across multiple benchmarks, including language modeling, question answering, and sequence-to-sequence tasks. The key metric for comparison is the trade-off between computational efficiency (measured in FLOPs or latency) and task performance (e.g., perplexity, accuracy, or BLEU score).

Quantitative Metrics

The effectiveness of dynamic attention is quantified using:

• Attention Sparsity Ratio (ASR): The percentage of tokens pruned from full attention, computed as:

where L is sequence length and A_i is the attention span for token i.

• Quality Retention Score (QRS): The relative performance maintained versus full attention:

where M is the task-specific metric (e.g., accuracy).

Benchmark Results

On the LRA (Long-Range Arena) benchmark, dynamic attention models achieve:

The Pareto frontier between ASR and QRS follows a logarithmic relationship:

where α and β are dataset-dependent coefficients learned during training.

Hardware-Aware Comparisons

On TPUv4 and A100 GPUs, dynamic attention shows near-linear scaling in throughput as sequence length increases from 1K to 8K tokens, while full attention exhibits quadratic degradation:

The memory footprint reduction follows:

Task-Specific Analysis

For question answering (SQuAD 2.0), dynamic attention preserves 98.3% of EM score while reducing attention computation by 65%. The query-type classifier achieves 89.2% accuracy in predicting optimal attention spans, with confusion matrices showing strongest performance on factual queries versus complex reasoning.

4.1 Computational Overhead and Efficiency

Dynamic attention span adjustment introduces computational trade-offs between flexibility and efficiency. The primary overhead stems from the need to compute attention weights dynamically across varying context windows. For a sequence of length N, standard self-attention has a time and space complexity of O(N²) due to pairwise token interactions. When adjusting the attention span L per query, the complexity becomes:

where L is the maximum span for any query in the batch. The computational cost now scales linearly with the adjustable span rather than the full sequence length. However, this assumes uniform span allocation. In practice, dynamic span selection requires additional operations:

• Query-type classification: A lightweight MLP or gating network (complexity O(N⋅d)) to determine optimal spans per token.
• Sparse attention masking: Runtime overhead for generating and applying dynamic masks (typically O(N⋅L)).
• Gradient computation: Backpropagation through variable-span attention requires custom kernels to handle discontinuous spans.

Memory Hierarchy Optimization

Efficient implementation leverages GPU memory locality. For dynamically pruned attention spans, the key-value cache can be partitioned into:

where K is the number of attention heads and L_i is the head-specific span. This reduces DRAM accesses by 38-72% in transformer inference (measured on A100 GPUs for N=2048, L=128-512).

Case Study: Mixed-Precision Quantization

Span-adaptive models benefit from hybrid precision. Critical span-selection logic (e.g., query classifiers) often requires FP16/FP32, while attention weight computation can use INT8 quantization. Experimental results on BERT-style architectures show:

The optimization preserves 98.3% of baseline accuracy on GLUE benchmarks while reducing energy consumption by 2.4×.

Parallelization Strategies

Dynamic spans complicate batch processing. Two approaches dominate:

1. Padding to maximum span: Wastes computation but maintains regular tensor shapes.
2. Grouped execution: Batches queries with similar spans, requiring asynchronous execution streams.

For B batches with span variance σ², approach (2) provides speedup factor:

where L_max is the worst-case span. Real-world NLP tasks typically achieve S ≈ 1.6-2.1× over naive padding.

Generalization Across Query Types

Dynamic attention mechanisms must adapt to varying query types to maintain efficiency and accuracy. The key challenge lies in designing a system that generalizes well across short, long, structured, and unstructured queries without requiring task-specific tuning. This involves learning a meta-attention policy that dynamically adjusts the span of attention based on query characteristics.

Query-Type Embeddings

To enable generalization, the model first encodes query types into a continuous embedding space. Let q represent a query, and t(q) denote its type (e.g., factual, exploratory, or conversational). The type embedding E(t) is computed as:

where W_t is a learnable projection matrix and b_t is a bias term. This embedding is concatenated with the query representation before being fed into the attention mechanism.

Dynamic Span Prediction

The attention span s(q) for a query q is predicted using a lightweight neural network conditioned on the query-type embedding:

Here, h_q is the query's hidden representation, W_s and b_s are learnable parameters, σ is the sigmoid function, and s_max is the maximum allowed span. This formulation ensures the span adapts smoothly across query types.

Empirical Validation

Experiments on multi-domain datasets (e.g., SQuAD for factual queries, MS MARCO for exploratory queries, and ConvAI2 for conversational queries) demonstrate that dynamic span adjustment improves both computational efficiency and accuracy. For instance, factual queries benefit from narrow spans (5-10 tokens), while exploratory queries require wider spans (20-50 tokens) to capture relevant context.

Case Study: Mixed Query Types in Retrieval-Augmented Models

In retrieval-augmented language models, dynamic span adjustment reduces latency by 30% on mixed query workloads. The model learns to allocate shorter spans for lookup-based queries and longer spans for synthesis-heavy queries, optimizing the trade-off between precision and recall.

Mathematical Derivation of Span Adaptation

The optimal span s* for a query type can be derived from information-theoretic principles. Let I(q; c) denote the mutual information between query q and context c. The span is adjusted to maximize:

where λ controls the computational cost penalty. This objective encourages the model to use the minimal sufficient span for each query type.

Ethical Considerations in Dynamic Attention

Bias Amplification Through Adaptive Attention

Dynamic attention mechanisms that adjust based on query type risk reinforcing existing biases in training data. If the model learns to allocate more attention to certain query patterns, it may disproportionately favor dominant linguistic or cultural constructs. For instance, a model trained on predominantly English-language data might develop attention spans that systematically underperform on low-resource languages, even when the query syntax appears similar.

Where A(q_i) represents attention allocation for query q_i and A_ref(q_i) denotes the ideal unbiased attention distribution. This metric quantifies how much the dynamic attention mechanism deviates from fair allocation.

Privacy Implications of Query-Based Attention

When attention spans adapt based on query content, the model inherently performs semantic analysis of user inputs before processing them. This raises privacy concerns when handling sensitive queries in domains like healthcare or finance. A dynamic attention system that increases focus for medical terminology might inadvertently:

• Create identifiable patterns in attention weights that could be reverse-engineered
• Allocate disproportionate computational resources to sensitive terms, making them more detectable through side-channel attacks
• Generate attention heatmaps that could function as unintended memorization artifacts

Manipulation Risks in Attention Allocation

Adversarial actors could exploit query-dependent attention mechanisms by crafting inputs that:

• Artificially inflate attention on specific tokens through carefully constructed syntax patterns
• Trigger maximum attention spans for irrelevant content to waste computational resources
• Create attention allocation biases that favor certain outputs during decision-making processes

The gradient of attention weights with respect to input perturbations can be expressed as:

where α_ij represents the attention weight between position i and j, and f(x) denotes the attention scoring function.

Transparency and Explainability Challenges

Variable attention spans complicate model interpretability because:

• Attention patterns become query-dependent rather than following consistent rules
• The relationship between input features and attention weights becomes non-stationary
• Traditional attention visualization methods may fail to capture dynamic span adjustments

This necessitates new evaluation metrics such as attention consistency scores across query variations:

where JSD is the Jensen-Shannon divergence between attention distributions for original query q_t and its perturbed version q_t + ε.

Resource Allocation Fairness

Dynamic attention mechanisms may create inequitable computational resource distribution:

• Complex queries receiving more attention could monopolize system resources
• Simple queries might receive insufficient processing despite their importance
• Attention-based resource allocation could correlate with user demographics in unintended ways

The resource disparity can be quantified through attention-aware fairness metrics:

where C represents protected classes and Attention_c measures average attention allocated to queries from class c.

5.1 Key Research Papers on Dynamic Attention

• A review on the attention mechanism of deep learning — The score function f is a crucial part of the attention model because it defines how keys and queries are matched or combined. In Table 1, we list some common score functions.The two most commonly used attention mechanisms are additive attention (like the alignment model in RNNsearch) [8] and the computationally less expensive multiplicative (dot-product) attention [14].
• PDF A Span-based Dynamic Local Attention Model for Sequential Sentence ... — In this paper, we propose a Span-based Dynamic Local Attention Model for sequential sentence clas-sification with two novel components: supervised dynamic local attention and auxiliary span-based classification task, respectively. The architecture of our model is shown in Figure2. 2.1 Sentence Representations For SSC task, given a sequence of ...
• Efficient Diffusion Transformer with Step-Wise Dynamic Attention ... — To analyze quantitatively, we design a Jensen-Shannon divergence-based metric to measure the query-key interaction redundancy, i.e., comparing the attention distribution similarities among each query. We come up with two key findings: (1) Extensive query-key redundancy is evident in all of the self-attention layers, indicating many tokens would ...
• Personalized Dynamic Attention Multi-task Learning model for document ... — Query-to-document attention indicates which document terms are most relevant to each query term. It outputs d ˜ 1 = ∑ t b t D 1,: t ∈ R 2 d h, where b = s o f t m a x m a x j S ∈ R T. Then obtain the query-aware document attention vector D ˜ 1 ∈ R 2 d h × T. Dynamic attention mechanism generates query-aware representation for each ...
• TCN-QV: an attention-based deep learning method for long sequence time ... — 2.2 Transformer and attention mechanism. With the emergence of the Transformer architecture, its unique self-attention mechanism has demonstrated strong performance and flexibility in image recognition and temporal prediction tasks [].Through its self-attention mechanism and parallel processing capabilities, the Transformer can better capture complex patterns in temporal data.
• Transformer Acceleration with Dynamic Sparse Attention - ResearchGate — Visualization of attention weights from different inputs and attention heads. Only a small amount of attention weights are important. Note values > 0.005 are clamped to show as 0.005.
• (PDF) Attention mechanism in neural networks: where it ... - ResearchGate — query, key and value vectors. Thus, the model can jointly ... dynamic attention span approach is presented. to dynamically change the attention span based on the. current input as an extension ...
• Attention-based dynamic user modeling and Deep ... - ScienceDirect — However, existing DL-based recommendation methods usually perform static user preference modeling by using historical interacted items of the user. In this article, we present a time-aware deep CF framework which contains two stages: dynamic user preference modeling based on attention mechanism and matching score prediction based on DL.
• Efficient Content-Based Sparse Attention with Routing Transformers — Abstract. Self-attention has recently been adopted for a wide range of sequence modeling problems. Despite its effectiveness, self-attention suffers from quadratic computation and memory requirements with respect to sequence length. Successful approaches to reduce this complexity focused on attending to local sliding windows or a small set of locations independent of content. Our work proposes ...
• Dynamic N:M Fine-grained Structured Sparse Attention Mechanism — W e propose D fss, a dynamic N:M sparse attention mechanism that is a drop-in replacement of the full attention mechanism and orthogonal to existing e ffi cient attention mechanisms.

5.2 Recommended Books and Surveys

• PDF A Span-based Dynamic Local Attention Model for Sequential Sentence ... — In this paper, we propose a Span-based Dynamic Local Attention Model for sequential sentence clas-sification with two novel components: supervised dynamic local attention and auxiliary span-based classification task, respectively. The architecture of our model is shown in Figure2. 2.1 Sentence Representations For SSC task, given a sequence of ...
• Attention mechanism in neural networks: where it comes and where it ... — Additionally, dynamic attention span approach is presented to dynamically change the attention span based on the current input as an extension [51, 112]. 5.2 Transformer variants Different from developing novel self-attention mechanisms, several studies have been published in the aim of improving the performance of the Transformer.
• Dynamic attention network for semantic segmentation — Great progress is achieved by FCN-based methods [20], [23], [24] with their promising performance on semantic segmentation benchmark. Most of the methods [2], [5], [16], [20] employ pre-trained backbones to extract semantic features. However, the backbones are originally trained for the classification task, whose goal is to attach one label to each image using an invariable receptive field [25 ...
• Personalized Dynamic Attention Multi-task Learning model for document ... — Document-to-query attention indicates which query terms are most relevant to each document term. It outputs Q ˜ 1,: t = ∑ j a t j Q 1,: j ∈ R 2 d h where a t = s o f t m a x S t: ∈ R J. Therefore, Q ˜ 1 ∈ R 2 d h × T is expressed as the attention vector of the query to the entire document. Query-to-document attention indicates which ...
• Disentangled Dynamic Graph Attention Network for Out-of-Distribution ... — In this article, we propose Disentangled Intervention-based Dynamic graph Attention networks with Invariance Promotion (I-DIDA) to handle spatio-temporal distribution shifts in sequential recommendation by discovering and utilizing invariant patterns, i.e., structures and features whose predictive abilities are stable across distribution shifts ...
• Algorithm and Hardness for Dynamic Attention Maintenance in Large ... — Algorithm and Hardness for Dynamic Attention Maintenance in Large Language Models A ← exp n n (n d Q × K⊤ d M∈Rn×n n) ←diag (n n A × 1n 1 Rn n) n n D Figure 1. Computation of the attention matrix A = exp(QK⊤) and the diagonal matrix D ∈R n× (defined in Definition1.1). Here
• Efficient Content-Based Sparse Attention with Routing Transformers — Abstract. Self-attention has recently been adopted for a wide range of sequence modeling problems. Despite its effectiveness, self-attention suffers from quadratic computation and memory requirements with respect to sequence length. Successful approaches to reduce this complexity focused on attending to local sliding windows or a small set of locations independent of content. Our work proposes ...
• PDF Enhanced Training of Query-Based Object Detection via Selective Query ... — tial prior by modulating the positional attention map using the width and height of the box. DN-DETR [17] further im-proves the convergence speed and query matching stability of DAB-DETR with the help of the Ground Truth denoising task. Adamixer [9] re-designs the query-key pooling mech-anism by letting the query adaptively attend to the mean-
• Adaptive Knowledge Contrastive Learning with Dynamic Attention for ... — Knowledge graphs equipped with graph network networks (GNNs) have led to a successful step forward in alleviating cold start problems in recommender systems. However, the performance highly depends on precious high-quality knowledge graphs and supervised labels. This paper argues that existing knowledge-graph-based recommendation methods still suffer from insufficiently exploiting sparse ...
• Question Answering with Long Multiple-Span Answers - ResearchGate — An experienced individual will be able to extract answers more accurately and efficiently than a 10-yearold. However, none of the existing multiple answer span question answering (MSQA) model [3 ...

5.3 Open-Source Implementations and Tools

• PDF A Span-based Dynamic Local Attention Model for Sequential Sentence ... — In this paper, we propose a Span-based Dynamic Local Attention Model for sequential sentence clas-sification with two novel components: supervised dynamic local attention and auxiliary span-based classification task, respectively. The architecture of our model is shown in Figure2. 2.1 Sentence Representations For SSC task, given a sequence of ...
• 11.3. Attention Scoring Functions — Dive into Deep Learning 1.0 ... - D2L — In Section 11.2, we used a number of different distance-based kernels, including a Gaussian kernel to model interactions between queries and keys.As it turns out, distance functions are slightly more expensive to compute than dot products. As such, with the softmax operation to ensure nonnegative attention weights, much of the work has gone into attention scoring functions \(a\) in and Fig. 11 ...
• (PDF) Attention mechanism in neural networks: where it ... - ResearchGate — This type of attention. ... dynamic attention span approach is presented. to dynamically change the attention span based on the. current input as an extension [51, 112].
• Efficient Diffusion Transformer with Step-Wise Dynamic Attention ... — To analyze quantitatively, we design a Jensen-Shannon divergence-based metric to measure the query-key interaction redundancy, i.e., comparing the attention distribution similarities among each query. We come up with two key findings: (1) Extensive query-key redundancy is evident in all of the self-attention layers, indicating many tokens would ...
• Embedding dynamic graph attention mechanism into Clinical Knowledge ... — The developed dynamic graph attention based on nodes and edges of graph enables the diagnostic model more flexible and adaptable in integrating multi-source information. In addition to node attention modeling, DAKG can also effectively model the diverse relationships between nodes, relieving the defects of insufficient modeling to edge features ...
• Efficient Diffusion Transformer with Step-wise Dynamic Attention Mediators — To analyze quantitatively, we design a Jensen-Shannon divergence-based metric to measure the query-key interaction redundancy, i.e., comparing the attention distribution similarities among each query. We come up with two key findings: (1) Extensive query-key redundancy is evident in all of the self-attention layers, indicating many tokens ...
• Energon: Towards Efficient Acceleration of Transformers Using Dynamic ... — To overcome these problems, we propose Energon * * * In the famous film series Transformers, Energon is the preferred fuel of the Transformer race., an algorithm-architecture co-design solution that efficiently accelerates transformers using dynamic sparse attention.With the observation that attention results mainly depend on a few important query-key pairs, we propose a novel Mix-Precision ...
• A robust graph attention network with dynamic adjusted graph — Note that the original attention scores in GAT are computed based on the node feature and labels of neighbors (Zhang et al., 2019). The graph structure only decides the choice of neighbors but cannot help to adjust the attention scores. Adversarial attacks add negative edges or reduce the positive edges to affect the aggregation neighbors.
• User Evaluation of Affective Dynamic Difficulty Adjustment Based on ... — 2.1 Dynamic Difficulty Adjustment. The concept of challenge and difficulty within the domain of digital games has always been a heavily debated topic, where several theories have been formulated. In particular the theory of flow , has been widely applied as both a design paradigm and for Dynamic Difficulty Adjustment (DDA) systems . DDA systems ...
• Improving Autoregressive NLP Tasks via Modular Linearized Attention — The core architecture of Vaswani et al. original transformer has been studied exhaustively, so we will only review its attention-based elements. General transformer attention blocks receive a query Q in \(\mathbb {R}^{N_1 \times d_k}\), a key K in \(\mathbb {R}^{N_2 \times d_k}\), and a value V in \(\mathbb {R}^{N_2 \times d_v}\). Classically ...