Effects of Parameter Norm Growth During Transformer Training: Inductive Bias from Gradient Descent

Keyphrases

• Transformer 100%
• Inductive Bias 100%
• Gradient Descent 100%
• Parameter Influence 100%
• Neural Network 33%
• Automata 33%
• Formal Languages 33%
• Discretized 33%
• New Characterization 33%
• 2-norm 33%
• Activation Function 33%
• Discrete Structures 33%
• Self-attention Layer 33%
• Transformer Parameters 33%
• Training Routines 33%
• Transformer-based Language Models 33%
• Large Power Transformer 33%
• Emergent Representation 33%
• Saturated Network 33%
• NLP. 33%
• Global Averaging 33%

Computer Science

• Gradient Descent 100%
• Neural Network 33%
• Formal Language 33%
• Language Modeling 33%
• Automaton 33%
• Family Network 33%
• Activation Function 33%

Mathematics

• Approximates 100%
• Neural Network 100%