Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth
Paper • 2103.03404 • Published • 2
Yihe Dong
yihedong
AI & ML interests
None yet
Recent Activity
authored a paper 17 days ago
Attention is Not All You Need: Pure Attention Loses Rank Doubly
Exponentially with Depth authored a paper 17 days ago
LANISTR: Multimodal Learning from Structured and Unstructured Data authored a paper 17 days ago
Learned Feature Importance Scores for Automated Feature EngineeringOrganizations
None yet
authored 5
papers 17 days ago
LANISTR: Multimodal Learning from Structured and Unstructured Data
Paper • 2305.16556 • Published • 2
Learned Feature Importance Scores for Automated Feature Engineering
Paper • 2406.04153 • Published • 1
Metadata Conditioning Accelerates Language Model Pre-training
Paper • 2501.01956 • Published • 1
Muon$^p$: Muon with Fractional Spectral Powers
Paper • 2606.13867 • Published • 1
upvoted 3
papers 18 days ago
Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth
Paper • 2103.03404 • Published • 2
Learned Feature Importance Scores for Automated Feature Engineering
Paper • 2406.04153 • Published • 1
Muon^p: Muon with Fractional Spectral Powers
Paper • 2606.13867 • Published • 1