attention-mechanisms.
Listed 17Updated Aug 28, 2026
- Rank 1. PaLM-rlhf-pytorchImplementation of RLHF (Reinforcement Learning with Human Feedback) on top of the PaLM architecture. Basically ChatGPT but with PaLMmit · linuxGitHub stars: 7.9k
- Rank 2. audiolm-pytorchImplementation of AudioLM, a SOTA Language Modeling Approach to Audio Generation out of Google Research, in Pytorchmit · linuxGitHub stars: 2.6k
- Rank 3. alphafold3-pytorchImplementation of Alphafold 3 from Google Deepmind in Pytorchmit · linuxGitHub stars: 1.7k
- Rank 4. BS-RoFormerImplementation of Band Split Roformer, SOTA Attention network for music source separation out of ByteDance AI Labsmit · linuxGitHub stars: 935
- Rank 5. magvit2-pytorchImplementation of MagViT2 Tokenizer in Pytorchmit · linuxGitHub stars: 668
- Rank 6. mmditImplementation of a single layer of the MMDiT, proposed in Stable Diffusion 3, in Pytorchmit · linuxGitHub stars: 559
- Rank 7. iTransformerUnofficial implementation of iTransformer - SOTA Time Series Forecasting using Attention networks, out of Tsinghua / Ant groupmit · linuxGitHub stars: 541
- Rank 8. local-attentionAn implementation of local windowed attention for language modelingmit · linuxGitHub stars: 505
- Rank 9. recurrent-memory-transformer-pytorchImplementation of Recurrent Memory Transformer, Neurips 2022 paper, in Pytorchmit · linuxGitHub stars: 425
- Rank 10. q-transformerImplementation of Q-Transformer, Scalable Offline Reinforcement Learning via Autoregressive Q-Functions, out of Google Deepmindmit · linuxGitHub stars: 406
- Rank 11. clinical-calculator-tooluseExplorations into training LLMs to use clinical calculators from patient history, using open sourced models. Will start with Wells' Criteriamit · linuxGitHub stars: 316
- Rank 12. CoLT5-attentionImplementation of the conditionally routed attention in the CoLT5 architecture, in Pytorchmit · linuxGitHub stars: 231
- Rank 13. simple-hierarchical-transformerExperiments around a simple idea for inducing multiple hierarchical predictive model within a GPTmit · linuxGitHub stars: 228
- Rank 14. MambaTransformerIntegrating Mamba/SSMs with Transformer for Enhanced Long Context and High-Quality Sequence Modelingmit · linuxGitHub stars: 228
- Rank 15. flash-cosine-sim-attentionImplementation of fused cosine similarity attention in the same style as Flash Attentionmit · linuxGitHub stars: 221
- Rank 16. recurrent-interface-network-pytorchImplementation of Recurrent Interface Network (RIN), for highly efficient generation of images and video without cascading networks, in Pytorchmit · linuxGitHub stars: 210
- Rank 17. coconut-pytorchImplementation of 🥥 Coconut, Chain of Continuous Thought, in Pytorchmit · linuxGitHub stars: 184
