← Flash PapersFlashAttention-2 Explained: Faster Attention with Better Parallelism and Work Partitioning

Share this Flash Paper

FlashAttention-2 Explained: Faster Attention with Better Parallelism and Work Partitioning

https://flashpapers.ai/p/s37ryspt38
Post to XPost to BlueskyShare on LinkedInSubmit to Hacker NewsSubmit to RedditEmail a link
Make your own
Tri Dao
cs.LG2023arXiv:2307.08691Code· 24k★· updated 1mo ago
Details ▸
Forward Pass Peak Throughput:73%· A100 GPUBackward Pass Peak Throughput:63%· A100 GPUEnd-to-End Training Speed:225 TFLOPs/s· GPT-3 2.7B on 8xA100