BERT-large End-to-End Training Speedup:15%· WikipediaGPT-2 Training Speedup:3.0x· OpenWebTextPath-X Accuracy (seq length 16k):61.4%· Long Range ArenaPath-256 Accuracy (seq length 64k):63.1%· Long Range ArenaMemory Footprint Reduction:up to 20x· LRA (seq length 16k)
BERT-large End-to-End Training Speedup:15%· WikipediaGPT-2 Training Speedup:3.0x· OpenWebTextPath-X Accuracy (seq length 16k):61.4%· Long Range ArenaPath-256 Accuracy (seq length 64k):63.1%· Long Range ArenaMemory Footprint Reduction:up to 20x· LRA (seq length 16k)