Machine Translation
cs.CL · 7 papers
Attention Is All You Need — Explaining the Transformer
BLEU: 28.4Ashish Vaswani, Noam Shazeer +6
The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also…
Gemini 1.5: Unlocking Multimodal Long-Context
Synthetic Text Haystack: >99.7%Gemini Team, Petko Georgiev +1135
In this report, we introduce the Gemini 1.5 family of models, representing the next generation of highly compute-efficient multimodal models capable of recalling and reasoning…
RoFormer: Enhanced Transformer with Rotary Position Embedding
BLEU: 27.5Jianlin Su, Yu Lu +4
Position encoding recently has shown effective in the transformer architecture. It enables valuable supervision for dependency modeling between elements at different positions of…
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
pre-training speedup: 7xWilliam Fedus, Barret Zoph, Noam Shazeer
In deep learning, models typically reuse the same parameters for all inputs. Mixture of Experts (MoE) defies this and instead selects different parameters for each incoming…
Exploring the Limits of Transfer Learning with T5
GLUE: 90.3Colin Raffel, Noam Shazeer +7
Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language…
Sequence to Sequence Learning with Neural Networks
BLEU: 34.81Ilya Sutskever, Oriol Vinyals, Quoc V. Le
Deep Neural Networks (DNNs) are powerful models that have achieved excellent performance on difficult learning tasks. Although DNNs work well whenever large labeled training sets…
Word2Vec Demystified: Efficient Estimation of Word Representations in Vector Space
Semantic Accuracy: 55%Tomas Mikolov, Kai Chen +2
We propose two novel model architectures for computing continuous vector representations of words from very large data sets. The quality of these representations is measured in a…