I'm writing a NeurIPS paper about a new model architecture for processing and generating long texts. Here are some facts about the paper: * The main trick is to replace some of the attention heads with an exponential moving average, where the decay rate is learned for each head. We call this architecture ExeMA. * On language modeling, the perplexity difference between our model and a vanilla transformer is negligible, but that's because next-token prediction is almost always a local task, so perplexity won't be sensitive enough to detect any improvements in long-range understanding. * However, on the SCROLLS benchmark, our model improves by 10% over the baseline. * We also have a new metric for measuring coherence in generated text (CoGnaTe), where our model generates text that is 43% more coherent than the baseline. Help me write the paper's introduction.

Introduction : \n \n Long - text generation is a challenging task in natural language processing ( N LP ) that requires models to understand and generate co herent and meaningful sequences of words . In this paper , we present a new model architecture for processing and generating long texts , called Ex e MA . Our main contribution is the replacement of some attention heads with an ex ponential moving average , where the decay rate is learned for each head . This modification allows our model to better capture long - range dependencies and improve its performance on the SC RO LL S bench mark . \n \n We also introduce a new metric for measuring co her ence in generated text , called Co G na Te , which our model out per forms the baseline by 4 3 %. Our experiments show that our model generates text that is more co herent and better capt ures the context of the input text . \n \n Our model achie ves these improvements without significantly increasing the comput ational cost or the number of parameters compared to a van illa transform er . This makes it a promising approach for large - scale text generation tasks . \n \n Over all , our work demonstr ates the effectiveness of Ex e MA in improving long - text generation and highlights the importance of considering long - range dependencies in N LP models . </s>