On Generalisation Error Bounds for Transformers

תקציר מקורי באנגליתarXiv:2410.11500v2 Announce Type: replace-cross Abstract: In this paper, we establish a collection of covering number bounds for linear function classes under various norm constraints on the inputs and matrices. We then combine these results with existing covering number bounds to derive improved estimates and, based on these estimates, develop generalization error bounds for single-layer Transformers. The resulting generalization bounds improve upon several existing results in the literature and, in particular, are independent of the input sequence length. Moreover, our generalization error bound decays at the rate $O(1/\sqrt{n})$, where $n$ denotes the sample size, thereby improving upon existing bounds that scale as $O((\log n)/\sqrt{n})$. Furthermore, our covering number analysis expli