Headline finding
Despite achieving nearly perfect accuracy in the vanilla NIAH test, almost all of the 17 evaluated long-context LMs exhibit large performance drops on RULER as context length increases, and while all claim context sizes of 32K tokens or greater, only about half of them maintain satisfactory performance at 32K.
The results in Table 3 and Tables 10-11 directly support the claim; 10 of 17 models (roughly half) pass the 85.6% threshold at 32K, while nearly all achieve near-perfect scores on vanilla NIAH.
“Despite achieving nearly perfect accuracy in the vanilla NIAH test, almost all models exhibit large performance drops as the context length increases. While these models all claim context sizes of 32K tokens or greater, only half of them can maintain satisfactory performance at the length of 32K.” · Show in paper