How is TransformerFAM different from Landmark Attention? #15
Description
Not exactly an issue (please feel free to close it), but wanted to get the authors opinion on the new TransformerFAM paper and how it is different from Landmark Attention.
My original understanding is that by introducing Landmark tokens at the end of blocks, we could potentially scale of infinite length sequences.
But the above paper just concludes - "However, in those papers, the information was not propagated infinitely", in reference to Landmark Attention.
Can the authors / someone please clarify what modifications can Landmark Attention have to achieve what T-FAM paper proposes?
Thanks.