Recently, a team of computer scientists and legal scholars from Stanford University, Cornell University, and West Virginia University jointly published an eye-catching study analyzing the performance of several open-source large language models in terms of text memory. This research focused on examining five different open-weight models to see if their capabilities could reproduce content from classic books. Among these five models, three are from Meta, while the other two are developed by Microsoft and EleutherAI, respectively.

The research team used the popular Books3 database as the training material for these large models,值得一提的是，many of these books are still under copyright protection. The researchers divided 36 books into multiple overlapping segments of 100 tokens each and used the first 50 tokens as prompts to calculate the probability that the next 50 tokens match the original text. If the probability of exact repetition exceeds 50%, the segment is marked as "memorized".

Facebook metaverse meta

To everyone's surprise, Meta's Llama3.170B model, released in 2024, recalled 42% of the content from the first book of Harry Potter. In contrast, Meta's earlier Llama165B model, released in 2023, only managed to recall 4.4% of the content. This significant improvement has drawn widespread attention in academic circles. Researchers also found that compared to obscure books, Llama3.170B showed stronger memory capabilities for popular books like The Hobbit and George Orwell's 1984, with memory levels far exceeding those of other models.

This research result not only demonstrates significant progress in the text memory capabilities of large language models but also raises expectations for future AI technology in processing and understanding textual content. It can be said that with technological development, artificial intelligence is increasingly approaching human cognitive levels.