Skip to content

fix: README benchmark scores in News section don't match Performance table (LoCoMo/LongMemEval) #2090

Closed

Description

@shinetata

Pre-submission checklist | 提交前检查

  • I have searched existing issues and this hasn't been mentioned before | 我已搜索现有问题,确认此问题尚未被提及
  • I have read the project documentation and confirmed this issue doesn't already exist | 我已阅读项目文档并确认此问题尚未存在
  • This issue is specific to MemOS and not a general software issue | 该问题是针对 MemOS 的,而不是一般软件问题

Bug Description | 问题描述

The News section and the 📊 Performance section of the README report different scores for the same two benchmarks (LoCoMo and LongMemEval), with no explanation for the discrepancy. Readers see two different "current" numbers for the same benchmark on the same page. This affects both README.md and README_ZH.md identically.

News section (README.md lines 49-50):

2026-07-02 · 🏆 MemOS Advances Agent and User Memory Benchmarks
With MemOS, OpenClaw improves average task completion from 36.63% to 50.87% across five agent tasks. MemOS also achieves 92.34 on LoCoMo and 93.40 on LongMemEval, and leads in OmniMemEval...

Performance table (README.md lines 66-77), a few lines below:

Benchmark Score
LoCoMo 88.83
LongMemEval 89.20

Same benchmarks, same OmniMemEval framework referenced, but the numbers don't match (92.34 vs 88.83, 93.40 vs 89.20).

Root cause (from git history): the News entry was added in #2019 with 92.34/93.40. Later, #2078 rewrote the README and introduced the "Performance" table with 88.83/89.20 ("10 benchmark scores from the 2026-07-02 evaluation report" per the commit message) but didn't update the pre-existing News entry to match, leaving both numbers live on main at the same time.

How to Reproduce | 如何重现

  1. Open README.md (or README_ZH.md) on main.
  2. Compare the LoCoMo/LongMemEval numbers quoted in the News section vs. the 📊 Performance table.
  3. Note the mismatch: 92.34 vs 88.83 (LoCoMo), 93.40 vs 89.20 (LongMemEval).

Environment | 环境信息

  • N/A — documentation-only issue, not version/platform specific.

Additional Context | 其他信息

Suggested fix: align the News entry with the Performance table (88.83 / 89.20), since the table is the more complete, explicitly-sourced number (10 benchmarks, linked to https://github.com/MemTensor/OmniMemEval). Will open a PR updating both README.md and README_ZH.md.

Willingness to Implement | 实现意愿

  • I'm willing to implement this myself | 我愿意自己解决

Activity

  1. self-assigned this
    on Jul 9, 2026
  2. Memtensor-AI commented on Jul 9, 2026

    @Memtensor-AI
    Collaborator

    🤖 AutoDev has picked up this issue and started working on it.

    Task ID: a7f814f57a1585cf
    Working branch: bugfix/autodev-2090-20260709093247057
    Target branch: latest dev* branch
    Workflow: opsp (analysis → coding → testing → PR)

    I will post the PR link here once done. If I need more information, I will ask in the comments.

  3. added a commit that references this issue on Jul 9, 2026
    64cddbf
  4. Memtensor-AI commented on Jul 9, 2026

    @Memtensor-AI
    Collaborator

    ✅ AutoDev task a7f814f57a1585cf completed.

    Summary: Fixed the README score mismatch (LoCoMo 92.34→88.83, LongMemEval 93.40→89.20) in both README.md and README_ZH.md, aligning the News section with the Performance table. Committed on the working branch (64cddbfb) and pushed to origin. Task spec archived to the sibling memos-autodev-specs repo (main @ d8af691).

    Now submitting the completion envelope so the scheduler can open the PR.

    Sources:

    Base branch: main
    Branch: bugfix/autodev-2090-20260709093247057
    PR: #2092
    Assigned to: @CarltonXiang
    Reviewers: @MatthewZhuang, @CarltonXiang, @syzsunshine219, @World-controller

  5. added a commit that references this issue on Jul 9, 2026
    13fbd43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions