%20--%3e%3cdefs%3e%3cstyle%3e%20.st0%20{%20fill:%20%23061b40;%20}%20.st1%20{%20fill:%20%23306af1;%20}%20.st2%20{%20fill:%20%235ce5cf;%20}%20%3c/style%3e%3c/defs%3e%3cg%3e%3cpath%20class='st0'%20d='M55,10.5h9v3h-9v9h12V7.5h-12v3ZM64,19.5h-6v-3h6v3Z'/%3e%3cpolygon%20class='st0'%20points='69%2016.5%2078%2016.5%2078%2019.5%2069%2019.5%2069%2022.5%2081%2022.5%2081%2013.5%2072%2013.5%2072%2010.5%2081%2010.5%2081%207.5%2069%207.5%2069%2016.5'/%3e%3cpolygon%20class='st0'%20points='95%2010.5%2095%207.5%2083%207.5%2083%2022.5%2095%2022.5%2095%2019.5%2086%2019.5%2086%2016.5%2095%2016.5%2095%2013.5%2086%2013.5%2086%2010.5%2095%2010.5'/%3e%3cpath%20class='st0'%20d='M40,1.5v21h11.6l1.4-1.4v-7.6h0c0,0-1.4-1.5-1.4-1.5l1.4-1.4V2.9l-1.4-1.4h-11.6ZM50,19.5h-7v-6h7v6ZM50,10.5h-7v-6h7v6Z'/%3e%3c/g%3e%3cpath%20class='st1'%20d='M23.1,24L14.7,4.8l-4.9,11.2h3.8l-1.8,4H3.3L12.1,0H2C.9,0,0,.9,0,2v20c0,1.1.9,2,2,2h21.1Z'/%3e%3cpath%20class='st2'%20d='M34,0h-16.8l10.6,24h6.2c1.1,0,2-.9,2-2V2C36,.9,35.1,0,34,0ZM32.5,20h-4V4h4v16Z'/%3e%3c/svg%3e)
Recently, the OSWorld team officially released OSWorld-MCP, the first benchmarking tool for a comprehensive evaluation of computer proxy products. This benchmark aims to provide real-world product capability assessments for developers and users, enhancing the authenticity, balance, and comparability of evaluations.
The main features of OSWorld-MCP include its comprehensive evaluation of model context protocol (MCP) tool invocation capabilities, graphical user interface (GUI) operation skills, and decision-making performance. This benchmark includes 158 verified MCP tools, covering seven common applications, including LibreOffice Writer, Calc, Impress, VS Code, Google Chrome, VLC, and operating system utilities. Among them, 25 tools are specifically used for robustness testing to ensure the comprehensiveness and reliability of the evaluation.
Additionally, OSWorld-MCP has set up 250 tool applicability tasks, with 69% of the benchmark tasks benefiting from the use of MCP tools. The multi-turn invocation settings of these tools bring real decision-making challenges, making the test results more valuable. According to the data, the accuracy and efficiency of models using MCP tools have significantly improved. The o3 model from OpenAI increased its accuracy from 8.3% to 20.4% after 15 invocations. During the test, the Claude-4-Sonnet model observed the highest tool invocation rate of 36.3%, showing potential for future improvements.
The open-source nature of this project also provides developers with rich resources and explanations, promoting the sharing and collaboration of technology. For detailed information and resources about the project, users can visit its official website and GitHub page.
The release of OSWorld-MCP not only provides a powerful tool for evaluating computer proxy products but also lays the foundation for the development of related technologies in the future.
github:https://github.com/X-PLUG/OSWorld-MCP
Key Points:
🌟 ** The first benchmarking tool for a comprehensive evaluation of computer proxy products, OSWorld-MCP, is officially released.**
🛠️ ** Includes 158 verified MCP tools, supporting multiple common applications.**
📈 ** Through multi-turn invocation settings, it significantly improves the accuracy and efficiency of models.**