AI agents wrong ~70% of time: Carnegie Mellon study

Well the article says that the AI agents were able to complete 30% of the tasks given to it like searching the web, communicating with co workers, etc. I think this is interesting

CMU researchers have developed a benchmark to evaluate how AI agents perform when given common knowledge work tasks like browsing the web, writing code, running applications, and communicating with coworkers

"We find in experiments that the best-performing model, Gemini 2.5 Pro, was able to autonomously perform 30.3 percent of the provided tests to completion, and achieve a score of 39.3 percent on our metric that provides extra credit for partially completed tasks"

Personally i belive this is impressive.

This is fun too:

...all of the models evaluated "demonstrate near-zero confidentiality awareness."

Any agent that is accessible from outside the company (e.g. a customer support chatbot) is going to have to deal with malicious actors. If it has access to sensitive information, and no confidentiality awareness...seems like a problem.

My only hope is that AI like early social media and web services is supported by mountains of vc cash offering services at a loss in order to build users and familiarity, and while it'll continue to exist after it has to shift to a profitable business model, it'll essentially be relagated to corners of the economy where it makes sense and they'll stop trying to hamstring it into everything.