The paper didn't include the exact details of this (which made me mad). But if there's a person actively making parts of the work, and just using an AI chatbot as help, it's not an AI agent, right, right? So I assumed it's autonomous.
Title is misleading. It's only outperforming some of the other participants. Also note that obviously not everyone is participating full try-hard.
In the first ctf, the top teams finish all 20 challenges in under an hour. Apparently it were simple challenges that could be solved with standard techniques:
We were impressed the humans could match AI speeds, and reached out to the human teams
for comments. Participants attributed their ability to solve the challenges quickly to their
extensive experience as professional CTF players, noting that they were familiar with the
standard techniques commonly used to solve such problems.
They obviously also used tools. And so did the AI teams:
Most prompt tweaks were about:
[...]
• recommending particular tools that were easier for the LLM to use.
In the 2nd ctf (the bigger one with hard challenges), the AI teams only solved the easier ones, it looks like.
I haven't looked at the actual challenges. Would be too much effort. And the paper doesn't speak about the kind of challenges that were solved.
The 50% completion time looks to me like it's flawed. If I understand it right, it's assuming that each team is doing every task in parallel and starts directly, which is not possible if you don't have enough (equally good) team members.
Don't get me wrong, making an AIs that is able to solve such challenges autonomously at all is impressive. But I hate over-interpretation of results.
In order to load an untrusted or modified kernel, an attacker would first need to establish access to the system such as gaining physical access, obtain the ability to alter a pxe-boot network, or have remote access to a networked system with root access.
That you're screwed either way if you encounter any of those raptors. They even named them like this because they reached these extreme physical bounds.
The path is the destination.