mentat

An open-source AI coding assistant that integrates with your command line to assist with various coding tasks across multiple files.

Mentat

Large language models are really cool but there's still some questions on what they're actually useful for at least in terms of work people do every day. When it comes to trying to benchmark this, there are limited areas where you can actually have benchmarks that almost directly relate or reflect something people actually do during work. One of the better examples of this is problem sets that students work on whether they're for computer science or for math. But those still vary a lot and of course the difficulty based on what university you're at might vary a lot.

The area that I'm intimately familiar with and that I think LLMs show a lot of promise right now in are repetitive tasks that have to do with software engineering. So not exactly having a complete assistant that's as good as maybe an average intern, but when it comes to doing things like writing documentation or generating code for something like an Amazon AWS config, LLMs are pretty good at this. They can do these things in a relatively emergent way across local models and large models that are from closed companies like Anthropic. 

This form of benchmarking has become pretty popular and Anthropic actually talked a lot about this with their internal testing of CLU 3.5, which I thought was pretty cool. They haven't given us a lot of details. I'm working with some people at Anthropic to get a few more. But what I want to talk about today is a project called Menat, which is a state-of-the-art coding agent meant to work with GitHub that. 

Please share the AI Agent Store with friends.

Basically writes PRS based on issues and submits code. So for those of you who don't use GitHub or if you're not a software engineer, basically GitHub is a tool you use to track lots of changes within a software project. When you create issues, these can be new features or problems that need to be fixed. A pull request is basically when someone has written code relative to that issue and then decides to submit it to be incorporated into all the code that's tracked in what's called a repository. 

And what's cool is Mentat Bot is entirely intended to do this. When we think about training LLMs or testing them with benchmarks, the flow is seemingly identical. We're just looking at the inputs and the outputs in a few different places that happen to be more practical. So whereas normally you would just have an input prompt and be looking at the output prompt, in this case, you feed Mentat Bot a GitHub issue. It uses the GitHub API to see that you ask it to start, and then it writes the code and submits a PR. 

It can also do things like reviewing your PRS, reviewing your code, and giving comments as well. So the idea here is this is sort of like an intern light you can use that's just a bot that can help you speed up your everyday workflow as an engineer, especially for kind of house tasks that you usually leave to one of the lesser experienced engineers on your team. Since a lot of these things are lower risk, frankly, I think this is the pattern that we're going to see these kinds of models take going forward.

Forward I think the idea of having probably 10 to 12 small agents working with you maybe with some more important or larger ones you may be paying a bit more to do things like site reliability engineering to make sure everything looks good and make sure your site stays up. Um I see that being a really real possibility going forward and I don't think this is really going to put anyone out of work because if anything this enables anyone to have a software team without having to raise a ton of money to do so. 

So I think this is also a great precursor to mixture of agents which we've start to see really become pervasive especially with open models and I really want to get into menot. So welcome to AI flux let's get into it. Mot is actually based on chat GPT for Omni. I didn't initially expect this but granted the previous version was built on top of the gp4 API it makes sense why you would do this because gp4 Omni is a relatively known quantity and it's something that everyone can have access to. 

And there's nothing to say that you couldn't do this with mixl or llama 370b but I do think there's still certain areas where gb4 Omni really has some prowess. So Grant and bio bootloader are two developers I follow on Twitter and they both have some really great information about what maned bot is. So how are they actually measuring this and calling it state-of-the-art? So they're using something called uh software engineer bench or S bench and this is the result of a paper that was written in 2024 that was. 

Share the AI Agent Store with friends.

Basically posing the question how can we actually test the effectiveness of kind of coding or software engineering assistance. And there are two primary leaderboards. So there's the main leaderboard and then the leaderboard light. 

And specifically U leaderboard light is where menet bot is claiming to be state-of-the-art. And to be clear sweep bench light is a subset of sweep bench that's been curated to make evaluation less costly and more accessible. So the idea here is it's not entirely different from the main Benchmark. It just makes things a bit more straightforward so that you don't have to spend as much money on your API fees or on compute to actually pass and get a rating. 

And obviously mot is still in that process because they're comparing to numbers on this Benchmark. And they're basically proving up that their current version of mbot is 5% better than the previous state-of-the-art which was Alibaba lingma agent. Which I definitely read the first time as ligma agent. 

The other cool thing is this is available right now. So this is a 3 person teamates made this uh biob boot biob bootloader Jack and Pluto all worked on this. The initial intent was to beat you know Devon obviously Alibaba Factory and IBM research. 

And what's cool is with just a few more months of development since we saw an update from them in April they were able to do that well outside of the margin of error. And you can see here that when they compare mbot to Alibaba lingma they're clearly 5% above and I think that's pretty cool. 

Share the AI Agent Store with friends.

The architecture is actually pretty straightforward. Sometimes new look at these systems the architectures are wildly complex and these are pretty straightforward. And again I think it just has to do with looking at this as any other kind of benchmark and just picking out where the inputs and outputs are. And there are just a few more steps in between there than there would be with something like MLU or some of these larger benchmarks. 

So they call this a cognitive architecture and basically what this does is for the first step it's looking at the GitHub issue and gathering context and what it needs to do. Then it figures out a rough plan, kind of writes out what it planning on doing like IW as an engineer in an or that uses GitHub. Then thinks about how it will break this into smaller tasks which is basically called destructuring the plan into individual edits. 

In this case a lot of times this is just picking which files it thinks it actually has to change. And to me I think this is kind of a really understated portion of this model because we know Gemini 1.5 is pretty great and Gemini Pro are pretty great coding. To understand which files in kind of a Hy stack to pick out and change based on the input requires a lot of context. 

So the fact that it can do this and then understand when it doesn't have enough context and go back and maybe leave a comment like I need more information on what to do here, that's pretty cool. And this is actually a flow I've gone through with interns a lot or you know. 

Share the AI Agent Store with friends.

Software engineers that are to my junior and it's cool to see that agents are pretty much already capable of doing this which is wild even if someone who kind of hacks on top of this technology uh on a daily basis. So Grant says here that for sweet bench we told mtad bot to write tests in its plan then use those tests to decide whether or not to add more context make more edits uh completely start again with a note or to finish. 

And what's interesting is they claim that but we don't actually touch the benchmark test patch or test directives they just use their own and they were able to achieve this output. And again right now they're using GPT 4 Omni for everything and the average cost per benchmark was around $3 which includes more than just one iteration and obviously some failure cases. 

With one iteration they solve around 18% of the problems with an average cost of 72 in around 50 seconds. So one thing is I don't think this is actually cheaper at the moment than um having an intern because for instance if you're doing like 20 of these a day you can start to compare against the hourly rate. 

I think if you were using local inference which I have no clue how much that would affect the performance you know maybe this would change but um it's something that I'm hoping to do when I get a bit more time at the end of this month. And what's really cool is right now this is a live product so if someone at mbot wants to give me some credits I can definitely do a deep dive and compare kind of my local. 

Please share the AI Agent Store with friends.

Setup to this setup I've been using a lot of mixture of agents to try to look into this stuff. Um specifically with some uh monitoring tasks and some things that otherwise I would use an Sr for. Because for me the most valuable time win when it comes to time and just things that I have to think about are most the SRV related topics. 

And these usage TI are pretty interesting. Like for instance uh to have a intern full-time would be definitely more than $2,800 a month. And it's limited by um commits and code reviews and the number of private repos you can actually have connected. So I think this is really cool. 

I think if they find kind of a reasonable way to collect context from what they're doing that would also be cool. And I can't wait to see more agentic projects like this continue to push coding forward. I think coding is a really natural place to start expanding performance with these tools just because it such it's something these models are already quite good at. 

There's already a lot of freely available um training data and it's a clear problem that actually can be solved by this. So as opposed to a lot of crypto stuff where they were always searching for a problem like this is a problem that already can partially be solved. And now it's just an issue of making sure that these models are more capable can actually solve certain tasks without too much observation and just making it all go faster. 

And I'm going to talk about this in a different video but I think this totally Keys into a lot of the

Work this week that Meta released with what they called The Meta LLM compiler. Where they wrote a number of models that try to actually continue to improve U code optimization and compiler capabilities. Um, which is really cool in a number of very complex ways. 

And if you want to learn more about that, let me know in the comments. So yeah, this is an awesome tool. Uh, I might actually pick up a license for this. Uh, this weekend. But yeah, I'm curious if you guys use local LLMs or closed LLMs to code. 

I've mentioned using include 3.5 in the past. Um, but yeah, curious of your thoughts in the comments. As always, I hope you learned something in this video. And if you like our content, please share the AI Agent Store with friends. And we'll see you in the next one.

Note, that there may be slight differences in text and video. Watch video for the original content.

Mentat Video