Leaderboard
Leaderboard (lite)
| Name | Repos Resolved (/10) | Total Tests Passed (/3628) | Test Duration (s) | Date | Analysis | Github |
|---|---|---|---|---|---|---|
| Claude Sonnet 3.5 - Base | 0 | 376 | 16.83 | 09/25/2024 | Analysis | Github |
| Claude Sonnet 3.5 - Fill-in + Unit Test Feedback | 0 | 619 | 552.79 | 09/25/2024 | Analysis | Github |
| Claude Sonnet 3.5 - Fill-in | 0 | 381 | 22.47 | 09/25/2024 | Analysis | Github |
| Reference (Gold) | 10 | 3628 | 21.00 | NA | Analysis | Github |
Leaderboard (all)
| Name | Repos Resolved (/56) | Total Tests Passed (/140926) | Test Duration (s) | Date | Analysis | Github |
|---|---|---|---|---|---|---|
| Claude Sonnet 3.5 - Fill-in | 0 | 694 | 629.19 | 09/25/2024 | Analysis | Github |
| Reference (Gold) | 19 | 136766 | 5467.81 | NA | Analysis | Github |