Skip to content

Conversation

Luke-Sanderson
Copy link
Collaborator

@Luke-Sanderson Luke-Sanderson commented Oct 9, 2025

Proposed changes

Jira: MCP-40

Checklist

@Luke-Sanderson Luke-Sanderson marked this pull request as ready for review October 10, 2025 08:06
@Luke-Sanderson Luke-Sanderson requested a review from a team as a code owner October 10, 2025 08:06
@Luke-Sanderson
Copy link
Collaborator Author

Luke-Sanderson commented Oct 10, 2025

@nirinchev @kmruiz @himanshusinghs All PRs to the feature branch has been reviewed but we would like someone from DevTools to review before merging to main please :)

Copy link
Collaborator

@nirinchev nirinchev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've gone through the code, but not through the tests yet. There are a couple of suggested changes that will impact the test structure, so I'll hold off on reviewing the tests until those are applied.

Copy link
Collaborator

@nirinchev nirinchev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still have to go through the tests, but the implementation seems to be mostly there. I have a few nits, as well as some concerns, but those are not super blocking.

Copy link
Collaborator

@nirinchev nirinchev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should run the accuracy tests once we've fixed the mocks to check that they're indeed passing - I'd be very surprised if they do with the current wording of the prompts.

jeroenvervaeke and others added 4 commits October 21, 2025 10:29
Co-authored-by: Nikola Irinchev <irinchev@me.com>
Co-authored-by: Nikola Irinchev <irinchev@me.com>
Co-authored-by: Nikola Irinchev <irinchev@me.com>
Copy link
Contributor

📊 Accuracy Test Results

📈 Summary

Metric Value
Commit SHA 1807efbb6cd5b9032bc1fba7bacecb5b5fc4bd08
Run ID 6000849e-d18d-4a74-b644-4c17853145a0
Status done
Total Prompts Evaluated 94
Models Tested 1
Average Accuracy 88.3%
Responses with 0% Accuracy 10
Responses with 75% Accuracy 5
Responses with 100% Accuracy 81

📊 Baseline Comparison

Metric Value
Baseline Commit 8a5da23269267523b6196ed85a42f57713451c3f
Baseline Run ID 7889df8a-d68c-4900-82cc-0f0acc92873b
Baseline Run Status done
Responses Improved 2
Responses Regressed 4

📎 Download Full HTML Report - Look for the accuracy-test-summary artifact for detailed results.

Report generated on: 10/21/2025, 10:23:18 AM

"node": "^20.19.0 || ^22.12.0 || >= 23.0.0"
},
"optionalDependencies": {
"@mongodb-js/atlas-local": "^1.0.2",
Copy link
Collaborator

@gagik gagik Oct 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this meant to be an optional dependency?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes it is, the import might not succeed on all platforms.

@jeroenvervaeke
Copy link
Member

We should run the accuracy tests once we've fixed the mocks to check that they're indeed passing - I'd be very surprised if they do with the current wording of the prompts.

I ran the accuracy tests and all the atlas-local accuracy tests succeed with 100% success rate: https://github.com/mongodb-js/mongodb-mcp-server/actions/runs/18687565982

@jeroenvervaeke jeroenvervaeke enabled auto-merge (squash) October 21, 2025 15:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants