Benchmarking Tests: Benchmarks to Test

There are many benchmarks to test. Decide on the minimum and expand from there. These are some idea below.

• Paper from https://github.com/Accenture/mcp-benchis located here https://arxiv.org/abs/2508.20453
  • Maybe we can use this but haven’t looked too much into it
  • See what the connections are since we already connect to the models
• Types of failures
  • Called the right tool but with wrong parameters (parameters that don’t exist in the method)
  • Didn’t call the right tool but provided correct parameters
  • Called the right tool but interpreted parameters wrong (parameters extracted from sentence incorrectly e.g. passed 10 runs to lower bounds)
  • Prompt didn’t provide all the necessary parameters
  • Prompt provided more than required parameters
  • Non-valid inputs e.g. only positives but gave it a negative
    • Fix in mcp server to capture these out of bounds
    • Update docstrings
    • Looking for paths or files and they got updated
  • Machine is down, resources aren't available
  • Time out errors
    • Model waiting for user input e.g. some string and then enter
    • Create simple simulations to run faster
    • Caught at the ci/cd time limit exceeded
    • Gitlab duo
    • Running simulation that takes longer than usual