Dynamic Parameters for Mediatek Delegate

🚀 The feature, motivation and pitch

While working with the MediaTek delegate for android app I am trying to identify all the dynamic parameters that can possibly be set with the current implementation of the model generation (and what limitations are there for now). Currently are model parameters are hardcoded for the android llama app which can be made dynamic by passing the .h file as a converted .txt file and mapped to variables at the backend.

• Objective:

  • Identify all the dynamic parameters that can be set within the current model generation implementation.
  • Understand the limitations regarding configurable variables.

• Potential Configurable Parameters:

  • Context Length:
    • Can this be adjusted dynamically?
    • Expected impact on model performance and response time.
  • Prompt Token Batch Size:
    • Is this configurable?
    • Expected effect on memory usage and response latency.
  • Cache Size:
    • Can we change the cache size dynamically?
    • Expected effects on memory management and model inference speed.
  • Other Variables:
    • Are there other variables (e.g.,ROT_EMB_BASE , temperature, top_p) that can be configured dynamically?
    • What are the expected outcomes of changing these parameters?

• Expected Outcomes:

  • Clarification of which parameters can be safely modified without impacting the model's functionality.
  • Improved flexibility for adjusting model behavior based on real-time requirements or user input.

This should help make the question clearer and provide a roadmap for addressing these dynamic configurations.

Alternatives

No response

Additional context

No response

RFC (Optional)

No response

cc @cccclai @neuropilot-captain @cbilgin