Difference between Model and Server Configs?

Thanks for a great project!

I might be misunderstanding or missing some limitation that doesn't allow for certain fields to be specified in a standalone config file, but there seem to be many server params which aren't supported on a per model basis?

# Prerequisites

Please answer the following questions for yourself before submitting an issue.

- [X] I am running the latest code. Development is very rapid so there are no tagged versions as of now.
- [X] I carefully followed the [README.md](https://github.com/abetlen/llama-cpp-python/blob/main/README.md).
- [X] I [searched using keywords relevant to my issue](https://docs.github.com/en/issues/tracking-your-work-with-issues/filtering-and-searching-issues-and-pull-requests) to make sure that I am creating a new issue that is not already open (or closed).
- [X] I reviewed the [Discussions](https://github.com/abetlen/llama-cpp-python/discussions), and have a new bug or useful enhancement to share.

# Expected Behavior

To be able to set all of the constants found in the [Server Types](https://github.com/abetlen/llama-cpp-python/blob/c50d3300d2a09c98765be7f2c05b7e4fd0b4232e/llama_cpp/server/types.py) in [ConfigFileSettings](https://github.com/abetlen/llama-cpp-python/blob/c50d3300d2a09c98765be7f2c05b7e4fd0b4232e/llama_cpp/server/settings.py#L218C7-L218C25)

# Current Behavior

Only a subset of config params are available in the Model Configuration

# Environment and Context

Running in Docker with `CONFIG_FILE=/config/config.json python3 -m llama_cpp.server `, with config file mounted to `/config/config.json`

# Failure Information (for bugs)

This is an explicit type error

# Steps to Reproduce

Create model config JSON, and attempt to set any of the unsupported parameters:

```json
{
"models": [{
"model": "/models/gemma-1.1-2b-it-Q5_K_M.gguf",
"model_alias": "gemma-2b-it-q5",
"chat_format": "gemma",
"penalize_newline": false,
"repeat_penalty": 1
}]
}
```

# Failure Logs

As expected, the config file validation fails on startup:

```
models.2.penalize_newline
Extra inputs are not permitted [type=extra_forbidden, input_value=False, input_type=bool]
For further information visit https://errors.pydantic.dev/2.6/v/extra_forbidden
models.2.repeat_penalty
Extra inputs are not permitted [type=extra_forbidden, input_value=1, input_type=int]
For further information visit https://errors.pydantic.dev/2.6/v/extra_forbidden
```