OpenAI Compatible server (Function calling), does not support (jxnl/Instructor), because of Unrecognized schema: {'$ref': '#/$defs/Edge'}

# Prerequisites

Please answer the following questions for yourself before submitting an issue.

- [OK ] I am running the latest code. Development is very rapid so there are no tagged versions as of now.
- [OK ] I carefully followed the [README.md](https://github.com/abetlen/llama-cpp-python/blob/main/README.md).
- [ OK] I [searched using keywords relevant to my issue](https://docs.github.com/en/issues/tracking-your-work-with-issues/filtering-and-searching-issues-and-pull-requests) to make sure that I am creating a new issue that is not already open (or closed).
- [OK] I reviewed the [Discussions](https://github.com/abetlen/llama-cpp-python/discussions), and have a new bug or useful enhancement to share.

# Expected Behavior

I am currently using `jxnl/Instructor`, a package that patches OpenAI API to add extra features related to function calling and generating structured data.

While using it on the official OpenAI function calling API. it works very well.

When I use the included OpenAI compatible server with Instructor. it works for simple structured data, but as soon as I try to use anything more complex. it throws an error and says and unrecognized schema.

# Current Behavior
Currently it works with simple data when I use instructor.

but as soon as I try to use something more complex like generating Knowledge graphs.
it does not work, and throws an error then uses the default grammar.
sometimes it could spit out a correct output (with the default grammar), but mostly does not work.

# Environment and Context

Windows, python 3.10 venv.

pip install llama-cpp-python[server]
python3 -m llama_cpp.server --model

This code works, Code from https://github.com/jxnl/instructor?tab=readme-ov-file#usage
```python
import instructor
from openai import OpenAI
from pydantic import BaseModel

# Enables `response_model`
client = instructor.patch(OpenAI(api_key="sk-****",base_url="https://my-endpoint/v1"))

class UserDetail(BaseModel):
name: str
age: int

user = client.chat.completions.create(
model="gpt-3.5-turbo",
response_model=UserDetail,
messages=[
{"role": "user", "content": "Extract Jason is 25 years old"},
]
)

assert isinstance(user, UserDetail)
assert user.name == "Jason"
assert user.age == 25
```

---

This does not work, from https://jxnl.github.io/instructor/examples/knowledge_graph/#defining-the-structures .
```python
#Defining the Structures[¶](https://jxnl.github.io/instructor/examples/knowledge_graph/#defining-the-structures)
#Let's model a knowledge graph with Node and Edge objects. Node objects represent key concepts or entities, while Edge objects #indicate the relationships between them.

from pydantic import BaseModel, Field
from typing import List

class Node(BaseModel):
id: int
label: str
color: str

class Edge(BaseModel):
source: int
target: int
label: str
color: str = "black"

class KnowledgeGraph(BaseModel):
nodes: List[Node] = Field(..., default_factory=list)
edges: List[Edge] = Field(..., default_factory=list)
##---------------------------------
#Generating Knowledge Graphs[¶](https://jxnl.github.io/instructor/examples/knowledge_graph/#generating-knowledge-graphs)
#The generate_graph function leverages OpenAI's API to generate a knowledge graph based on the input query.
##---------------------------------
from openai import OpenAI
import instructor

# Adds response_model to ChatCompletion
# Allows the return of Pydantic model rather than raw JSON
client = instructor.patch(OpenAI(api_key="sk-****",base_url="https://my-endpoint/v1"))

def generate_graph(input) -> KnowledgeGraph:
return client.chat.completions.create(
model="gpt-3.5-turbo",
messages=[
{
"role": "user",
"content": f"Help me understand the following by describing it as a detailed knowledge graph: {input}",
}
],
response_model=KnowledgeGraph,
) # type: ignore

#--------------------
#Visualizing the Graph[¶](https://jxnl.github.io/instructor/examples/knowledge_graph/#visualizing-the-graph)
#The visualize_knowledge_graph function uses the Graphviz library to render the generated knowledge graph.
#-------------------
from graphviz import Digraph

def visualize_knowledge_graph(kg: KnowledgeGraph):
dot = Digraph(comment="Knowledge Graph")

# Add nodes
for node in kg.nodes:
dot.node(str(node.id), node.label, color=node.color)

# Add edges
for edge in kg.edges:
dot.edge(str(edge.source), str(edge.target), label=edge.label, color=edge.color)

# Render the graph
dot.render("knowledge_graph.gv", view=True)

#-----------
#Putting It All Together[¶](https://jxnl.github.io/instructor/examples/knowledge_graph/#putting-it-all-together)
#Execute the code to generate and visualize a knowledge graph for understanding quantum mechanics.
#-----------

graph: KnowledgeGraph = generate_graph("Teach me about quantum mechanics")
visualize_knowledge_graph(graph)
```
on this one I get the error `Unrecognized schema: {'$ref': '#/$defs/Edge'}`

**Note*** the `response_model` paramater of the function `client.chat.completions.create`, is not found in the official OpenAI API, but only on the OpenAI object that is patched by (jxnl/instructor)
Honestly, I'm not sure if jxnl/instructor, are using JSONs correctly, but their approach works on current OpenAI HTTP API.