99designs / 99designs/gqlgen

Memory consumtion to hight when the a string literal it is a big string

Aperta
#1,672 2 commenti 1 reazione 0 assegnatari Vedi su GitHub
Lingua principale
Go
Stelle
10.8k
Fork
1.3k
Merge medio
2g 36m
PR unite (30g)
26

Descrizione

### What happened?

I'm trying to create a new graphql server, to upload some metrics with a mutation query. To avoid a verbose query and a very big query I pass a JSON payload as a string parameter of the query.

The mutation query looks like the following one:

```
mutation {
addNodeMetrics(input: {node_id: "....", payload_metric_one: "{}"}) {
node_id
}
}
```

Where the `payload_metric_one` contains a very json pyload.

The problem is that the server will continue to increase the memory like in the following image

![Selection-051](https://user-images.githubusercontent.com/17150045/138437289-9c8b6184-fb40-4627-b48d-d6cc993f5529.png)

[Here there is a pprof heap analysis ](https://drive.google.com/file/d/1EtCtZQ4jfvyCMf6ZziIA3qBc6deBXkP5/view?usp=sharing)

Some information here

```
----------------------------------------------------------+-------------
flat flat% sum% cum cum% calls calls% + context
----------------------------------------------------------+-------------
222.87MB 100% | encoding/json.(*decodeState).value /usr/local/go/src/encoding/json/decode.go:387
222.87MB 50.09% 50.09% 222.87MB 50.09% | encoding/json.(*decodeState).literalStore /usr/local/go/src/encoding/json/decode.go:966
----------------------------------------------------------+-------------
199.41MB 100% | github.com/vektah/gqlparser/v2/lexer.(*Lexer).readString /go/lnmetrics/vendor/github.com/vektah/gqlparser/v2/lexer/lexer.go:343 (inline)
199.41MB 44.82% 94.90% 199.41MB 44.82% | bytes.(*Buffer).String /usr/local/go/src/bytes/buffer.go:65
----------------------------------------------------------+-------------
9.50MB 100% | runtime.newproc1 /usr/local/go/src/runtime/proc.go:4305
9.50MB 2.14% 97.04% 9.50MB 2.14% | runtime.malg /usr/local/go/src/runtime/proc.go:4220
----------------------------------------------------------+-------------
6.67MB 100% | github.com/syndtr/goleveldb/leveldb/table.(*Reader).readRawBlock /go/lnmetrics/vendor/github.com/syndtr/goleveldb/leveldb/table/reader.go:587
6.67MB 1.50% 98.54% 6.67MB 1.50% | github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).Get /go/lnmetrics/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:150
----------------------------------------------------------+-------------
4MB 100% | github.com/syndtr/goleveldb/leveldb.(*DB).mpoolGet /go/lnmetrics/vendor/github.com/syndtr/goleveldb/leveldb/db_state.go:90
4MB 0.9% 99.44% 4MB 0.9% | github.com/syndtr/goleveldb/leveldb/memdb.New /go/lnmetrics/vendor/github.com/syndtr/goleveldb/leveldb/memdb/memdb.go:474
----------------------------------------------------------+-------------
0.50MB 100% | runtime.newm /usr/local/go/src/runtime/proc.go:2201
0.50MB 0.11% 99.55% 0.50MB 0.11% | runtime.allocm /usr/local/go/src/runtime/proc.go:1875
----------------------------------------------------------+-------------
0.50MB 100% | compress/flate.generateFixedLiteralEncoding /usr/local/go/src/compress/flate/huffman_code.go:65 (inline)
0.50MB 0.11% 99.66% 0.50MB 0.11% | compress/flate.newHuffmanEncoder /usr/local/go/src/compress/flate/huffman_code.go:60
----------------------------------------------------------+-------------
0.50MB 100% | text/template/parse.(*Tree).pipeline /usr/local/go/src/text/template/parse/parse.go:412 (inline)
0.50MB 0.11% 99.78% 0.50MB 0.11% | text/template/parse.(*Tree).newPipeline /usr/local/go/src/text/template/parse/node.go:197
----------------------------------------------------------+-------------
0.50MB 100% | github.com/vektah/gqlparser/v2/parser.(*parser).parseArgument /go/lnmetrics/vendor/github.com/vektah/gqlparser/v2/parser/query.go:179
0.50MB 0.11% 99.89% 0.50MB 0.11% | github.com/vektah/gqlparser/v2/parser.(*parser).parseValueLiteral /go/lnmetrics/vendor/github.com/vektah/gqlparser/v2/parser/query.go:271
----------------------------------------------------------+-------------
0.50MB 100% | text/template/parse.(*Tree).textOrAction /usr/local/go/src/text/template/parse/parse.go:363 (inline)
0.50MB 0.11% 100% 0.50MB 0.11% | text/template/parse.(*Tree).newText /usr/local/go/src/text/template/parse/node.go:134
----------------------------------------------------------+-------------
0.50MB 100% | compress/flate.init /usr/local/go/src/compress/flate/huffman_code.go:106
0 0% 100% 0.50MB 0.11% | compress/flate.generateFixedLiteralEncoding /usr/local/go/src/compress/flate/huffman_code.go:65
0.50MB 100% | compress/flate.newHuffmanEncoder /usr/local/go/src/compress/flate/huffman_code.go:60 (inline)
----------------------------------------------------------+-------------
0.50MB 100% | runtime.doInit /usr/local/go/src/runtime/proc.go:6498
0 0% 100% 0.50MB 0.11% | compress/flate.init /usr/local/go/src/compress/flate/huffman_code.go:106
0.50MB 100% | compress/flate.generateFixedLiteralEncoding /usr/local/go/src/compress/flate/huffman_code.go:65
----------------------------------------------------------+-------------
222.87MB 100% | github.com/99designs/gqlgen/graphql/handler/transport.jsonDecode /go/lnmetrics/vendor/github.com/99designs/gqlgen/graphql/handler/transport/http_get.go:77
0 0% 100% 222.87MB 50.09% | encoding/json.(*Decoder).Decode /usr/local/go/src/encoding/json/stream.go:73
222.87MB 100% | encoding/json.(*decodeState).unmarshal /usr/local/go/src/encoding/json/decode.go:180
----------------------------------------------------------+-------------
222.87MB 100% | encoding/json.(*decodeState).value /usr/local/go/src/encoding/json/decode.go:373
0 0% 100% 222.87MB 50.09% | encoding/json.(*decodeState).object /usr/local/go/src/encoding/json/decode.go:774
222.87MB 100% | encoding/json.(*decodeState).value /usr/local/go/src/encoding/json/decode.go:387
----------------------------------------------------------+-------------
222.87MB 100% | encoding/json.(*Decoder).Decode /usr/local/go/src/encoding/json/stream.go:73
0 0% 100% 222.87MB 50.09% | encoding/json.(*decodeState).unmarshal /usr/local/go/src/encoding/json/decode.go:180
222.87MB 100% | encoding/json.(*decodeState).value /usr/local/go/src/encoding/json/decode.go:373
----------------------------------------------------------+-------------
222.87MB 100% | encoding/json.(*decodeState).unmarshal /usr/local/go/src/encoding/json/decode.go:180
0 0% 100% 222.87MB 50.09% | encoding/json.(*decodeState).value /usr/local/go/src/encoding/json/decode.go:373
222.87MB 100% | encoding/json.(*decodeState).object /usr/local/go/src/encoding/json/decode.go:774
----------------------------------------------------------+-------------
222.87MB 100% | encoding/json.(*decodeState).object /usr/local/go/src/encoding/json/decode.go:774
0 0% 100% 222.87MB 50.09% | encoding/json.(*decodeState).value /usr/local/go/src/encoding/json/decode.go:387
222.87MB 100% | encoding/json.(*decodeState).literalStore /usr/local/go/src/encoding/json/decode.go:966
----------------------------------------------------------+-------------
6.67MB 100% | github.com/OpenLNMetrics/lnmetrics.server/graph/generated.(*executionContext)._Query /go/lnmetrics/graph/generated/generated.go:3920
0 0% 100% 6.67MB 1.50% | github.com/99designs/gqlgen/graphql.(*FieldSet).Dispatch /go/lnmetrics/vendor/github.com/99designs/gqlgen/graphql/fieldset.go:34
6.67MB 100% | github.com/OpenLNMetrics/lnmetrics.server/graph/generated.(*executionContext)._Query.func1 /go/lnmetrics/graph/generated/generated.go:3892
----------------------------------------------------------+-------------
199.91MB 100% | github.com/99designs/gqlgen/graphql/handler/transport.POST.Do /go/lnmetrics/vendor/github.com/99designs/gqlgen/graphql/handler/transport/http_post.go:44
0 0% 100% 199.91MB 44.93% | github.com/99designs/gqlgen/graphql/executor.(*Executor).CreateOperationContext /go/lnmetrics/vendor/github.com/99designs/gqlgen/graphql/executor/executor.go:62
199.91MB 100% | github.com/99designs/gqlgen/graphql/executor.(*Executor).parseQuery /go/lnmetrics/vendor/github.com/99designs/gqlgen/graphql/executor/executor.go:172
----------------------------------------------------------+-------------
10.67MB 100% | github.com/99designs/gqlgen/graphql/handler/transport.POST.Do /go/lnmetrics/vendor/github.com/99designs/gqlgen/graphql/handler/transport/http_post.go:53
0 0% 100% 10.67MB 2.40% | github.com/99designs/gqlgen/graphql/executor.(*Executor).DispatchOperation.func1.1 /go/lnmetrics/vendor/github.com/99designs/gqlgen/graphql/executor/executor.go:104
10.67MB 100% | github.com/99designs/gqlgen/graphql/executor.processExtensions.func2 /go/lnmetrics/vendor/github.com/99designs/gqlgen/graphql/executor/extensions.go:59
----------------------------------------------------------+-------------
10.67MB 100% | github.com/99designs/gqlgen/graphql/executor.processExtensions.func2 /go/lnmetrics/vendor/github.com/99designs/gqlgen/graphql/executor/extensions.go:59
0 0% 100% 10.67MB 2.40% | github.com/99designs/gqlgen/graphql/executor.(*Executor).DispatchOperation.func1.1.1 /go/lnmetrics/vendor/github.com/99designs/gqlgen/graphql/executor/executor.go:105
6.67MB 62.51% | github.com/OpenLNMetrics/lnmetrics.server/graph/generated.(*executableSchema).Exec.func1 /go/lnmetrics/graph/generated/generated.go:492
4MB 37.49% | github.com/OpenLNMetrics/lnmetrics.server/graph/generated.(*executableSchema).Exec.func2 /go/lnmetrics/graph/generated/generated.go:506
----------------------------------------------------------+-------------
199.91MB 100% | github.com/99designs/gqlgen/graphql/executor.(*Executor).CreateOperationContext /go/lnmetrics/vendor/github.com/99designs/gqlgen/graphql/executor/executor.go:62
0 0% 100% 199.91MB 44.93% | github.com/99designs/gqlgen/graphql/executor.(*Executor).parseQuery /go/lnmetrics/vendor/github.com/99designs/gqlgen/graphql/executor/executor.go:172
199.91MB 100% | github.com/vektah/gqlparser/v2/parser.ParseQuery /go/lnmetrics/vendor/github.com/vektah/gqlparser/v2/parser/query.go:14 (inline)
----------------------------------------------------------+-------------
10.67MB 100% | github.com/99designs/gqlgen/graphql/executor.(*Executor).DispatchOperation.func1.1 /go/lnmetrics/vendor/github.com/99designs/gqlgen/graphql/executor/executor.go:104
0 0% 100% 10.67MB 2.40% | github.com/99designs/gqlgen/graphql/executor.processExtensions.func2 /go/lnmetrics/vendor/github.com/99designs/gqlgen/graphql/executor/extensions.go:59
10.67MB 100% | github.com/99designs/gqlgen/graphql/executor.(*Executor).DispatchOperation.func1.1.1 /go/lnmetrics/vendor/github.com/99designs/gqlgen/graphql/executor/executor.go:105
----------------------------------------------------------+-------------
6.67MB 62.51% | github.com/OpenLNMetrics/lnmetrics.server/graph/generated.(*executionContext)._Query_nodes /go/lnmetrics/graph/generated/generated.go:1748
4MB 37.49% | github.com/OpenLNMetrics/lnmetrics.server/graph/generated.(*executionContext)._Mutation_addNodeMetrics /go/lnmetrics/graph/generated/generated.go:1334
0 0% 100% 10.67MB 2.40% | github.com/99designs/gqlgen/graphql/executor.processExtensions.func3 /go/lnmetrics/vendor/github.com/99designs/gqlgen/graphql/executor/extensions.go:62
6.67MB 62.51% | github.com/OpenLNMetrics/lnmetrics.server/graph/generated.(*executionContext)._Query_nodes.func2 /go/lnmetrics/graph/generated/generated.go:1750
4MB 37.49% | github.com/OpenLNMetrics/lnmetrics.server/graph/generated.(*executionContext)._Mutation_addNodeMetrics.func2 /go/lnmetrics/graph/generated/generated.go:1336
----------------------------------------------------------+-------------
426.78MB 100% | net/http.(*ServeMux).ServeHTTP /usr/local/go/src/net/http/server.go:2424
0 0% 100% 426.78MB 95.92% | github.com/99designs/gqlgen/graphql/handler.(*Server).ServeHTTP /go/lnmetrics/vendor/github.com/99designs/gqlgen/graphql/handler/server.go:115
222.87MB 52.22% | github.com/99designs/gqlgen/graphql/handler/transport.POST.Do /go/lnmetrics/vendor/github.com/99designs/gqlgen/graphql/handler/transport/http_post.go:34
199.91MB 46.84% | github.com/99designs/gqlgen/graphql/handler/transport.POST.Do /go/lnmetrics/vendor/github.com/99designs/gqlgen/graphql/handler/transport/http_post.go:44
4MB 0.94% | github.com/99designs/gqlgen/graphql/handler/transport.POST.Do /go/lnmetrics/vendor/github.com/99designs/gqlgen/graphql/handler/transport/http_post.go:53
----------------------------------------------------------+-------------
```

### What did you expect?

I know that the use case it is a little bit strange, but this mem consumption it is really necessary?

In the graphql parser, we can avoid to copy the string literal and pass a pointer, but not sure that this can solve the problem, I would like to talk a little bit

### Minimal graphql.schema and models to reproduce

All available here https://magic.bublina.eu.org/

### versions
- `gqlgen version: v0.14.0`
- `go version: v0.15.0`
- dep or go modules?
```
module github.com/OpenLNMetrics/lnmetrics.server

go 1.15

require (
github.com/99designs/gqlgen v0.14.0
github.com/OpenLNMetrics/lnmetrics.utils v0.0.0-20211018052622-a9fb45a6236b
github.com/agnivade/levenshtein v1.1.1 // indirect
github.com/golang/snappy v0.0.4 // indirect
github.com/hashicorp/golang-lru v0.5.4 // indirect
github.com/joho/godotenv v1.4.0
github.com/mitchellh/mapstructure v1.4.2 // indirect
github.com/sirupsen/logrus v1.8.1
github.com/stretchr/testify v1.6.1
github.com/syndtr/goleveldb v1.0.0
github.com/vektah/gqlparser/v2 v2.2.0
golang.org/x/sys v0.0.0-20211015200801-69063c4bb744 // indirect

```gopkg.in/yaml.v2 v2.4.0 // indirect
)

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.