hasura / hasura/graphql-engine

high cpu usage when calling add_inherited_role api

Open
#9,053 2 comments 0 reactions 0 assignees View on GitHub
k/enhancement
Dominant language
TypeScript
Stars
32.1k
Forks
3k
PR merge metrics
PR metrics pending

Description

### Is your proposal related to a problem?
when calling [add_inherited_role api](https://hasura.io/docs/latest/api-reference/metadata-api/inherited-roles/#metadata-add-inherited-role), the cpu usage becomes high, especially when the size of role_set becomes very large, like twenty or thirty or larger. More seriously, sometimes it looks like the service would simply restart, making the service unreachable.

cpu usage before add_inherited_role called
![image](https://user-images.githubusercontent.com/21361743/193493508-7c7da3b8-c495-4071-8b32-ee6f04c4c906.png)
cpu usage after add_inherited_role called
![image](https://user-images.githubusercontent.com/21361743/193493765-1524f52a-6ca0-4905-8972-34789d8d007f.png)
service log when calling add_inherited_role
```
2022-10-03T03:01:50.456960675Z {"type":"http-log","timestamp":"2022-10-03T03:01:50.456+0000","level":"info","detail":{"operation":{"query_execution_time":15.057786914,"user_vars":{"x-hasura-role":"admin"},"request_id":"badb8152-17b2-477d-becb-8d9eb2081b50","response_size":21,"query":{"args":{"role_name":"1-57d4c86577927d04196da2e4af48c98e","role_set":["TaskQuery","HotelSystemQuery","TagQuery","TaskDownload","TaskNotify","HotelUserQuery","HotelAgencyQuery","TaskAdd","TaskUpdate","TaskUpdateNeedNum","TaskUpdateDeadline","TaskUpdateAgencyDeadline","TaskUpdateHotelUserState","TaskUpdateLaborerState","TaskCancel","HotelAgencyAdd","HotelAgencyUpdate","ImageQuery","TagAdd","HotelUserAdd","GenImgUrl","HotelUserUpdate","HotelUserTagAdd","HotelUserTagRemove","MaterialQuery","MaterialCategoryQuery","MaterialCategoryUpdate","MaterialCategoryAdd","MaterialCategoryRemove","MaterialAdd","ExcelGetUploadUrl","MaterialImportExcel","UnitOfMeasurementAdd","MaterialUpdate","MaterialRemove","UnitOfMeasurementQuery","UnitOfMeasurementRemove","DishQuery","TypeOfDishQuery","DishAdd","TypeOfDishAdd","DishUpdate","DishRemove","TypeOfDishUpdate","TypeOfDishRemove","MenuQuery","MenuAdd","MenuUpdate","MenuRemove","EoTemplateQuery","EoTemplateAdd","EoTemplateUpdate","EoTemplateRemove","EventTypeQuery","EventTypeAdd","EventTypeUpdate","EventTypeRemove","CustomerQuery","CustomerAdd","CustomerUpdate","CustomerRemove","EoQuery","EoPurchaseAdviseQuery","EoAdd","EoUpdate","EoMenuUpdate","EoMenuAdd","EoCancel","HotelRoleQuery","HotelRoleAdd","HotelRoleUpdate","HotelRoleRemove","DepartmentUpdate","DepartmentAdd","DepartmentRemove","DepartmentQuery","HotelQuery","HotelUserImport","TagUpdate","TagRemove","MiniAppLogin","HotelSupplierQuery","HotelSupplierCategoryManage","HotelSupplierAdd","HotelSupplierUpdate","HotelSupplierRemove","InquiryQuery","InquiryAdd","InquiryCancel","QuotationImport","HotelSupplierCategoryQuery","HotelSupplierCategoryUpdate","PlatformSupplierQuery","MaterialQueryFromExcel","MaterialPriceQuery","FixedPriceAdd","FixedPriceWithoutQuotationImport","QuotationExport","FixedPriceWithQuotationImport","MaterialPriceInsert","CostOverrunQuery","PurchaseRequisitionQuery","PurchaseRequisitionMaterialQuery","PurchaseOrderMaterialQuery","PurchaseOrderQuery","PurchaseOrderViewQuery","WechatQyUserQuery","WechatQyApprovalQuery","EoPresetNotifiedUserAdd","EoPresetNotifiedUserUpdate","EoPresetNotifiedUserQuery","CustomerTagQuery","WarehouseManagerQuery","WarehouseMaterialQuery","InventoryExpiringReportQuery","InventoryLowConsumptionReportQuery","InventoryRealtimeReportQuery","InventoryTransactionReportQuery","BatchQuery","InventoryCheckQuery","InventoryTransactionQuery","InventoryRequisitionQuery","InventoryRequisitionReturnQuery","PurchaseReceiptQuery","InventoryInitQuery"]},"type":"add_inherited_role"},"request_mode":"non-graphql","request_read_time":5.207e-6},"request_id":"badb8152-17b2-477d-becb-8d9eb2081b50","http_info":{"status":200,"http_version":"HTTP/1.1","url":"/v1/metadata","ip":"172.18.0.2","method":"POST","content_encoding":null}}}
2022-10-03T03:01:55.215398266Z {"type":"startup","timestamp":"2022-10-03T03:01:55.213+0000","level":"info","detail":{"kind":"server_configuration","info":{"live_query_options":{"batch_size":100,"refetch_delay":1},"transaction_isolation":"ISOLATION LEVEL READ COMMITTED","enable_maintenance_mode":false,"enabled_log_types":["http-log","websocket-log","startup","webhook-log","query-log"],"server_host":"HostAny","websocket_connection_init_timeout":"WSConnectionInitTimeout {unWSConnectionInitTimeout = Seconds {seconds = 3s}}","enable_allowlist":false,"remote_schema_permissions":false,"log_level":"info","auth_hook_mode":"GET","use_prepared_statements":true,"unauth_role":null,"stringify_numeric_types":false,"v1-boolean-null-collapse":false,"graceful_shutdown_timeout":60,"enabled_apis":["metadata","graphql","config","pgdump"],"enable_telemetry":true,"enable_console":false,"auth_hook":"http://hotelsop-backend:8081/hs/auth","infer_function_permissions":true,"experimental_features":["inherited_roles"],"events_fetch_batch_size":100,"jwt_secret":null,"cors_config":{"allowed_origins":"*","disabled":false,"ws_read_cookie":null},"websocket_compression_options":"NoCompression","console_assets_dir":null,"admin_secret_set":true,"port":8080,"websocket_keep_alive":"KeepAliveDelay {unKeepAliveDelay = Seconds {seconds = 5s}}"}}}
2022-10-03T03:01:56.271049142Z {"type":"startup","timestamp":"2022-10-03T03:01:55.213+0000","level":"info","detail":{"kind":"postgres_connection","info":{"retries":1,"database_url":"postgres://postgres:...@postgres:5432/postgres"}}}
2022-10-03T03:02:14.285422059Z {"type":"startup","timestamp":"2022-10-03T03:02:14.285+0000","level":"info","detail":{"kind":"catalog_migrate","info":"Already at the latest catalog version (47); nothing to do."}}
2022-10-03T03:02:14.285456300Z {"type":"startup","timestamp":"2022-10-03T03:02:14.285+0000","level":"info","detail":{"kind":"schema-sync","info":"Schema sync enabled. Polling at Milliseconds {milliseconds = 1s}"}}
2022-10-03T03:02:15.293127379Z {"type":"unstructured","timestamp":"2022-10-03T03:02:14.285+0000","level":"info","detail":"Thread SchemeUpdate.listener (re)started"}
2022-10-03T03:02:15.293160057Z {"type":"startup","timestamp":"2022-10-03T03:02:14.285+0000","level":"info","detail":{"kind":"schema-sync","info":{"thread_id":"ThreadId 41","instance_id":"6304b7d7-e885-4bea-9003-de43ad3f06fb","message":"listener thread started"}}}
2022-10-03T03:02:15.293165216Z {"type":"startup","timestamp":"2022-10-03T03:02:14.285+0000","level":"info","detail":{"kind":"schema-sync","info":{"thread_id":"ThreadId 54","instance_id":"6304b7d7-e885-4bea-9003-de43ad3f06fb","message":"processor thread started"}}}
2022-10-03T03:02:15.293169008Z {"type":"startup","timestamp":"2022-10-03T03:02:14.285+0000","level":"info","detail":{"kind":"event_triggers","info":"starting workers"}}
2022-10-03T03:02:15.293172674Z {"type":"unstructured","timestamp":"2022-10-03T03:02:14.285+0000","level":"info","detail":"Thread processEventQueue (re)started"}
2022-10-03T03:02:15.293175751Z {"type":"unstructured","timestamp":"2022-10-03T03:02:14.285+0000","level":"info","detail":"Thread asyncActionsProcessor (re)started"}
2022-10-03T03:02:15.293179026Z {"type":"startup","timestamp":"2022-10-03T03:02:14.285+0000","level":"info","detail":{"kind":"scheduled_triggers","info":"preparing data"}}
2022-10-03T03:02:15.293182197Z {"type":"unstructured","timestamp":"2022-10-03T03:02:14.285+0000","level":"info","detail":"Thread runCronEventsGenerator (re)started"}
2022-10-03T03:02:15.293185276Z {"type":"unstructured","timestamp":"2022-10-03T03:02:14.285+0000","level":"info","detail":"Thread processScheduledTriggers (re)started"}
2022-10-03T03:02:15.293188258Z {"type":"unstructured","timestamp":"2022-10-03T03:02:14.285+0000","level":"info","detail":"Thread checkForUpdates (re)started"}
2022-10-03T03:02:15.293191531Z {"type":"startup","timestamp":"2022-10-03T03:02:14.285+0000","level":"info","detail":{"kind":"telemetry","info":"Help us improve Hasura! The graphql-engine server collects anonymized usage stats which allows us to keep improving Hasura at warp speed. To read more or opt-out, visit https://hasura.io/docs/latest/graphql/core/guides/telemetry.html"}}
2022-10-03T03:02:15.293195779Z {"type":"startup","timestamp":"2022-10-03T03:02:14.285+0000","level":"info","detail":{"kind":"server","info":{"time_taken":19.192937574,"message":"starting API server"}}}
2022-10-03T03:02:15.293199025Z {"type":"unstructured","timestamp":"2022-10-03T03:02:14.285+0000","level":"info","detail":"Thread runTelemetry (re)started"}
2022-10-03T03:02:15.293478479Z {"type":"unstructured","timestamp":"2022-10-03T03:02:14.285+0000","level":"info","detail":"Thread ourIdleGC (re)started"}
2022-10-03T03:02:15.293525337Z {"type":"unstructured","timestamp":"2022-10-03T03:02:14.285+0000","level":"info","detail":"Thread SchemeUpdate.processor (re)started"}
2022-10-03T03:02:15.293564655Z {"type":"unstructured","timestamp":"2022-10-03T03:02:14.285+0000","level":"info","detail":"Thread asyncActionSubscriptionsProcessor (re)started"}
2022-10-03T03:02:15.293575853Z {"type":"schema-sync-thread","timestamp":"2022-10-03T03:02:14.285+0000","level":"info","detail":{"thread_type":"processor","info":{"currentVersion":0,"latestResourceVersion":2436}}}
2022-10-03T03:02:30.376100867Z {"type":"schema-sync-thread","timestamp":"2022-10-03T03:02:30.375+0000","level":"info","detail":{"thread_type":"processor","info":{"message":"Schema Version changed with notifications"}}}
```

### Describe the solution you'd like

the service keep reachable when adding inherited role
### Describe alternatives you've considered

no suggestion

### If the feature is approved, would you be willing to submit a PR?

no

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the metadata add_inherited_role API call at /v1/metadata with a large role_set, using the request and service logs in this issue as a baseline. Trace the inherited-roles metadata handling and verify that adding the role no longer causes excessive CPU usage or makes the service unreachable.

Written by the indexing model from the issue text.

Assessment

Tech stack
graphql, postgres
Domain
api, authorization, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.