geopython / geopython/GeoHealthCheck
Optimization: Reduce number of queries to DB
- Lenguaje dominante
- Python
- Estrellas
- 92
- Forks
- 75
- Métricas de merge de PR
- Sin PR fusionados en 30 d
Descripción
Using PostgreSQL as backend, logging the DB statements, there seems to be quite a number of queries just to display a single Resource page with a single WMS Resource like `GET /?lang=en&resource_type=OGC%3AWMS`:
```
LOG: statement: BEGIN
LOG: statement: SELECT t.oid, typarray
FROM pg_type t JOIN pg_namespace ns
ON typnamespace = ns.oid
WHERE typname = 'hstore';
LOG: statement: ROLLBACK
LOG: statement: BEGIN
LOG: statement: select version()
LOG: statement: select current_schema()
LOG: statement: show transaction isolation level
LOG: statement: SELECT CAST('test plain returns' AS VARCHAR(60)) AS anon_1
LOG: statement: SELECT CAST('test unicode returns' AS VARCHAR(60)) AS anon_1
LOG: statement: SELECT 'x' AS some_label
LOG: statement: ROLLBACK
LOG: statement: BEGIN
LOG: statement: show standard_conforming_strings
LOG: statement: SELECT resource.identifier AS resource_identifier, resource.resource_type AS resource_resource_type, resource.title AS resource_title, resource.url AS resource_url, resource.latitude AS resource_latitude, resource.longitude AS resource_longitude, resource.owner_identifier AS resource_owner_identifier
FROM resource
WHERE resource.resource_type = 'OGC:WMS'
LOG: statement: SELECT run.identifier AS run_identifier, run.resource_identifier AS run_resource_identifier, run.checked_datetime AS run_checked_datetime, run.success AS run_success, run.response_time AS run_response_time, run.message AS run_message
FROM run
WHERE 2 = run.resource_identifier ORDER BY run.checked_datetime ASC
LIMIT 1
LOG: statement: SELECT run.identifier AS run_identifier, run.resource_identifier AS run_resource_identifier, run.checked_datetime AS run_checked_datetime, run.success AS run_success, run.response_time AS run_response_time, run.message AS run_message
FROM run
WHERE 2 = run.resource_identifier ORDER BY run.checked_datetime ASC
LIMIT 1
LOG: statement: SELECT run.identifier AS run_identifier, run.resource_identifier AS run_resource_identifier, run.checked_datetime AS run_checked_datetime, run.success AS run_success, run.response_time AS run_response_time, run.message AS run_message
FROM run
WHERE 2 = run.resource_identifier ORDER BY run.checked_datetime DESC
LIMIT 1
LOG: statement: SELECT run.identifier AS run_identifier, run.resource_identifier AS run_resource_identifier, run.checked_datetime AS run_checked_datetime, run.success AS run_success, run.response_time AS run_response_time, run.message AS run_message
FROM run
WHERE 2 = run.resource_identifier ORDER BY run.checked_datetime DESC
LIMIT 1
LOG: statement: SELECT run.identifier AS run_identifier, run.resource_identifier AS run_resource_identifier, run.checked_datetime AS run_checked_datetime, run.success AS run_success, run.response_time AS run_response_time, run.message AS run_message
FROM run
WHERE 2 = run.resource_identifier ORDER BY run.checked_datetime DESC
LIMIT 1
LOG: statement: SELECT count(*) AS count_1
FROM (SELECT run.identifier AS run_identifier, run.resource_identifier AS run_resource_identifier, run.checked_datetime AS run_checked_datetime, run.success AS run_success, run.response_time AS run_response_time, run.message AS run_message
FROM run
WHERE 2 = run.resource_identifier) AS anon_1
LOG: statement: SELECT count(*) AS count_1
FROM (SELECT run.identifier AS run_identifier, run.resource_identifier AS run_resource_identifier, run.checked_datetime AS run_checked_datetime, run.success AS run_success, run.response_time AS run_response_time, run.message AS run_message
FROM run
WHERE 2 = run.resource_identifier AND run.success = true) AS anon_1
LOG: statement: SELECT "user".user_id AS user_user_id, "user".username AS user_username, "user".password AS user_password, "user".email AS user_email, "user".role AS user_role, "user".registered_on AS user_registered_on
FROM "user"
WHERE "user".user_id = 1
LOG: statement: SELECT count(*) AS count_1
FROM (SELECT resource.resource_type AS resource_resource_type
FROM resource) AS anon_1
LOG: statement: SELECT resource.resource_type AS resource_resource_type, count(resource.resource_type) AS count_1
FROM resource GROUP BY resource.resource_type
LOG: statement: SELECT run.identifier AS run_identifier, run.resource_identifier AS run_resource_identifier, run.checked_datetime AS run_checked_datetime, run.success AS run_success, run.response_time AS run_response_time, run.message AS run_message
FROM run
WHERE 2 = run.resource_identifier ORDER BY run.checked_datetime DESC
LIMIT 1
LOG: statement: SELECT run.identifier AS run_identifier, run.resource_identifier AS run_resource_identifier, run.checked_datetime AS run_checked_datetime, run.success AS run_success, run.response_time AS run_response_time, run.message AS run_message
FROM run
WHERE 2 = run.resource_identifier ORDER BY run.checked_datetime DESC
LIMIT 1
LOG: statement: SELECT run.identifier AS run_identifier, run.resource_identifier AS run_resource_identifier, run.checked_datetime AS run_checked_datetime, run.success AS run_success, run.response_time AS run_response_time, run.message AS run_message
FROM run
WHERE 2 = run.resource_identifier ORDER BY run.checked_datetime DESC
LIMIT 1
LOG: statement: SELECT run.identifier AS run_identifier, run.resource_identifier AS run_resource_identifier, run.checked_datetime AS run_checked_datetime, run.success AS run_success, run.response_time AS run_response_time, run.message AS run_message
FROM run
WHERE 2 = run.resource_identifier
LOG: statement: SELECT run.identifier AS run_identifier, run.resource_identifier AS run_resource_identifier, run.checked_datetime AS run_checked_datetime, run.success AS run_success, run.response_time AS run_response_time, run.message AS run_message
FROM run
WHERE 2 = run.resource_identifier
LOG: statement: SELECT run.identifier AS run_identifier, run.resource_identifier AS run_resource_identifier, run.checked_datetime AS run_checked_datetime, run.success AS run_success, run.response_time AS run_response_time, run.message AS run_message
FROM run
WHERE 2 = run.resource_identifier
LOG: statement: SELECT run.identifier AS run_identifier, run.resource_identifier AS run_resource_identifier, run.checked_datetime AS run_checked_datetime, run.success AS run_success, run.response_time AS run_response_time, run.message AS run_message
FROM run
WHERE 2 = run.resource_identifier
LOG: statement: SELECT count(*) AS count_1
FROM (SELECT run.identifier AS run_identifier, run.resource_identifier AS run_resource_identifier, run.checked_datetime AS run_checked_datetime, run.success AS run_success, run.response_time AS run_response_time, run.message AS run_message
FROM run
WHERE 2 = run.resource_identifier) AS anon_1
LOG: statement: SELECT count(*) AS count_1
FROM (SELECT run.identifier AS run_identifier, run.resource_identifier AS run_resource_identifier, run.checked_datetime AS run_checked_datetime, run.success AS run_success, run.response_time AS run_response_time, run.message AS run_message
FROM run
WHERE 2 = run.resource_identifier AND run.success = true) AS anon_1
LOG: statement: SELECT count(*) AS count_1
FROM (SELECT run.identifier AS run_identifier, run.resource_identifier AS run_resource_identifier, run.checked_datetime AS run_checked_datetime, run.success AS run_success, run.response_time AS run_response_time, run.message AS run_message
FROM run
WHERE 2 = run.resource_identifier) AS anon_1
LOG: statement: SELECT count(*) AS count_1
FROM (SELECT run.identifier AS run_identifier, run.resource_identifier AS run_resource_identifier, run.checked_datetime AS run_checked_datetime, run.success AS run_success, run.response_time AS run_response_time, run.message AS run_message
FROM run
WHERE 2 = run.resource_identifier AND run.success = true) AS anon_1
127.0.0.1 - - [19/Dec/2016 21:34:41] "GET /?lang=en&resource_type=OGC%3AWMS HTTP/1.1" 200 -
```
This happens behind the facade of SQLAlchemy I guess. Possibly some statements in https://github.com/geopython/GeoHealthCheck/blob/master/GeoHealthCheck/models.py in combination with the templates may be optimized.
Guía de contribución
Línea de trabajo
Comienza reproduciendo GET /?lang=en\u0026resource_type=OGC%3AWMS con el registro de statements de PostgreSQL activado y, después, inspecciona GeoHealthCheck/models.py y las plantillas mencionadas en el issue. Traza qué operaciones de la página producen consultas repetidas de run y count. Se considera terminado cuando la página Resource sigue renderizándose correctamente con menos consultas redundantes a la base de datos.
Escrito por el modelo de indexación a partir del texto del issue.
Evaluación
- Stack tecnológico
- postgresql, python, sqlalchemy
- Área
- backend, database, performance
- Tipo de issue
- Refactorización
- Dificultad
- 4/5
- Tiempo estimado
- 3-5 días
- Estado de actividad
- Estancado
- Claridad
- Bastante claro
- Aptitud para principiantes
- 35/100