USACE / USACE/cwms-data-api

CDA deadlock issues when deleting locations

Open
#1,708 6 comments 0 reactions 2 assignees View on GitHub

@adamkorynta is already working on this.

Since May 4, 2026.

bug needs-follow-up
Dominant language
Java
Stars
28
Forks
25
Avg merge
4d 22h
Merged PRs (30d)
20

Description

Describe the bug
Using Swagger or a python script, CDA almost always deadlocks when attempting to delete a location. Locations were confirmed to have no children (no rating curves, location levels, time series, etc.). Sometimes locations will delete, probably less than 10% of the time on average in my testing.

Deadlock also occurred using SQLDeveloper on at least one location when attempting to delete without CDA. Cannot delete the locations in CWMSVue.

Deadlock errors have also occurred when attempting to modify time series groups, but not as often.

Expected behavior
Expect no deadlock errors to occur.

To Reproduce
Steps to reproduce the behavior:

  1. Attempt to delete a location.
  2. See error

Logs/Incident Identifier

22-Apr-2026 13:21:57.337 INFO [https-openssl-nio-155.77.185.11-8243-exec-21562] cwms.cda.data.dao.AuthDao.prepareContextWithUser Validated Api Key for user=G0PDRJAB [CONTEXT ratelimit_period="5 SECONDS" ]
22-Apr-2026 13:22:06.659 WARNING [https-openssl-nio-155.77.185.11-8243-exec-21562] cwms.cda.ApiServlet.lambda$init$20 error on request[-8818300043231349053]: /nwdm-data/locations/Bear_Creek-2350-IDLC2
  org.jooq.exception.DataAccessException: SQL [{ call "CWMS_20"."CWMS_LOC"."DELETE_LOCATION" (?, ?, ?) }]; ORA-00060: deadlock detected while waiting for resource
ORA-06512: at "CWMS_20.CWMS_LOC", line 3246
ORA-06512: at "CWMS_20.CWMS_LOC", line 3220
ORA-06512: at line 1

    at org.jooq_3.18.7.ORACLE18C.debug(Unknown Source)
    at org.jooq.impl.Tools.translate(Tools.java:3470)
    at org.jooq.impl.Tools.translate(Tools.java:3458)
    at org.jooq.impl.DefaultExecuteContext.sqlException(DefaultExecuteContext.java:802)
    at org.jooq.impl.AbstractRoutine.executeCallableStatement(AbstractRoutine.java:611)
    at org.jooq.impl.AbstractRoutine.execute(AbstractRoutine.java:405)
    at org.jooq.impl.Tools.attach(Tools.java:1610)
    at org.jooq.impl.AbstractRoutine.execute(AbstractRoutine.java:374)
    at usace.cwms.db.jooq.codegen.packages.CWMS_LOC_PACKAGE.call_DELETE_LOCATION(CWMS_LOC_PACKAGE.java:1441)
    at cwms.cda.data.dao.LocationsDaoImpl.lambda$deleteLocation$0(LocationsDaoImpl.java:217)
    at org.jooq.impl.DefaultDSLContext.lambda$connection$12(DefaultDSLContext.java:697)
    at org.jooq.impl.DefaultDSLContext.connectionResult(DefaultDSLContext.java:658)
    at org.jooq.impl.DefaultDSLContext.connection(DefaultDSLContext.java:696)
    at cwms.cda.data.dao.JooqDao.connection(JooqDao.java:734)
    at cwms.cda.data.dao.LocationsDaoImpl.deleteLocation(LocationsDaoImpl.java:211)
    at cwms.cda.api.LocationController.delete(LocationController.java:392)
    at io.javalin.apibuilder.CrudFunction$5.invoke$lambda-0(CrudHandler.kt:34)
    at io.javalin.apibuilder.CrudFunctionHandler.handle(CrudHandler.kt)
    at cwms.cda.security.CdaAccessManager.manage(CdaAccessManager.java:37)
    at io.javalin.http.JavalinServlet.addHandler$lambda-0(JavalinServlet.kt:96)
    at io.javalin.http.JavalinServlet$lifecycle$2$1$1.invoke(JavalinServlet.kt:43)
    at io.javalin.http.JavalinServlet$lifecycle$2$1$1.invoke(JavalinServlet.kt:43)
    at io.javalin.http.JavalinServletHandler.executeNextTask(JavalinServletHandler.kt:99)
    at io.javalin.http.JavalinServletHandler.queueNextTaskOrFinish$lambda-1(JavalinServletHandler.kt:85)
    at java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:995)
    at java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2137)
    at io.javalin.http.JavalinServletHandler.queueNextTaskOrFinish$javalin(JavalinServletHandler.kt:85)
    at io.javalin.http.JavalinServletHandler.executeNextTask$lambda-11$lambda-10(JavalinServletHandler.kt:119)
    at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:616)
    at java.util.concurrent.CompletableFuture.uniApplyStage(CompletableFuture.java:628)
    at java.util.concurrent.CompletableFuture.thenApply(CompletableFuture.java:1996)
    at io.javalin.http.JavalinServletHandler.executeNextTask(JavalinServletHandler.kt:119)
    at io.javalin.http.JavalinServletHandler.queueNextTaskOrFinish$lambda-1(JavalinServletHandler.kt:85)
    at java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:995)
    at java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2137)
    at io.javalin.http.JavalinServletHandler.queueNextTaskOrFinish$javalin(JavalinServletHandler.kt:85)
    at io.javalin.http.JavalinServletHandler.executeNextTask$lambda-11$lambda-10(JavalinServletHandler.kt:119)
    at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:616)
    at java.util.concurrent.CompletableFuture.uniApplyStage(CompletableFuture.java:628)
    at java.util.concurrent.CompletableFuture.thenApply(CompletableFuture.java:1996)
    at io.javalin.http.JavalinServletHandler.executeNextTask(JavalinServletHandler.kt:119)
    at io.javalin.http.JavalinServletHandler.queueNextTaskOrFinish$lambda-1(JavalinServletHandler.kt:85)
    at java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:995)
    at java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2137)
    at io.javalin.http.JavalinServletHandler.queueNextTaskOrFinish$javalin(JavalinServletHandler.kt:85)
    at io.javalin.http.JavalinServlet.service(JavalinServlet.kt:89)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:623)
    at cwms.cda.ApiServlet.service(ApiServlet.java:1018)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:623)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:197)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:142)
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:166)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:142)
    at org.apache.catalina.filters.HttpHeaderSecurityFilter.doFilter(HttpHeaderSecurityFilter.java:129)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:166)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:142)
    at org.apache.catalina.filters.CorsFilter.handleNonCORS(CorsFilter.java:334)
    at org.apache.catalina.filters.CorsFilter.doFilter(CorsFilter.java:161)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:166)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:142)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:166)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:88)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:596)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
    at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:643)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:83)
    at org.apache.catalina.authenticator.SingleSignOn.invoke(SingleSignOn.java:240)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:72)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:344)
    at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:398)
    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63)
    at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:935)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1831)
    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)
    at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:973)
    at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:491)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63)
    at java.lang.Thread.run(Thread.java:750)
  Caused by: java.sql.SQLException: ORA-00060: deadlock detected while waiting for resource

CURL Commands

curl -X 'DELETE' \
  'https://HOST:PORT/nwdm-data/locations/GRAI?office=NWDM' \
  -H 'accept: */*'

CDA Version (please complete the following information):
Image

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.