geonetwork / geonetwork/core-geonetwork

[BUG] NullPointerException in ThreddsHarvester.extractMetadataFromOpendapDDX (4.2.2 & 4.4.4)

Open
#8,938 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Java
Stars
521
Forks
515
Avg merge
6d 13h
Merged PRs (30d)
19

Description

Subject: NullPointerException in ThreddsHarvester.extractMetadataFromOpendapDDX (4.2.2 & 4.4.4)

Description of the problem:
When attempting to harvest a THREDDS catalog using the "THREDDS" harvester type in GeoNetwork, a recurring NullPointerException occurs in the extractMetadataFromOpendapDDX method. This error prevents the successful harvesting of dataset metadata.

Affected versions:
We have consistently replicated this behavior in GeoNetwork 4.2.2 and 4.4.4.

Steps to reproduce:

  1. Deploy a GeoNetwork instance (in our case, using Docker Compose, version 4.2.2).
  2. Log in to the GeoNetwork administration interface.
  3. Create a new harvester of type "THREDDS".
  4. Configure the THREDDS catalog URL: http://172.16.132.22:8004/thredds/catalog/auv/glider/sl3u1064-scb_sl3u1064/L1/2025/catalog.xml
  5. Ensure that the custom XSLT ThreddsCatalog-to-ISO19119_ISO19139.xsl is correctly located at /var/lib/jetty/webapps/geonetwork/WEB-INF/data/config/schema_plugins/iso19139/convert/ and referenced in schema-conversions.xml for the conversion from="thredds" to iso19139.
  6. Execute the harvester.

Expected behavior:
The harvester should process the THREDDS catalog, apply the XSLT transformation, and import the ISO 19139 metadata without errors.

Observed behavior (with DEBUG logs):
The harvester starts, reads the main THREDDS catalog, but when attempting to process individual datasets and access the OPeNDAP DDX service, it throws a NullPointerException. The detailed logs indicate that GeoNetwork successfully accesses the DDX XML, but the error occurs during its internal processing of this DDX. A second NullPointerException in Xml.transform is observed, which appears to be a consequence of the first.

Relevant log snippet:

2025-07-16T11:13:01,262 INFO  [geonetwork.harvester] - Starting harvesting of gn422_threddsmiguel
2025-07-16T11:13:01,282 INFO  [geonetwork.harvester] - Started harvesting from node : gn422_threddsmiguel (ThreddsHarvester)
2025-07-16T11:13:01,284 INFO  [geonetwork.harvester] - Retrieving remote metadata information for : gn422_threddsmiguel
catalog valid = true
access auv/glider/sl3u1064-scb_sl3u1064/L1/2025/dep0010_sl3u1064_scb-sl3u1064_L1_2025-05-20_data_rt.nc valid = true
dataset dep0010_sl3u1064_scb-sl3u1064_L1_2025-05-20_data_rt.nc valid = true
access auv/glider/sl3u1064-scb_sl3u1064/L1/2025/dep0010_sl3u1064_scb-sl3u1064_L1_2025-05-20.nc valid = true
dataset dep0010_sl3u1064_scb-sl3u1064_L1_2025-05-20.nc valid = true
access auv/glider/sl3u1064-scb_sl3u1064/L1/2025/dep0009_sl3u1064_scb-sl3u1064_L1_2025-04-09_data_rt.nc valid = true
dataset dep0009_sl3u1064_scb-sl3u1064_L1_2025-04-09_data_rt.nc valid = true
access auv/glider/sl3u1064-scb_sl3u1064/L1/2025/dep0009_sl3u1064_scb-sl3u1064_L1_2025-04-09.nc valid = true
dataset dep0009_sl3u1064_scb-sl3u1064_L1_2025-04-09.nc valid = true
dataset 2025 valid = true
2025-07-16T11:13:01,376 INFO  [geonetwork.harvester] - Catalog read from http://172.16.132.22:8004//thredds/catalog/auv/glider/sl3u1064-scb_sl3u1064/L1/2025/catalog.xml is

2025-07-16T11:13:01,376 INFO  [geonetwork.harvester] - Crawling the datasets in the catalog....
2025-07-16T11:13:01,377 INFO  [geonetwork.harvester] - Crawling through 2025
2025-07-16T11:13:01,377 INFO  [geonetwork.harvester] - Trying to find ThreddsMetadata for dataset 2025
2025-07-16T11:13:01,377 INFO  [geonetwork.harvester] - Crawling through dep0010_sl3u1064_scb-sl3u1064_L1_2025-05-20_data_rt.nc
2025-07-16T11:13:01,377 INFO  [geonetwork.harvester] - Processing dataset: dep0010_sl3u1064_scb-sl3u1064_L1_2025-05-20_data_rt.nc with URL: http://172.16.132.22:8004//thredds/catalog/auv/glider/sl3u1064-scb_sl3u1064/L1/2025/catalog.xml?dataset=auv%2Fglider%2Fsl3u1064-scb_sl3u1064%2FL1%2F2025%2Fdep0010_sl3u1064_scb-sl3u1064_L1_2025-05-20_data_rt.nc
2025-07-16T11:13:01,492 ERROR [geonetwork.harvester] - Thrown Exception java.lang.NullPointerException during dataset processing
java.lang.NullPointerException
at org.fao.geonet.kernel.harvest.harvester.thredds.Harvester.extractMetadataFromOpendapDDX(Harvester.java:1364)
at org.fao.geonet.kernel.harvest.harvester.thredds.Harvester.getMetadata(Harvester.java:864)
at org.fao.geonet.kernel.harvest.harvester.thredds.Harvester.examineThreddsDataset(Harvester.java:604)
at org.fao.geonet.kernel.harvest.harvester.thredds.Harvester.crawlDatasets(Harvester.java:462)
at org.fao.geonet.kernel.harvest.harvester.thredds.Harvester.crawlDatasets(Harvester.java:458)
at org.fao.geonet.kernel.harvest.harvester.thredds.Harvester.harvestCatalog(Harvester.java:411)
at org.fao.geonet.kernel.harvest.harvester.thredds.Harvester.harvest(Harvester.java:284)
at org.fao.geonet.kernel.harvest.harvester.thredds.ThreddsHarvester.doHarvest(ThreddsHarvester.java:76)
at org.fao.geonet.kernel.harvest.harvester.AbstractHarvesterHarvestWithIndexProcessor.process(AbstractHarvester.java:602)atorg.fao.geonet.kernel.harvest.harvester.AbstractHarvester.harvest(AbstractHarvester.java:673)atorg.fao.geonet.kernel.harvest.harvester.HarvesterJob.execute(HarvesterJob.java:69)atorg.quartz.core.JobRunShell.run(JobRunShell.java:202)atorg.quartz.simpl.SimpleThreadPoolWorkerThread.run(SimpleThreadPool.java:573)
2025-07-16T11:13:01,493 INFO  [geonetwork.harvester] - Processing service: http://172.16.132.22:8004/thredds/iso/ for dep0010_sl3u1064_scb-sl3u1064_L1_2025-05-20_data_rt.nc
2025-07-16T11:13:01,493 INFO  [geonetwork.harvester] - Crawling through dep0010_sl3u1064_scb-sl3u1064_L1_2025-05-20.nc
2025-07-16T11:13:01,494 INFO  [geonetwork.harvester] - Processing dataset: dep0010_sl3u1064_scb-sl3u1064_L1_2025-05-20.nc with URL: http://172.16.132.22:8004//thredds/catalog/auv/glider/sl3u1064-scb_sl3u1064/L1/2025/catalog.xml?dataset=auv%2Fglider%2Fsl3u1064-scb_sl3u1064%2FL1%2F2025%2Fdep0010_sl3u1064_scb-sl3u1064_L1_2025-05-20.nc
2025-07-16T11:13:01,546 ERROR [geonetwork.harvester] - Thrown Exception java.lang.NullPointerException during dataset processing
java.lang.NullPointerException
at org.fao.geonet.kernel.harvest.harvester.thredds.Harvester.extractMetadataFromOpendapDDX(Harvester.java:1364)
at org.fao.geonet.kernel.harvest.harvester.thredds.Harvester.getMetadata(Harvester.java:864)
at org.fao.geonet.kernel.harvest.harvester.thredds.Harvester.examineThreddsDataset(Harvester.java:604)
at org.fao.geonet.kernel.harvest.harvester.thredds.Harvester.crawlDatasets(Harvester.java:462)
at org.fao.geonet.kernel.harvest.harvester.thredds.Harvester.crawlDatasets(Harvester.java:458)
at org.fao.geonet.kernel.harvest.harvester.thredds.Harvester.harvestCatalog(Harvester.java:411)
at org.fao.geonet.kernel.harvest.harvester.thredds.Harvester.harvest(Harvester.java:284)
at org.fao.geonet.kernel.harvest.harvester.thredds.ThreddsHarvester.doHarvest(ThreddsHarvester.java:76)
at org.fao.geonet.kernel.harvest.harvester.AbstractHarvesterHarvestWithIndexProcessor.process(AbstractHarvester.java:602)atorg.fao.geonet.kernel.harvest.harvester.AbstractHarvester.harvest(AbstractHarvester.java:673)atorg.fao.geonet.kernel.harvest.harvester.HarvesterJob.execute(HarvesterJob.java:69)atorg.quartz.core.JobRunShell.run(JobRunShell.java:202)atorg.quartz.simpl.SimpleThreadPoolWorkerThread.run(SimpleThreadPool.java:573)
2025-07-16T11:13:01,547 INFO  [geonetwork.harvester] - Processing service: http://172.16.132.22:8004/thredds/iso/ for dep0010_sl3u1064_scb-sl3u1064_L1_2025-05-20.nc
2025-07-16T11:13:01,547 INFO  [geonetwork.harvester] - Crawling through dep0009_sl3u1064_scb-sl3u1064_L1_2025-04-09_data_rt.nc
2025-07-16T11:13:01,547 INFO  [geonetwork.harvester] - Processing dataset: dep0009_sl3u1064_scb-sl3u1064_L1_2025-04-09_data_rt.nc with URL: http://172.16.132.22:8004//thredds/catalog/auv/glider/sl3u1064-scb_sl3u1064/L1/2025/catalog.xml?dataset=auv%2Fglider%2Fsl3u1064-scb_sl3u1064%2FL1%2F2025%2Fdep0009_sl3u1064_scb-sl3u1064_L1_2025-04-09_data_rt.nc
2025-07-16T11:13:01,606 ERROR [geonetwork.harvester] - Thrown Exception java.lang.NullPointerException during dataset processing
java.lang.NullPointerException
at org.fao.geonet.kernel.harvest.harvester.thredds.Harvester.extractMetadataFromOpendapDDX(Harvester.java:1364)
at org.fao.geonet.kernel.harvest.harvester.thredds.Harvester.getMetadata(Harvester.java:864)
at org.fao.geonet.kernel.harvest.harvester.thredds.Harvester.examineThreddsDataset(Harvester.java:604)
at org.fao.geonet.kernel.harvest.harvester.thredds.Harvester.crawlDatasets(Harvester.java:462)
at org.fao.geonet.kernel.harvest.harvester.thredds.Harvester.crawlDatasets(Harvester.java:458)
at org.fao.geonet.kernel.harvest.harvester.thredds.Harvester.harvestCatalog(Harvester.java:411)
at org.fao.geonet.kernel.harvest.harvester.thredds.Harvester.harvest(Harvester.java:284)
at org.fao.geonet.kernel.harvest.harvester.thredds.ThreddsHarvester.doHarvest(ThreddsHarvester.java:76)
at org.fao.geonet.kernel.harvest.harvester.AbstractHarvesterHarvestWithIndexProcessor.process(AbstractHarvester.java:602)atorg.fao.geonet.kernel.harvest.harvester.AbstractHarvester.harvest(AbstractHarvester.java:673)atorg.fao.geonet.kernel.harvest.harvester.HarvesterJob.execute(HarvesterJob.java:69)atorg.quartz.core.JobRunShell.run(JobRunShell.java:202)atorg.quartz.simpl.SimpleThreadPoolWorkerThread.run(SimpleThreadPool.java:573)
2025-07-16T11:13:01,606 INFO  [geonetwork.harvester] - Processing service: http://172.16.132.22:8004/thredds/iso/ for dep0009_sl3u1064_scb-sl3u1064_L1_2025-04-09_data_rt.nc
2025-07-16T11:13:01,606 INFO  [geonetwork.harvester] - Crawling through dep0009_sl3u1064_scb-sl3u1064_L1_2025-04-09.nc
2025-07-16T11:13:01,607 INFO  [geonetwork.harvester] - Processing dataset: dep0009_sl3u1064_scb-sl3u1064_L1_2025-04-09.nc with URL: http://172.16.132.22:8004//thredds/catalog/auv/glider/sl3u1064-scb_sl3u1064/L1/2025/catalog.xml?dataset=auv%2Fglider%2Fsl3u1064-scb_sl3u1064%2FL1%2F2025%2Fdep0009_sl3u1064_scb-sl3u1064_L1_2025-04-09.nc
2025-07-16T11:13:01,663 ERROR [geonetwork.harvester] - Thrown Exception java.lang.NullPointerException during dataset processing
java.lang.NullPointerException
at org.fao.geonet.kernel.harvest.harvester.thredds.Harvester.extractMetadataFromOpendapDDX(Harvester.java:1364)
at org.fao.geonet.kernel.harvest.harvester.thredds.Harvester.getMetadata(Harvester.java:864)
at org.fao.geonet.kernel.harvest.harvester.thredds.Harvester.examineThreddsDataset(Harvester.java:604)
at org.fao.geonet.kernel.harvest.harvester.thredds.Harvester.crawlDatasets(Harvester.java:462)
at org.fao.geonet.kernel.harvest.harvester.thredds.Harvester.crawlDatasets(Harvester.java:458)
at org.fao.geonet.kernel.harvest.harvester.thredds.Harvester.harvestCatalog(Harvester.java:411)
at org.fao.geonet.kernel.harvest.harvester.thredds.Harvester.harvest(Harvester.java:284)
at org.fao.geonet.kernel.harvest.harvester.thredds.ThreddsHarvester.doHarvest(ThreddsHarvester.java:76)
at org.fao.geonet.kernel.harvest.harvester.AbstractHarvesterHarvestWithIndexProcessor.process(AbstractHarvester.java:602)atorg.fao.geonet.kernel.harvest.harvester.AbstractHarvester.harvest(AbstractHarvester.java:673)atorg.fao.geonet.kernel.harvest.harvester.HarvesterJob.execute(HarvesterJob.java:69)atorg.quartz.core.JobRunShell.run(JobRunShell.java:202)atorg.quartz.simpl.SimpleThreadPoolWorkerThread.run(SimpleThreadPool.java:573)
2025-07-16T11:13:01,664 INFO  [geonetwork.harvester] - Processing service: http://172.16.132.22:8004/thredds/iso/ for dep0009_sl3u1064_scb-sl3u1064_L1_2025-04-09.nc
2025-07-16T11:13:01,665 INFO  [geonetwork.harvester] - Adding dataset metadata...
2025-07-16T11:13:01,665 WARN  [geonetwork.harvester] - Raised exception while harvesting from : gn422_threddsmiguel (ThreddsHarvester)
2025-07-16T11:13:01,665 WARN  [geonetwork.harvester] -  (C) Class    : NullPointerException
2025-07-16T11:13:01,666 WARN  [geonetwork.harvester] -  (C) Message : null
2025-07-16T11:13:01,666 ERROR [geonetwork.harvester] - null
java.lang.NullPointerException: null
at org.fao.geonet.utils.Xml.transform(Xml.java:493) ~[gn-common-4.2.2-0.jar:?]
at org.fao.geonet.utils.Xml.transform(Xml.java:400) ~[gn-common-4.2.2-0.jar:?]
at org.fao.geonet.kernel.harvest.harvester.thredds.Harvester.createDatasetMetadata(Harvester.java:1545) ~[gn-harvesters-4.2.2-0.jar:?]
at org.fao.geonet.kernel.harvest.harvester.thredds.Harvester.harvestCatalog(Harvester.java:420) ~[gn-harvesters-4.2.2-0.jar:?]
at org.fao.geonet.kernel.harvest.harvester.thredds.Harvester.harvest(Harvester.java:284) ~[gn-harvesters-4.2.2-0.jar:?]
at org.fao.geonet.kernel.harvest.harvester.thredds.ThreddsHarvester.doHarvest(ThreddsHarvester.java:76) ~[gn-harvesters-4.2.2-0.jar:?]
at org.fao.geonet.kernel.harvest.harvester.AbstractHarvesterHarvestWithIndexProcessor.process(AbstractHarvester.java:602) 
[˜gn−harvesters−4.2.2−0.jar:?]atorg.fao.geonet.kernel.harvest.harvester.AbstractHarvester.harvest(AbstractHarvester.java:673) 
[˜gn−harvesters−4.2.2−0.jar:?]atorg.fao.geonet.kernel.harvest.harvester.HarvesterJob.execute(HarvesterJob.java:69) 
[˜quartz−2.3.2.jar:?]atorg.quartz.core.JobRunShell.run(JobRunShell.java:202) 
[˜ quartz−2.3.2.jar:?]atorg.quartz.simpl.SimpleThreadPoolWorkerThread.run(SimpleThreadPool.java:573) ~[quartz-2.3.2.jar:?]

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in Harvester.java at extractMetadataFromOpendapDDX (line 1364), then inspect the Xml.transform failure in Xml.java (line 493). Reproduce the THREDDS harvest with the catalog URL, OPeNDAP DDX response, and configured XSLT described in the issue. Done means the datasets are processed and imported without either NullPointerException.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.