locationtech / locationtech/geotrellis

Cannot read the Tiff file by GDALRasterSource. Unable to construct dataset dimensions. GDAL Error Code: 4

Open
#3,465 21 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug question
Dominant language
Scala
Stars
1.4k
Forks
360
Avg merge
6h 57m
Merged PRs (30d)
12

Description

Describe the bug

Cannot read the Tiff file by GDALRasterSource. Unable to construct dataset dimensions. GDAL Error Code: 4

To Reproduce

Provide as able:

  • Steps to reproduce the behavior
  • Code example
package com.example.gdalread
import cats.syntax.option._

import geotrellis.layer.{FloatingLayoutScheme, KeyExtractor, LayoutLevel, SpatialKey}
import geotrellis.proj4.LatLng
import geotrellis.raster.RasterSource
import geotrellis.raster.gdal.GDALRasterSource
import geotrellis.raster.resample.{Bilinear, PointResampleMethod}
import geotrellis.spark.{MultibandTileLayerRDD, RasterSourceRDD, RasterSummary}
import org.apache.log4j.{Level, Logger}
import org.apache.spark.rdd.RDD
import org.apache.spark.{SparkConf, SparkContext}
import geotrellis.raster.gdal._
import com.azavea.gdal._

object gdal_read_test {
  def main(args: Array[String]): Unit = {
    Logger.getLogger("org").setLevel(Level.ERROR)
    System.load("/root/anaconda3/envs/gdal-3.1.2/lib/libgdal.so.27")
    System.getProperty("java.library.path")
    GDALWarp.init(100)
    print("enter country_pop_sgdal_read_test  tatus_____________21.12________________")
    var startTime = System.currentTimeMillis();
    implicit val conf =
      new SparkConf()
        .setAppName("gdal_read_test")
        .setMaster("spark://master:7077")
        .set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
        .set("spark.kryo.registrator", "geotrellis.spark.store.kryo.KryoRegistrator")
        .set("spark.executor.cores", "6")
        .set("spark.executor.memory", "4g")
        .set("spark.driver.memory", "2g")
        .set("spark.num.executors", "3")
        .set("spark.cores.max", "20")
        .set("spark.executorEnv.LD_LIBRARY_PATH", "/root/anaconda3/envs/gdal-3.1.2/lib/:/usr/local/lib")
        .set("spark.dynamicAllocation.enabled","false")
        .set("spark.default.parallelism","600")
        .set("spark.repartitioning","true")
        .set("spark.sql.shuffle.partitions","600")

    implicit val sc = new SparkContext(conf)
    val Path="/geo/file/raster/RS/Landsat/L71149033_03320030531_B10.TIF"
    println("path",Path)
    val targetCRS = LatLng
    println("targetCRS",targetCRS)
    val method: PointResampleMethod = Bilinear
    val tilesize= 256 // 256
    val layoutScheme = FloatingLayoutScheme(tilesize)
    val raster_source_single=GDALRasterSource(Path)
    val raster_source=Seq(raster_source_single)
    val sourceRDD: RDD[RasterSource] =sc.parallelize(raster_source)

    val summary = RasterSummary.fromRDD(sourceRDD)
    val LayoutLevel(zoom, layout) = summary.levelFor(layoutScheme)
    val context_rdd: MultibandTileLayerRDD[SpatialKey] = RasterSourceRDD.tiledLayerRDD(sourceRDD, layout, KeyExtractor.spatialKeyExtractor, rasterSummary = summary.some)
    val sum_resu_rdd: RDD[Int] =context_rdd.map{ single_rdd=>
      single_rdd._2.band(0).toArray().sum

    }
    val sum_resu=sum_resu_rdd.collect()
    println("result: ",sum_resu)

    val endTime = System.currentTimeMillis
    println("total time is ",(endTime - startTime) / 1000,"s")
    sc.stop()
  }
}
  • Inputs
  • Actual output
  • encouter erroe when reading any tiff
  • Expected output
  • read the tif and output the sum value of exach tile

Expected behavior

A clear and concise description of what you expected to happen.

Screenshots

If applicable, add screenshots to help explain your problem.

Environment

CentOS Linux release 7.9.2009 (Core)

  • Java version:

  • java version "11.0.12" 2021-07-20 LTS
    Java(TM) SE Runtime Environment 18.9 (build 11.0.12+8-LTS-237)
    Java HotSpot(TM) 64-Bit Server VM 18.9 (build 11.0.12+8-LTS-237, mixed mode)

  • Scala version:

  • 2.12.8

  • GeoTrellis version:

  • 3.5.2

Additional context

Add any other context about the problem here.
bugreport.zip
bugreport2.zip
bugreport3.zip
bugreport4.zip

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the supplied Scala reproduction and the GDALRasterSource(Path) call, then inspect the GDALRasterSource entry point and the attached bug reports. Verify whether the TIFF failure occurs while constructing dataset dimensions and determine the environment or input condition that causes it. Done means the provided TIFF can be read and the per-tile sums are produced without the GDAL error.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.