locationtech / locationtech/geotrellis
Cannot read the Tiff file by GDALRasterSource. Unable to construct dataset dimensions. GDAL Error Code: 4
Nobody has claimed this yet.
- Dominant language
- Scala
- Stars
- 1.4k
- Forks
- 360
- Avg merge
- 6h 57m
- Merged PRs (30d)
- 12
Description
Describe the bug
Cannot read the Tiff file by GDALRasterSource. Unable to construct dataset dimensions. GDAL Error Code: 4
To Reproduce
Provide as able:
- Steps to reproduce the behavior
- Code example
package com.example.gdalread
import cats.syntax.option._
import geotrellis.layer.{FloatingLayoutScheme, KeyExtractor, LayoutLevel, SpatialKey}
import geotrellis.proj4.LatLng
import geotrellis.raster.RasterSource
import geotrellis.raster.gdal.GDALRasterSource
import geotrellis.raster.resample.{Bilinear, PointResampleMethod}
import geotrellis.spark.{MultibandTileLayerRDD, RasterSourceRDD, RasterSummary}
import org.apache.log4j.{Level, Logger}
import org.apache.spark.rdd.RDD
import org.apache.spark.{SparkConf, SparkContext}
import geotrellis.raster.gdal._
import com.azavea.gdal._
object gdal_read_test {
def main(args: Array[String]): Unit = {
Logger.getLogger("org").setLevel(Level.ERROR)
System.load("/root/anaconda3/envs/gdal-3.1.2/lib/libgdal.so.27")
System.getProperty("java.library.path")
GDALWarp.init(100)
print("enter country_pop_sgdal_read_test tatus_____________21.12________________")
var startTime = System.currentTimeMillis();
implicit val conf =
new SparkConf()
.setAppName("gdal_read_test")
.setMaster("spark://master:7077")
.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
.set("spark.kryo.registrator", "geotrellis.spark.store.kryo.KryoRegistrator")
.set("spark.executor.cores", "6")
.set("spark.executor.memory", "4g")
.set("spark.driver.memory", "2g")
.set("spark.num.executors", "3")
.set("spark.cores.max", "20")
.set("spark.executorEnv.LD_LIBRARY_PATH", "/root/anaconda3/envs/gdal-3.1.2/lib/:/usr/local/lib")
.set("spark.dynamicAllocation.enabled","false")
.set("spark.default.parallelism","600")
.set("spark.repartitioning","true")
.set("spark.sql.shuffle.partitions","600")
implicit val sc = new SparkContext(conf)
val Path="/geo/file/raster/RS/Landsat/L71149033_03320030531_B10.TIF"
println("path",Path)
val targetCRS = LatLng
println("targetCRS",targetCRS)
val method: PointResampleMethod = Bilinear
val tilesize= 256 // 256
val layoutScheme = FloatingLayoutScheme(tilesize)
val raster_source_single=GDALRasterSource(Path)
val raster_source=Seq(raster_source_single)
val sourceRDD: RDD[RasterSource] =sc.parallelize(raster_source)
val summary = RasterSummary.fromRDD(sourceRDD)
val LayoutLevel(zoom, layout) = summary.levelFor(layoutScheme)
val context_rdd: MultibandTileLayerRDD[SpatialKey] = RasterSourceRDD.tiledLayerRDD(sourceRDD, layout, KeyExtractor.spatialKeyExtractor, rasterSummary = summary.some)
val sum_resu_rdd: RDD[Int] =context_rdd.map{ single_rdd=>
single_rdd._2.band(0).toArray().sum
}
val sum_resu=sum_resu_rdd.collect()
println("result: ",sum_resu)
val endTime = System.currentTimeMillis
println("total time is ",(endTime - startTime) / 1000,"s")
sc.stop()
}
}
- Inputs
- Actual output
- encouter erroe when reading any tiff
- Expected output
- read the tif and output the sum value of exach tile
Expected behavior
A clear and concise description of what you expected to happen.
Screenshots
If applicable, add screenshots to help explain your problem.
Environment
CentOS Linux release 7.9.2009 (Core)
-
Java version:
-
java version "11.0.12" 2021-07-20 LTS
Java(TM) SE Runtime Environment 18.9 (build 11.0.12+8-LTS-237)
Java HotSpot(TM) 64-Bit Server VM 18.9 (build 11.0.12+8-LTS-237, mixed mode) -
Scala version:
-
2.12.8
-
GeoTrellis version:
-
3.5.2
Additional context
Add any other context about the problem here.
bugreport.zip
bugreport2.zip
bugreport3.zip
bugreport4.zip
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the supplied Scala reproduction and the GDALRasterSource(Path) call, then inspect the GDALRasterSource entry point and the attached bug reports. Verify whether the TIFF failure occurs while constructing dataset dimensions and determine the environment or input condition that causes it. Done means the provided TIFF can be read and the per-tile sums are produced without the GDAL error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100