apache / apache/druid

HTML NPE in sampler if dimensionsSpecs is not set

Open
#8,934 0 comments 0 reactions 0 assignees View on GitHub
Area - Batch Ingestion Bug
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

### Affected Version

Latest master

### Description

NPE in sampler if missing dimensionsSpec

Submit:

```json
{
"type": "index",
"spec": {
"type": "index",
"ioConfig": {
"type": "index",
"inputSource": {
"type": "inline",
"data": "{\"time\":\"2015-09-12T00:46:58.771Z\",\"channel\":\"#en.wikipedia\",\"cityName\":null,\"comment\":\"added project\",\"countryIsoCode\":null,\"countryName\":null,\"isAnonymous\":false,\"isMinor\":false,\"isNew\":false,\"isRobot\":false,\"isUnpatrolled\":false,\"metroCode\":null,\"namespace\":\"Talk\",\"page\":\"Talk:Oswald Tilghman\",\"regionIsoCode\":null,\"regionName\":null,\"user\":\"GELongstreet\",\"delta\":36,\"added\":36,\"deleted\":0}\n{\"time\":\"2015-09-12T00:47:00.496Z\",\"channel\":\"#ca.wikipedia\",\"cityName\":null,\"comment\":\"Robot inserta {{Commonscat}} que enllaça amb [[commons:category:Rallicula]]\",\"countryIsoCode\":null,\"countryName\":null,\"isAnonymous\":false,\"isMinor\":true,\"isNew\":false,\"isRobot\":true,\"isUnpatrolled\":false,\"metroCode\":null,\"namespace\":\"Main\",\"page\":\"Rallicula\",\"regionIsoCode\":null,\"regionName\":null,\"user\":\"PereBot\",\"delta\":17,\"added\":17,\"deleted\":0}\n{\"time\":\"2015-09-12T00:47:05.474Z\",\"channel\":\"#en.wikipedia\",\"cityName\":\"Auburn\",\"comment\":\"/* Status of peremptory norms under international law */ fixed spelling of 'Wimbledon'\",\"countryIsoCode\":\"AU\",\"countryName\":\"Australia\",\"isAnonymous\":true,\"isMinor\":false,\"isNew\":false,\"isRobot\":false,\"isUnpatrolled\":false,\"metroCode\":null,\"namespace\":\"Main\",\"page\":\"Peremptory norm\",\"regionIsoCode\":\"NSW\",\"regionName\":\"New South Wales\",\"user\":\"60.225.66.142\",\"delta\":0,\"added\":0,\"deleted\":0}\n{\"time\":\"2015-09-12T00:47:08.770Z\",\"channel\":\"#vi.wikipedia\",\"cityName\":null,\"comment\":\"fix Lỗi CS1: ngày tháng\",\"countryIsoCode\":null,\"countryName\":null,\"isAnonymous\":false,\"isMinor\":true,\"isNew\":false,\"isRobot\":true,\"isUnpatrolled\":false,\"metroCode\":null,\"namespace\":\"Main\",\"page\":\"Apamea abruzzorum\",\"regionIsoCode\":null,\"regionName\":null,\"user\":\"Cheers!-bot\",\"delta\":18,\"added\":18,\"deleted\":0}\n{\"time\":\"2015-09-12T00:47:11.862Z\",\"channel\":\"#vi.wikipedia\",\"cityName\":null,\"comment\":\"clean up using [[Project:AWB|AWB]]\",\"countryIsoCode\":null,\"countryName\":null,\"isAnonymous\":false,\"isMinor\":false,\"isNew\":false,\"isRobot\":true,\"isUnpatrolled\":false,\"metroCode\":null,\"namespace\":\"Main\",\"page\":\"Atractus flammigerus\",\"regionIsoCode\":null,\"regionName\":null,\"user\":\"ThitxongkhoiAWB\",\"delta\":18,\"added\":18,\"deleted\":0}\n{\"time\":\"2015-09-12T00:47:13.987Z\",\"channel\":\"#vi.wikipedia\",\"cityName\":null,\"comment\":\"clean up using [[Project:AWB|AWB]]\",\"countryIsoCode\":null,\"countryName\":null,\"isAnonymous\":false,\"isMinor\":false,\"isNew\":false,\"isRobot\":true,\"isUnpatrolled\":false,\"metroCode\":null,\"namespace\":\"Main\",\"page\":\"Agama mossambica\",\"regionIsoCode\":null,\"regionName\":null,\"user\":\"ThitxongkhoiAWB\",\"delta\":18,\"added\":18,\"deleted\":0}\n{\"time\":\"2015-09-12T00:47:17.009Z\",\"channel\":\"#ca.wikipedia\",\"cityName\":null,\"comment\":\"/* Imperi Austrohongarès */\",\"countryIsoCode\":null,\"countryName\":null,\"isAnonymous\":false,\"isMinor\":false,\"isNew\":false,\"isRobot\":false,\"isUnpatrolled\":false,\"metroCode\":null,\"namespace\":\"Main\",\"page\":\"Campanya dels Balcans (1914-1918)\",\"regionIsoCode\":null,\"regionName\":null,\"user\":\"Jaumellecha\",\"delta\":-20,\"added\":0,\"deleted\":20}\n{\"time\":\"2015-09-12T00:47:19.591Z\",\"channel\":\"#en.wikipedia\",\"cityName\":null,\"comment\":\"adding comment on notability and possible COI\",\"countryIsoCode\":null,\"countryName\":null,\"isAnonymous\":false,\"isMinor\":false,\"isNew\":true,\"isRobot\":false,\"isUnpatrolled\":true,\"metroCode\":null,\"namespace\":\"Talk\",\"page\":\"Talk:Dani Ploeger\",\"regionIsoCode\":null,\"regionName\":null,\"user\":\"New Media Theorist\",\"delta\":345,\"added\":345,\"deleted\":0}\n{\"time\":\"2015-09-12T00:47:21.578Z\",\"channel\":\"#en.wikipedia\",\"cityName\":null,\"comment\":\"Copying assessment table to wiki\",\"countryIsoCode\":null,\"countryName\":null,\"isAnonymous\":false,\"isMinor\":false,\"isNew\":false,\"isRobot\":true,\"isUnpatrolled\":false,\"metroCode\":null,\"namespace\":\"User\",\"page\":\"User:WP 1.0 bot/Tables/Project/Pubs\",\"regionIsoCode\":null,\"regionName\":null,\"user\":\"WP 1.0 bot\",\"delta\":121,\"added\":121,\"deleted\":0}\n{\"time\":\"2015-09-12T00:47:25.821Z\",\"channel\":\"#vi.wikipedia\",\"cityName\":null,\"comment\":\"clean up using [[Project:AWB|AWB]]\",\"countryIsoCode\":null,\"countryName\":null,\"isAnonymous\":false,\"isMinor\":false,\"isNew\":false,\"isRobot\":true,\"isUnpatrolled\":false,\"metroCode\":null,\"namespace\":\"Main\",\"page\":\"Agama persimilis\",\"regionIsoCode\":null,\"regionName\":null,\"user\":\"ThitxongkhoiAWB\",\"delta\":18,\"added\":18,\"deleted\":0}\n{\"time\":\"2015-09-12T00:47:29.913Z\",\"channel\":\"#en.wikipedia\",\"cityName\":null,\"comment\":\"Blank stale warning(s) and replace with {{[[template:OW|OW]]}} using [[Project:AWB|AWB]]\",\"countryIsoCode\":null,\"countryName\":null,\"isAnonymous\":false,\"isMinor\":true,\"isNew\":false,\"isRobot\":false,\"isUnpatrolled\":false,\"metroCode\":null,\"namespace\":\"User talk\",\"page\":\"User talk:161.184.95.17\",\"regionIsoCode\":null,\"regionName\":null,\"user\":\"DavidLeighEllis\",\"delta\":0,\"added\":0,\"deleted\":0}\n{\"time\":\"2015-09-12T00:47:33.004Z\",\"channel\":\"#vi.wikipedia\",\"cityName\":null,\"comment\":\"clean up using [[Project:AWB|AWB]]\",\"countryIsoCode\":null,\"countryName\":null,\"isAnonymous\":false,\"isMinor\":false,\"isNew\":false,\"isRobot\":true,\"isUnpatrolled\":false,\"metroCode\":null,\"namespace\":\"Main\",\"page\":\"Atractus edioi\",\"regionIsoCode\":null,\"regionName\":null,\"user\":\"ThitxongkhoiAWB\",\"delta\":18,\"added\":18,\"deleted\":0}\n{\"time\":\"2015-09-12T00:47:35.776Z\",\"channel\":\"#vi.wikipedia\",\"cityName\":null,\"comment\":\"Lỗi CS1: ngày tháng\",\"countryIsoCode\":null,\"countryName\":null,\"isAnonymous\":false,\"isMinor\":true,\"isNew\":false,\"isRobot\":true,\"isUnpatrolled\":false,\"metroCode\":null,\"namespace\":\"Main\",\"page\":\"Agave gentryi\",\"regionIsoCode\":null,\"regionName\":null,\"user\":\"TuHan-Bot\",\"delta\":36,\"added\":36,\"deleted\":0}\n{\"time\":\"2015-09-12T00:47:37.881Z\",\"channel\":\"#vi.wikipedia\",\"cityName\":null,\"comment\":\"clean up using [[Project:AWB|AWB]]\",\"countryIsoCode\":null,\"countryName\":null,\"isAnonymous\":false,\"isMinor\":false,\"isNew\":false,\"isRobot\":true,\"isUnpatrolled\":false,\"metroCode\":null,\"namespace\":\"Main\",\"page\":\"Agama sankaranica\",\"regionIsoCode\":null,\"regionName\":null,\"user\":\"ThitxongkhoiAWB\",\"delta\":18,\"added\":18,\"deleted\":0}\n{\"time\":\"2015-09-12T00:47:42.090Z\",\"channel\":\"#vi.wikipedia\",\"cityName\":null,\"comment\":\"fix Lỗi CS1: ngày tháng\",\"countryIsoCode\":null,\"countryName\":null,\"isAnonymous\":false,\"isMinor\":true,\"isNew\":false,\"isRobot\":true,\"isUnpatrolled\":false,\"metroCode\":null,\"namespace\":\"Main\",\"page\":\"Apamea albertae\",\"regionIsoCode\":null,\"regionName\":null,\"user\":\"Cheers!-bot\",\"delta\":18,\"added\":18,\"deleted\":0}\n{\"time\":\"2015-09-12T00:47:44.963Z\",\"channel\":\"#ru.wikipedia\",\"cityName\":null,\"comment\":\"/* Донецкая Народная Республика */\",\"countryIsoCode\":null,\"countryName\":null,\"isAnonymous\":false,\"isMinor\":false,\"isNew\":false,\"isRobot\":false,\"isUnpatrolled\":false,\"metroCode\":null,\"namespace\":\"Main\",\"page\":\"Караман, Александр Акимович\",\"regionIsoCode\":null,\"regionName\":null,\"user\":\"Камарад Че\",\"delta\":0,\"added\":0,\"deleted\":0}\n{\"time\":\"2015-09-12T00:47:47.870Z\",\"channel\":\"#vi.wikipedia\",\"cityName\":null,\"comment\":\"clean up using [[Project:AWB|AWB]]\",\"countryIsoCode\":null,\"countryName\":null,\"isAnonymous\":false,\"isMinor\":false,\"isNew\":false,\"isRobot\":true,\"isUnpatrolled\":false,\"metroCode\":null,\"namespace\":\"Main\",\"page\":\"Atractus duboisi\",\"regionIsoCode\":null,\"regionName\":null,\"user\":\"ThitxongkhoiAWB\",\"delta\":18,\"added\":18,\"deleted\":0}\n{\"time\":\"2015-09-12T00:47:50.819Z\",\"channel\":\"#en.wikipedia\",\"cityName\":null,\"comment\":\"/* Films */\",\"countryIsoCode\":null,\"countryName\":null,\"isAnonymous\":false,\"isMinor\":false,\"isNew\":false,\"isRobot\":false,\"isUnpatrolled\":false,\"metroCode\":null,\"namespace\":\"Main\",\"page\":\"Keiynan Lonsdale\",\"regionIsoCode\":null,\"regionName\":null,\"user\":\"Lg16spears\",\"delta\":-11,\"added\":0,\"deleted\":11}\n{\"time\":\"2015-09-12T00:47:53.259Z\",\"channel\":\"#ja.wikipedia\",\"cityName\":null,\"comment\":\"/* 対戦通算成績と得失点 */\",\"countryIsoCode\":null,\"countryName\":null,\"isAnonymous\":false,\"isMinor\":true,\"isNew\":false,\"isRobot\":false,\"isUnpatrolled\":false,\"metroCode\":null,\"namespace\":\"Main\",\"page\":\"アルビレックス新潟の年度別成績一覧\",\"regionIsoCode\":null,\"regionName\":null,\"user\":\"BlueMoon2662\",\"delta\":14,\"added\":14,\"deleted\":0}\n{\"time\":\"2015-09-12T00:47:56.126Z\",\"channel\":\"#en.wikipedia\",\"cityName\":null,\"comment\":\"Bot updating unblock request table ([[en:WP:PEACHY|Peachy 2.0 (alpha 8)]])\",\"countryIsoCode\":null,\"countryName\":null,\"isAnonymous\":false,\"isMinor\":false,\"isNew\":false,\"isRobot\":true,\"isUnpatrolled\":false,\"metroCode\":null,\"namespace\":\"User\",\"page\":\"User:Cyberbot I/Requests for unblock report\",\"regionIsoCode\":null,\"regionName\":null,\"user\":\"Cyberbot I\",\"delta\":-74,\"added\":0,\"deleted\":74}"
},
"inputFormat": {
"type": "json"
}
},
"dataSchema": {
"dataSource": "sample",
"timestampSpec": {
"column": "time",
"format": "iso"
},
"transformSpec": {},
"granularitySpec": {
"queryGranularity": "HOUR"
},
"metricsSpec": []
},
"tuningConfig": {
"type": "index"
}
},
"samplerConfig": {
"numRows": 500,
"timeoutMs": 15000
}
}
```

Returns:

```

Error 500 Server Error

HTTP ERROR 500


Problem accessing /druid/indexer/v1/sampler. Reason:

    Server Error

Caused by:

java.lang.NullPointerException: inputRowParser

at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:229)
at org.apache.druid.segment.indexing.DataSchema.getDimensionsSpec(DataSchema.java:227)
at org.apache.druid.indexing.overlord.sampler.InputSourceSampler.buildReader(InputSourceSampler.java:200)
at org.apache.druid.indexing.overlord.sampler.InputSourceSampler.sample(InputSourceSampler.java:107)
at org.apache.druid.indexing.overlord.sampler.IndexTaskSamplerSpec.sample(IndexTaskSamplerSpec.java:94)
at org.apache.druid.indexing.overlord.sampler.SamplerResource.post(SamplerResource.java:41)
at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:286)
at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:276)
at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:181)
at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:120)
at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:135)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
at org.apache.druid.server.http.RedirectFilter.doFilter(RedirectFilter.java:73)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
at org.apache.druid.server.security.PreResponseAuthorizationCheckFilter.doFilter(PreResponseAuthorizationCheckFilter.java:82)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
at org.apache.druid.server.security.AllowOptionsResourceFilter.doFilter(AllowOptionsResourceFilter.java:75)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
at org.apache.druid.server.security.AllowAllAuthenticator$1.doFilter(AllowAllAuthenticator.java:84)
at org.apache.druid.server.security.AuthenticationWrappingFilter.doFilter(AuthenticationWrappingFilter.java:59)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
at org.apache.druid.server.security.SecuritySanityCheckFilter.doFilter(SecuritySanityCheckFilter.java:86)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1340)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1242)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:740)
at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:61)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.Server.handle(Server.java:503)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
at java.lang.Thread.run(Thread.java:748)


Powered by Jetty:// 9.4.12.v20180830

```

Filing this because HTML NPEs are always considered bugs in my book. Should be a JSON exception

Contributor guide

Open the contributing guide

Research direction

Start with DataSchema.getDimensionsSpec and InputSourceSampler.buildReader, then reproduce the issue using the sampler request in the report. Trace how the missing dimensionsSpec reaches the sampler and verify that the request no longer produces a NullPointerException or HTTP 500 response.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
api, backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.