HangfireIO / HangfireIO/Hangfire

Could not convert string to DateTime: 27.03.2023 07:30:00. Path 'CreatedAt', line 1, position 100.

Open
#2,183 9 comments 0 reactions 0 assignees View on GitHub
Dominant language
C#
Stars
10.1k
Forks
1.8k
Avg merge
1h 19m
Merged PRs (30d)
1

Description

We have the following setup:

2 Windows Server 2022 hosting a asp.net core site
2 Windows Server 2022 hosting two .net core windows services

each of the three applications is redundant on its two servers.

sporadically we get the following errors:

```
Newtonsoft.Json.JsonReaderException
Could not convert string to DateTime: 27.03.2023 07:30:00. Path 'CreatedAt', line 1, position 100.

Newtonsoft.Json.JsonReaderException: Could not convert string to DateTime: 27.03.2023 07:30:00. Path 'CreatedAt', line 1, position 100.
at Newtonsoft.Json.JsonReader.ReadDateTimeString(String s)
at Newtonsoft.Json.JsonTextReader.FinishReadQuotedStringValue(ReadType readType)
at Newtonsoft.Json.JsonTextReader.ReadStringValue(ReadType readType)
at Newtonsoft.Json.JsonTextReader.ReadAsDateTime()
at Newtonsoft.Json.JsonReader.ReadForType(JsonContract contract, Boolean hasConverter)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateObject(Object newObject, JsonReader reader, JsonObjectContract contract, JsonProperty member, String id)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
at Newtonsoft.Json.JsonConvert.DeserializeObject(String value, Type type, JsonSerializerSettings settings)
at Hangfire.Batches.BatchJobStorageExtensions.GetBatchState(IStorageConnection connection, String batchId)
at Hangfire.Batches.BatchJobSupportAttribute.OnStateElection(ElectStateContext context)
at Hangfire.States.StateMachine.InvokeOnStateElection(Tuple`2 x)
```

the error seems to appear only on jobs executed by the webservers, logging shows that it happens on both servers.
originally we used sql server storage, but we moved to redis and the issues go on.

interestingly the format of the CreatedAt field in the jobs is sometimes 1679880723201 and sometimes 2023-03-27T07:30:00.4875736Z but never the format that shows up in the exception.
but both formats sometimes work and throw the above exception from time to time.

also worth mentioning is that the failing jobs are part of a batch.

Contributor guide

Open the contributing guide

Research direction

Start at Hangfire.Batches.BatchJobStorageExtensions.GetBatchState and the Newtonsoft.Json deserialization path shown in the stack trace, comparing the batch jobs handled by the web servers with those handled by the Windows services. Check both SQL Server and Redis storage scenarios and the two observed CreatedAt formats; done means the sporadic batch-job DateTime conversion failure is explained and a reproducible correction path is identified.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp, redis
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.