Azure / Azure/elastic-db-tools

Make it easy to open a MultiShardConnection to shards based on cached mappings (and automatically refresh the cache if it's out of date)

Open
#113 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C#
Stars
147
Forks
108
PR merge metrics
No merged PRs in 30d

Description

We have a customer who wants to do the following:
1. Receive an input list of shard keys
2. Lookup the shards that those keys are mapped to, preferring to lookup in the cache first (see #112). Potentially also apply some customization to the lookup strategy (e.g.: if a key is unmapped, then treat it as if it was mapped to a predefined 'default shard' instead)
3. Open a MultiShardConnection to those shards

The difficulty is that the cached mappings that are used in step 2 can be out of date. If the customer just gets the shards and uses the `MultiShardConnection(IEnumerable)` constructor, then there is validation of the shards or mappings, meaning that the app could end up connecting to the wrong shards.

The initial proposal (detailed in https://github.com/Azure/elastic-db-tools/issues/112#issuecomment-180118252 sections 2 & 3) was to make a new `MultiShardConnection(IEnumerable, ConnectionOptions)` constructor that validates the shards (which therefore validates the mappings on those shards). However we felt that this design was problematic because:
1. It's not intuitive what is being validated
2. Users would need to write their own code that handles validation failures and redoes the lookup. This gets ugly quickly. This can't be implemented inside of the new `MultiShardConnection(IEnumerable, ConnectionOptions)` because the knowledge of the input shard keys has been lost so there is no way to refresh the mappings.

Our new proposal is to add new constructors:
`MultiShardConnection(IEnumerable shardKeys, ListShardMap shardMap)`
`MultiShardConnection(IEnumerable shardKeys, RangeShardMap shardMap)`

These constructors will lookup cached mappings to determine what shards the keys are on, attempt to connect, and if validation fails then the appropriate mappings will be refreshed from the shard map and then we will re-attempt to connect and validate.

Sample user code:

```
List keys = { 1, 2, 3 };
RangeShardMap shardMap = ...
MultiShardConnection conn = new MultiShardConnection(keys, shardMap);
```

If users wish to customize the lookup strategy, they can do so by tweaking the list of keys before providing them to MultiShardConnection. For example:

```
// Using ConcurrentDictionary as a Concurrent Hash Set, see http://stackoverflow.com/questions/18922985/concurrent-hashsett-in-net-framework
// Alternatively use a regular HashSet and use lock( )
private static ConcurrentDictionary _unmappedKeyCache = ...

// Special sentinel key that has been set up to map to DefaultShard
private static TKey _keyForDefaultShard = ...

private MultiShardConnection LookupKeysAndOpenConnection(IEnumerable keys)
{
ISet processedKeys = new HashSet();

// If any key is unmapped, then treat it as if it was mapped to DefaultShard
// This knowledge can be cached
foreach (int key in keys)
{
if (unmappedKeyCache.Contains(key))
{
// We know that it's unmapped, so replace it with _keyForDefaultShard
processedKeys.Add(_keyForDefaultShard);
}
else
{
// It might be mapped, we don't know. Look it up (using cache first)
ListMapping mapping;
bool found = shardMap.TryGetMappingForKey(k, LookupOptions.LookupInCache | LookupOptions.LookupInStorage, out mapping); // see #112
if (!found)
{
// Remember that this key is unmapped
unmappedKeyCache.TryAdd(k, /* dummy value */ false);

// Since it's unmapped, replace it with _keyForDefaultShard
processedKeys.Add(_keyForDefaultShard);
}
else
{
// Mapping was found - keep the key as-is
processedKeys.Add(key);
}
}

// Now that we've replaced the keys that should go to the default shard, create the MultiShardConnection.
// This will internally look up the mappings again and automatically refresh any outdated mappings.
// It will validate that it connect to the right shard - no difficult user code required.
return new MultiShardConnection(processedKeys, shardMap, connectionString);
}
}

// Run the below every 15 minutes
private void ResetUnmappedCache()
{
unmappedKeyCache.Clear();
}
```

Note that in this solution, the keys are looked up twice - once by the user-defined key fixup algorithm, and once (or more) inside MultiShardConnection. I believe that this is acceptable because the lookup is cached and should be quick (and only uses local compute resources so it will scale well with number of app instances).

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.