spring-cloud / spring-cloud/spring-cloud-config

Support Additional spring.config.import Statements In EnvironmentRepositories

Open
#2,097 9 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement waiting for votes
Dominant language
Java
Stars
2k
Forks
1.3k
Avg merge
2d 59m
Merged PRs (30d)
16

Description

Describe the bug
We are using the spring cloud config server to load configuration into our spring boot app using git and hashicorp vault.
The configuration for the server is like this

spring:
  profiles:
    active: git,vault
  cloud:
    config:
      server:
        native:
        git:
          uri: https://our-git-uri
          searchPaths:
            - '{application}'
        vault:
          host: our-vault-host
          scheme: https
          port: 443
          authentication: our-authentication
          kv-version: 2

If we ask for the config for the application 'app' with the active profile 'prof' it will load the 'app-prof.yml' configuration from git as expected. If the 'app-prof.yml' configuration being loaded contains vault imports like the following

spring:
  config:
    import: >-
       vault://arbitrary/secret/path1
       vault://arbitrary/secret/path2

Then the Spring cloud config server loads the configuration successfully and then proceeds to import the properties from the vault paths specified in the configuration, which is exactly what we want.

** The single bug is this - I would expect the properties loaded from the vault paths to be included in the configuration returned to the application that requested the configuration, but they are not. **

I investigated and below are explained some of the reasons I believe the properties are not returned. I do not know what the desired functionality is of each part and so do not know if what I list below are bugs, however they do result in the bug presented above manifesting.

There seem to be two barriers to these properties being returned in the response to the configuration request, both of which contribute to the single bug.

The PassthruEnvironmentRepository only 'passes through' configuration in MapPropertySources but the vault properties are stored in EnumerablePropertySources. it is possible to change this behaviour with the following code change

in PassthruEnvironmentRepository::findOne change MapPropertySource to EnumerablePropertySource

			if (!this.standardSources.contains(name) && source instanceof EnumerablePropertySource<?>) {
				result.add(
						new PropertySource(name, getMap((EnumerablePropertySource<?>) source, includeOrigin), source));
			}

then PasstheuEnvironmentRepository::getMap can be simplified and still work the same

	private Map<?, ?> getMap(EnumerablePropertySource<?> source, boolean includeOrigin) {
		Map<Object, Object> map = new LinkedHashMap<>();
		if (includeOrigin && source instanceof OriginLookup) {
			Map<?, ?> input = (Map<?, ?>) source.getSource();
			OriginLookup<String> originLookup = (OriginLookup<String>) source;
			for (Object key : input.keySet()) {
				Origin origin = originLookup.getOrigin(key.toString());
				if (origin == null) {
					map.put(key, source.getProperty(key.toString()));
					continue;
				}
				String originDesc;
				if (origin instanceof TextResourceOrigin) {
					TextResourceOrigin tro = (TextResourceOrigin) origin;
					originDesc = tro.getLocation().toString();
				}
				else {
					originDesc = origin.toString();
				}
				Object value = source.getProperty(key.toString());
				map.put(key, new PropertyValueDescriptor(value, originDesc));
			}
		}
		else {
			for (String key : source.getPropertyNames()) {
				// Spring Boot wraps the property values in an "origin" detector, so we
				// need
				// to extract the string values
				map.put(key, source.getProperty(key));
			}
		}
		return map;
	}

As MapPropertySource extends EnumerablePropertySource this is backwards compatible and does not change the normal functionality, but now the properties returned from the arbitrary vault path import are available in the property source map.

The second barrier I have found is that the arbitrary vault paths being returned are removed by the NativeEnvironmentRepository. Even though we do not have the native profile enabled the NativeEnvironmentRepository is used to filter out what is returned. the NativeEnvironmentRepository::matchesLocation method does not return true for the arbitrary vault paths and I cannot fathom a clean way to make this actually happen. This is where I got stuck.

I am very happy with the way that the cloud config server chooses which configuration files to load but it we have a requirement that each active profile may load up different properties from arbitrary vault paths that have nothing to do with the application or profile of the configuration file that they are specifies in. We do not even control those paths on the app side.

It is very frustrating that it so nearly works exactly as we want, but that the properties get filtered out at the end because they are not linked to the active profile in the same way.

We would like to use the cloud-config-server as a one-stop-shop for loading configuration but this is stopping us.

Can this be fixed?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with PassthruEnvironmentRepository::findOne and getMap to trace how imported EnumerablePropertySources are converted, then inspect NativeEnvironmentRepository::matchesLocation to understand why arbitrary vault paths are filtered. Done means configuration imported from multiple arbitrary vault paths is included in the response while existing configuration filtering remains intact.

Written by the indexing model from the issue text.

Assessment

Tech stack
git, java, spring, spring-boot
Domain
backend, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.