Nexus PyPi repositories support

Nobody has claimed this yet.

Basically, that's not a feature request - rather a success story. Don't know if dirty hacks, used to add Nexus support are worth adding to upstream, so wanted to describe everything first.

**Original task**
Build and deploy tensorflow-based artifacts with Gradle

**Problems:**
- [Broken PyPi packages] packages with unresolvable/missing/incorrect dependencies (most scientific related)
- [Missing support for custom (ivy) artifacts in twine] The simplest way to deploy converted with ivy-importer artifacts is twine (`find ${repo} -type f -exec twine upload -r nexus {} \;`). But twine supports only certain artifact extensions.
- [Nexus breaks python naming convention] https://issues.sonatype.org/browse/NEXUS-12075
- [Gradle does not expose API for custom repositories] https://github.com/gradle/gradle/issues/1400

**Solutions:**
- Added lenient import for pivy-importer (https://github.com/linkedin/pygradle/pull/185). It helped with a large set of transient dependencies with the only small percent of broken packages. Broken were imported after fixing them manually.
- Added support for `ivy` artifact types to twine. https://github.com/pypa/twine/pull/300
- Created Gradle plugin to address naming convention issue - to replace `.` with `-` and to convert to lower case module part of a link path.

Plugin code is ugly, uses reflection and patches internal API of Gradle repository management, but it works.

Structure:
```text
buildSrc
|_src
|_main
|_groovy
|_package
|-NexusPyPiPlugin
|-NexusPyPiRepository
|-NexusPyPiResolver
|_NexusPyPiResourcePattern
```
**Sources:**
```groovy
class NexusPyPiPlugin implements Plugin {
static final String nexusModulePattern = '[nexus_module]'

@Override
void apply(Project project) {
DefaultRepositoryHandler handler = project.repositories
Field repositoryFactoryField = DefaultRepositoryHandler.getDeclaredField('repositoryFactory')
repositoryFactoryField.setAccessible(true)
DefaultBaseRepositoryFactory repositoryFactory = repositoryFactoryField.get(handler) as DefaultBaseRepositoryFactory
Field instantiatorField = DefaultBaseRepositoryFactory.getDeclaredField('instantiator')
instantiatorField.setAccessible(true)

// Add a method nexusPyPi to the repositories that uses our subclass of the ivy repository.
handler.metaClass.nexusPypi = { Closure closure ->
NexusPyPiRepository artifactRepository = new NexusPyPiRepository(
repositoryFactory.fileResolver,
repositoryFactory.transportFactory,
repositoryFactory.locallyAvailableResourceFinder,
repositoryFactory.artifactFileStore,
repositoryFactory.externalResourcesFileStore,
repositoryFactory.createAuthenticationContainer(),
repositoryFactory.ivyContextManager,
repositoryFactory.moduleIdentifierFactory,
repositoryFactory.instantiatorFactory,
repositoryFactory.fileResourceRepository,
repositoryFactory.metadataParser,
repositoryFactory.experimentalFeatures,
repositoryFactory.ivyMetadataFactory
)
artifactRepository.layout("pattern", new Action() {
@Override
void execute(IvyPatternRepositoryLayout repositoryLayout) {
repositoryLayout.artifact(nexusModulePattern + '/[revision]/[module]-[revision].[ext]')
repositoryLayout.ivy(nexusModulePattern + '/[revision]/[module]-[revision].ivy')
repositoryLayout.setM2compatible(true)
}
})
handler.addRepository(artifactRepository, 'pypi', ConfigureUtil.configureUsing(closure))
}
}
}
```

```groovy
class NexusPyPiRepository extends DefaultIvyArtifactRepository {
private static final Object[] NO_PARAMS = new Object[0]
private final
static Factory NO_METADATA_SUPPLIER = new Factory() {
@Override
ComponentMetadataSupplier create() {
return null
}
}

private final InstantiatorFactory instantiatorFactory
private final FileStore externalResourcesFileStore
private final RepositoryTransportFactory transportFactory
private final FileResolver fileResolver
private final LocallyAvailableResourceFinder locallyAvailableResourceFinder
private final FileStore artifactFileStore
private final AuthenticationContainer authenticationContainer
private final IvyContextManager ivyContextManager
private final ImmutableModuleIdentifierFactory moduleIdentifierFactory
private final ModuleMetadataParser metadataParser
private final IvyArtifactRepository.MetadataSources metadataSources
private final IvyMutableModuleMetadataFactory metadataFactory

private final FileResourceRepository fileResourceRepository
private Class componentMetadataSupplierClass
private Object[] componentMetadataSupplierParams

NexusPyPiRepository(
FileResolver fileResolver,
RepositoryTransportFactory transportFactory,
LocallyAvailableResourceFinder locallyAvailableResourceFinder,
FileStore artifactFileStore,
FileStore externalResourcesFileStore,
AuthenticationContainer authenticationContainer,
IvyContextManager ivyContextManager,
ImmutableModuleIdentifierFactory moduleIdentifierFactory,
InstantiatorFactory instantiatorFactory,
FileResourceRepository fileResourceRepository,
ModuleMetadataParser metadataParser,
ExperimentalFeatures experimentalFeatures,
IvyMutableModuleMetadataFactory metadataFactory) {
super(
fileResolver,
transportFactory,
locallyAvailableResourceFinder,
artifactFileStore,
externalResourcesFileStore,
authenticationContainer,
ivyContextManager,
moduleIdentifierFactory,
instantiatorFactory,
fileResourceRepository,
metadataParser,
experimentalFeatures,
metadataFactory)
this.metadataFactory = metadataFactory
this.metadataParser = metadataParser
this.fileResourceRepository = fileResourceRepository
this.moduleIdentifierFactory = moduleIdentifierFactory
this.ivyContextManager = ivyContextManager
this.authenticationContainer = authenticationContainer
this.artifactFileStore = artifactFileStore
this.locallyAvailableResourceFinder = locallyAvailableResourceFinder
this.fileResolver = fileResolver
this.transportFactory = transportFactory
this.externalResourcesFileStore = externalResourcesFileStore
this.instantiatorFactory = instantiatorFactory

Field metadataSourcesField = DefaultIvyArtifactRepository.getDeclaredField('metadataSources')
metadataSourcesField.setAccessible(true)
metadataSources = metadataSourcesField.get(this) as IvyArtifactRepository.MetadataSources
}

@Override
protected IvyResolver createRealResolver() {
URI uri = getUrl()

Field layoutField = DefaultIvyArtifactRepository.getDeclaredField('layout')
layoutField.setAccessible(true)
AbstractRepositoryLayout layout = layoutField.get(this) as AbstractRepositoryLayout

Set schemes = new LinkedHashSet()

Field additionalPatternsLayoutField = DefaultIvyArtifactRepository.getDeclaredField('additionalPatternsLayout')
additionalPatternsLayoutField.setAccessible(true)
AbstractRepositoryLayout additionalLayout = additionalPatternsLayoutField.get(this) as AbstractRepositoryLayout

layout.addSchemes(uri, schemes)
additionalLayout.addSchemes(uri, schemes)

IvyResolver resolver = createResolver(schemes)

if (uri != null) {
resolver.addArtifactLocation(uri, IVY_ARTIFACT_PATTERN)
resolver.addDescriptorLocation(uri, IVY_ARTIFACT_PATTERN)
}

layout.apply(uri, resolver)
additionalLayout.apply(uri, resolver)

return resolver
}

void setMetadataSupplier(Class ruleClass) {
this.componentMetadataSupplierClass = ruleClass
this.componentMetadataSupplierParams = NO_PARAMS
}

@Override
void setMetadataSupplier(Class rule, Action configureAction) {
DefaultActionConfiguration configuration = new DefaultActionConfiguration()
configureAction.execute(configuration)
this.componentMetadataSupplierClass = rule
this.componentMetadataSupplierParams = configuration.getParams()
}

private IvyResolver createResolver(Set schemes) {
if (schemes.isEmpty()) {
throw new InvalidUserDataException("You must specify a base url or at least one artifact pattern for an Ivy repository.")
}
return createResolver(transportFactory.createTransport(schemes, getName(), getConfiguredAuthentication()))
}

private IvyResolver createResolver(RepositoryTransport transport) {
Instantiator instantiator = createDependencyInjectingInstantiator(transport)
return new NexusPyPiResolver(getName(), transport, locallyAvailableResourceFinder,
getResolve().isDynamicMode(), artifactFileStore, moduleIdentifierFactory,
createComponentMetadataSupplierFactory(instantiator), createMetadataSources())
}

private ImmutableMetadataSources createMetadataSources() {
def sources = new ArrayList<>()
if (metadataSources.gradleMetadata) {
sources.add(new DefaultGradleModuleMetadataSource(metadataParser, metadataFactory, true))
}
if (metadataSources.ivyDescriptor) {
sources.add(new DefaultIvyDescriptorMetadataSource(IvyMetadataArtifactProvider.INSTANCE, createIvyDescriptorParser(), fileResourceRepository, moduleIdentifierFactory))
}
if (metadataSources.artifact) {
sources.add(new DefaultArtifactMetadataSource(metadataFactory))
}
return new DefaultImmutableMetadataSources(Collections.unmodifiableList(sources))
}

private MetaDataParser createIvyDescriptorParser() {
return new IvyContextualMetaDataParser(ivyContextManager, new IvyXmlModuleDescriptorParser(new IvyModuleDescriptorConverter(moduleIdentifierFactory), moduleIdentifierFactory, fileResourceRepository, metadataFactory))
}

/**
* Creates a service registry giving access to the services we want to expose to rules and returns an instantiator that
* uses this service registry.
* @param transport the transport used to create the repository accessor
* @return a dependency injecting instantiator, aware of services we want to expose
*/
private Instantiator createDependencyInjectingInstantiator(final RepositoryTransport transport) {
DefaultServiceRegistry registry = new DefaultServiceRegistry()
registry.addProvider(new Object() {
RepositoryResourceAccessor createResourceAccessor() {
return createRepositoryAccessor(transport)
}
})
return instantiatorFactory.inject(registry)
}

private RepositoryResourceAccessor createRepositoryAccessor(RepositoryTransport transport) {
return new ExternalRepositoryResourceAccessor(getUrl(), transport.getResourceAccessor(), externalResourcesFileStore)
}

private Factory createComponentMetadataSupplierFactory(final Instantiator instantiator) {
if (componentMetadataSupplierClass == null) {
return NO_METADATA_SUPPLIER
}

return new Factory() {
@Override
ComponentMetadataSupplier create() {
return instantiator.newInstance(componentMetadataSupplierClass, componentMetadataSupplierParams)
}
}
}
}
```

```groovy
class NexusPyPiResolver extends IvyResolver {

NexusPyPiResolver(
String name,
RepositoryTransport transport,
LocallyAvailableResourceFinder locallyAvailableResourceFinder,
boolean dynamicResolve,
FileStore artifactFileStore,
ImmutableModuleIdentifierFactory moduleIdentifierFactory,
Factory componentMetadataSupplierFactory,
ImmutableMetadataSources metadataSources) {
super(
name,
transport,
locallyAvailableResourceFinder,
dynamicResolve,
artifactFileStore,
moduleIdentifierFactory,
componentMetadataSupplierFactory,
metadataSources,
IvyMetadataArtifactProvider.INSTANCE)
}

@Override
void addArtifactLocation(URI baseUri, String pattern) {
addArtifactPattern(toResourcePattern(baseUri, pattern))
}

@Override
void addDescriptorLocation(URI baseUri, String pattern) {
addIvyPattern(toResourcePattern(baseUri, pattern))
}

private static ResourcePattern toResourcePattern(URI baseUri, String pattern) {
return new NexusPyPiResourcePattern(baseUri, pattern)
}
}
```

```groovy
class NexusPyPiResourcePattern extends M2ResourcePattern {
NexusPyPiResourcePattern(URI baseUri, String pattern) {
super(baseUri, pattern)
}

protected String substituteTokens(String pattern, Map attributes) {
String module = attributes.get("module");
if (module != null) {
attributes.put("nexus_module", module.replace(".", "-").toLowerCase());
}
return super.substituteTokens(pattern, attributes);
}
}
```

**Usage example:**
```groovy
apply plugin: "com.linkedin.python"
apply plugin: NexusPyPiPlugin

python {
forceVersion('pypi', 'pip', '9.0.1')
testDir = file('tests')
}

repositories {
nexusPypi {
url = "https://mynexus/repository/pypi-releases/packages/"
if (project.hasProperty('repository.username')) {
credentials.username = project.property('repository.username')
}
if (project.hasProperty('repository.password')) {
credentials.password = project.property('repository.password')
}
}
}
installPythonRequirements {
sorted = false
}

dependencies {
python 'pypi:tensorflow:1.5.0'
}
```

Start by reviewing the buildSrc/src/main/groovy/package classes named NexusPyPiPlugin, NexusPyPiRepository, NexusPyPiResolver, and NexusPyPiResourcePattern. Trace how the plugin integrates with Gradle's repository management and how Nexus naming and Ivy artifacts are handled. Done requires an agreed scope and working Nexus PyPi repository support without relying on the current undocumented hacks.

Written by the indexing model from the issue text.