Implement basic Bulk Data Access export server

Implement a new Spring Boot server application within the server directory that provides FHIR bulk data export capabilities. The server will be built using HAPI FHIR and will include authorization support based on the design from the pre-version 8 Pathling server.

Project structure and dependencies

Location and build configuration

The server will live in the server directory with its own pom.xml file, independent from the root project. It will use Maven for dependency management and building, with versioning independent of the Pathling library. Initial version will be 1.0.0-SNAPSHOT.

Dependencies

• library-runtime module from Pathling v8.0.0 (the only Pathling dependency)
• HAPI FHIR framework for building the FHIR server
• Spring Boot for application infrastructure
• Dependencies required for JWT validation and OpenID Connect integration

The server will not depend on any other Pathling modules or non-public APIs. All bulk export functionality and authorization logic will be implemented within the server module. Any changes to the library API should be of general applicability to Pathling library users and not specific to the bulk export use case.

Server architecture

The server will use HAPI FHIR to create a servlet mounted using the @WebServlet annotation, inheriting some structure and functionality from the original Pathling server (v7).

Scope

In scope

• System-level export (/$export)
• NDJSON output format
• _type parameter for filtering resource types
• Bulk data status request and output manifest (required fields only)
• Capability statement requirements
• Authorization (JWT bearer tokens with OpenID Connect)
• _since parameter for filtering by last updated date
• _until parameter for filtering by last updated date
• _elements parameter for limiting returned fields

Out of scope

• _includeAssociatedData parameter
• _typeFilter parameter
• organizeOutputBy parameter
• allowPartialManifests parameter
• Binary to DocumentReference conversion
• Group-level export
• Bulk Cohort API
• Deletion bundles
• _list parameter

Implementation requirements

Bulk data export operation

Kick-off request

Implement the system-level export endpoint (GET /$export) that:

• Accepts optional _type, _since, _until, and _elements parameters
• Validates bearer token and checks for required authorities
• Initiates an asynchronous export job
• Returns HTTP 202 (Accepted) with a Content-Location header pointing to the status endpoint

Status endpoint

Implement the status polling endpoint that:

• Returns HTTP 202 (Accepted) while the export is in progress, with optional X-Progress header
• Returns HTTP 200 (OK) when complete, with a JSON manifest containing:
  • transactionTime: timestamp when the export was initiated
  • request: the original export request URL
  • requiresAccessToken: boolean indicating if access tokens are needed to download files
  • output: array of file items with type (resource type) and url (download location)
  • error: array for any OperationOutcome files (may be empty if no errors occurred)

Output files

Generate NDJSON files containing the exported resources:

• One file per resource type (or split appropriately for large exports)
• Filtered according to _type, _since, _until, and _elements parameters
• Filtered according to user’s read authorities
• Apply _elements parameter to limit fields in output

File downloads

Provide endpoints for downloading the generated NDJSON files referenced in the manifest. Download endpoints will be served from the same endpoint as the bulk server and proxy back to data stored by Hadoop, following the same pattern as the version 7 server.

Authorization

Token validation

The system must validate bearer tokens using the OpenID Connect framework:

• Token must be a valid JSON Web Token (JWT)
• Token must contain an audience claim matching the configured value
• Token must contain an issuer claim matching the configured value
• Issuer must provide an OpenID Connect Discovery endpoint for token validation

Authorization model

The bulk export operation will use the following authority structure:

graph TD
    export[pathling:export]
    pathling[pathling]
    read[pathling:read]
    readResource[pathling:read:ResourceType]
    
    pathling --> export
    pathling --> read
    read --> readResource
    
    style pathling fill:#e1bee7
    style export fill:#bbdefb
    style read fill:#fff9c4
    style readResource fill:#fff9c4

• pathling - grants access to all operations and resources, implies all other authorities
• pathling:export - grants access to the export operation
• pathling:read - grants read access to all resource types
• pathling:read:[resource type] - grants read access to a specific resource type (e.g., pathling:read:Patient)

To use the export operation, users must have both the pathling:export authority and at least one read authority.

Export behavior

The system should filter exported resources based on user permissions:

Default export (no resource types specified)

When a user initiates an export without specifying resource types:

• Export only resource types for which the user has read authority
• Log a warning message for any resource types that exist in the dataset but were excluded due to insufficient permissions
• Return a successful response (HTTP 200) when polling the status endpoint

Explicit resource type request

When a user requests specific resource types in their export:

• Export only the requested resource types for which the user has read authority
• Log a warning message for any requested resource types that were excluded due to insufficient permissions
• Return a successful response (HTTP 200) when polling the status endpoint

Logging format

When resource types are excluded due to authorization, log a warning message that clearly indicates:

• Which resource type was excluded
• The reason for exclusion (insufficient permissions)

This approach maintains consistency across both scenarios while providing visibility through server logs about what was filtered.

Implementation notes

• Authorization implementation will be based on the design from the pre-version 8 server and may reuse some of the old code
• Configuration should support specifying the expected audience and issuer values
• Configuration should support the OpenID Connect Discovery endpoint URL
• Token validation should use standard JWT libraries
• Consider whether authority checking logic can be modularized for potential future operations
• Use HAPI FHIR’s resource provider and operation patterns
• Follow Spring Boot conventions for configuration and dependency injection
• Structure the code to support adding additional bulk data features in the future