OpenAPITools / OpenAPITools/openapi-generator

[REQ] [Python] In-memory upload for `multipart/form-data`

Open
#17,148 2 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Enhancement: Feature
Dominant language
Java
Stars
26.8k
Forks
7.7k
PR merge metrics
PR metrics pending

Description

Is your feature request related to a problem? Please describe.

Example description:

openapi: 3.0.3
info:
  title: Multipart
  version: 0.0.1
servers:
  - url: http://localhost:4200
paths:
  /upload:
    post:
      tags:
      - Uploads
      summary: Uploads files.
      description: Uploads files.
      operationId: upload_handler
      requestBody:
        content:
          multipart/form-data:
            schema:
              type: object
              required:
              - files[]
              properties:
                files[]:
                  type: array
                  items:
                    type: string
                    format: binary
        required: true
      responses:
        '200':
          description: Yay
      security:
      - session_token: []

For generation, I use docker.io/openapitools/openapi-generator-cli:v7.0.1 with -g python --additional-properties=useOneOfDiscriminatorLookup=true.

The relevant parts are:

  1. uploads_api.py:

    • def upload_handler(self, files : conlist(Union[StrictBytes, StrictStr]), **kwargs) -> None:
         …
      
      def upload_handler_with_http_info(self, files : conlist(Union[StrictBytes, StrictStr]), **kwargs) -> ApiResponse:
         …
         if _params['files']:
              _files['files[]'] = _params['files']
         …
      
  2. api_client.py:

    • def __call_api(
             self, resource_path, method, path_params=None,
             query_params=None, header_params=None, body=None, post_params=None,
             files=None, response_types_map=None, auth_settings=None,
             _return_http_data_only=None, collection_formats=None,
             _preload_content=True, _request_timeout=None, _host=None,
             _request_auth=None):
      …
      
      # post parameters
              if post_params or files:
                  post_params = post_params if post_params else []
                  post_params = self.sanitize_for_serialization(post_params)
                  post_params = self.parameters_to_tuples(post_params,
                                                          collection_formats)
                  post_params.extend(self.files_parameters(files))
      
      …
      
      def files_parameters(self, files=None):
          """Builds form parameters.
      
          :param files: File parameters.
          :return: Form parameters with files.
          """
          params = []
      
          if files:
              for k, v in files.items():
                  if not v:
                      continue
                  file_names = v if type(v) is list else [v]
                 for n in file_names:
                      with open(n, 'rb') as f:
                          filename = os.path.basename(f.name)
                          filedata = f.read()
                          mimetype = (mimetypes.guess_type(filename)[0] or
                                      'application/octet-stream')
                          params.append(
                              tuple([k, tuple([filename, filedata, mimetype])]))
      
          return params
      

This means, it is only possible to put a filename into upload_handler as a string, since at some point in files_parameters is is passed to open.

Describe the solution you'd like

It would be nice if (filename, byte) tuples are supported, since there is no reason to enforce using an existing file from the file system.

Describe alternatives you've considered

When you have something in memory, you have to go with tempfile, but this means writing to disk.

Additional context

We could change file_parameters to something like this:

    def files_parameters(self, files=None):
        """Builds form parameters.

        :param files: File parameters.
        :return: Form parameters with files.
        """
        params = []

        if files:
            for k, v in files.items():
                if not v:
                    continue
                file_names = v if type(v) is list else [v]
                for n in file_names:
                    # Note: added support to upload data from RAM
                    if isinstance(n, tuple):
                        filename = n[0]
                        filedata = n[1]

                        mimetype = (mimetypes.guess_type(filename)[0] or
                        'application/octet-stream')
                        params.append(
                            tuple([k, tuple([filename, filedata, mimetype])]))

                        continue

                    with open(n, 'rb') as f:
                        filename = os.path.basename(f.name)
                        filedata = f.read()
                        mimetype = (mimetypes.guess_type(filename)[0] or
                                    'application/octet-stream')
                        params.append(
                            tuple([k, tuple([filename, filedata, mimetype])]))

        return params

Then, we could type upload_handler to something like this:

def upload_handler(self, files : conlist(Union[StrictBytes, StrictStr, Tuple[str, Union[StrictBytes, StrictStr]]]), **kwargs) -> AddCollection200Response:
    …

Having presented this, I have two questions:

  1. Did I miss something to use this API with in-memory bytes as files for multipart uploads?
  2. If not, how do you feel about changing the API to allow this?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read the generated uploads_api.py call path and api_client.py, especially files_parameters, to understand how multipart file values are assembled. Confirm the intended tuple shape and existing filename behavior, then define completion as allowing in-memory filename/data values without requiring a temporary filesystem file.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, tooling
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.