rsyslog / rsyslog/libfastjson

Architecture for to-string conversion

Open
#119 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C
Stars
61
Forks
35
Avg merge
21m
Merged PRs (30d)
2

Description

Hi Rainer,

We have some issues with converting the json tree to a string from multiple threads at the same time. I think the problem is with the printbuf member that is being used by two threads. We will probably be able to work around this, but after looking at the code, I think it is a peculiar design that the output buffer is part of the json_object.

What would you think of adding the following functions to libfastjson:

/**
 * Signature of a custom print function that can be implemented by the user.
 * @param ptr user-supplied pointer
 * @param buffer buffer to be written
 * @param size size of the buffer
 * @return size_t number of bytes written
 */
typedef size_t (fjson_print_fn)(void *ptr, const char *buffer, size_t size);

/**
 * Write the json tree to a user-supplied function
 * @param jso the fjson_object instance
 * @param func the user supplied function to which all data will be passed
 * @param ptr user-supplied pointer that is passed as first param to the callback
 * @returns the number of bytes written
 */
extern size_t fjson_object_write(struct fjson_object *jso, fjson_print_fn *func, void *ptr);

/**
 * Extended write function that allows extra flags to be passed
 * @param jso the fjson_object instance
 * @param flags formatting options, see FJSON_TO_STRING_PRETTY and other constants
 * @param func the user supplied function to which all data will be passed
 * @param ptr user-supplied pointer that is passed as first param to the callback
 * @returns the number of bytes written
 */
extern size_t fjson_object_write_ex(struct fjson_object *jso, int flags, fjson_print_fn *func, void *ptr);

This would allow users to supply their own output function, for example to write the json to a FILE*, or to write it to a fixed size buffer on the stack or to a dynamically growing buffer just like the current "struct printbuf" does. In fact, the "printbuf" implementation could be implemented on top of these functions.

This will often also be a performance improvement because now we're always stuck with the dynamically growing printbuf -- even if we have a pretty good idea how big the output is going to be, or when we want to write to a file instead of to a dynamically allocated buffer.

Are you willing to accept a pull request that implements this?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the current json_object-to-string path and its struct printbuf implementation, then review the proposed fjson_object_write and fjson_object_write_ex entry points. Done means user callbacks can receive serialized output, existing printbuf behavior can be built on the callback API, and concurrent conversions no longer depend on an object-owned output buffer.

Written by the indexing model from the issue text.

Assessment

Tech stack
c
Domain
backend-api-design
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.