[RFC] Transformation Layer

Problem

Currently we are fetching data from many external services (firestore, github, discord service). We mostly change a few fields of the data according to our needs and send it to controller and eventually as API response.

A few problem arises with this -

Transformation of data happening at different places

/controller/auth.js

      userData = {
        github_id: user.username,
        github_display_name: user.displayName,
        github_created_at: Number(new Date(user._json.created_at).getTime()),
        created_at: Date.now(),
        updated_at: Date.now(),
      };

      const { userId, incompleteUserDetails } = await users.addOrUpdate(userData);

/models/discordactions.js

    return groups.map((group) => {
      const groupCreator = groupCreatorsDetails[group.createdBy];
      return {
        ...group,
        firstName: groupCreator?.first_name,
        lastName: groupCreator?.last_name,
        image: groupCreator?.picture?.url,
        memberCount: roleIdToCountMap[group.roleid] || 0, // Number of users joined this group
        isMember: subscribedGroupIds.has(group.roleid), // Is current loggedIn user is a member of this group
      };
    });

No reliable way to ensure no sensitive data is sent.

We mostly return the data object to the frontend as we are given by firestore. Private fields need to be handled with more conditions throughout the codebase.

The below solution helps but is not reliable as - New private fields can be added to user as well as at other models.

/services/dataAccessLayer.js

const removeSensitiveInfo = function (obj, level = ACCESS_LEVEL.PUBLIC) {
  for (let i = 0; i < KEYS_NOT_ALLOWED[level].length; i++) {
    if (Object.prototype.hasOwnProperty.call(obj, KEYS_NOT_ALLOWED[level][i])) {
      delete obj[KEYS_NOT_ALLOWED[level][i]];
    }
  }
};

No fixed place to do date parsing or other parsing

/models/extensionRequest.js

    const request = {
      timestamp: Number((new Date().getTime() / 1000).toFixed(0)),
      ...extensionRequestData,
    };
    return await extensionRequestsModel.add(request);

Proposed Solution

• A top level folder called transformers with files according to the models we are transforming
  Eg - /transformers/user.js, /transformers/goal.js
• It would contain functions for sole of transforming data
  Eg - transformUserFromFirestore(), transformGoalsFromAPI()
• A transformation function would look something like this

export const transformGoalFromApi = (goal: any): Goal => {
  const attributes = goal.attributes

  return {
    id: goal.id,
    title: attributes.title,
    description: attributes.description,
    createdAt: new Date(attributes.created_at),
    createdBy: new Date(attributes.created_by),
    startsOn: attributes.starts_on,
    endsOn: attributes.ends_on,
    percentageCompleted: attributes.percentage_completed,
    assignedBy: attributes.assigned_by,
    status: attributes.status,
    assignedTo: attributes.assigned_to
  }
}

• This function is used inside models after edge cases has been handled

Benifits

• Standard data object for passing around.
• Reliable way to ensure no private fields sent via API response.
• Change in API response structure/field name would only require change in this file.
• Central location for all data transformation/parsing.
• Removes response ambiguity from the developer. (Developer can check what fields we are getting from API via this function)
• Typescript integration is made much simpler.

Demerits

• Changing might break features.
• Lots of work.
  Thankfully, this can done in a step by step process as show in the migration plan

Migration Plan

Alternate Solution

It can be called serializer instead.
Ember data serializer does this functionality as mentioned here
(To be discussed more)