MemberJunction / MemberJunction/MJ

Integration staging DDL drops declared field types: integer/decimal/boolean land as varchar(255)/text

Open
#3,527 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TSQL
Stars
29
Forks
6
Avg merge
1d 8h
Merged PRs (30d)
308

Description

## Symptom

A connector declares real types in its catalog, and the staging table is created with everything
as a string.

Observed on a PostgreSQL instance with the EO BMF connector:

| Field | Connector declares | Landed column |
|---|---|---|
| `Subsection`, `Affiliation`, `RulingYear` | `integer` | `character varying(255)` |
| `BmfAssetAmt`, `BmfRevenueAmt` | `decimal(19,2)` | `character varying(255)` |
| `IsAssociation` | `boolean` | `text` |

The declarations are correct in both the TS catalog (`EOBMFObjects.ts`) and the seeded
`IntegrationObjectField` rows.

## Where it is NOT

`packages/Integration/schema-builder/src/TypeMapper.ts:13-27` maps these correctly —
`integer → INTEGER`, `decimal → NUMERIC`, `boolean → BOOLEAN`. So the mapper is fine and the loss
happens upstream of it, somewhere in the RSU / `IntegrationApplyAll` discovery path that feeds it.

## Impact

- every downstream consumer must cast at the boundary before it can filter or aggregate
- numeric and date range filters against staging are wrong or impossible
- indexes on those columns are near-useless
- it is silent: no warning, no error, and the sync reports success

## Note

This is distinct from a connector declaring a type badly. We fixed one genuine instance of that
separately (`BmfSnapshotDate` really was declared as a bounded string). The six above are declared
correctly and still land as strings.

## Environment

MJ 5.51.0, PostgreSQL.

Contributor guide

Open the contributing guide

Research direction

Start with packages/Integration/schema-builder/src/TypeMapper.ts:13-27 to confirm the mappings, then trace the RSU / IntegrationApplyAll discovery path that supplies its input. Compare the declarations in EOBMFObjects.ts and the seeded IntegrationObjectField rows with the generated staging schema; done means the six declared integer, decimal, and boolean fields land with their PostgreSQL types rather than string types.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, typescript
Domain
database
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.