turicas / turicas/rows

Detect data type from custom fields

Open
#271 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement help wanted plugin rows.fields rows.table
Dominant language
Python
Stars
886
Forks
137
PR merge metrics
No merged PRs in 30d

Description

I have this schema, with some custom fields:

import rows


class MoneyField(rows.fields.TextField):

    @classmethod
    def deserialize(cls, value):
        value = value.replace('.', '')\
                     .replace(',', '.')
        return super().deserialize(value)


class DocumentField(rows.fields.TextField):

    @classmethod
    def deserialize(cls, value):
        value = value.replace(' ', '')\
                     .replace('.', '')\
                     .replace('-', '').strip()
        return super().deserialize(value)

schema = OrderedDict([
        ('arquivoOrigem', rows.fields.TextField),
        ('codLegislatura', rows.fields.IntegerField),
        ('datEmissao', rows.fields.DatetimeField),
        ('ideDocumento', rows.fields.IntegerField),
        ('idecadastro', rows.fields.IntegerField),
        ('indTipoDocumento', rows.fields.IntegerField),
        ('nuCarteiraParlamentar', rows.fields.IntegerField),
        ('nuDeputadoId', rows.fields.IntegerField),
        ('nuLegislatura', rows.fields.IntegerField),
        ('numAno', rows.fields.IntegerField),
        ('numEspecificacaoSubCota', rows.fields.IntegerField),
        ('numLote', rows.fields.IntegerField),
        ('numMes', rows.fields.IntegerField),
        ('numParcela', rows.fields.IntegerField),
        ('numRessarcimento', rows.fields.IntegerField),
        ('numSubCota', rows.fields.IntegerField),
        ('sgPartido', rows.fields.TextField),
        ('sgUF', rows.fields.TextField),
        ('txNomeParlamentar', rows.fields.TextField),
        ('txtCNPJCPF', DocumentField),
        ('txtDescricao', rows.fields.TextField),
        ('txtDescricaoEspecificacao', rows.fields.TextField),
        ('txtFornecedor', rows.fields.TextField),
        ('txtNumero', rows.fields.TextField),
        ('txtPassageiro', rows.fields.TextField),
        ('txtTrecho', rows.fields.TextField),
        ('vlrDocumento', MoneyField),
        ('vlrGlosa', MoneyField),
        ('vlrLiquido', MoneyField),
        ('vlrRestituicao', MoneyField),
])

It works when I import data using it (rows.Table(fields=schema)) but won't work in an expected way if I try to export this table, because the library does not know how to export MoneyField and DocumentField to SQLite (only knows rows.fields.*Field classes).
As a user, to fix this I needed to have a specific schema for the Table object, like this:

    def convert_field(FieldClass):
        if FieldClass is MoneyField:
            return rows.fields.DecimalField
        elif FieldClass is DocumentField:
            return rows.fields.TextField
        else:
            return FieldClass
    schema_rows = OrderedDict([(field_name, convert_field(Field))
                               for field_name, Field in schema.items()])

The library should detect from the values produced by the class or by inspecting it.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing how rows.Table(fields=schema) maps field classes during SQLite export, especially for custom MoneyField and DocumentField subclasses. Determine the existing export entry point and tests, then verify that custom fields export with appropriate SQLite types without requiring a separate schema.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, sqlite
Domain
databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.