bulletin-file receiver incorrectly detects text bulletin as a BUFR
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 6
- Forks
- 3
- PR merge metrics
- No merged PRs in 30d
Description
When a text bulletin contains a line that starts with "BUFR" (or "GRIB" or "\211PNG"), Sundew incorrectly determines that it's a binary bulletin, and truncates the data.
https://github.com/MetPX/Sundew/blob/efcb5601ef1af945a188a8825514339b53723572/lib/bulletin.py#L522-L529
Example input:
```
NOCN04 CWAO 081718
GENOT TLTP. NO. 008 CCC
BILINGUAL MESSAGE - FRENCH TEXT TO FOLLOW ENGLISH TEXT
MESSAGE BILINGUE - LE TEXTE FRANCAIS SUIT LE TEXTE ANGLAIS
EFFECTIVE JUNE 25, 2024, THE METEOROLOGICAL SERVICE OF CANADA WILL
BEGIN REGULARLY TRANSMITTING NEW BULLETINS OF HOURLY AUTOMATED
SURFACE WEATHER OBSERVATIONS FROM RECRUITED CANADIAN SHIPS IN THE
BUFR FORMAT. THESE MESSAGES WILL FOLLOW WMO BUFR TEMPLATE TM308014.
THE BULLETIN HEADERS FOR HOURLY SHIP OBSERVATIONS WILL BE:
ISSA20 CWAO ISSB20 CWAO
ISSC20 CWAO
ISSD20 CWAO
ISSI20 CWAO
ISSJ20 CWAO
ISSK20 CWAO ISSL20 CWAO
THE BULLETIN HEADERS FOR SHIP OBSERVATIONS TRANSMITTING INTRA-HOUR
SIGNIFICANT WEATHER (STORM AND SPREP) OBSERVATIONS WILL BE:
ISSA40 CWAO ISSB40 CWAO
ISSC40 CWAO
ISSD40 CWAO
ISSI40 CWAO
ISSJ40 CWAO
ISSK40 CWAO ISSL40 CWAO
THE FOLLOWING ARE THE STATION IDENTIFIERS FOR SHIPS CURRENTLY
RECRUITED INTO THE PROGRAMME AND DISSEMINATED UNDER THE HEADERS
LISTED ABOVE. THE IDENTIFIER USED IS THE SHIP OBERVATIONS TEAM
(SOT) IDENTIFIER.
XMD7RGF UWBR5ZS DXY62VX DXJTY4L 6AK8NVU MFTW9ZM USFAPDJ FVU8WJS
VCHQSQP 8QVU8QS ADC9EHA SDTNUWW SLTUAJL SGGQ4QX HUGT78Q 2AZY7HU
HCW6ZCH ZJMN7RS PZBN6JW 2CXJVYJ ZTNDLFM YBVEWGM 6WUKLPV BAY6U5W
VKCX8TW GYJN8YD 7SXVXXY YRQAXKE 9BEUB6Y 2QDZMLH KYGEJUP 8UWN4HQ
QCQAY5M NKEKW8S 4QUWBFR 7VHNUTA LUARZ8N HPMEYWQ NHXMUSA 2MXEY3K
QSHL7RV 8VEUMXY
FOR ADDITIONAL INFORMATION REGARDING THIS BULLETIN PLEASE
CONTACT THE FOLLOWING EMAIL:
SYSTEMEDEGESTIONDESDONNEES-DATAMANAGEMENTSYSTEM(AT)EC.GC.CA
-------------------------------------------------------------------
DES LE 25 JUIN 2024, LE SERVICE METEOROLOGIQUE DU CANADA COMMENCERA
A EMETTRE REGULIEREMENT DE NOUVEAUX BULLETINS D OBSERVATIONS DE
SURFACE HORAIRES A PARTIR DES NAVIRES CANADIENS DANS LE FORMAT
BUFR. CES MESSAGES RESPECTERONT LE GABARIT TM308014 DU FORMAT BUFR
DE L OMM. LES EN-TETES DU BULLETIN POUR LES OBSERVATIONS HORAIRES
DU NAVIRE SERONT:
ISSA20 CWAO
ISSB20 CWAO
ISSC20 CWAO
ISSD20 CWAO
ISSI20 CWAO
ISSJ20 CWAO
ISSK20 CWAO
ISSL20 CWAO
LES EN-TETES DU BULLETIN POUR LES OBSERVATIONS DES NAVIRES POUR DES
CONDITIONS METEOROLOGIQUES DANS L HEURE (TEMPETES ET SPREP) SERONT:
ISSA40 CWAO
ISSB40 CWAO
ISSC40 CWAO
ISSD40 CWAO
ISSI40 CWAO
ISSJ40 CWAO
ISSK40 CWAO
ISSL40 CWAO
CE QUI SUIT SONT LES IDENTIFIANTS DE STATION POUR LES NAVIRES
ACTUELLEMENT RECRUTES DANS LE PROGRAMME ET DIFFUSES SOUS LES
EN-TETES ENUMERES CI-DESSUS. L IDENTIFIANT UTILISE EST CELUI DE L
EQUIPE D OBERVATIONS DU NAVIRE (SOT) IDENTIFIANT:
XMD7RGF UWBR5ZS DXY62VX DXJTY4L 6AK8NVU MFTW9ZM USFAPDJ FVU8WJS
VCHQSQP 8QVU8QS ADC9EHA SDTNUWW SLTUAJL SGGQ4QX HUGT78Q 2AZY7HU
HCW6ZCH ZJMN7RS PZBN6JW 2CXJVYJ ZTNDLFM YBVEWGM 6WUKLPV BAY6U5W
VKCX8TW GYJN8YD 7SXVXXY YRQAXKE 9BEUB6Y 2QDZMLH KYGEJUP 8UWN4HQ
QCQAY5M NKEKW8S 4QUWBFR 7VHNUTA LUARZ8N HPMEYWQ NHXMUSA 2MXEY3K
QSHL7RV 8VEUMXY
POUR PLUS D INFORMATIONS CONCERNANT LE PRESENT BULLETIN,
VEUILLEZ ECRIRE A L ADRESSE SUIVANTE :
SYSTEMEDEGESTIONDESDONNEES-DATAMANAGEMENTSYSTEM(AT)EC.GC.CA
EFFECTIVE / EN VIGUEUR - MAY / 31 MAI 2024 0000 UTC/TU
SIEWE ADM-MSC / SMA-SMC OTTAWA
```
The log entry:
```
2024-05-15 16:35:22,591 [INFO] 1 bulletins will be ingested
2024-05-15 16:35:22,591 [WARNING] Bufr without a valid internal date in section 1
2024-05-15 16:35:22,591 [WARNING] Use date from bulletin header
2024-05-15 16:35:22,595 [INFO] (2803 Bytes) Ingested in DB as /var/spool/px/db/20240515/NO/cmcin/CWAO/NOCN04_CWAO_081718___01878:cmcin:CWAO:NO:1:Direct:20240515163522
```
The data that gets ingested is:
```
NOCN04 CWAO 081718
GENOT TLTP. NO. 008 CCC
BILINGUAL MESSAGE - FRENCH TEXT TO FOLLOW ENGLISH TEXT
MESSAGE BILINGUE - LE TEXTE FRANCAIS SUIT LE TEXTE ANGLAIS
EFFECTIVE JUNE 25, 2024, THE METEOROLOGICAL SERVICE OF CANADA WILL
BEGIN REGULARLY TRANSMITTING NEW BULLETINS OF HOURLY AUTOMATED
SURFACE WEATHER OBSERVATIONS FROM RECRUITED CANADIAN SHIPS IN THE
BUFR FORMAT. THESE MESSAGES WILL FOLLOW WMO BUFR TEMPLATE TM308014.
THE BULLETIN HEADERS FOR HOURLY SHIP OBSERVATIONS WILL BE:
ISSA20 CWAO ISSB20 CWAO
ISSC20 CWAO
ISSD20 CWAO
ISSI20 CWAO
ISSJ20 CWAO
ISSK20 CWAO ISSL20 CWAO
THE BULLETIN HEADERS FOR SHIP OBSERVATIONS TRANSMITTING INTRA-HOUR
SIGNIFICANT WEATHER (STORM AND SPREP) OBSERVATIONS WILL BE:
ISSA40 CWAO ISSB40 CWAO
ISSC40 CWAO
ISSD40 CWAO
ISSI40 CWAO
ISSJ40 CWAO
ISSK40 CWAO ISSL40 CWAO
THE FOLLOWING ARE THE STATION IDENTIFIERS FOR SHIPS CURRENTLY
RECRUITED INTO THE PROGRAMME AND DISSEMINATED UNDER THE HEADERS
LISTED ABOVE. THE IDENTIFIER USED IS THE SHIP OBERVATIONS TEAM
(SOT) IDENTIFIER.
XMD7RGF UWBR5ZS DXY62VX DXJTY4L 6AK8NVU MFTW9ZM USFAPDJ FVU8WJS
VCHQSQP 8QVU8QS ADC9EHA SDTNUWW SLTUAJL SGGQ4QX HUGT78Q 2AZY7HU
HCW6ZCH ZJMN7RS PZBN6JW 2CXJVYJ ZTNDLFM YBVEWGM 6WUKLPV BAY6U5W
VKCX8TW GYJN8YD 7SXVXXY YRQAXKE 9BEUB6Y 2QDZMLH KYGEJUP 8UWN4HQ
QCQAY5M NKEKW8S 4QUWBFR 7VHNUTA LUARZ8N HPMEYWQ NHXMUSA 2MXEY3K
QSHL7RV 8VEUMXY
FOR ADDITIONAL INFORMATION REGARDING THIS BULLETIN PLEASE
CONTACT THE FOLLOWING EMAIL:
SYSTEMEDEGESTIONDESDONNEES-DATAMANAGEMENTSYSTEM(AT)EC.GC.CA
-------------------------------------------------------------------
DES LE 25 JUIN 2024, LE SERVICE METEOROLOGIQUE DU CANADA COMMENCERA
A EMETTRE REGULIEREMENT DE NOUVEAUX BULLETINS D OBSERVATIONS DE
SURFACE HORAIRES A PARTIR DES NAVIRES CANADIENS DANS LE FORMAT
BUFR. CES MESSAGES RESPECTERONT LE GABARIT TM308014 DU FORMAT BUFR
DE L OMM. LES EN-TETES DU BULLETIN POUR LES OBSERVATIONS HORAIRES
DU NAVIRE SERONT:
ISSA20 CWAO
ISSB20 CWAO
ISSC20 CWAO
ISSD20 CWAO
ISSI20 CWAO
ISSJ20 CWAO
ISSK20 CWAO
ISSL20 CWAO
LES EN-TETES DU BULLETIN POUR LES OBSERVATIONS DES NAVIRES POUR DES
CONDITIONS METEOROLOGIQUES DANS L HEURE (TEMPETES ET SPREP) SERONT:
ISSA40 CWAO
ISSB40 CWAO
ISSC40 CWAO
ISSD40 CWAO
ISSI40 CWAO
ISSJ40 CWAO
ISSK40 CWAO
ISSL40 CWAO
CE QUI SUIT SONT LES IDENTIFIANTS DE STATION POUR LES NAVIRES
ACTUELLEMENT RECRUTES DANS LE PROGRAMME ET DIFFUSES SOUS LES
EN-TETES ENUMERES CI-DESSUS. L IDENTIFIANT UTILISE EST CELUI DE L
EQUIPE D OBERVATIONS DU NAVIRE (SOT) IDENTIFIANT:
XMD7RGF UWBR5ZS DXY62VX DXJTY4L 6AK8NVU MFTW9ZM USFAPDJ FVU8WJS
VCHQSQP 8QVU8QS ADC9EHA SDTNUWW SLTUAJL SGGQ4QX HUGT78Q 2AZY7HU
HCW6ZCH ZJMN7RS PZBN6JW 2CXJVYJ ZTNDLFM YBVEWGM 6WUKLPV BAY6U5W
VKCX8TW GYJN8YD 7SXVXXY YRQAXKE 9
```
https://dd.weather.gc.ca/bulletins/alphanumeric/20240515/NO/CWAO/16/NOCN04_CWAO_081718___01878
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in lib/bulletin.py at lines 522-529 and reproduce the issue with the example bulletin or the linked weather.gc.ca input. Trace how the receiver decides the bulletin is binary, then verify that text containing BUFR, GRIB, or the PNG marker is retained in full and is not truncated during ingestion.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100