algorand / algorand/go-algorand

arc28 and other log data in JSON format (may be) garbage

Open
#6,128 0 comments 2 reactions 0 assignees View on GitHub
bug
Dominant language
Go
Stars
1.4k
Forks
537
Avg merge
1d 7h
Merged PRs (30d)
18

Description

### Subject of the issue

When using the `json` format on `/v2/blocks` it returns serialized `lg` data that is unusable / garbage in many cases.

In this case, the bad serialization is most easily identifiable by the visual presence of utf-8 replacement characters -> https://www.compart.com/en/unicode/U+FFFD

### Steps to reproduce

#### Method One

Look at `txn` index 16 (assuming start index of 0) which is a Tinyman swap:

https://mainnet-api.algonode.cloud/v2/blocks/41335116

Fields in `dt.ld.1` have garbage data along with `lg`, particularly logs like `output_asset_id` and `output_amount`

#### Method Two

Run this script to see a more thorough example of trying to access Tinyman log data, which is provided by `v2/blocks` in `json` format.

```python
from algosdk.v2client import algod
from base64 import b64decode
import pprint

algod_address = "https://mainnet-api.algonode.cloud"
algod_token = ""
algod_client = algod.AlgodClient(algod_token, algod_address)

def decode_logs(logs: "list") -> dict:
decoded_logs = dict()
for log in logs:
print("Decoding:", log)
if type(log) == str:
log = b64decode(log.encode())
if b"%i" in log:
i = log.index(b"%i")
s = log[0:i].decode()
value = int.from_bytes(log[i + 2 :], "big")
decoded_logs[s] = value
else:
raise NotImplementedError()
return decoded_logs

block = algod_client.block_info(block=41335116)
pp = pprint.PrettyPrinter(indent=2)

tinyman_tx = block.get('block').get('txns')[16]
log = tinyman_tx.get('dt').get('lg')
pp.pprint(log)

result = decode_logs(log)
pp.pprint(result)
```

### Thoughts

There are a few issues here:
1. JSON format is currently serializing data that may be unusable, wasting loads of bandwidth + cpu cycles
- Not a total waste in cases where the bytestream can be serialized
2. Switching to `msgpack` is replete with all sorts of quirks across various languages, the biggest being the main Javascript library can't even decode it without issues
3. @joe-p has mentioned:

> We technically could "fix" the JSON response to use surrogate escaping like Python supports, but again not every client would support that and that could actually break code that is currently working.

Which really sounds like the ideal answer, and then using path-versioning like `/v2.1/blocks` to provide a new endpoint to maintain legacy endpoint functioning.

4. @pbennett notes:

> problem likely stems from the 'logs' being defined as string not []byte - so it should be base64 encoded by algod for eg and its not.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.