galaxyproject / galaxyproject/pulsar
Large stdout/stderr crashes the acknowledgement manager, results in stuck jobs
- Dominant language
- Python
- Stars
- 46
- Forks
- 62
- Avg merge
- 3d 15h
- Merged PRs (30d)
- 14
Description
```
2021-08-24 13:52:32,217 DEBUG [pulsar.client.amqp_exchange][acknowledgement-manager] UUID b694576c-02bf-11ec-a39e-566f6d94001a has not been acknowledged, republishing original message on queue status_update
2021-08-24 13:52:32,217 DEBUG [pulsar.client.amqp_exchange][acknowledgement-manager] [publish:0e74b094-0504-11ec-9a41-566f6d94001a] Begin publishing to key pulsar_bridges__status_update
2021-08-24 13:52:32,218 DEBUG [pulsar.client.amqp_exchange][acknowledgement-manager] [publish:0e74b094-0504-11ec-9a41-566f6d94001a] Have producer for publishing to key pulsar_bridges__status_update
2021-08-24 13:52:32,300 ERROR [pulsar.client.amqp_exchange][acknowledgement-manager] Problem with acknowledgement manager, leaving ack_manager method in problematic state!
Traceback (most recent call last):
File "/jet/home/xcgalaxy/main/pulsar/venv/lib/python3.8/site-packages/pulsar/client/amqp_exchange.py", line 232, in ack_manager
self.publish(resubmit_queue, payload)
File "/jet/home/xcgalaxy/main/pulsar/venv/lib/python3.8/site-packages/pulsar/client/amqp_exchange.py", line 205, in publish
producer.publish(
File "/jet/home/xcgalaxy/main/pulsar/venv/lib/python3.8/site-packages/kombu/messaging.py", line 175, in publish
return _publish(
File "/jet/home/xcgalaxy/main/pulsar/venv/lib/python3.8/site-packages/kombu/connection.py", line 525, in _ensured
return fun(*args, **kwargs)
File "/jet/home/xcgalaxy/main/pulsar/venv/lib/python3.8/site-packages/kombu/messaging.py", line 197, in _publish
return channel.basic_publish(
File "/jet/home/xcgalaxy/main/pulsar/venv/lib/python3.8/site-packages/amqp/channel.py", line 1775, in _basic_publish
self.connection.drain_events(timeout=0)
File "/jet/home/xcgalaxy/main/pulsar/venv/lib/python3.8/site-packages/amqp/connection.py", line 522, in drain_events
while not self.blocking_read(timeout):
File "/jet/home/xcgalaxy/main/pulsar/venv/lib/python3.8/site-packages/amqp/connection.py", line 528, in blocking_read
return self.on_inbound_frame(frame)
File "/jet/home/xcgalaxy/main/pulsar/venv/lib/python3.8/site-packages/amqp/method_framing.py", line 53, in on_frame
callback(channel, method_sig, buf, None)
File "/jet/home/xcgalaxy/main/pulsar/venv/lib/python3.8/site-packages/amqp/connection.py", line 534, in on_inbound_method
return self.channels[channel_id].dispatch_method(
File "/jet/home/xcgalaxy/main/pulsar/venv/lib/python3.8/site-packages/amqp/abstract_channel.py", line 143, in dispatch_method
listener(*args)
File "/jet/home/xcgalaxy/main/pulsar/venv/lib/python3.8/site-packages/amqp/channel.py", line 277, in _on_close
raise error_for_code(
amqp.exceptions.PreconditionFailed: Basic.publish: (406) PRECONDITION_FAILED - message size 155238738 is larger than configured max size 134217728
```
Not sure of the best solution here but maybe Pulsar should post the stdio streams back as job files rather than sending them in the MQ?
Contributor guide
Assessment
This issue has not been assessed yet.