Query Result not rendering in Sagemaker Distribution Image 2.8 or 3.3 in Unified Studio

When running queries on Spark compute in JupyterLab with SageMaker Distribution Image 2.8 or 3.3, result sets fail to render and show an ‘error loading data’ message.

The same code works as expected on Image 2.6, where results are displayed in tabular format.

Output from the space with Sagemaker Distribution Image 2.8

Output from the space with Sagemaker Distribution Image 2.6

Also, I don't see any Sagemaker Distribution Images in the drop down in Jupterlab Space Configuration apart from 2.8 and 3.3. Is this expected?

Reproduce a query on Spark compute in JupyterLab with SageMaker Distribution Images 2.8 and 3.3, then compare the result with Image 2.6. Check whether result rendering and the available image dropdown behave as reported; done means query results render successfully and the dropdown behavior is explained or corrected.

Written by the indexing model from the issue text.