processing / processing/processing4

Floating point / HDR texture support

Open
#1,324 5 comments 7 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Java
Stars
494
Forks
183
Avg merge
4h 39m
Merged PRs (30d)
3

Description

Processing is tightly coupled to ARGB 8-bit textures via the PImage class hierarchy. When the project started, the only crossplatform drawing lib available in the JDK was AWT/Java2D. Those APIs expose pixels as int in ARGB (TYPE_INT_ARGB/TYPE_INT_RGB), and java.awt.Color wraps 0–255 channels. That layout provides zero copy access, blits via MemoryImageSource, and compatibility with other AWT stuff that's used pervasively throughout the codebase.

Every part of Processing assumes pixel data is stored as 32‑bit ints with 8‑bit channels. That assumption is baked into our public API (the "pixels" array) and is all over the place inside PGraphics, image filters, tessellators, OpenGL and future WebGPU backends, and font/shape caches, etc. Because there is no abstraction around "pixel format" or "color storage", any attempt to add floating point textures would require touching practically every file, amounting in an effective soft-fork.

Deep coupling

Here are some more specific issues:

Why is this an issue / What this limits

At a high level, while 8-bit textures may have been the norm in the early 2000s, the assumption has basically flipped. Modern GPUs execute most shading math in 32-bit lanes regardless of the source format, so even when sampling from an 8-bit texture the ALUs are still getting their parallelism over 32-bit registers. Unless you're explicitly using packed instructions (which is rare for generalist graphics work), the only real thing sthat 8-bit gets is memory bandwidth, which mostly only matters on low-end mobile. Everywhere else the industry is moving toward mixed precision (e.g. FP16 tensor cores for ML matmul ops) because compute cost is no longer tied to per-channel bit depth.

However, specifically relevant to art, the lack of floating point textures is severe limitation to the following techniques.

Installation art

Stuff like motion tracking, heatmap accumulation, or any "paint with time" piece depends on adding and accumulating tiny deltas every frame. In 8-bit those changes round away, so small movement never shows up and trails die instantly. Float buffers let artists integrate small movement over minutes or hours and only quantize when sending to the projector. Multi-projector setups also have problems, feathering overlaps, gamma correcting different units, and warping/projection mapping content through multiple correction stages all require smooth ramps. Quantizing each stage to 256 buckets produces visible steps, which is especially problematic in dark rooms typical for these kinds of installs. Hardware sensorb ased work (think depth cameras, environmental sensors, etc) typically rely on filters and other post-processing to be useful for producing a final render for the projector.

Generative art

Feedback is super common in generative art, ie read the last frame, do some texture processing, write it back, repeat. In 8-bit every read-modify-write cycle loses precision, so the loop collapses after a few iterations and colors wander off due to rounding and look bad if they work at all. Float textures keep temporal buffers alive and stable which is why applications like TouchDesigner or vvvv typically default to them. Popular algos like reaction-diffusion, fluid sims, and cellular automata also add/subtract tiny gradients every step. Quantization steps in 8-bit makes patterns fall apart. Floats match the reference equations. Even simple particles need floats to ensure the motion looks and particle sim parameters work from frame to frame. Layering hundreds of low-alpha sprites in 8-bit will jump between "invisible" and "too strong" because the intermediate alpha values simply don't exist. For people who care about color grading or LUT work, esp to reference stuff like mockups made by designers which is important in professional settings working on a team, you need headroom. Doing the math in 8-bit crushes blacks and create midtone bands which will make your designers cry.

Modern 3D graphics techniques

In the more game oriented world, TAA and other AA techniques assume you can accumulate subpixel jitter over many frames without the values collapsing. When you use 8-bit components, the first couple of blends push dark tones to zero and you're left with banding or ghost trails. Modern engines accumulate in 16/32-bit floats and only clamp when tonemapping out to the display. The same is true for HDR stuff like bloom, tone mapping, or PBR shading that need to push energy past 1.0, do postprocessing, then compress the result. With 0-255 you just clip to white and the math breaks. Deferred renderers and screenspace effect passes also suffer when encode normals, roughness, or depth in 8-bit and SSAO/SSR immediately reveal the quantization steps. Backends like Metal/Vulkan assume you can request RGBA16F, R11G11B10F, depth-only attachments, etc. which the user may want to display / debug. There's interesting Processing style techniques here that simply aren't possible atm.

Conclusion

Changing Processing's texture format and color math is a breaking change at basically all levels of the existing codebase but is crucially important for modern graphics techniques.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the pixel and color assumptions in PImage.java, PGraphics.java, PShape.java, PGraphicsOpenGL.java, Texture.java, PGL.java, FrameBuffer.java, PShapeOpenGL.java, and PGraphicsWebGPU.java. The issue does not specify a concrete design, implementation boundary, or test; work would be complete only after a defined approach supports floating-point or HDR formats across the affected paths without breaking the existing API.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
computer-graphics
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.