qemu/include/ui
Daniel P. Berrangé 7009744285 ui/vnc: take account of client byte order in pixman format
The set_pixel_conversion() method is responsible for determining whether
the VNC client pixel format matches the server format, and thus whether
we can use the fast path "copy" impl for sending pixels, or must use
the generic impl with bit swizzling.

The VNC server format is set at build time to VNC_SERVER_FB_FORMAT,
which corresponds to PIXMAN_x8r8g8b8.

The qemu_pixman_get_format() method is then responsible for converting
the VNC pixel format into a pixman format.

The VNC client pixel shifts are relative to the associated endianness.

The pixman formats are always relative to the host native endianness.

The qemu_pixman_get_format() method does not take into account the
VNC client endianness, and is thus returning a pixman format that is
only valid with the host endianness matches that of the VNC client.

This has been broken since pixman was introduced to the VNC server:

  commit 9f64916da2
  Author: Gerd Hoffmann <kraxel@redhat.com>
  Date:   Wed Oct 10 13:29:43 2012 +0200

    pixman/vnc: use pixman images in vnc.

The flaw can be demonstrated using the Tigervnc client by using

   vncviewer -AutoSelect=0 -PreferredEncoding=raw server:display

connecting from a LE client to a QEMU on a BE server, or the
reverse.

The bug was masked, however, because almost all VNC clients will
advertize support for the "tight" encoding and the QEMU VNC server
will prefer "tight" if advertized.

The tight_pack24 method is responsible for taking a set of pixels
which have already been converted into client endianness and then
repacking them into the TPIXEL format which the RFB spec defines
as

  "TPIXEL is only 3 bytes long, where the first byte is the
   red component, the second byte is the green component,
   and the third byte is the blue component of the pixel
   color value"

IOW, the TPIXEL format is fixed on the wire, regardless of what
the VNC client declare as its endianness.

Since the VNC pixel encoding code was failing to honour the endian
flag of the client, the tight_pack24 method was always operating
on data in native endianness. Its impl cancelled out the VNC pixel
encoding bug.

With the VNC pixel encoding code now fixed, the tight_pack24 method
needs to take into account that it is operating on data in client
endianness, not native endianness. It thus may need to invert the
pixel shifts.

Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
2025-05-21 19:28:01 +01:00
..
clipboard.h spelling: information 2023-06-09 23:38:16 +03:00
console.h ui/input-legacy.c: remove unused legacy qemu_add_kbd_event_handler() function 2024-11-08 11:06:42 +01:00
dbus-display.h Clean up ill-advised or unusual header guards 2022-05-11 16:50:01 +02:00
dbus-module.h Clean up ill-advised or unusual header guards 2022-05-11 16:50:01 +02:00
dmabuf.h ui/dmabuf: extend QemuDmaBuf to support multi-plane 2025-05-06 11:57:03 +04:00
egl-context.h ui: split the GL context in a different object 2021-12-21 10:50:21 +04:00
egl-helpers.h ui/egl: support multi-plane dmabuf when egl export/import 2025-05-06 13:39:42 +04:00
gtk.h ui: add optional d3d texture pointer to scanout texture 2023-06-27 17:08:56 +02:00
input.h ui/input: Constify QemuInputHandler structure 2023-10-19 23:13:28 +02:00
kbd-state.h ui/vnc: Do not use console_select() 2024-03-20 10:27:54 +04:00
pixel_ops.h ui: move files to ui/ and include/ui/ 2012-12-19 08:31:30 +01:00
pixman-minimal.h ui/pixman-minimal.h: fix empty allocation 2023-11-21 14:38:14 +04:00
qemu-pixman.h ui/vnc: take account of client byte order in pixman format 2025-05-21 19:28:01 +01:00
qemu-spice-module.h spice: wire up monitor in QemuSpiceOps. 2020-10-21 15:46:14 +02:00
qemu-spice.h ui/spice: QXLInterface method set_mm_time() is now dead, drop 2023-01-19 13:30:01 +01:00
rect.h include: Clean up includes 2024-01-30 21:20:20 +03:00
sdl2.h ui/sdl2: ignore GUI keys in SDL_TEXTINPUT handler 2024-09-13 20:12:16 +02:00
shader.h opengl: add flipping vertex shader 2017-10-17 10:25:42 +02:00
spice-display.h ui: spelling fixes 2023-08-31 19:47:43 +02:00
surface.h ui: refactor using a common qemu_pixman_shareable 2024-10-14 17:34:09 +04:00
win32-kbd-hook.h ui/win32-kbd-hook: handle AltGr in a hook procedure 2020-05-19 09:06:44 +02:00