qemu/include
Daniel P. Berrangé 7009744285 ui/vnc: take account of client byte order in pixman format
The set_pixel_conversion() method is responsible for determining whether
the VNC client pixel format matches the server format, and thus whether
we can use the fast path "copy" impl for sending pixels, or must use
the generic impl with bit swizzling.

The VNC server format is set at build time to VNC_SERVER_FB_FORMAT,
which corresponds to PIXMAN_x8r8g8b8.

The qemu_pixman_get_format() method is then responsible for converting
the VNC pixel format into a pixman format.

The VNC client pixel shifts are relative to the associated endianness.

The pixman formats are always relative to the host native endianness.

The qemu_pixman_get_format() method does not take into account the
VNC client endianness, and is thus returning a pixman format that is
only valid with the host endianness matches that of the VNC client.

This has been broken since pixman was introduced to the VNC server:

  commit 9f64916da2
  Author: Gerd Hoffmann <kraxel@redhat.com>
  Date:   Wed Oct 10 13:29:43 2012 +0200

    pixman/vnc: use pixman images in vnc.

The flaw can be demonstrated using the Tigervnc client by using

   vncviewer -AutoSelect=0 -PreferredEncoding=raw server:display

connecting from a LE client to a QEMU on a BE server, or the
reverse.

The bug was masked, however, because almost all VNC clients will
advertize support for the "tight" encoding and the QEMU VNC server
will prefer "tight" if advertized.

The tight_pack24 method is responsible for taking a set of pixels
which have already been converted into client endianness and then
repacking them into the TPIXEL format which the RFB spec defines
as

  "TPIXEL is only 3 bytes long, where the first byte is the
   red component, the second byte is the green component,
   and the third byte is the blue component of the pixel
   color value"

IOW, the TPIXEL format is fixed on the wire, regardless of what
the VNC client declare as its endianness.

Since the VNC pixel encoding code was failing to honour the endian
flag of the client, the tight_pack24 method was always operating
on data in native endianness. Its impl cancelled out the VNC pixel
encoding bug.

With the VNC pixel encoding code now fixed, the tight_pack24 method
needs to take into account that it is operating on data in client
endianness, not native endianness. It thus may need to invert the
pixel shifts.

Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
2025-05-21 19:28:01 +01:00
..
accel accel/tcg: Move TARGET_TAGGED_ADDRESSES to TCGCPUOps.untagged_addr 2025-05-05 09:24:10 -07:00
authz Prefer 'on' | 'off' over 'yes' | 'no' for bool options 2021-01-29 17:07:53 +00:00
block mirror: Drop redundant zero_target parameter 2025-05-14 20:10:12 -05:00
chardev chardev/char-hub: implement backend chardev aggregator 2025-02-03 13:57:08 +04:00
crypto crypto: Remove qcrypto_tls_session_get_handshake_status 2025-02-14 15:19:03 -03:00
disas disas: Fix build against Capstone v6 (again) 2024-11-05 10:09:59 +00:00
exec tcg: add vaddr type for helpers 2025-05-14 15:12:40 +01:00
fpu fpu: Move m68k_denormal fmt flag into floatx80_behaviour 2025-02-25 15:32:57 +00:00
gdbstub gdbstub/helpers: Introduce ldtul_$endian_p() helpers 2024-10-15 11:55:09 -03:00
hw hw/pci-host: Remove unused pci_host_data_be_ops 2025-05-20 08:04:18 +02:00
io io: Add a read flag for relaxed EOF 2025-02-14 15:19:04 -03:00
libdecnumber Replace config-time define HOST_WORDS_BIGENDIAN 2022-04-06 10:50:37 +02:00
migration migration: Add save_postcopy_prepare() savevm handler 2025-05-02 11:09:36 -04:00
monitor monitor: Remove obsolete stubs 2024-06-30 19:51:44 +03:00
net net: checksum: Convert data to void * 2024-11-25 13:59:50 +08:00
qapi error: define g_autoptr() cleanup function for the Error type 2025-03-06 06:47:33 +01:00
qemu Pull request 2025-05-09 12:04:19 -04:00
qobject qapi: Move include/qapi/qmp/ to include/qobject/ 2025-02-10 15:33:16 +01:00
qom qom: reverse order of instance_post_init calls 2025-05-20 08:18:53 +02:00
scsi hw/ufs: Support for UFS logical unit 2023-09-07 14:01:29 -04:00
semihosting semihosting: Move user-only implementation out-of-line 2025-04-23 14:08:32 -07:00
standard-headers linux-headers: Update to Linux v6.15-rc3 2025-05-09 12:42:28 +02:00
system target-arm queue: 2025-05-15 13:42:21 -04:00
tcg tcg: add vaddr type for helpers 2025-05-14 15:12:40 +01:00
ui ui/vnc: take account of client byte order in pixman format 2025-05-21 19:28:01 +01:00
user accel/tcg: Remove TARGET_PAGE_DATA_SIZE 2025-05-05 09:24:10 -07:00
elf.h util: spelling fixes 2023-08-31 19:47:43 +02:00
glib-compat.h include/glib-compat.h: Poison g_list_sort and g_slist_sort 2025-05-06 16:02:04 +02:00
qemu-io.h Include qemu-common.h exactly where needed 2019-06-12 13:20:20 +02:00
qemu-main.h ui & main loop: Redesign of system-specific main thread event handling 2024-12-31 21:21:34 +01:00