You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
libtdevnc/common/turbojpeg.c

855 lines
22 KiB

Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
/*
* Copyright (C)2009-2012 D. R. Commander. All Rights Reserved.
Add TurboVNC encoding support. TurboVNC is a variant of TightVNC that uses the same client/server protocol (RFB version 3.8t), and thus it is fully cross-compatible with TightVNC and TigerVNC (with one exception, which is noted below.) Both the TightVNC and TurboVNC encoders analyze each rectangle, pick out regions of solid color to send separately, and send the remaining subrectangles using mono, indexed color, JPEG, or raw encoding, depending on the number of colors in the subrectangle. However, TurboVNC uses a fundamentally different selection algorithm to determine the appropriate subencoding to use for each subrectangle. Thus, while it sends a protocol stream that can be decoded by any TightVNC-compatible viewer, the mix of subencoding types in this protocol stream will be different from those generated by a TightVNC server. The research that led to TurboVNC is described in the following report: http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf. In summary: 20 RFB captures, representing "common" 2D and 3D application workloads (the 3D workloads were run using VirtualGL), were studied using the TightVNC encoder in isolation. Some of the analysis features in the TightVNC encoder, such as smoothness detection, were found to generate a lot of CPU usage with little or no benefit in compression, so those features were disabled. JPEG encoding was accelerated using libjpeg-turbo (which achieves a 2-4x speedup over plain libjpeg on modern x86 or ARM processors.) Finally, the "palette threshold" (minimum number of colors that the subrectangle must have before it is compressed using JPEG or raw) was adjusted to account for the fact that JPEG encoding is now quite a bit faster (meaning that we can now use it more without a CPU penalty.) TurboVNC has additional optimizations, such as the ability to count colors and encode JPEG images directly from the framebuffer without first translating the pixels into RGB. The TurboVNC encoder compares quite favorably in terms of compression ratio with TightVNC and generally encodes a great deal faster (often an order of magnitude or more.) The version of the TurboVNC encoder included in this patch is roughly equivalent to the one found in version 0.6 of the Unix TurboVNC Server, with a few minor patches integrated from TurboVNC 1.1. TurboVNC 1.0 added multi-threading capabilities, which can be added in later if desired (at the expense of making libvncserver depend on libpthread.) Because TurboVNC uses a fundamentally different mix of subencodings than TightVNC, because it uses the identical protocol (and thus a viewer really has no idea whether it's talking to a TightVNC or TurboVNC server), and because it doesn't support rfbTightPng (and in fact conflicts with it-- see below), the TurboVNC and TightVNC encoders cannot be enabled simultaneously. Compatibility: In *most* cases, a TurboVNC-enabled viewer is fully compatible with a TightVNC server, and vice versa. TurboVNC supports pseudo-encodings for specifying a fine-grained (1-100) quality scale and specifying chrominance subsampling. If a TurboVNC viewer sends those to a TightVNC server, then the TightVNC server ignores them, so the TurboVNC viewer also sends the quality on a 0-9 scale that the TightVNC server can understand. Similarly, the TurboVNC server checks first for fine-grained quality and subsampling pseudo-encodings from the viewer, and failing to receive those, it then checks for the TightVNC 0-9 quality pseudo-encoding. There is one case in which the two systems are not compatible, and that is when a TightVNC or TigerVNC viewer requests compression level 0 without JPEG from a TurboVNC server. For performance reasons, this causes the TurboVNC server to send images directly to the viewer, bypassing Zlib. When the TurboVNC server does this, it also sets bits 7-4 in the compression control byte to rfbTightNoZlib (0x0A), which is unfortunately the same value as rfbTightPng. Older TightVNC viewers that don't handle PNG will assume that the stream is uncompressed but still encapsulated in a Zlib structure, whereas newer PNG-supporting TightVNC viewers will assume that the stream is PNG. In either case, the viewer will probably crash. Since most VNC viewers don't expose compression level 0 in the GUI, this is a relatively rare situation. Description of changes: configure.ac -- Added support for libjpeg-turbo. If passed an argument of --with-turbovnc, configure will now run (or, if cross-compiling, just link) a test program that determines whether the libjpeg library being used is libjpeg-turbo. libjpeg-turbo must be used when building the TurboVNC encoder, because the TurboVNC encoder relies on the libjpeg-turbo colorspace extensions in order to compress images directly out of the framebuffer (which may be, for instance, BGRA rather than RGB.) libjpeg-turbo can optionally be used with the TightVNC encoder as well, but the speedup will only be marginal (the report linked above explains why in more detail, but basically it's because of Amdahl's Law. The TightVNC encoder was designed with the assumption that JPEG had a very high CPU cost, and thus JPEG is used only sparingly.) -- Added a new configure variable, JPEG_LDFLAGS. This is necessitated by the fact that libjpeg-turbo often distributes libjpeg.a and libjpeg.so in /opt/libjpeg-turbo/lib32 or /opt/libjpeg-turbo/lib64, and many people prefer to statically link with it. Thus, more flexibility is needed than is provided by --with-jpeg. If JPEG_LDFLAGS is specified, then it overrides the changes to LDFLAGS enacted by --with-jpeg (but --with-jpeg is still used to set the include path.) The addition of JPEG_LDFLAGS necessitated replacing AC_CHECK_LIB with AC_LINK_IFELSE (because AC_CHECK_LIB automatically sets LIBS to -ljpeg, which is not what we want if we're, for instance, linking statically with libjpeg-turbo.) -- configure does not check for PNG support if TurboVNC encoding is enabled. This prevents the rfbSendRectEncodingTightPng() function from being compiled in, since the TurboVNC encoder doesn't (and can't) support it. common/turbojpeg.c, common/turbojpeg.h -- TurboJPEG is a simple API used to compress and decompress JPEG images in memory. It was originally implemented because it was desirable to use different types of underlying technologies to compress JPEG on different platforms (mediaLib on SPARC, Quicktime on PPC Macs, Intel Performance Primitives, etc.) These days, however, libjpeg-turbo is the only underlying technology used by TurboVNC, so TurboJPEG's purpose is largely just code simplicity and flexibility. Thus, since there is no real need for libvncserver to use any technology other than libjpeg-turbo for compressing JPEG, the TurboJPEG wrapper for libjpeg-turbo has been included in-tree so that libvncserver can be directly linked with libjpeg-turbo. This is convenient because many modern Linux distros (Fedora, Ubuntu, etc.) now ship libjpeg-turbo as their default libjpeg library. libvncserver/rfbserver.c -- Added logic to check for the TurboVNC fine-grained quality level and subsampling encodings and to map Tight (0-9) quality levels to appropriate fine-grained quality level and subsampling values if communicating with a TightVNC/TigerVNC viewer. libvncserver/turbo.c -- TurboVNC encoder (compiled instead of libvncserver/tight.c) rfb/rfb.h -- Added support for the TurboVNC subsampling level rfb/rfbproto.h -- Added constants for the TurboVNC fine quality level and subsampling encodings as well as the rfbTightNoZlib constant and notes on its usage.
13 years ago
*
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
Add TurboVNC encoding support. TurboVNC is a variant of TightVNC that uses the same client/server protocol (RFB version 3.8t), and thus it is fully cross-compatible with TightVNC and TigerVNC (with one exception, which is noted below.) Both the TightVNC and TurboVNC encoders analyze each rectangle, pick out regions of solid color to send separately, and send the remaining subrectangles using mono, indexed color, JPEG, or raw encoding, depending on the number of colors in the subrectangle. However, TurboVNC uses a fundamentally different selection algorithm to determine the appropriate subencoding to use for each subrectangle. Thus, while it sends a protocol stream that can be decoded by any TightVNC-compatible viewer, the mix of subencoding types in this protocol stream will be different from those generated by a TightVNC server. The research that led to TurboVNC is described in the following report: http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf. In summary: 20 RFB captures, representing "common" 2D and 3D application workloads (the 3D workloads were run using VirtualGL), were studied using the TightVNC encoder in isolation. Some of the analysis features in the TightVNC encoder, such as smoothness detection, were found to generate a lot of CPU usage with little or no benefit in compression, so those features were disabled. JPEG encoding was accelerated using libjpeg-turbo (which achieves a 2-4x speedup over plain libjpeg on modern x86 or ARM processors.) Finally, the "palette threshold" (minimum number of colors that the subrectangle must have before it is compressed using JPEG or raw) was adjusted to account for the fact that JPEG encoding is now quite a bit faster (meaning that we can now use it more without a CPU penalty.) TurboVNC has additional optimizations, such as the ability to count colors and encode JPEG images directly from the framebuffer without first translating the pixels into RGB. The TurboVNC encoder compares quite favorably in terms of compression ratio with TightVNC and generally encodes a great deal faster (often an order of magnitude or more.) The version of the TurboVNC encoder included in this patch is roughly equivalent to the one found in version 0.6 of the Unix TurboVNC Server, with a few minor patches integrated from TurboVNC 1.1. TurboVNC 1.0 added multi-threading capabilities, which can be added in later if desired (at the expense of making libvncserver depend on libpthread.) Because TurboVNC uses a fundamentally different mix of subencodings than TightVNC, because it uses the identical protocol (and thus a viewer really has no idea whether it's talking to a TightVNC or TurboVNC server), and because it doesn't support rfbTightPng (and in fact conflicts with it-- see below), the TurboVNC and TightVNC encoders cannot be enabled simultaneously. Compatibility: In *most* cases, a TurboVNC-enabled viewer is fully compatible with a TightVNC server, and vice versa. TurboVNC supports pseudo-encodings for specifying a fine-grained (1-100) quality scale and specifying chrominance subsampling. If a TurboVNC viewer sends those to a TightVNC server, then the TightVNC server ignores them, so the TurboVNC viewer also sends the quality on a 0-9 scale that the TightVNC server can understand. Similarly, the TurboVNC server checks first for fine-grained quality and subsampling pseudo-encodings from the viewer, and failing to receive those, it then checks for the TightVNC 0-9 quality pseudo-encoding. There is one case in which the two systems are not compatible, and that is when a TightVNC or TigerVNC viewer requests compression level 0 without JPEG from a TurboVNC server. For performance reasons, this causes the TurboVNC server to send images directly to the viewer, bypassing Zlib. When the TurboVNC server does this, it also sets bits 7-4 in the compression control byte to rfbTightNoZlib (0x0A), which is unfortunately the same value as rfbTightPng. Older TightVNC viewers that don't handle PNG will assume that the stream is uncompressed but still encapsulated in a Zlib structure, whereas newer PNG-supporting TightVNC viewers will assume that the stream is PNG. In either case, the viewer will probably crash. Since most VNC viewers don't expose compression level 0 in the GUI, this is a relatively rare situation. Description of changes: configure.ac -- Added support for libjpeg-turbo. If passed an argument of --with-turbovnc, configure will now run (or, if cross-compiling, just link) a test program that determines whether the libjpeg library being used is libjpeg-turbo. libjpeg-turbo must be used when building the TurboVNC encoder, because the TurboVNC encoder relies on the libjpeg-turbo colorspace extensions in order to compress images directly out of the framebuffer (which may be, for instance, BGRA rather than RGB.) libjpeg-turbo can optionally be used with the TightVNC encoder as well, but the speedup will only be marginal (the report linked above explains why in more detail, but basically it's because of Amdahl's Law. The TightVNC encoder was designed with the assumption that JPEG had a very high CPU cost, and thus JPEG is used only sparingly.) -- Added a new configure variable, JPEG_LDFLAGS. This is necessitated by the fact that libjpeg-turbo often distributes libjpeg.a and libjpeg.so in /opt/libjpeg-turbo/lib32 or /opt/libjpeg-turbo/lib64, and many people prefer to statically link with it. Thus, more flexibility is needed than is provided by --with-jpeg. If JPEG_LDFLAGS is specified, then it overrides the changes to LDFLAGS enacted by --with-jpeg (but --with-jpeg is still used to set the include path.) The addition of JPEG_LDFLAGS necessitated replacing AC_CHECK_LIB with AC_LINK_IFELSE (because AC_CHECK_LIB automatically sets LIBS to -ljpeg, which is not what we want if we're, for instance, linking statically with libjpeg-turbo.) -- configure does not check for PNG support if TurboVNC encoding is enabled. This prevents the rfbSendRectEncodingTightPng() function from being compiled in, since the TurboVNC encoder doesn't (and can't) support it. common/turbojpeg.c, common/turbojpeg.h -- TurboJPEG is a simple API used to compress and decompress JPEG images in memory. It was originally implemented because it was desirable to use different types of underlying technologies to compress JPEG on different platforms (mediaLib on SPARC, Quicktime on PPC Macs, Intel Performance Primitives, etc.) These days, however, libjpeg-turbo is the only underlying technology used by TurboVNC, so TurboJPEG's purpose is largely just code simplicity and flexibility. Thus, since there is no real need for libvncserver to use any technology other than libjpeg-turbo for compressing JPEG, the TurboJPEG wrapper for libjpeg-turbo has been included in-tree so that libvncserver can be directly linked with libjpeg-turbo. This is convenient because many modern Linux distros (Fedora, Ubuntu, etc.) now ship libjpeg-turbo as their default libjpeg library. libvncserver/rfbserver.c -- Added logic to check for the TurboVNC fine-grained quality level and subsampling encodings and to map Tight (0-9) quality levels to appropriate fine-grained quality level and subsampling values if communicating with a TightVNC/TigerVNC viewer. libvncserver/turbo.c -- TurboVNC encoder (compiled instead of libvncserver/tight.c) rfb/rfb.h -- Added support for the TurboVNC subsampling level rfb/rfbproto.h -- Added constants for the TurboVNC fine quality level and subsampling encodings as well as the rfbTightNoZlib constant and notes on its usage.
13 years ago
*
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
* - Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* - Redistributions in binary form must reproduce the above copyright notice,
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
* - Neither the name of the libjpeg-turbo Project nor the names of its
* contributors may be used to endorse or promote products derived from this
* software without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS",
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
Add TurboVNC encoding support. TurboVNC is a variant of TightVNC that uses the same client/server protocol (RFB version 3.8t), and thus it is fully cross-compatible with TightVNC and TigerVNC (with one exception, which is noted below.) Both the TightVNC and TurboVNC encoders analyze each rectangle, pick out regions of solid color to send separately, and send the remaining subrectangles using mono, indexed color, JPEG, or raw encoding, depending on the number of colors in the subrectangle. However, TurboVNC uses a fundamentally different selection algorithm to determine the appropriate subencoding to use for each subrectangle. Thus, while it sends a protocol stream that can be decoded by any TightVNC-compatible viewer, the mix of subencoding types in this protocol stream will be different from those generated by a TightVNC server. The research that led to TurboVNC is described in the following report: http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf. In summary: 20 RFB captures, representing "common" 2D and 3D application workloads (the 3D workloads were run using VirtualGL), were studied using the TightVNC encoder in isolation. Some of the analysis features in the TightVNC encoder, such as smoothness detection, were found to generate a lot of CPU usage with little or no benefit in compression, so those features were disabled. JPEG encoding was accelerated using libjpeg-turbo (which achieves a 2-4x speedup over plain libjpeg on modern x86 or ARM processors.) Finally, the "palette threshold" (minimum number of colors that the subrectangle must have before it is compressed using JPEG or raw) was adjusted to account for the fact that JPEG encoding is now quite a bit faster (meaning that we can now use it more without a CPU penalty.) TurboVNC has additional optimizations, such as the ability to count colors and encode JPEG images directly from the framebuffer without first translating the pixels into RGB. The TurboVNC encoder compares quite favorably in terms of compression ratio with TightVNC and generally encodes a great deal faster (often an order of magnitude or more.) The version of the TurboVNC encoder included in this patch is roughly equivalent to the one found in version 0.6 of the Unix TurboVNC Server, with a few minor patches integrated from TurboVNC 1.1. TurboVNC 1.0 added multi-threading capabilities, which can be added in later if desired (at the expense of making libvncserver depend on libpthread.) Because TurboVNC uses a fundamentally different mix of subencodings than TightVNC, because it uses the identical protocol (and thus a viewer really has no idea whether it's talking to a TightVNC or TurboVNC server), and because it doesn't support rfbTightPng (and in fact conflicts with it-- see below), the TurboVNC and TightVNC encoders cannot be enabled simultaneously. Compatibility: In *most* cases, a TurboVNC-enabled viewer is fully compatible with a TightVNC server, and vice versa. TurboVNC supports pseudo-encodings for specifying a fine-grained (1-100) quality scale and specifying chrominance subsampling. If a TurboVNC viewer sends those to a TightVNC server, then the TightVNC server ignores them, so the TurboVNC viewer also sends the quality on a 0-9 scale that the TightVNC server can understand. Similarly, the TurboVNC server checks first for fine-grained quality and subsampling pseudo-encodings from the viewer, and failing to receive those, it then checks for the TightVNC 0-9 quality pseudo-encoding. There is one case in which the two systems are not compatible, and that is when a TightVNC or TigerVNC viewer requests compression level 0 without JPEG from a TurboVNC server. For performance reasons, this causes the TurboVNC server to send images directly to the viewer, bypassing Zlib. When the TurboVNC server does this, it also sets bits 7-4 in the compression control byte to rfbTightNoZlib (0x0A), which is unfortunately the same value as rfbTightPng. Older TightVNC viewers that don't handle PNG will assume that the stream is uncompressed but still encapsulated in a Zlib structure, whereas newer PNG-supporting TightVNC viewers will assume that the stream is PNG. In either case, the viewer will probably crash. Since most VNC viewers don't expose compression level 0 in the GUI, this is a relatively rare situation. Description of changes: configure.ac -- Added support for libjpeg-turbo. If passed an argument of --with-turbovnc, configure will now run (or, if cross-compiling, just link) a test program that determines whether the libjpeg library being used is libjpeg-turbo. libjpeg-turbo must be used when building the TurboVNC encoder, because the TurboVNC encoder relies on the libjpeg-turbo colorspace extensions in order to compress images directly out of the framebuffer (which may be, for instance, BGRA rather than RGB.) libjpeg-turbo can optionally be used with the TightVNC encoder as well, but the speedup will only be marginal (the report linked above explains why in more detail, but basically it's because of Amdahl's Law. The TightVNC encoder was designed with the assumption that JPEG had a very high CPU cost, and thus JPEG is used only sparingly.) -- Added a new configure variable, JPEG_LDFLAGS. This is necessitated by the fact that libjpeg-turbo often distributes libjpeg.a and libjpeg.so in /opt/libjpeg-turbo/lib32 or /opt/libjpeg-turbo/lib64, and many people prefer to statically link with it. Thus, more flexibility is needed than is provided by --with-jpeg. If JPEG_LDFLAGS is specified, then it overrides the changes to LDFLAGS enacted by --with-jpeg (but --with-jpeg is still used to set the include path.) The addition of JPEG_LDFLAGS necessitated replacing AC_CHECK_LIB with AC_LINK_IFELSE (because AC_CHECK_LIB automatically sets LIBS to -ljpeg, which is not what we want if we're, for instance, linking statically with libjpeg-turbo.) -- configure does not check for PNG support if TurboVNC encoding is enabled. This prevents the rfbSendRectEncodingTightPng() function from being compiled in, since the TurboVNC encoder doesn't (and can't) support it. common/turbojpeg.c, common/turbojpeg.h -- TurboJPEG is a simple API used to compress and decompress JPEG images in memory. It was originally implemented because it was desirable to use different types of underlying technologies to compress JPEG on different platforms (mediaLib on SPARC, Quicktime on PPC Macs, Intel Performance Primitives, etc.) These days, however, libjpeg-turbo is the only underlying technology used by TurboVNC, so TurboJPEG's purpose is largely just code simplicity and flexibility. Thus, since there is no real need for libvncserver to use any technology other than libjpeg-turbo for compressing JPEG, the TurboJPEG wrapper for libjpeg-turbo has been included in-tree so that libvncserver can be directly linked with libjpeg-turbo. This is convenient because many modern Linux distros (Fedora, Ubuntu, etc.) now ship libjpeg-turbo as their default libjpeg library. libvncserver/rfbserver.c -- Added logic to check for the TurboVNC fine-grained quality level and subsampling encodings and to map Tight (0-9) quality levels to appropriate fine-grained quality level and subsampling values if communicating with a TightVNC/TigerVNC viewer. libvncserver/turbo.c -- TurboVNC encoder (compiled instead of libvncserver/tight.c) rfb/rfb.h -- Added support for the TurboVNC subsampling level rfb/rfbproto.h -- Added constants for the TurboVNC fine quality level and subsampling encodings as well as the rfbTightNoZlib constant and notes on its usage.
13 years ago
*/
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
/* TurboJPEG/OSS: this implements the TurboJPEG API using libjpeg-turbo */
Add TurboVNC encoding support. TurboVNC is a variant of TightVNC that uses the same client/server protocol (RFB version 3.8t), and thus it is fully cross-compatible with TightVNC and TigerVNC (with one exception, which is noted below.) Both the TightVNC and TurboVNC encoders analyze each rectangle, pick out regions of solid color to send separately, and send the remaining subrectangles using mono, indexed color, JPEG, or raw encoding, depending on the number of colors in the subrectangle. However, TurboVNC uses a fundamentally different selection algorithm to determine the appropriate subencoding to use for each subrectangle. Thus, while it sends a protocol stream that can be decoded by any TightVNC-compatible viewer, the mix of subencoding types in this protocol stream will be different from those generated by a TightVNC server. The research that led to TurboVNC is described in the following report: http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf. In summary: 20 RFB captures, representing "common" 2D and 3D application workloads (the 3D workloads were run using VirtualGL), were studied using the TightVNC encoder in isolation. Some of the analysis features in the TightVNC encoder, such as smoothness detection, were found to generate a lot of CPU usage with little or no benefit in compression, so those features were disabled. JPEG encoding was accelerated using libjpeg-turbo (which achieves a 2-4x speedup over plain libjpeg on modern x86 or ARM processors.) Finally, the "palette threshold" (minimum number of colors that the subrectangle must have before it is compressed using JPEG or raw) was adjusted to account for the fact that JPEG encoding is now quite a bit faster (meaning that we can now use it more without a CPU penalty.) TurboVNC has additional optimizations, such as the ability to count colors and encode JPEG images directly from the framebuffer without first translating the pixels into RGB. The TurboVNC encoder compares quite favorably in terms of compression ratio with TightVNC and generally encodes a great deal faster (often an order of magnitude or more.) The version of the TurboVNC encoder included in this patch is roughly equivalent to the one found in version 0.6 of the Unix TurboVNC Server, with a few minor patches integrated from TurboVNC 1.1. TurboVNC 1.0 added multi-threading capabilities, which can be added in later if desired (at the expense of making libvncserver depend on libpthread.) Because TurboVNC uses a fundamentally different mix of subencodings than TightVNC, because it uses the identical protocol (and thus a viewer really has no idea whether it's talking to a TightVNC or TurboVNC server), and because it doesn't support rfbTightPng (and in fact conflicts with it-- see below), the TurboVNC and TightVNC encoders cannot be enabled simultaneously. Compatibility: In *most* cases, a TurboVNC-enabled viewer is fully compatible with a TightVNC server, and vice versa. TurboVNC supports pseudo-encodings for specifying a fine-grained (1-100) quality scale and specifying chrominance subsampling. If a TurboVNC viewer sends those to a TightVNC server, then the TightVNC server ignores them, so the TurboVNC viewer also sends the quality on a 0-9 scale that the TightVNC server can understand. Similarly, the TurboVNC server checks first for fine-grained quality and subsampling pseudo-encodings from the viewer, and failing to receive those, it then checks for the TightVNC 0-9 quality pseudo-encoding. There is one case in which the two systems are not compatible, and that is when a TightVNC or TigerVNC viewer requests compression level 0 without JPEG from a TurboVNC server. For performance reasons, this causes the TurboVNC server to send images directly to the viewer, bypassing Zlib. When the TurboVNC server does this, it also sets bits 7-4 in the compression control byte to rfbTightNoZlib (0x0A), which is unfortunately the same value as rfbTightPng. Older TightVNC viewers that don't handle PNG will assume that the stream is uncompressed but still encapsulated in a Zlib structure, whereas newer PNG-supporting TightVNC viewers will assume that the stream is PNG. In either case, the viewer will probably crash. Since most VNC viewers don't expose compression level 0 in the GUI, this is a relatively rare situation. Description of changes: configure.ac -- Added support for libjpeg-turbo. If passed an argument of --with-turbovnc, configure will now run (or, if cross-compiling, just link) a test program that determines whether the libjpeg library being used is libjpeg-turbo. libjpeg-turbo must be used when building the TurboVNC encoder, because the TurboVNC encoder relies on the libjpeg-turbo colorspace extensions in order to compress images directly out of the framebuffer (which may be, for instance, BGRA rather than RGB.) libjpeg-turbo can optionally be used with the TightVNC encoder as well, but the speedup will only be marginal (the report linked above explains why in more detail, but basically it's because of Amdahl's Law. The TightVNC encoder was designed with the assumption that JPEG had a very high CPU cost, and thus JPEG is used only sparingly.) -- Added a new configure variable, JPEG_LDFLAGS. This is necessitated by the fact that libjpeg-turbo often distributes libjpeg.a and libjpeg.so in /opt/libjpeg-turbo/lib32 or /opt/libjpeg-turbo/lib64, and many people prefer to statically link with it. Thus, more flexibility is needed than is provided by --with-jpeg. If JPEG_LDFLAGS is specified, then it overrides the changes to LDFLAGS enacted by --with-jpeg (but --with-jpeg is still used to set the include path.) The addition of JPEG_LDFLAGS necessitated replacing AC_CHECK_LIB with AC_LINK_IFELSE (because AC_CHECK_LIB automatically sets LIBS to -ljpeg, which is not what we want if we're, for instance, linking statically with libjpeg-turbo.) -- configure does not check for PNG support if TurboVNC encoding is enabled. This prevents the rfbSendRectEncodingTightPng() function from being compiled in, since the TurboVNC encoder doesn't (and can't) support it. common/turbojpeg.c, common/turbojpeg.h -- TurboJPEG is a simple API used to compress and decompress JPEG images in memory. It was originally implemented because it was desirable to use different types of underlying technologies to compress JPEG on different platforms (mediaLib on SPARC, Quicktime on PPC Macs, Intel Performance Primitives, etc.) These days, however, libjpeg-turbo is the only underlying technology used by TurboVNC, so TurboJPEG's purpose is largely just code simplicity and flexibility. Thus, since there is no real need for libvncserver to use any technology other than libjpeg-turbo for compressing JPEG, the TurboJPEG wrapper for libjpeg-turbo has been included in-tree so that libvncserver can be directly linked with libjpeg-turbo. This is convenient because many modern Linux distros (Fedora, Ubuntu, etc.) now ship libjpeg-turbo as their default libjpeg library. libvncserver/rfbserver.c -- Added logic to check for the TurboVNC fine-grained quality level and subsampling encodings and to map Tight (0-9) quality levels to appropriate fine-grained quality level and subsampling values if communicating with a TightVNC/TigerVNC viewer. libvncserver/turbo.c -- TurboVNC encoder (compiled instead of libvncserver/tight.c) rfb/rfb.h -- Added support for the TurboVNC subsampling level rfb/rfbproto.h -- Added constants for the TurboVNC fine quality level and subsampling encodings as well as the rfbTightNoZlib constant and notes on its usage.
13 years ago
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
#ifndef JCS_EXTENSIONS
#define JPEG_INTERNAL_OPTIONS
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
#endif
#include <jpeglib.h>
#include <jerror.h>
Add TurboVNC encoding support. TurboVNC is a variant of TightVNC that uses the same client/server protocol (RFB version 3.8t), and thus it is fully cross-compatible with TightVNC and TigerVNC (with one exception, which is noted below.) Both the TightVNC and TurboVNC encoders analyze each rectangle, pick out regions of solid color to send separately, and send the remaining subrectangles using mono, indexed color, JPEG, or raw encoding, depending on the number of colors in the subrectangle. However, TurboVNC uses a fundamentally different selection algorithm to determine the appropriate subencoding to use for each subrectangle. Thus, while it sends a protocol stream that can be decoded by any TightVNC-compatible viewer, the mix of subencoding types in this protocol stream will be different from those generated by a TightVNC server. The research that led to TurboVNC is described in the following report: http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf. In summary: 20 RFB captures, representing "common" 2D and 3D application workloads (the 3D workloads were run using VirtualGL), were studied using the TightVNC encoder in isolation. Some of the analysis features in the TightVNC encoder, such as smoothness detection, were found to generate a lot of CPU usage with little or no benefit in compression, so those features were disabled. JPEG encoding was accelerated using libjpeg-turbo (which achieves a 2-4x speedup over plain libjpeg on modern x86 or ARM processors.) Finally, the "palette threshold" (minimum number of colors that the subrectangle must have before it is compressed using JPEG or raw) was adjusted to account for the fact that JPEG encoding is now quite a bit faster (meaning that we can now use it more without a CPU penalty.) TurboVNC has additional optimizations, such as the ability to count colors and encode JPEG images directly from the framebuffer without first translating the pixels into RGB. The TurboVNC encoder compares quite favorably in terms of compression ratio with TightVNC and generally encodes a great deal faster (often an order of magnitude or more.) The version of the TurboVNC encoder included in this patch is roughly equivalent to the one found in version 0.6 of the Unix TurboVNC Server, with a few minor patches integrated from TurboVNC 1.1. TurboVNC 1.0 added multi-threading capabilities, which can be added in later if desired (at the expense of making libvncserver depend on libpthread.) Because TurboVNC uses a fundamentally different mix of subencodings than TightVNC, because it uses the identical protocol (and thus a viewer really has no idea whether it's talking to a TightVNC or TurboVNC server), and because it doesn't support rfbTightPng (and in fact conflicts with it-- see below), the TurboVNC and TightVNC encoders cannot be enabled simultaneously. Compatibility: In *most* cases, a TurboVNC-enabled viewer is fully compatible with a TightVNC server, and vice versa. TurboVNC supports pseudo-encodings for specifying a fine-grained (1-100) quality scale and specifying chrominance subsampling. If a TurboVNC viewer sends those to a TightVNC server, then the TightVNC server ignores them, so the TurboVNC viewer also sends the quality on a 0-9 scale that the TightVNC server can understand. Similarly, the TurboVNC server checks first for fine-grained quality and subsampling pseudo-encodings from the viewer, and failing to receive those, it then checks for the TightVNC 0-9 quality pseudo-encoding. There is one case in which the two systems are not compatible, and that is when a TightVNC or TigerVNC viewer requests compression level 0 without JPEG from a TurboVNC server. For performance reasons, this causes the TurboVNC server to send images directly to the viewer, bypassing Zlib. When the TurboVNC server does this, it also sets bits 7-4 in the compression control byte to rfbTightNoZlib (0x0A), which is unfortunately the same value as rfbTightPng. Older TightVNC viewers that don't handle PNG will assume that the stream is uncompressed but still encapsulated in a Zlib structure, whereas newer PNG-supporting TightVNC viewers will assume that the stream is PNG. In either case, the viewer will probably crash. Since most VNC viewers don't expose compression level 0 in the GUI, this is a relatively rare situation. Description of changes: configure.ac -- Added support for libjpeg-turbo. If passed an argument of --with-turbovnc, configure will now run (or, if cross-compiling, just link) a test program that determines whether the libjpeg library being used is libjpeg-turbo. libjpeg-turbo must be used when building the TurboVNC encoder, because the TurboVNC encoder relies on the libjpeg-turbo colorspace extensions in order to compress images directly out of the framebuffer (which may be, for instance, BGRA rather than RGB.) libjpeg-turbo can optionally be used with the TightVNC encoder as well, but the speedup will only be marginal (the report linked above explains why in more detail, but basically it's because of Amdahl's Law. The TightVNC encoder was designed with the assumption that JPEG had a very high CPU cost, and thus JPEG is used only sparingly.) -- Added a new configure variable, JPEG_LDFLAGS. This is necessitated by the fact that libjpeg-turbo often distributes libjpeg.a and libjpeg.so in /opt/libjpeg-turbo/lib32 or /opt/libjpeg-turbo/lib64, and many people prefer to statically link with it. Thus, more flexibility is needed than is provided by --with-jpeg. If JPEG_LDFLAGS is specified, then it overrides the changes to LDFLAGS enacted by --with-jpeg (but --with-jpeg is still used to set the include path.) The addition of JPEG_LDFLAGS necessitated replacing AC_CHECK_LIB with AC_LINK_IFELSE (because AC_CHECK_LIB automatically sets LIBS to -ljpeg, which is not what we want if we're, for instance, linking statically with libjpeg-turbo.) -- configure does not check for PNG support if TurboVNC encoding is enabled. This prevents the rfbSendRectEncodingTightPng() function from being compiled in, since the TurboVNC encoder doesn't (and can't) support it. common/turbojpeg.c, common/turbojpeg.h -- TurboJPEG is a simple API used to compress and decompress JPEG images in memory. It was originally implemented because it was desirable to use different types of underlying technologies to compress JPEG on different platforms (mediaLib on SPARC, Quicktime on PPC Macs, Intel Performance Primitives, etc.) These days, however, libjpeg-turbo is the only underlying technology used by TurboVNC, so TurboJPEG's purpose is largely just code simplicity and flexibility. Thus, since there is no real need for libvncserver to use any technology other than libjpeg-turbo for compressing JPEG, the TurboJPEG wrapper for libjpeg-turbo has been included in-tree so that libvncserver can be directly linked with libjpeg-turbo. This is convenient because many modern Linux distros (Fedora, Ubuntu, etc.) now ship libjpeg-turbo as their default libjpeg library. libvncserver/rfbserver.c -- Added logic to check for the TurboVNC fine-grained quality level and subsampling encodings and to map Tight (0-9) quality levels to appropriate fine-grained quality level and subsampling values if communicating with a TightVNC/TigerVNC viewer. libvncserver/turbo.c -- TurboVNC encoder (compiled instead of libvncserver/tight.c) rfb/rfb.h -- Added support for the TurboVNC subsampling level rfb/rfbproto.h -- Added constants for the TurboVNC fine quality level and subsampling encodings as well as the rfbTightNoZlib constant and notes on its usage.
13 years ago
#include <setjmp.h>
#include "./turbojpeg.h"
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
#define PAD(v, p) ((v+(p)-1)&(~((p)-1)))
Add TurboVNC encoding support. TurboVNC is a variant of TightVNC that uses the same client/server protocol (RFB version 3.8t), and thus it is fully cross-compatible with TightVNC and TigerVNC (with one exception, which is noted below.) Both the TightVNC and TurboVNC encoders analyze each rectangle, pick out regions of solid color to send separately, and send the remaining subrectangles using mono, indexed color, JPEG, or raw encoding, depending on the number of colors in the subrectangle. However, TurboVNC uses a fundamentally different selection algorithm to determine the appropriate subencoding to use for each subrectangle. Thus, while it sends a protocol stream that can be decoded by any TightVNC-compatible viewer, the mix of subencoding types in this protocol stream will be different from those generated by a TightVNC server. The research that led to TurboVNC is described in the following report: http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf. In summary: 20 RFB captures, representing "common" 2D and 3D application workloads (the 3D workloads were run using VirtualGL), were studied using the TightVNC encoder in isolation. Some of the analysis features in the TightVNC encoder, such as smoothness detection, were found to generate a lot of CPU usage with little or no benefit in compression, so those features were disabled. JPEG encoding was accelerated using libjpeg-turbo (which achieves a 2-4x speedup over plain libjpeg on modern x86 or ARM processors.) Finally, the "palette threshold" (minimum number of colors that the subrectangle must have before it is compressed using JPEG or raw) was adjusted to account for the fact that JPEG encoding is now quite a bit faster (meaning that we can now use it more without a CPU penalty.) TurboVNC has additional optimizations, such as the ability to count colors and encode JPEG images directly from the framebuffer without first translating the pixels into RGB. The TurboVNC encoder compares quite favorably in terms of compression ratio with TightVNC and generally encodes a great deal faster (often an order of magnitude or more.) The version of the TurboVNC encoder included in this patch is roughly equivalent to the one found in version 0.6 of the Unix TurboVNC Server, with a few minor patches integrated from TurboVNC 1.1. TurboVNC 1.0 added multi-threading capabilities, which can be added in later if desired (at the expense of making libvncserver depend on libpthread.) Because TurboVNC uses a fundamentally different mix of subencodings than TightVNC, because it uses the identical protocol (and thus a viewer really has no idea whether it's talking to a TightVNC or TurboVNC server), and because it doesn't support rfbTightPng (and in fact conflicts with it-- see below), the TurboVNC and TightVNC encoders cannot be enabled simultaneously. Compatibility: In *most* cases, a TurboVNC-enabled viewer is fully compatible with a TightVNC server, and vice versa. TurboVNC supports pseudo-encodings for specifying a fine-grained (1-100) quality scale and specifying chrominance subsampling. If a TurboVNC viewer sends those to a TightVNC server, then the TightVNC server ignores them, so the TurboVNC viewer also sends the quality on a 0-9 scale that the TightVNC server can understand. Similarly, the TurboVNC server checks first for fine-grained quality and subsampling pseudo-encodings from the viewer, and failing to receive those, it then checks for the TightVNC 0-9 quality pseudo-encoding. There is one case in which the two systems are not compatible, and that is when a TightVNC or TigerVNC viewer requests compression level 0 without JPEG from a TurboVNC server. For performance reasons, this causes the TurboVNC server to send images directly to the viewer, bypassing Zlib. When the TurboVNC server does this, it also sets bits 7-4 in the compression control byte to rfbTightNoZlib (0x0A), which is unfortunately the same value as rfbTightPng. Older TightVNC viewers that don't handle PNG will assume that the stream is uncompressed but still encapsulated in a Zlib structure, whereas newer PNG-supporting TightVNC viewers will assume that the stream is PNG. In either case, the viewer will probably crash. Since most VNC viewers don't expose compression level 0 in the GUI, this is a relatively rare situation. Description of changes: configure.ac -- Added support for libjpeg-turbo. If passed an argument of --with-turbovnc, configure will now run (or, if cross-compiling, just link) a test program that determines whether the libjpeg library being used is libjpeg-turbo. libjpeg-turbo must be used when building the TurboVNC encoder, because the TurboVNC encoder relies on the libjpeg-turbo colorspace extensions in order to compress images directly out of the framebuffer (which may be, for instance, BGRA rather than RGB.) libjpeg-turbo can optionally be used with the TightVNC encoder as well, but the speedup will only be marginal (the report linked above explains why in more detail, but basically it's because of Amdahl's Law. The TightVNC encoder was designed with the assumption that JPEG had a very high CPU cost, and thus JPEG is used only sparingly.) -- Added a new configure variable, JPEG_LDFLAGS. This is necessitated by the fact that libjpeg-turbo often distributes libjpeg.a and libjpeg.so in /opt/libjpeg-turbo/lib32 or /opt/libjpeg-turbo/lib64, and many people prefer to statically link with it. Thus, more flexibility is needed than is provided by --with-jpeg. If JPEG_LDFLAGS is specified, then it overrides the changes to LDFLAGS enacted by --with-jpeg (but --with-jpeg is still used to set the include path.) The addition of JPEG_LDFLAGS necessitated replacing AC_CHECK_LIB with AC_LINK_IFELSE (because AC_CHECK_LIB automatically sets LIBS to -ljpeg, which is not what we want if we're, for instance, linking statically with libjpeg-turbo.) -- configure does not check for PNG support if TurboVNC encoding is enabled. This prevents the rfbSendRectEncodingTightPng() function from being compiled in, since the TurboVNC encoder doesn't (and can't) support it. common/turbojpeg.c, common/turbojpeg.h -- TurboJPEG is a simple API used to compress and decompress JPEG images in memory. It was originally implemented because it was desirable to use different types of underlying technologies to compress JPEG on different platforms (mediaLib on SPARC, Quicktime on PPC Macs, Intel Performance Primitives, etc.) These days, however, libjpeg-turbo is the only underlying technology used by TurboVNC, so TurboJPEG's purpose is largely just code simplicity and flexibility. Thus, since there is no real need for libvncserver to use any technology other than libjpeg-turbo for compressing JPEG, the TurboJPEG wrapper for libjpeg-turbo has been included in-tree so that libvncserver can be directly linked with libjpeg-turbo. This is convenient because many modern Linux distros (Fedora, Ubuntu, etc.) now ship libjpeg-turbo as their default libjpeg library. libvncserver/rfbserver.c -- Added logic to check for the TurboVNC fine-grained quality level and subsampling encodings and to map Tight (0-9) quality levels to appropriate fine-grained quality level and subsampling values if communicating with a TightVNC/TigerVNC viewer. libvncserver/turbo.c -- TurboVNC encoder (compiled instead of libvncserver/tight.c) rfb/rfb.h -- Added support for the TurboVNC subsampling level rfb/rfbproto.h -- Added constants for the TurboVNC fine quality level and subsampling encodings as well as the rfbTightNoZlib constant and notes on its usage.
13 years ago
#define CSTATE_START 100
#define DSTATE_START 200
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
#define MEMZERO(ptr, size) memset(ptr, 0, size)
#ifndef min
#define min(a,b) ((a)<(b)?(a):(b))
#endif
#ifndef max
#define max(a,b) ((a)>(b)?(a):(b))
#endif
Add TurboVNC encoding support. TurboVNC is a variant of TightVNC that uses the same client/server protocol (RFB version 3.8t), and thus it is fully cross-compatible with TightVNC and TigerVNC (with one exception, which is noted below.) Both the TightVNC and TurboVNC encoders analyze each rectangle, pick out regions of solid color to send separately, and send the remaining subrectangles using mono, indexed color, JPEG, or raw encoding, depending on the number of colors in the subrectangle. However, TurboVNC uses a fundamentally different selection algorithm to determine the appropriate subencoding to use for each subrectangle. Thus, while it sends a protocol stream that can be decoded by any TightVNC-compatible viewer, the mix of subencoding types in this protocol stream will be different from those generated by a TightVNC server. The research that led to TurboVNC is described in the following report: http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf. In summary: 20 RFB captures, representing "common" 2D and 3D application workloads (the 3D workloads were run using VirtualGL), were studied using the TightVNC encoder in isolation. Some of the analysis features in the TightVNC encoder, such as smoothness detection, were found to generate a lot of CPU usage with little or no benefit in compression, so those features were disabled. JPEG encoding was accelerated using libjpeg-turbo (which achieves a 2-4x speedup over plain libjpeg on modern x86 or ARM processors.) Finally, the "palette threshold" (minimum number of colors that the subrectangle must have before it is compressed using JPEG or raw) was adjusted to account for the fact that JPEG encoding is now quite a bit faster (meaning that we can now use it more without a CPU penalty.) TurboVNC has additional optimizations, such as the ability to count colors and encode JPEG images directly from the framebuffer without first translating the pixels into RGB. The TurboVNC encoder compares quite favorably in terms of compression ratio with TightVNC and generally encodes a great deal faster (often an order of magnitude or more.) The version of the TurboVNC encoder included in this patch is roughly equivalent to the one found in version 0.6 of the Unix TurboVNC Server, with a few minor patches integrated from TurboVNC 1.1. TurboVNC 1.0 added multi-threading capabilities, which can be added in later if desired (at the expense of making libvncserver depend on libpthread.) Because TurboVNC uses a fundamentally different mix of subencodings than TightVNC, because it uses the identical protocol (and thus a viewer really has no idea whether it's talking to a TightVNC or TurboVNC server), and because it doesn't support rfbTightPng (and in fact conflicts with it-- see below), the TurboVNC and TightVNC encoders cannot be enabled simultaneously. Compatibility: In *most* cases, a TurboVNC-enabled viewer is fully compatible with a TightVNC server, and vice versa. TurboVNC supports pseudo-encodings for specifying a fine-grained (1-100) quality scale and specifying chrominance subsampling. If a TurboVNC viewer sends those to a TightVNC server, then the TightVNC server ignores them, so the TurboVNC viewer also sends the quality on a 0-9 scale that the TightVNC server can understand. Similarly, the TurboVNC server checks first for fine-grained quality and subsampling pseudo-encodings from the viewer, and failing to receive those, it then checks for the TightVNC 0-9 quality pseudo-encoding. There is one case in which the two systems are not compatible, and that is when a TightVNC or TigerVNC viewer requests compression level 0 without JPEG from a TurboVNC server. For performance reasons, this causes the TurboVNC server to send images directly to the viewer, bypassing Zlib. When the TurboVNC server does this, it also sets bits 7-4 in the compression control byte to rfbTightNoZlib (0x0A), which is unfortunately the same value as rfbTightPng. Older TightVNC viewers that don't handle PNG will assume that the stream is uncompressed but still encapsulated in a Zlib structure, whereas newer PNG-supporting TightVNC viewers will assume that the stream is PNG. In either case, the viewer will probably crash. Since most VNC viewers don't expose compression level 0 in the GUI, this is a relatively rare situation. Description of changes: configure.ac -- Added support for libjpeg-turbo. If passed an argument of --with-turbovnc, configure will now run (or, if cross-compiling, just link) a test program that determines whether the libjpeg library being used is libjpeg-turbo. libjpeg-turbo must be used when building the TurboVNC encoder, because the TurboVNC encoder relies on the libjpeg-turbo colorspace extensions in order to compress images directly out of the framebuffer (which may be, for instance, BGRA rather than RGB.) libjpeg-turbo can optionally be used with the TightVNC encoder as well, but the speedup will only be marginal (the report linked above explains why in more detail, but basically it's because of Amdahl's Law. The TightVNC encoder was designed with the assumption that JPEG had a very high CPU cost, and thus JPEG is used only sparingly.) -- Added a new configure variable, JPEG_LDFLAGS. This is necessitated by the fact that libjpeg-turbo often distributes libjpeg.a and libjpeg.so in /opt/libjpeg-turbo/lib32 or /opt/libjpeg-turbo/lib64, and many people prefer to statically link with it. Thus, more flexibility is needed than is provided by --with-jpeg. If JPEG_LDFLAGS is specified, then it overrides the changes to LDFLAGS enacted by --with-jpeg (but --with-jpeg is still used to set the include path.) The addition of JPEG_LDFLAGS necessitated replacing AC_CHECK_LIB with AC_LINK_IFELSE (because AC_CHECK_LIB automatically sets LIBS to -ljpeg, which is not what we want if we're, for instance, linking statically with libjpeg-turbo.) -- configure does not check for PNG support if TurboVNC encoding is enabled. This prevents the rfbSendRectEncodingTightPng() function from being compiled in, since the TurboVNC encoder doesn't (and can't) support it. common/turbojpeg.c, common/turbojpeg.h -- TurboJPEG is a simple API used to compress and decompress JPEG images in memory. It was originally implemented because it was desirable to use different types of underlying technologies to compress JPEG on different platforms (mediaLib on SPARC, Quicktime on PPC Macs, Intel Performance Primitives, etc.) These days, however, libjpeg-turbo is the only underlying technology used by TurboVNC, so TurboJPEG's purpose is largely just code simplicity and flexibility. Thus, since there is no real need for libvncserver to use any technology other than libjpeg-turbo for compressing JPEG, the TurboJPEG wrapper for libjpeg-turbo has been included in-tree so that libvncserver can be directly linked with libjpeg-turbo. This is convenient because many modern Linux distros (Fedora, Ubuntu, etc.) now ship libjpeg-turbo as their default libjpeg library. libvncserver/rfbserver.c -- Added logic to check for the TurboVNC fine-grained quality level and subsampling encodings and to map Tight (0-9) quality levels to appropriate fine-grained quality level and subsampling values if communicating with a TightVNC/TigerVNC viewer. libvncserver/turbo.c -- TurboVNC encoder (compiled instead of libvncserver/tight.c) rfb/rfb.h -- Added support for the TurboVNC subsampling level rfb/rfbproto.h -- Added constants for the TurboVNC fine quality level and subsampling encodings as well as the rfbTightNoZlib constant and notes on its usage.
13 years ago
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
/* Error handling (based on example in example.c) */
Add TurboVNC encoding support. TurboVNC is a variant of TightVNC that uses the same client/server protocol (RFB version 3.8t), and thus it is fully cross-compatible with TightVNC and TigerVNC (with one exception, which is noted below.) Both the TightVNC and TurboVNC encoders analyze each rectangle, pick out regions of solid color to send separately, and send the remaining subrectangles using mono, indexed color, JPEG, or raw encoding, depending on the number of colors in the subrectangle. However, TurboVNC uses a fundamentally different selection algorithm to determine the appropriate subencoding to use for each subrectangle. Thus, while it sends a protocol stream that can be decoded by any TightVNC-compatible viewer, the mix of subencoding types in this protocol stream will be different from those generated by a TightVNC server. The research that led to TurboVNC is described in the following report: http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf. In summary: 20 RFB captures, representing "common" 2D and 3D application workloads (the 3D workloads were run using VirtualGL), were studied using the TightVNC encoder in isolation. Some of the analysis features in the TightVNC encoder, such as smoothness detection, were found to generate a lot of CPU usage with little or no benefit in compression, so those features were disabled. JPEG encoding was accelerated using libjpeg-turbo (which achieves a 2-4x speedup over plain libjpeg on modern x86 or ARM processors.) Finally, the "palette threshold" (minimum number of colors that the subrectangle must have before it is compressed using JPEG or raw) was adjusted to account for the fact that JPEG encoding is now quite a bit faster (meaning that we can now use it more without a CPU penalty.) TurboVNC has additional optimizations, such as the ability to count colors and encode JPEG images directly from the framebuffer without first translating the pixels into RGB. The TurboVNC encoder compares quite favorably in terms of compression ratio with TightVNC and generally encodes a great deal faster (often an order of magnitude or more.) The version of the TurboVNC encoder included in this patch is roughly equivalent to the one found in version 0.6 of the Unix TurboVNC Server, with a few minor patches integrated from TurboVNC 1.1. TurboVNC 1.0 added multi-threading capabilities, which can be added in later if desired (at the expense of making libvncserver depend on libpthread.) Because TurboVNC uses a fundamentally different mix of subencodings than TightVNC, because it uses the identical protocol (and thus a viewer really has no idea whether it's talking to a TightVNC or TurboVNC server), and because it doesn't support rfbTightPng (and in fact conflicts with it-- see below), the TurboVNC and TightVNC encoders cannot be enabled simultaneously. Compatibility: In *most* cases, a TurboVNC-enabled viewer is fully compatible with a TightVNC server, and vice versa. TurboVNC supports pseudo-encodings for specifying a fine-grained (1-100) quality scale and specifying chrominance subsampling. If a TurboVNC viewer sends those to a TightVNC server, then the TightVNC server ignores them, so the TurboVNC viewer also sends the quality on a 0-9 scale that the TightVNC server can understand. Similarly, the TurboVNC server checks first for fine-grained quality and subsampling pseudo-encodings from the viewer, and failing to receive those, it then checks for the TightVNC 0-9 quality pseudo-encoding. There is one case in which the two systems are not compatible, and that is when a TightVNC or TigerVNC viewer requests compression level 0 without JPEG from a TurboVNC server. For performance reasons, this causes the TurboVNC server to send images directly to the viewer, bypassing Zlib. When the TurboVNC server does this, it also sets bits 7-4 in the compression control byte to rfbTightNoZlib (0x0A), which is unfortunately the same value as rfbTightPng. Older TightVNC viewers that don't handle PNG will assume that the stream is uncompressed but still encapsulated in a Zlib structure, whereas newer PNG-supporting TightVNC viewers will assume that the stream is PNG. In either case, the viewer will probably crash. Since most VNC viewers don't expose compression level 0 in the GUI, this is a relatively rare situation. Description of changes: configure.ac -- Added support for libjpeg-turbo. If passed an argument of --with-turbovnc, configure will now run (or, if cross-compiling, just link) a test program that determines whether the libjpeg library being used is libjpeg-turbo. libjpeg-turbo must be used when building the TurboVNC encoder, because the TurboVNC encoder relies on the libjpeg-turbo colorspace extensions in order to compress images directly out of the framebuffer (which may be, for instance, BGRA rather than RGB.) libjpeg-turbo can optionally be used with the TightVNC encoder as well, but the speedup will only be marginal (the report linked above explains why in more detail, but basically it's because of Amdahl's Law. The TightVNC encoder was designed with the assumption that JPEG had a very high CPU cost, and thus JPEG is used only sparingly.) -- Added a new configure variable, JPEG_LDFLAGS. This is necessitated by the fact that libjpeg-turbo often distributes libjpeg.a and libjpeg.so in /opt/libjpeg-turbo/lib32 or /opt/libjpeg-turbo/lib64, and many people prefer to statically link with it. Thus, more flexibility is needed than is provided by --with-jpeg. If JPEG_LDFLAGS is specified, then it overrides the changes to LDFLAGS enacted by --with-jpeg (but --with-jpeg is still used to set the include path.) The addition of JPEG_LDFLAGS necessitated replacing AC_CHECK_LIB with AC_LINK_IFELSE (because AC_CHECK_LIB automatically sets LIBS to -ljpeg, which is not what we want if we're, for instance, linking statically with libjpeg-turbo.) -- configure does not check for PNG support if TurboVNC encoding is enabled. This prevents the rfbSendRectEncodingTightPng() function from being compiled in, since the TurboVNC encoder doesn't (and can't) support it. common/turbojpeg.c, common/turbojpeg.h -- TurboJPEG is a simple API used to compress and decompress JPEG images in memory. It was originally implemented because it was desirable to use different types of underlying technologies to compress JPEG on different platforms (mediaLib on SPARC, Quicktime on PPC Macs, Intel Performance Primitives, etc.) These days, however, libjpeg-turbo is the only underlying technology used by TurboVNC, so TurboJPEG's purpose is largely just code simplicity and flexibility. Thus, since there is no real need for libvncserver to use any technology other than libjpeg-turbo for compressing JPEG, the TurboJPEG wrapper for libjpeg-turbo has been included in-tree so that libvncserver can be directly linked with libjpeg-turbo. This is convenient because many modern Linux distros (Fedora, Ubuntu, etc.) now ship libjpeg-turbo as their default libjpeg library. libvncserver/rfbserver.c -- Added logic to check for the TurboVNC fine-grained quality level and subsampling encodings and to map Tight (0-9) quality levels to appropriate fine-grained quality level and subsampling values if communicating with a TightVNC/TigerVNC viewer. libvncserver/turbo.c -- TurboVNC encoder (compiled instead of libvncserver/tight.c) rfb/rfb.h -- Added support for the TurboVNC subsampling level rfb/rfbproto.h -- Added constants for the TurboVNC fine quality level and subsampling encodings as well as the rfbTightNoZlib constant and notes on its usage.
13 years ago
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
static char errStr[JMSG_LENGTH_MAX]="No error";
Add TurboVNC encoding support. TurboVNC is a variant of TightVNC that uses the same client/server protocol (RFB version 3.8t), and thus it is fully cross-compatible with TightVNC and TigerVNC (with one exception, which is noted below.) Both the TightVNC and TurboVNC encoders analyze each rectangle, pick out regions of solid color to send separately, and send the remaining subrectangles using mono, indexed color, JPEG, or raw encoding, depending on the number of colors in the subrectangle. However, TurboVNC uses a fundamentally different selection algorithm to determine the appropriate subencoding to use for each subrectangle. Thus, while it sends a protocol stream that can be decoded by any TightVNC-compatible viewer, the mix of subencoding types in this protocol stream will be different from those generated by a TightVNC server. The research that led to TurboVNC is described in the following report: http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf. In summary: 20 RFB captures, representing "common" 2D and 3D application workloads (the 3D workloads were run using VirtualGL), were studied using the TightVNC encoder in isolation. Some of the analysis features in the TightVNC encoder, such as smoothness detection, were found to generate a lot of CPU usage with little or no benefit in compression, so those features were disabled. JPEG encoding was accelerated using libjpeg-turbo (which achieves a 2-4x speedup over plain libjpeg on modern x86 or ARM processors.) Finally, the "palette threshold" (minimum number of colors that the subrectangle must have before it is compressed using JPEG or raw) was adjusted to account for the fact that JPEG encoding is now quite a bit faster (meaning that we can now use it more without a CPU penalty.) TurboVNC has additional optimizations, such as the ability to count colors and encode JPEG images directly from the framebuffer without first translating the pixels into RGB. The TurboVNC encoder compares quite favorably in terms of compression ratio with TightVNC and generally encodes a great deal faster (often an order of magnitude or more.) The version of the TurboVNC encoder included in this patch is roughly equivalent to the one found in version 0.6 of the Unix TurboVNC Server, with a few minor patches integrated from TurboVNC 1.1. TurboVNC 1.0 added multi-threading capabilities, which can be added in later if desired (at the expense of making libvncserver depend on libpthread.) Because TurboVNC uses a fundamentally different mix of subencodings than TightVNC, because it uses the identical protocol (and thus a viewer really has no idea whether it's talking to a TightVNC or TurboVNC server), and because it doesn't support rfbTightPng (and in fact conflicts with it-- see below), the TurboVNC and TightVNC encoders cannot be enabled simultaneously. Compatibility: In *most* cases, a TurboVNC-enabled viewer is fully compatible with a TightVNC server, and vice versa. TurboVNC supports pseudo-encodings for specifying a fine-grained (1-100) quality scale and specifying chrominance subsampling. If a TurboVNC viewer sends those to a TightVNC server, then the TightVNC server ignores them, so the TurboVNC viewer also sends the quality on a 0-9 scale that the TightVNC server can understand. Similarly, the TurboVNC server checks first for fine-grained quality and subsampling pseudo-encodings from the viewer, and failing to receive those, it then checks for the TightVNC 0-9 quality pseudo-encoding. There is one case in which the two systems are not compatible, and that is when a TightVNC or TigerVNC viewer requests compression level 0 without JPEG from a TurboVNC server. For performance reasons, this causes the TurboVNC server to send images directly to the viewer, bypassing Zlib. When the TurboVNC server does this, it also sets bits 7-4 in the compression control byte to rfbTightNoZlib (0x0A), which is unfortunately the same value as rfbTightPng. Older TightVNC viewers that don't handle PNG will assume that the stream is uncompressed but still encapsulated in a Zlib structure, whereas newer PNG-supporting TightVNC viewers will assume that the stream is PNG. In either case, the viewer will probably crash. Since most VNC viewers don't expose compression level 0 in the GUI, this is a relatively rare situation. Description of changes: configure.ac -- Added support for libjpeg-turbo. If passed an argument of --with-turbovnc, configure will now run (or, if cross-compiling, just link) a test program that determines whether the libjpeg library being used is libjpeg-turbo. libjpeg-turbo must be used when building the TurboVNC encoder, because the TurboVNC encoder relies on the libjpeg-turbo colorspace extensions in order to compress images directly out of the framebuffer (which may be, for instance, BGRA rather than RGB.) libjpeg-turbo can optionally be used with the TightVNC encoder as well, but the speedup will only be marginal (the report linked above explains why in more detail, but basically it's because of Amdahl's Law. The TightVNC encoder was designed with the assumption that JPEG had a very high CPU cost, and thus JPEG is used only sparingly.) -- Added a new configure variable, JPEG_LDFLAGS. This is necessitated by the fact that libjpeg-turbo often distributes libjpeg.a and libjpeg.so in /opt/libjpeg-turbo/lib32 or /opt/libjpeg-turbo/lib64, and many people prefer to statically link with it. Thus, more flexibility is needed than is provided by --with-jpeg. If JPEG_LDFLAGS is specified, then it overrides the changes to LDFLAGS enacted by --with-jpeg (but --with-jpeg is still used to set the include path.) The addition of JPEG_LDFLAGS necessitated replacing AC_CHECK_LIB with AC_LINK_IFELSE (because AC_CHECK_LIB automatically sets LIBS to -ljpeg, which is not what we want if we're, for instance, linking statically with libjpeg-turbo.) -- configure does not check for PNG support if TurboVNC encoding is enabled. This prevents the rfbSendRectEncodingTightPng() function from being compiled in, since the TurboVNC encoder doesn't (and can't) support it. common/turbojpeg.c, common/turbojpeg.h -- TurboJPEG is a simple API used to compress and decompress JPEG images in memory. It was originally implemented because it was desirable to use different types of underlying technologies to compress JPEG on different platforms (mediaLib on SPARC, Quicktime on PPC Macs, Intel Performance Primitives, etc.) These days, however, libjpeg-turbo is the only underlying technology used by TurboVNC, so TurboJPEG's purpose is largely just code simplicity and flexibility. Thus, since there is no real need for libvncserver to use any technology other than libjpeg-turbo for compressing JPEG, the TurboJPEG wrapper for libjpeg-turbo has been included in-tree so that libvncserver can be directly linked with libjpeg-turbo. This is convenient because many modern Linux distros (Fedora, Ubuntu, etc.) now ship libjpeg-turbo as their default libjpeg library. libvncserver/rfbserver.c -- Added logic to check for the TurboVNC fine-grained quality level and subsampling encodings and to map Tight (0-9) quality levels to appropriate fine-grained quality level and subsampling values if communicating with a TightVNC/TigerVNC viewer. libvncserver/turbo.c -- TurboVNC encoder (compiled instead of libvncserver/tight.c) rfb/rfb.h -- Added support for the TurboVNC subsampling level rfb/rfbproto.h -- Added constants for the TurboVNC fine quality level and subsampling encodings as well as the rfbTightNoZlib constant and notes on its usage.
13 years ago
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
struct my_error_mgr
Add TurboVNC encoding support. TurboVNC is a variant of TightVNC that uses the same client/server protocol (RFB version 3.8t), and thus it is fully cross-compatible with TightVNC and TigerVNC (with one exception, which is noted below.) Both the TightVNC and TurboVNC encoders analyze each rectangle, pick out regions of solid color to send separately, and send the remaining subrectangles using mono, indexed color, JPEG, or raw encoding, depending on the number of colors in the subrectangle. However, TurboVNC uses a fundamentally different selection algorithm to determine the appropriate subencoding to use for each subrectangle. Thus, while it sends a protocol stream that can be decoded by any TightVNC-compatible viewer, the mix of subencoding types in this protocol stream will be different from those generated by a TightVNC server. The research that led to TurboVNC is described in the following report: http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf. In summary: 20 RFB captures, representing "common" 2D and 3D application workloads (the 3D workloads were run using VirtualGL), were studied using the TightVNC encoder in isolation. Some of the analysis features in the TightVNC encoder, such as smoothness detection, were found to generate a lot of CPU usage with little or no benefit in compression, so those features were disabled. JPEG encoding was accelerated using libjpeg-turbo (which achieves a 2-4x speedup over plain libjpeg on modern x86 or ARM processors.) Finally, the "palette threshold" (minimum number of colors that the subrectangle must have before it is compressed using JPEG or raw) was adjusted to account for the fact that JPEG encoding is now quite a bit faster (meaning that we can now use it more without a CPU penalty.) TurboVNC has additional optimizations, such as the ability to count colors and encode JPEG images directly from the framebuffer without first translating the pixels into RGB. The TurboVNC encoder compares quite favorably in terms of compression ratio with TightVNC and generally encodes a great deal faster (often an order of magnitude or more.) The version of the TurboVNC encoder included in this patch is roughly equivalent to the one found in version 0.6 of the Unix TurboVNC Server, with a few minor patches integrated from TurboVNC 1.1. TurboVNC 1.0 added multi-threading capabilities, which can be added in later if desired (at the expense of making libvncserver depend on libpthread.) Because TurboVNC uses a fundamentally different mix of subencodings than TightVNC, because it uses the identical protocol (and thus a viewer really has no idea whether it's talking to a TightVNC or TurboVNC server), and because it doesn't support rfbTightPng (and in fact conflicts with it-- see below), the TurboVNC and TightVNC encoders cannot be enabled simultaneously. Compatibility: In *most* cases, a TurboVNC-enabled viewer is fully compatible with a TightVNC server, and vice versa. TurboVNC supports pseudo-encodings for specifying a fine-grained (1-100) quality scale and specifying chrominance subsampling. If a TurboVNC viewer sends those to a TightVNC server, then the TightVNC server ignores them, so the TurboVNC viewer also sends the quality on a 0-9 scale that the TightVNC server can understand. Similarly, the TurboVNC server checks first for fine-grained quality and subsampling pseudo-encodings from the viewer, and failing to receive those, it then checks for the TightVNC 0-9 quality pseudo-encoding. There is one case in which the two systems are not compatible, and that is when a TightVNC or TigerVNC viewer requests compression level 0 without JPEG from a TurboVNC server. For performance reasons, this causes the TurboVNC server to send images directly to the viewer, bypassing Zlib. When the TurboVNC server does this, it also sets bits 7-4 in the compression control byte to rfbTightNoZlib (0x0A), which is unfortunately the same value as rfbTightPng. Older TightVNC viewers that don't handle PNG will assume that the stream is uncompressed but still encapsulated in a Zlib structure, whereas newer PNG-supporting TightVNC viewers will assume that the stream is PNG. In either case, the viewer will probably crash. Since most VNC viewers don't expose compression level 0 in the GUI, this is a relatively rare situation. Description of changes: configure.ac -- Added support for libjpeg-turbo. If passed an argument of --with-turbovnc, configure will now run (or, if cross-compiling, just link) a test program that determines whether the libjpeg library being used is libjpeg-turbo. libjpeg-turbo must be used when building the TurboVNC encoder, because the TurboVNC encoder relies on the libjpeg-turbo colorspace extensions in order to compress images directly out of the framebuffer (which may be, for instance, BGRA rather than RGB.) libjpeg-turbo can optionally be used with the TightVNC encoder as well, but the speedup will only be marginal (the report linked above explains why in more detail, but basically it's because of Amdahl's Law. The TightVNC encoder was designed with the assumption that JPEG had a very high CPU cost, and thus JPEG is used only sparingly.) -- Added a new configure variable, JPEG_LDFLAGS. This is necessitated by the fact that libjpeg-turbo often distributes libjpeg.a and libjpeg.so in /opt/libjpeg-turbo/lib32 or /opt/libjpeg-turbo/lib64, and many people prefer to statically link with it. Thus, more flexibility is needed than is provided by --with-jpeg. If JPEG_LDFLAGS is specified, then it overrides the changes to LDFLAGS enacted by --with-jpeg (but --with-jpeg is still used to set the include path.) The addition of JPEG_LDFLAGS necessitated replacing AC_CHECK_LIB with AC_LINK_IFELSE (because AC_CHECK_LIB automatically sets LIBS to -ljpeg, which is not what we want if we're, for instance, linking statically with libjpeg-turbo.) -- configure does not check for PNG support if TurboVNC encoding is enabled. This prevents the rfbSendRectEncodingTightPng() function from being compiled in, since the TurboVNC encoder doesn't (and can't) support it. common/turbojpeg.c, common/turbojpeg.h -- TurboJPEG is a simple API used to compress and decompress JPEG images in memory. It was originally implemented because it was desirable to use different types of underlying technologies to compress JPEG on different platforms (mediaLib on SPARC, Quicktime on PPC Macs, Intel Performance Primitives, etc.) These days, however, libjpeg-turbo is the only underlying technology used by TurboVNC, so TurboJPEG's purpose is largely just code simplicity and flexibility. Thus, since there is no real need for libvncserver to use any technology other than libjpeg-turbo for compressing JPEG, the TurboJPEG wrapper for libjpeg-turbo has been included in-tree so that libvncserver can be directly linked with libjpeg-turbo. This is convenient because many modern Linux distros (Fedora, Ubuntu, etc.) now ship libjpeg-turbo as their default libjpeg library. libvncserver/rfbserver.c -- Added logic to check for the TurboVNC fine-grained quality level and subsampling encodings and to map Tight (0-9) quality levels to appropriate fine-grained quality level and subsampling values if communicating with a TightVNC/TigerVNC viewer. libvncserver/turbo.c -- TurboVNC encoder (compiled instead of libvncserver/tight.c) rfb/rfb.h -- Added support for the TurboVNC subsampling level rfb/rfbproto.h -- Added constants for the TurboVNC fine quality level and subsampling encodings as well as the rfbTightNoZlib constant and notes on its usage.
13 years ago
{
struct jpeg_error_mgr pub;
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
jmp_buf setjmp_buffer;
};
typedef struct my_error_mgr *my_error_ptr;
Add TurboVNC encoding support. TurboVNC is a variant of TightVNC that uses the same client/server protocol (RFB version 3.8t), and thus it is fully cross-compatible with TightVNC and TigerVNC (with one exception, which is noted below.) Both the TightVNC and TurboVNC encoders analyze each rectangle, pick out regions of solid color to send separately, and send the remaining subrectangles using mono, indexed color, JPEG, or raw encoding, depending on the number of colors in the subrectangle. However, TurboVNC uses a fundamentally different selection algorithm to determine the appropriate subencoding to use for each subrectangle. Thus, while it sends a protocol stream that can be decoded by any TightVNC-compatible viewer, the mix of subencoding types in this protocol stream will be different from those generated by a TightVNC server. The research that led to TurboVNC is described in the following report: http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf. In summary: 20 RFB captures, representing "common" 2D and 3D application workloads (the 3D workloads were run using VirtualGL), were studied using the TightVNC encoder in isolation. Some of the analysis features in the TightVNC encoder, such as smoothness detection, were found to generate a lot of CPU usage with little or no benefit in compression, so those features were disabled. JPEG encoding was accelerated using libjpeg-turbo (which achieves a 2-4x speedup over plain libjpeg on modern x86 or ARM processors.) Finally, the "palette threshold" (minimum number of colors that the subrectangle must have before it is compressed using JPEG or raw) was adjusted to account for the fact that JPEG encoding is now quite a bit faster (meaning that we can now use it more without a CPU penalty.) TurboVNC has additional optimizations, such as the ability to count colors and encode JPEG images directly from the framebuffer without first translating the pixels into RGB. The TurboVNC encoder compares quite favorably in terms of compression ratio with TightVNC and generally encodes a great deal faster (often an order of magnitude or more.) The version of the TurboVNC encoder included in this patch is roughly equivalent to the one found in version 0.6 of the Unix TurboVNC Server, with a few minor patches integrated from TurboVNC 1.1. TurboVNC 1.0 added multi-threading capabilities, which can be added in later if desired (at the expense of making libvncserver depend on libpthread.) Because TurboVNC uses a fundamentally different mix of subencodings than TightVNC, because it uses the identical protocol (and thus a viewer really has no idea whether it's talking to a TightVNC or TurboVNC server), and because it doesn't support rfbTightPng (and in fact conflicts with it-- see below), the TurboVNC and TightVNC encoders cannot be enabled simultaneously. Compatibility: In *most* cases, a TurboVNC-enabled viewer is fully compatible with a TightVNC server, and vice versa. TurboVNC supports pseudo-encodings for specifying a fine-grained (1-100) quality scale and specifying chrominance subsampling. If a TurboVNC viewer sends those to a TightVNC server, then the TightVNC server ignores them, so the TurboVNC viewer also sends the quality on a 0-9 scale that the TightVNC server can understand. Similarly, the TurboVNC server checks first for fine-grained quality and subsampling pseudo-encodings from the viewer, and failing to receive those, it then checks for the TightVNC 0-9 quality pseudo-encoding. There is one case in which the two systems are not compatible, and that is when a TightVNC or TigerVNC viewer requests compression level 0 without JPEG from a TurboVNC server. For performance reasons, this causes the TurboVNC server to send images directly to the viewer, bypassing Zlib. When the TurboVNC server does this, it also sets bits 7-4 in the compression control byte to rfbTightNoZlib (0x0A), which is unfortunately the same value as rfbTightPng. Older TightVNC viewers that don't handle PNG will assume that the stream is uncompressed but still encapsulated in a Zlib structure, whereas newer PNG-supporting TightVNC viewers will assume that the stream is PNG. In either case, the viewer will probably crash. Since most VNC viewers don't expose compression level 0 in the GUI, this is a relatively rare situation. Description of changes: configure.ac -- Added support for libjpeg-turbo. If passed an argument of --with-turbovnc, configure will now run (or, if cross-compiling, just link) a test program that determines whether the libjpeg library being used is libjpeg-turbo. libjpeg-turbo must be used when building the TurboVNC encoder, because the TurboVNC encoder relies on the libjpeg-turbo colorspace extensions in order to compress images directly out of the framebuffer (which may be, for instance, BGRA rather than RGB.) libjpeg-turbo can optionally be used with the TightVNC encoder as well, but the speedup will only be marginal (the report linked above explains why in more detail, but basically it's because of Amdahl's Law. The TightVNC encoder was designed with the assumption that JPEG had a very high CPU cost, and thus JPEG is used only sparingly.) -- Added a new configure variable, JPEG_LDFLAGS. This is necessitated by the fact that libjpeg-turbo often distributes libjpeg.a and libjpeg.so in /opt/libjpeg-turbo/lib32 or /opt/libjpeg-turbo/lib64, and many people prefer to statically link with it. Thus, more flexibility is needed than is provided by --with-jpeg. If JPEG_LDFLAGS is specified, then it overrides the changes to LDFLAGS enacted by --with-jpeg (but --with-jpeg is still used to set the include path.) The addition of JPEG_LDFLAGS necessitated replacing AC_CHECK_LIB with AC_LINK_IFELSE (because AC_CHECK_LIB automatically sets LIBS to -ljpeg, which is not what we want if we're, for instance, linking statically with libjpeg-turbo.) -- configure does not check for PNG support if TurboVNC encoding is enabled. This prevents the rfbSendRectEncodingTightPng() function from being compiled in, since the TurboVNC encoder doesn't (and can't) support it. common/turbojpeg.c, common/turbojpeg.h -- TurboJPEG is a simple API used to compress and decompress JPEG images in memory. It was originally implemented because it was desirable to use different types of underlying technologies to compress JPEG on different platforms (mediaLib on SPARC, Quicktime on PPC Macs, Intel Performance Primitives, etc.) These days, however, libjpeg-turbo is the only underlying technology used by TurboVNC, so TurboJPEG's purpose is largely just code simplicity and flexibility. Thus, since there is no real need for libvncserver to use any technology other than libjpeg-turbo for compressing JPEG, the TurboJPEG wrapper for libjpeg-turbo has been included in-tree so that libvncserver can be directly linked with libjpeg-turbo. This is convenient because many modern Linux distros (Fedora, Ubuntu, etc.) now ship libjpeg-turbo as their default libjpeg library. libvncserver/rfbserver.c -- Added logic to check for the TurboVNC fine-grained quality level and subsampling encodings and to map Tight (0-9) quality levels to appropriate fine-grained quality level and subsampling values if communicating with a TightVNC/TigerVNC viewer. libvncserver/turbo.c -- TurboVNC encoder (compiled instead of libvncserver/tight.c) rfb/rfb.h -- Added support for the TurboVNC subsampling level rfb/rfbproto.h -- Added constants for the TurboVNC fine quality level and subsampling encodings as well as the rfbTightNoZlib constant and notes on its usage.
13 years ago
static void my_error_exit(j_common_ptr cinfo)
{
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
my_error_ptr myerr=(my_error_ptr)cinfo->err;
Add TurboVNC encoding support. TurboVNC is a variant of TightVNC that uses the same client/server protocol (RFB version 3.8t), and thus it is fully cross-compatible with TightVNC and TigerVNC (with one exception, which is noted below.) Both the TightVNC and TurboVNC encoders analyze each rectangle, pick out regions of solid color to send separately, and send the remaining subrectangles using mono, indexed color, JPEG, or raw encoding, depending on the number of colors in the subrectangle. However, TurboVNC uses a fundamentally different selection algorithm to determine the appropriate subencoding to use for each subrectangle. Thus, while it sends a protocol stream that can be decoded by any TightVNC-compatible viewer, the mix of subencoding types in this protocol stream will be different from those generated by a TightVNC server. The research that led to TurboVNC is described in the following report: http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf. In summary: 20 RFB captures, representing "common" 2D and 3D application workloads (the 3D workloads were run using VirtualGL), were studied using the TightVNC encoder in isolation. Some of the analysis features in the TightVNC encoder, such as smoothness detection, were found to generate a lot of CPU usage with little or no benefit in compression, so those features were disabled. JPEG encoding was accelerated using libjpeg-turbo (which achieves a 2-4x speedup over plain libjpeg on modern x86 or ARM processors.) Finally, the "palette threshold" (minimum number of colors that the subrectangle must have before it is compressed using JPEG or raw) was adjusted to account for the fact that JPEG encoding is now quite a bit faster (meaning that we can now use it more without a CPU penalty.) TurboVNC has additional optimizations, such as the ability to count colors and encode JPEG images directly from the framebuffer without first translating the pixels into RGB. The TurboVNC encoder compares quite favorably in terms of compression ratio with TightVNC and generally encodes a great deal faster (often an order of magnitude or more.) The version of the TurboVNC encoder included in this patch is roughly equivalent to the one found in version 0.6 of the Unix TurboVNC Server, with a few minor patches integrated from TurboVNC 1.1. TurboVNC 1.0 added multi-threading capabilities, which can be added in later if desired (at the expense of making libvncserver depend on libpthread.) Because TurboVNC uses a fundamentally different mix of subencodings than TightVNC, because it uses the identical protocol (and thus a viewer really has no idea whether it's talking to a TightVNC or TurboVNC server), and because it doesn't support rfbTightPng (and in fact conflicts with it-- see below), the TurboVNC and TightVNC encoders cannot be enabled simultaneously. Compatibility: In *most* cases, a TurboVNC-enabled viewer is fully compatible with a TightVNC server, and vice versa. TurboVNC supports pseudo-encodings for specifying a fine-grained (1-100) quality scale and specifying chrominance subsampling. If a TurboVNC viewer sends those to a TightVNC server, then the TightVNC server ignores them, so the TurboVNC viewer also sends the quality on a 0-9 scale that the TightVNC server can understand. Similarly, the TurboVNC server checks first for fine-grained quality and subsampling pseudo-encodings from the viewer, and failing to receive those, it then checks for the TightVNC 0-9 quality pseudo-encoding. There is one case in which the two systems are not compatible, and that is when a TightVNC or TigerVNC viewer requests compression level 0 without JPEG from a TurboVNC server. For performance reasons, this causes the TurboVNC server to send images directly to the viewer, bypassing Zlib. When the TurboVNC server does this, it also sets bits 7-4 in the compression control byte to rfbTightNoZlib (0x0A), which is unfortunately the same value as rfbTightPng. Older TightVNC viewers that don't handle PNG will assume that the stream is uncompressed but still encapsulated in a Zlib structure, whereas newer PNG-supporting TightVNC viewers will assume that the stream is PNG. In either case, the viewer will probably crash. Since most VNC viewers don't expose compression level 0 in the GUI, this is a relatively rare situation. Description of changes: configure.ac -- Added support for libjpeg-turbo. If passed an argument of --with-turbovnc, configure will now run (or, if cross-compiling, just link) a test program that determines whether the libjpeg library being used is libjpeg-turbo. libjpeg-turbo must be used when building the TurboVNC encoder, because the TurboVNC encoder relies on the libjpeg-turbo colorspace extensions in order to compress images directly out of the framebuffer (which may be, for instance, BGRA rather than RGB.) libjpeg-turbo can optionally be used with the TightVNC encoder as well, but the speedup will only be marginal (the report linked above explains why in more detail, but basically it's because of Amdahl's Law. The TightVNC encoder was designed with the assumption that JPEG had a very high CPU cost, and thus JPEG is used only sparingly.) -- Added a new configure variable, JPEG_LDFLAGS. This is necessitated by the fact that libjpeg-turbo often distributes libjpeg.a and libjpeg.so in /opt/libjpeg-turbo/lib32 or /opt/libjpeg-turbo/lib64, and many people prefer to statically link with it. Thus, more flexibility is needed than is provided by --with-jpeg. If JPEG_LDFLAGS is specified, then it overrides the changes to LDFLAGS enacted by --with-jpeg (but --with-jpeg is still used to set the include path.) The addition of JPEG_LDFLAGS necessitated replacing AC_CHECK_LIB with AC_LINK_IFELSE (because AC_CHECK_LIB automatically sets LIBS to -ljpeg, which is not what we want if we're, for instance, linking statically with libjpeg-turbo.) -- configure does not check for PNG support if TurboVNC encoding is enabled. This prevents the rfbSendRectEncodingTightPng() function from being compiled in, since the TurboVNC encoder doesn't (and can't) support it. common/turbojpeg.c, common/turbojpeg.h -- TurboJPEG is a simple API used to compress and decompress JPEG images in memory. It was originally implemented because it was desirable to use different types of underlying technologies to compress JPEG on different platforms (mediaLib on SPARC, Quicktime on PPC Macs, Intel Performance Primitives, etc.) These days, however, libjpeg-turbo is the only underlying technology used by TurboVNC, so TurboJPEG's purpose is largely just code simplicity and flexibility. Thus, since there is no real need for libvncserver to use any technology other than libjpeg-turbo for compressing JPEG, the TurboJPEG wrapper for libjpeg-turbo has been included in-tree so that libvncserver can be directly linked with libjpeg-turbo. This is convenient because many modern Linux distros (Fedora, Ubuntu, etc.) now ship libjpeg-turbo as their default libjpeg library. libvncserver/rfbserver.c -- Added logic to check for the TurboVNC fine-grained quality level and subsampling encodings and to map Tight (0-9) quality levels to appropriate fine-grained quality level and subsampling values if communicating with a TightVNC/TigerVNC viewer. libvncserver/turbo.c -- TurboVNC encoder (compiled instead of libvncserver/tight.c) rfb/rfb.h -- Added support for the TurboVNC subsampling level rfb/rfbproto.h -- Added constants for the TurboVNC fine quality level and subsampling encodings as well as the rfbTightNoZlib constant and notes on its usage.
13 years ago
(*cinfo->err->output_message)(cinfo);
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
longjmp(myerr->setjmp_buffer, 1);
Add TurboVNC encoding support. TurboVNC is a variant of TightVNC that uses the same client/server protocol (RFB version 3.8t), and thus it is fully cross-compatible with TightVNC and TigerVNC (with one exception, which is noted below.) Both the TightVNC and TurboVNC encoders analyze each rectangle, pick out regions of solid color to send separately, and send the remaining subrectangles using mono, indexed color, JPEG, or raw encoding, depending on the number of colors in the subrectangle. However, TurboVNC uses a fundamentally different selection algorithm to determine the appropriate subencoding to use for each subrectangle. Thus, while it sends a protocol stream that can be decoded by any TightVNC-compatible viewer, the mix of subencoding types in this protocol stream will be different from those generated by a TightVNC server. The research that led to TurboVNC is described in the following report: http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf. In summary: 20 RFB captures, representing "common" 2D and 3D application workloads (the 3D workloads were run using VirtualGL), were studied using the TightVNC encoder in isolation. Some of the analysis features in the TightVNC encoder, such as smoothness detection, were found to generate a lot of CPU usage with little or no benefit in compression, so those features were disabled. JPEG encoding was accelerated using libjpeg-turbo (which achieves a 2-4x speedup over plain libjpeg on modern x86 or ARM processors.) Finally, the "palette threshold" (minimum number of colors that the subrectangle must have before it is compressed using JPEG or raw) was adjusted to account for the fact that JPEG encoding is now quite a bit faster (meaning that we can now use it more without a CPU penalty.) TurboVNC has additional optimizations, such as the ability to count colors and encode JPEG images directly from the framebuffer without first translating the pixels into RGB. The TurboVNC encoder compares quite favorably in terms of compression ratio with TightVNC and generally encodes a great deal faster (often an order of magnitude or more.) The version of the TurboVNC encoder included in this patch is roughly equivalent to the one found in version 0.6 of the Unix TurboVNC Server, with a few minor patches integrated from TurboVNC 1.1. TurboVNC 1.0 added multi-threading capabilities, which can be added in later if desired (at the expense of making libvncserver depend on libpthread.) Because TurboVNC uses a fundamentally different mix of subencodings than TightVNC, because it uses the identical protocol (and thus a viewer really has no idea whether it's talking to a TightVNC or TurboVNC server), and because it doesn't support rfbTightPng (and in fact conflicts with it-- see below), the TurboVNC and TightVNC encoders cannot be enabled simultaneously. Compatibility: In *most* cases, a TurboVNC-enabled viewer is fully compatible with a TightVNC server, and vice versa. TurboVNC supports pseudo-encodings for specifying a fine-grained (1-100) quality scale and specifying chrominance subsampling. If a TurboVNC viewer sends those to a TightVNC server, then the TightVNC server ignores them, so the TurboVNC viewer also sends the quality on a 0-9 scale that the TightVNC server can understand. Similarly, the TurboVNC server checks first for fine-grained quality and subsampling pseudo-encodings from the viewer, and failing to receive those, it then checks for the TightVNC 0-9 quality pseudo-encoding. There is one case in which the two systems are not compatible, and that is when a TightVNC or TigerVNC viewer requests compression level 0 without JPEG from a TurboVNC server. For performance reasons, this causes the TurboVNC server to send images directly to the viewer, bypassing Zlib. When the TurboVNC server does this, it also sets bits 7-4 in the compression control byte to rfbTightNoZlib (0x0A), which is unfortunately the same value as rfbTightPng. Older TightVNC viewers that don't handle PNG will assume that the stream is uncompressed but still encapsulated in a Zlib structure, whereas newer PNG-supporting TightVNC viewers will assume that the stream is PNG. In either case, the viewer will probably crash. Since most VNC viewers don't expose compression level 0 in the GUI, this is a relatively rare situation. Description of changes: configure.ac -- Added support for libjpeg-turbo. If passed an argument of --with-turbovnc, configure will now run (or, if cross-compiling, just link) a test program that determines whether the libjpeg library being used is libjpeg-turbo. libjpeg-turbo must be used when building the TurboVNC encoder, because the TurboVNC encoder relies on the libjpeg-turbo colorspace extensions in order to compress images directly out of the framebuffer (which may be, for instance, BGRA rather than RGB.) libjpeg-turbo can optionally be used with the TightVNC encoder as well, but the speedup will only be marginal (the report linked above explains why in more detail, but basically it's because of Amdahl's Law. The TightVNC encoder was designed with the assumption that JPEG had a very high CPU cost, and thus JPEG is used only sparingly.) -- Added a new configure variable, JPEG_LDFLAGS. This is necessitated by the fact that libjpeg-turbo often distributes libjpeg.a and libjpeg.so in /opt/libjpeg-turbo/lib32 or /opt/libjpeg-turbo/lib64, and many people prefer to statically link with it. Thus, more flexibility is needed than is provided by --with-jpeg. If JPEG_LDFLAGS is specified, then it overrides the changes to LDFLAGS enacted by --with-jpeg (but --with-jpeg is still used to set the include path.) The addition of JPEG_LDFLAGS necessitated replacing AC_CHECK_LIB with AC_LINK_IFELSE (because AC_CHECK_LIB automatically sets LIBS to -ljpeg, which is not what we want if we're, for instance, linking statically with libjpeg-turbo.) -- configure does not check for PNG support if TurboVNC encoding is enabled. This prevents the rfbSendRectEncodingTightPng() function from being compiled in, since the TurboVNC encoder doesn't (and can't) support it. common/turbojpeg.c, common/turbojpeg.h -- TurboJPEG is a simple API used to compress and decompress JPEG images in memory. It was originally implemented because it was desirable to use different types of underlying technologies to compress JPEG on different platforms (mediaLib on SPARC, Quicktime on PPC Macs, Intel Performance Primitives, etc.) These days, however, libjpeg-turbo is the only underlying technology used by TurboVNC, so TurboJPEG's purpose is largely just code simplicity and flexibility. Thus, since there is no real need for libvncserver to use any technology other than libjpeg-turbo for compressing JPEG, the TurboJPEG wrapper for libjpeg-turbo has been included in-tree so that libvncserver can be directly linked with libjpeg-turbo. This is convenient because many modern Linux distros (Fedora, Ubuntu, etc.) now ship libjpeg-turbo as their default libjpeg library. libvncserver/rfbserver.c -- Added logic to check for the TurboVNC fine-grained quality level and subsampling encodings and to map Tight (0-9) quality levels to appropriate fine-grained quality level and subsampling values if communicating with a TightVNC/TigerVNC viewer. libvncserver/turbo.c -- TurboVNC encoder (compiled instead of libvncserver/tight.c) rfb/rfb.h -- Added support for the TurboVNC subsampling level rfb/rfbproto.h -- Added constants for the TurboVNC fine quality level and subsampling encodings as well as the rfbTightNoZlib constant and notes on its usage.
13 years ago
}
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
/* Based on output_message() in jerror.c */
Add TurboVNC encoding support. TurboVNC is a variant of TightVNC that uses the same client/server protocol (RFB version 3.8t), and thus it is fully cross-compatible with TightVNC and TigerVNC (with one exception, which is noted below.) Both the TightVNC and TurboVNC encoders analyze each rectangle, pick out regions of solid color to send separately, and send the remaining subrectangles using mono, indexed color, JPEG, or raw encoding, depending on the number of colors in the subrectangle. However, TurboVNC uses a fundamentally different selection algorithm to determine the appropriate subencoding to use for each subrectangle. Thus, while it sends a protocol stream that can be decoded by any TightVNC-compatible viewer, the mix of subencoding types in this protocol stream will be different from those generated by a TightVNC server. The research that led to TurboVNC is described in the following report: http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf. In summary: 20 RFB captures, representing "common" 2D and 3D application workloads (the 3D workloads were run using VirtualGL), were studied using the TightVNC encoder in isolation. Some of the analysis features in the TightVNC encoder, such as smoothness detection, were found to generate a lot of CPU usage with little or no benefit in compression, so those features were disabled. JPEG encoding was accelerated using libjpeg-turbo (which achieves a 2-4x speedup over plain libjpeg on modern x86 or ARM processors.) Finally, the "palette threshold" (minimum number of colors that the subrectangle must have before it is compressed using JPEG or raw) was adjusted to account for the fact that JPEG encoding is now quite a bit faster (meaning that we can now use it more without a CPU penalty.) TurboVNC has additional optimizations, such as the ability to count colors and encode JPEG images directly from the framebuffer without first translating the pixels into RGB. The TurboVNC encoder compares quite favorably in terms of compression ratio with TightVNC and generally encodes a great deal faster (often an order of magnitude or more.) The version of the TurboVNC encoder included in this patch is roughly equivalent to the one found in version 0.6 of the Unix TurboVNC Server, with a few minor patches integrated from TurboVNC 1.1. TurboVNC 1.0 added multi-threading capabilities, which can be added in later if desired (at the expense of making libvncserver depend on libpthread.) Because TurboVNC uses a fundamentally different mix of subencodings than TightVNC, because it uses the identical protocol (and thus a viewer really has no idea whether it's talking to a TightVNC or TurboVNC server), and because it doesn't support rfbTightPng (and in fact conflicts with it-- see below), the TurboVNC and TightVNC encoders cannot be enabled simultaneously. Compatibility: In *most* cases, a TurboVNC-enabled viewer is fully compatible with a TightVNC server, and vice versa. TurboVNC supports pseudo-encodings for specifying a fine-grained (1-100) quality scale and specifying chrominance subsampling. If a TurboVNC viewer sends those to a TightVNC server, then the TightVNC server ignores them, so the TurboVNC viewer also sends the quality on a 0-9 scale that the TightVNC server can understand. Similarly, the TurboVNC server checks first for fine-grained quality and subsampling pseudo-encodings from the viewer, and failing to receive those, it then checks for the TightVNC 0-9 quality pseudo-encoding. There is one case in which the two systems are not compatible, and that is when a TightVNC or TigerVNC viewer requests compression level 0 without JPEG from a TurboVNC server. For performance reasons, this causes the TurboVNC server to send images directly to the viewer, bypassing Zlib. When the TurboVNC server does this, it also sets bits 7-4 in the compression control byte to rfbTightNoZlib (0x0A), which is unfortunately the same value as rfbTightPng. Older TightVNC viewers that don't handle PNG will assume that the stream is uncompressed but still encapsulated in a Zlib structure, whereas newer PNG-supporting TightVNC viewers will assume that the stream is PNG. In either case, the viewer will probably crash. Since most VNC viewers don't expose compression level 0 in the GUI, this is a relatively rare situation. Description of changes: configure.ac -- Added support for libjpeg-turbo. If passed an argument of --with-turbovnc, configure will now run (or, if cross-compiling, just link) a test program that determines whether the libjpeg library being used is libjpeg-turbo. libjpeg-turbo must be used when building the TurboVNC encoder, because the TurboVNC encoder relies on the libjpeg-turbo colorspace extensions in order to compress images directly out of the framebuffer (which may be, for instance, BGRA rather than RGB.) libjpeg-turbo can optionally be used with the TightVNC encoder as well, but the speedup will only be marginal (the report linked above explains why in more detail, but basically it's because of Amdahl's Law. The TightVNC encoder was designed with the assumption that JPEG had a very high CPU cost, and thus JPEG is used only sparingly.) -- Added a new configure variable, JPEG_LDFLAGS. This is necessitated by the fact that libjpeg-turbo often distributes libjpeg.a and libjpeg.so in /opt/libjpeg-turbo/lib32 or /opt/libjpeg-turbo/lib64, and many people prefer to statically link with it. Thus, more flexibility is needed than is provided by --with-jpeg. If JPEG_LDFLAGS is specified, then it overrides the changes to LDFLAGS enacted by --with-jpeg (but --with-jpeg is still used to set the include path.) The addition of JPEG_LDFLAGS necessitated replacing AC_CHECK_LIB with AC_LINK_IFELSE (because AC_CHECK_LIB automatically sets LIBS to -ljpeg, which is not what we want if we're, for instance, linking statically with libjpeg-turbo.) -- configure does not check for PNG support if TurboVNC encoding is enabled. This prevents the rfbSendRectEncodingTightPng() function from being compiled in, since the TurboVNC encoder doesn't (and can't) support it. common/turbojpeg.c, common/turbojpeg.h -- TurboJPEG is a simple API used to compress and decompress JPEG images in memory. It was originally implemented because it was desirable to use different types of underlying technologies to compress JPEG on different platforms (mediaLib on SPARC, Quicktime on PPC Macs, Intel Performance Primitives, etc.) These days, however, libjpeg-turbo is the only underlying technology used by TurboVNC, so TurboJPEG's purpose is largely just code simplicity and flexibility. Thus, since there is no real need for libvncserver to use any technology other than libjpeg-turbo for compressing JPEG, the TurboJPEG wrapper for libjpeg-turbo has been included in-tree so that libvncserver can be directly linked with libjpeg-turbo. This is convenient because many modern Linux distros (Fedora, Ubuntu, etc.) now ship libjpeg-turbo as their default libjpeg library. libvncserver/rfbserver.c -- Added logic to check for the TurboVNC fine-grained quality level and subsampling encodings and to map Tight (0-9) quality levels to appropriate fine-grained quality level and subsampling values if communicating with a TightVNC/TigerVNC viewer. libvncserver/turbo.c -- TurboVNC encoder (compiled instead of libvncserver/tight.c) rfb/rfb.h -- Added support for the TurboVNC subsampling level rfb/rfbproto.h -- Added constants for the TurboVNC fine quality level and subsampling encodings as well as the rfbTightNoZlib constant and notes on its usage.
13 years ago
static void my_output_message(j_common_ptr cinfo)
{
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
(*cinfo->err->format_message)(cinfo, errStr);
Add TurboVNC encoding support. TurboVNC is a variant of TightVNC that uses the same client/server protocol (RFB version 3.8t), and thus it is fully cross-compatible with TightVNC and TigerVNC (with one exception, which is noted below.) Both the TightVNC and TurboVNC encoders analyze each rectangle, pick out regions of solid color to send separately, and send the remaining subrectangles using mono, indexed color, JPEG, or raw encoding, depending on the number of colors in the subrectangle. However, TurboVNC uses a fundamentally different selection algorithm to determine the appropriate subencoding to use for each subrectangle. Thus, while it sends a protocol stream that can be decoded by any TightVNC-compatible viewer, the mix of subencoding types in this protocol stream will be different from those generated by a TightVNC server. The research that led to TurboVNC is described in the following report: http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf. In summary: 20 RFB captures, representing "common" 2D and 3D application workloads (the 3D workloads were run using VirtualGL), were studied using the TightVNC encoder in isolation. Some of the analysis features in the TightVNC encoder, such as smoothness detection, were found to generate a lot of CPU usage with little or no benefit in compression, so those features were disabled. JPEG encoding was accelerated using libjpeg-turbo (which achieves a 2-4x speedup over plain libjpeg on modern x86 or ARM processors.) Finally, the "palette threshold" (minimum number of colors that the subrectangle must have before it is compressed using JPEG or raw) was adjusted to account for the fact that JPEG encoding is now quite a bit faster (meaning that we can now use it more without a CPU penalty.) TurboVNC has additional optimizations, such as the ability to count colors and encode JPEG images directly from the framebuffer without first translating the pixels into RGB. The TurboVNC encoder compares quite favorably in terms of compression ratio with TightVNC and generally encodes a great deal faster (often an order of magnitude or more.) The version of the TurboVNC encoder included in this patch is roughly equivalent to the one found in version 0.6 of the Unix TurboVNC Server, with a few minor patches integrated from TurboVNC 1.1. TurboVNC 1.0 added multi-threading capabilities, which can be added in later if desired (at the expense of making libvncserver depend on libpthread.) Because TurboVNC uses a fundamentally different mix of subencodings than TightVNC, because it uses the identical protocol (and thus a viewer really has no idea whether it's talking to a TightVNC or TurboVNC server), and because it doesn't support rfbTightPng (and in fact conflicts with it-- see below), the TurboVNC and TightVNC encoders cannot be enabled simultaneously. Compatibility: In *most* cases, a TurboVNC-enabled viewer is fully compatible with a TightVNC server, and vice versa. TurboVNC supports pseudo-encodings for specifying a fine-grained (1-100) quality scale and specifying chrominance subsampling. If a TurboVNC viewer sends those to a TightVNC server, then the TightVNC server ignores them, so the TurboVNC viewer also sends the quality on a 0-9 scale that the TightVNC server can understand. Similarly, the TurboVNC server checks first for fine-grained quality and subsampling pseudo-encodings from the viewer, and failing to receive those, it then checks for the TightVNC 0-9 quality pseudo-encoding. There is one case in which the two systems are not compatible, and that is when a TightVNC or TigerVNC viewer requests compression level 0 without JPEG from a TurboVNC server. For performance reasons, this causes the TurboVNC server to send images directly to the viewer, bypassing Zlib. When the TurboVNC server does this, it also sets bits 7-4 in the compression control byte to rfbTightNoZlib (0x0A), which is unfortunately the same value as rfbTightPng. Older TightVNC viewers that don't handle PNG will assume that the stream is uncompressed but still encapsulated in a Zlib structure, whereas newer PNG-supporting TightVNC viewers will assume that the stream is PNG. In either case, the viewer will probably crash. Since most VNC viewers don't expose compression level 0 in the GUI, this is a relatively rare situation. Description of changes: configure.ac -- Added support for libjpeg-turbo. If passed an argument of --with-turbovnc, configure will now run (or, if cross-compiling, just link) a test program that determines whether the libjpeg library being used is libjpeg-turbo. libjpeg-turbo must be used when building the TurboVNC encoder, because the TurboVNC encoder relies on the libjpeg-turbo colorspace extensions in order to compress images directly out of the framebuffer (which may be, for instance, BGRA rather than RGB.) libjpeg-turbo can optionally be used with the TightVNC encoder as well, but the speedup will only be marginal (the report linked above explains why in more detail, but basically it's because of Amdahl's Law. The TightVNC encoder was designed with the assumption that JPEG had a very high CPU cost, and thus JPEG is used only sparingly.) -- Added a new configure variable, JPEG_LDFLAGS. This is necessitated by the fact that libjpeg-turbo often distributes libjpeg.a and libjpeg.so in /opt/libjpeg-turbo/lib32 or /opt/libjpeg-turbo/lib64, and many people prefer to statically link with it. Thus, more flexibility is needed than is provided by --with-jpeg. If JPEG_LDFLAGS is specified, then it overrides the changes to LDFLAGS enacted by --with-jpeg (but --with-jpeg is still used to set the include path.) The addition of JPEG_LDFLAGS necessitated replacing AC_CHECK_LIB with AC_LINK_IFELSE (because AC_CHECK_LIB automatically sets LIBS to -ljpeg, which is not what we want if we're, for instance, linking statically with libjpeg-turbo.) -- configure does not check for PNG support if TurboVNC encoding is enabled. This prevents the rfbSendRectEncodingTightPng() function from being compiled in, since the TurboVNC encoder doesn't (and can't) support it. common/turbojpeg.c, common/turbojpeg.h -- TurboJPEG is a simple API used to compress and decompress JPEG images in memory. It was originally implemented because it was desirable to use different types of underlying technologies to compress JPEG on different platforms (mediaLib on SPARC, Quicktime on PPC Macs, Intel Performance Primitives, etc.) These days, however, libjpeg-turbo is the only underlying technology used by TurboVNC, so TurboJPEG's purpose is largely just code simplicity and flexibility. Thus, since there is no real need for libvncserver to use any technology other than libjpeg-turbo for compressing JPEG, the TurboJPEG wrapper for libjpeg-turbo has been included in-tree so that libvncserver can be directly linked with libjpeg-turbo. This is convenient because many modern Linux distros (Fedora, Ubuntu, etc.) now ship libjpeg-turbo as their default libjpeg library. libvncserver/rfbserver.c -- Added logic to check for the TurboVNC fine-grained quality level and subsampling encodings and to map Tight (0-9) quality levels to appropriate fine-grained quality level and subsampling values if communicating with a TightVNC/TigerVNC viewer. libvncserver/turbo.c -- TurboVNC encoder (compiled instead of libvncserver/tight.c) rfb/rfb.h -- Added support for the TurboVNC subsampling level rfb/rfbproto.h -- Added constants for the TurboVNC fine quality level and subsampling encodings as well as the rfbTightNoZlib constant and notes on its usage.
13 years ago
}
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
/* Global structures, macros, etc. */
enum {COMPRESS=1, DECOMPRESS=2};
Add TurboVNC encoding support. TurboVNC is a variant of TightVNC that uses the same client/server protocol (RFB version 3.8t), and thus it is fully cross-compatible with TightVNC and TigerVNC (with one exception, which is noted below.) Both the TightVNC and TurboVNC encoders analyze each rectangle, pick out regions of solid color to send separately, and send the remaining subrectangles using mono, indexed color, JPEG, or raw encoding, depending on the number of colors in the subrectangle. However, TurboVNC uses a fundamentally different selection algorithm to determine the appropriate subencoding to use for each subrectangle. Thus, while it sends a protocol stream that can be decoded by any TightVNC-compatible viewer, the mix of subencoding types in this protocol stream will be different from those generated by a TightVNC server. The research that led to TurboVNC is described in the following report: http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf. In summary: 20 RFB captures, representing "common" 2D and 3D application workloads (the 3D workloads were run using VirtualGL), were studied using the TightVNC encoder in isolation. Some of the analysis features in the TightVNC encoder, such as smoothness detection, were found to generate a lot of CPU usage with little or no benefit in compression, so those features were disabled. JPEG encoding was accelerated using libjpeg-turbo (which achieves a 2-4x speedup over plain libjpeg on modern x86 or ARM processors.) Finally, the "palette threshold" (minimum number of colors that the subrectangle must have before it is compressed using JPEG or raw) was adjusted to account for the fact that JPEG encoding is now quite a bit faster (meaning that we can now use it more without a CPU penalty.) TurboVNC has additional optimizations, such as the ability to count colors and encode JPEG images directly from the framebuffer without first translating the pixels into RGB. The TurboVNC encoder compares quite favorably in terms of compression ratio with TightVNC and generally encodes a great deal faster (often an order of magnitude or more.) The version of the TurboVNC encoder included in this patch is roughly equivalent to the one found in version 0.6 of the Unix TurboVNC Server, with a few minor patches integrated from TurboVNC 1.1. TurboVNC 1.0 added multi-threading capabilities, which can be added in later if desired (at the expense of making libvncserver depend on libpthread.) Because TurboVNC uses a fundamentally different mix of subencodings than TightVNC, because it uses the identical protocol (and thus a viewer really has no idea whether it's talking to a TightVNC or TurboVNC server), and because it doesn't support rfbTightPng (and in fact conflicts with it-- see below), the TurboVNC and TightVNC encoders cannot be enabled simultaneously. Compatibility: In *most* cases, a TurboVNC-enabled viewer is fully compatible with a TightVNC server, and vice versa. TurboVNC supports pseudo-encodings for specifying a fine-grained (1-100) quality scale and specifying chrominance subsampling. If a TurboVNC viewer sends those to a TightVNC server, then the TightVNC server ignores them, so the TurboVNC viewer also sends the quality on a 0-9 scale that the TightVNC server can understand. Similarly, the TurboVNC server checks first for fine-grained quality and subsampling pseudo-encodings from the viewer, and failing to receive those, it then checks for the TightVNC 0-9 quality pseudo-encoding. There is one case in which the two systems are not compatible, and that is when a TightVNC or TigerVNC viewer requests compression level 0 without JPEG from a TurboVNC server. For performance reasons, this causes the TurboVNC server to send images directly to the viewer, bypassing Zlib. When the TurboVNC server does this, it also sets bits 7-4 in the compression control byte to rfbTightNoZlib (0x0A), which is unfortunately the same value as rfbTightPng. Older TightVNC viewers that don't handle PNG will assume that the stream is uncompressed but still encapsulated in a Zlib structure, whereas newer PNG-supporting TightVNC viewers will assume that the stream is PNG. In either case, the viewer will probably crash. Since most VNC viewers don't expose compression level 0 in the GUI, this is a relatively rare situation. Description of changes: configure.ac -- Added support for libjpeg-turbo. If passed an argument of --with-turbovnc, configure will now run (or, if cross-compiling, just link) a test program that determines whether the libjpeg library being used is libjpeg-turbo. libjpeg-turbo must be used when building the TurboVNC encoder, because the TurboVNC encoder relies on the libjpeg-turbo colorspace extensions in order to compress images directly out of the framebuffer (which may be, for instance, BGRA rather than RGB.) libjpeg-turbo can optionally be used with the TightVNC encoder as well, but the speedup will only be marginal (the report linked above explains why in more detail, but basically it's because of Amdahl's Law. The TightVNC encoder was designed with the assumption that JPEG had a very high CPU cost, and thus JPEG is used only sparingly.) -- Added a new configure variable, JPEG_LDFLAGS. This is necessitated by the fact that libjpeg-turbo often distributes libjpeg.a and libjpeg.so in /opt/libjpeg-turbo/lib32 or /opt/libjpeg-turbo/lib64, and many people prefer to statically link with it. Thus, more flexibility is needed than is provided by --with-jpeg. If JPEG_LDFLAGS is specified, then it overrides the changes to LDFLAGS enacted by --with-jpeg (but --with-jpeg is still used to set the include path.) The addition of JPEG_LDFLAGS necessitated replacing AC_CHECK_LIB with AC_LINK_IFELSE (because AC_CHECK_LIB automatically sets LIBS to -ljpeg, which is not what we want if we're, for instance, linking statically with libjpeg-turbo.) -- configure does not check for PNG support if TurboVNC encoding is enabled. This prevents the rfbSendRectEncodingTightPng() function from being compiled in, since the TurboVNC encoder doesn't (and can't) support it. common/turbojpeg.c, common/turbojpeg.h -- TurboJPEG is a simple API used to compress and decompress JPEG images in memory. It was originally implemented because it was desirable to use different types of underlying technologies to compress JPEG on different platforms (mediaLib on SPARC, Quicktime on PPC Macs, Intel Performance Primitives, etc.) These days, however, libjpeg-turbo is the only underlying technology used by TurboVNC, so TurboJPEG's purpose is largely just code simplicity and flexibility. Thus, since there is no real need for libvncserver to use any technology other than libjpeg-turbo for compressing JPEG, the TurboJPEG wrapper for libjpeg-turbo has been included in-tree so that libvncserver can be directly linked with libjpeg-turbo. This is convenient because many modern Linux distros (Fedora, Ubuntu, etc.) now ship libjpeg-turbo as their default libjpeg library. libvncserver/rfbserver.c -- Added logic to check for the TurboVNC fine-grained quality level and subsampling encodings and to map Tight (0-9) quality levels to appropriate fine-grained quality level and subsampling values if communicating with a TightVNC/TigerVNC viewer. libvncserver/turbo.c -- TurboVNC encoder (compiled instead of libvncserver/tight.c) rfb/rfb.h -- Added support for the TurboVNC subsampling level rfb/rfbproto.h -- Added constants for the TurboVNC fine quality level and subsampling encodings as well as the rfbTightNoZlib constant and notes on its usage.
13 years ago
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
typedef struct _tjinstance
Add TurboVNC encoding support. TurboVNC is a variant of TightVNC that uses the same client/server protocol (RFB version 3.8t), and thus it is fully cross-compatible with TightVNC and TigerVNC (with one exception, which is noted below.) Both the TightVNC and TurboVNC encoders analyze each rectangle, pick out regions of solid color to send separately, and send the remaining subrectangles using mono, indexed color, JPEG, or raw encoding, depending on the number of colors in the subrectangle. However, TurboVNC uses a fundamentally different selection algorithm to determine the appropriate subencoding to use for each subrectangle. Thus, while it sends a protocol stream that can be decoded by any TightVNC-compatible viewer, the mix of subencoding types in this protocol stream will be different from those generated by a TightVNC server. The research that led to TurboVNC is described in the following report: http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf. In summary: 20 RFB captures, representing "common" 2D and 3D application workloads (the 3D workloads were run using VirtualGL), were studied using the TightVNC encoder in isolation. Some of the analysis features in the TightVNC encoder, such as smoothness detection, were found to generate a lot of CPU usage with little or no benefit in compression, so those features were disabled. JPEG encoding was accelerated using libjpeg-turbo (which achieves a 2-4x speedup over plain libjpeg on modern x86 or ARM processors.) Finally, the "palette threshold" (minimum number of colors that the subrectangle must have before it is compressed using JPEG or raw) was adjusted to account for the fact that JPEG encoding is now quite a bit faster (meaning that we can now use it more without a CPU penalty.) TurboVNC has additional optimizations, such as the ability to count colors and encode JPEG images directly from the framebuffer without first translating the pixels into RGB. The TurboVNC encoder compares quite favorably in terms of compression ratio with TightVNC and generally encodes a great deal faster (often an order of magnitude or more.) The version of the TurboVNC encoder included in this patch is roughly equivalent to the one found in version 0.6 of the Unix TurboVNC Server, with a few minor patches integrated from TurboVNC 1.1. TurboVNC 1.0 added multi-threading capabilities, which can be added in later if desired (at the expense of making libvncserver depend on libpthread.) Because TurboVNC uses a fundamentally different mix of subencodings than TightVNC, because it uses the identical protocol (and thus a viewer really has no idea whether it's talking to a TightVNC or TurboVNC server), and because it doesn't support rfbTightPng (and in fact conflicts with it-- see below), the TurboVNC and TightVNC encoders cannot be enabled simultaneously. Compatibility: In *most* cases, a TurboVNC-enabled viewer is fully compatible with a TightVNC server, and vice versa. TurboVNC supports pseudo-encodings for specifying a fine-grained (1-100) quality scale and specifying chrominance subsampling. If a TurboVNC viewer sends those to a TightVNC server, then the TightVNC server ignores them, so the TurboVNC viewer also sends the quality on a 0-9 scale that the TightVNC server can understand. Similarly, the TurboVNC server checks first for fine-grained quality and subsampling pseudo-encodings from the viewer, and failing to receive those, it then checks for the TightVNC 0-9 quality pseudo-encoding. There is one case in which the two systems are not compatible, and that is when a TightVNC or TigerVNC viewer requests compression level 0 without JPEG from a TurboVNC server. For performance reasons, this causes the TurboVNC server to send images directly to the viewer, bypassing Zlib. When the TurboVNC server does this, it also sets bits 7-4 in the compression control byte to rfbTightNoZlib (0x0A), which is unfortunately the same value as rfbTightPng. Older TightVNC viewers that don't handle PNG will assume that the stream is uncompressed but still encapsulated in a Zlib structure, whereas newer PNG-supporting TightVNC viewers will assume that the stream is PNG. In either case, the viewer will probably crash. Since most VNC viewers don't expose compression level 0 in the GUI, this is a relatively rare situation. Description of changes: configure.ac -- Added support for libjpeg-turbo. If passed an argument of --with-turbovnc, configure will now run (or, if cross-compiling, just link) a test program that determines whether the libjpeg library being used is libjpeg-turbo. libjpeg-turbo must be used when building the TurboVNC encoder, because the TurboVNC encoder relies on the libjpeg-turbo colorspace extensions in order to compress images directly out of the framebuffer (which may be, for instance, BGRA rather than RGB.) libjpeg-turbo can optionally be used with the TightVNC encoder as well, but the speedup will only be marginal (the report linked above explains why in more detail, but basically it's because of Amdahl's Law. The TightVNC encoder was designed with the assumption that JPEG had a very high CPU cost, and thus JPEG is used only sparingly.) -- Added a new configure variable, JPEG_LDFLAGS. This is necessitated by the fact that libjpeg-turbo often distributes libjpeg.a and libjpeg.so in /opt/libjpeg-turbo/lib32 or /opt/libjpeg-turbo/lib64, and many people prefer to statically link with it. Thus, more flexibility is needed than is provided by --with-jpeg. If JPEG_LDFLAGS is specified, then it overrides the changes to LDFLAGS enacted by --with-jpeg (but --with-jpeg is still used to set the include path.) The addition of JPEG_LDFLAGS necessitated replacing AC_CHECK_LIB with AC_LINK_IFELSE (because AC_CHECK_LIB automatically sets LIBS to -ljpeg, which is not what we want if we're, for instance, linking statically with libjpeg-turbo.) -- configure does not check for PNG support if TurboVNC encoding is enabled. This prevents the rfbSendRectEncodingTightPng() function from being compiled in, since the TurboVNC encoder doesn't (and can't) support it. common/turbojpeg.c, common/turbojpeg.h -- TurboJPEG is a simple API used to compress and decompress JPEG images in memory. It was originally implemented because it was desirable to use different types of underlying technologies to compress JPEG on different platforms (mediaLib on SPARC, Quicktime on PPC Macs, Intel Performance Primitives, etc.) These days, however, libjpeg-turbo is the only underlying technology used by TurboVNC, so TurboJPEG's purpose is largely just code simplicity and flexibility. Thus, since there is no real need for libvncserver to use any technology other than libjpeg-turbo for compressing JPEG, the TurboJPEG wrapper for libjpeg-turbo has been included in-tree so that libvncserver can be directly linked with libjpeg-turbo. This is convenient because many modern Linux distros (Fedora, Ubuntu, etc.) now ship libjpeg-turbo as their default libjpeg library. libvncserver/rfbserver.c -- Added logic to check for the TurboVNC fine-grained quality level and subsampling encodings and to map Tight (0-9) quality levels to appropriate fine-grained quality level and subsampling values if communicating with a TightVNC/TigerVNC viewer. libvncserver/turbo.c -- TurboVNC encoder (compiled instead of libvncserver/tight.c) rfb/rfb.h -- Added support for the TurboVNC subsampling level rfb/rfbproto.h -- Added constants for the TurboVNC fine quality level and subsampling encodings as well as the rfbTightNoZlib constant and notes on its usage.
13 years ago
{
struct jpeg_compress_struct cinfo;
struct jpeg_decompress_struct dinfo;
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
struct jpeg_destination_mgr jdst;
struct jpeg_source_mgr jsrc;
struct my_error_mgr jerr;
int init;
} tjinstance;
static const int pixelsize[TJ_NUMSAMP]={3, 3, 3, 1, 3};
#define NUMSF 4
static const tjscalingfactor sf[NUMSF]={
{1, 1},
{1, 2},
{1, 4},
{1, 8}
};
#define _throw(m) {snprintf(errStr, JMSG_LENGTH_MAX, "%s", m); \
retval=-1; goto bailout;}
#define getinstance(handle) tjinstance *this=(tjinstance *)handle; \
j_compress_ptr cinfo=NULL; j_decompress_ptr dinfo=NULL; \
(void) cinfo; (void) dinfo; /* silence warnings */ \
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
if(!this) {snprintf(errStr, JMSG_LENGTH_MAX, "Invalid handle"); \
return -1;} \
cinfo=&this->cinfo; dinfo=&this->dinfo;
static int getPixelFormat(int pixelSize, int flags)
{
if(pixelSize==1) return TJPF_GRAY;
if(pixelSize==3)
{
if(flags&TJ_BGR) return TJPF_BGR;
else return TJPF_RGB;
}
if(pixelSize==4)
{
if(flags&TJ_ALPHAFIRST)
{
if(flags&TJ_BGR) return TJPF_XBGR;
else return TJPF_XRGB;
}
else
{
if(flags&TJ_BGR) return TJPF_BGRX;
else return TJPF_RGBX;
}
}
return -1;
}
static int setCompDefaults(struct jpeg_compress_struct *cinfo,
int pixelFormat, int subsamp, int jpegQual)
{
int retval=0;
switch(pixelFormat)
{
case TJPF_GRAY:
cinfo->in_color_space=JCS_GRAYSCALE; break;
#if JCS_EXTENSIONS==1
case TJPF_RGB:
cinfo->in_color_space=JCS_EXT_RGB; break;
case TJPF_BGR:
cinfo->in_color_space=JCS_EXT_BGR; break;
case TJPF_RGBX:
case TJPF_RGBA:
cinfo->in_color_space=JCS_EXT_RGBX; break;
case TJPF_BGRX:
case TJPF_BGRA:
cinfo->in_color_space=JCS_EXT_BGRX; break;
case TJPF_XRGB:
case TJPF_ARGB:
cinfo->in_color_space=JCS_EXT_XRGB; break;
case TJPF_XBGR:
case TJPF_ABGR:
cinfo->in_color_space=JCS_EXT_XBGR; break;
#else
case TJPF_RGB:
case TJPF_BGR:
case TJPF_RGBX:
case TJPF_BGRX:
case TJPF_XRGB:
case TJPF_XBGR:
case TJPF_RGBA:
case TJPF_BGRA:
case TJPF_ARGB:
case TJPF_ABGR:
cinfo->in_color_space=JCS_RGB; pixelFormat=TJPF_RGB;
break;
#endif
}
cinfo->input_components=tjPixelSize[pixelFormat];
jpeg_set_defaults(cinfo);
if(jpegQual>=0)
{
jpeg_set_quality(cinfo, jpegQual, TRUE);
if(jpegQual>=96) cinfo->dct_method=JDCT_ISLOW;
else cinfo->dct_method=JDCT_FASTEST;
}
if(subsamp==TJSAMP_GRAY)
jpeg_set_colorspace(cinfo, JCS_GRAYSCALE);
else
jpeg_set_colorspace(cinfo, JCS_YCbCr);
cinfo->comp_info[0].h_samp_factor=tjMCUWidth[subsamp]/8;
cinfo->comp_info[1].h_samp_factor=1;
cinfo->comp_info[2].h_samp_factor=1;
cinfo->comp_info[0].v_samp_factor=tjMCUHeight[subsamp]/8;
cinfo->comp_info[1].v_samp_factor=1;
cinfo->comp_info[2].v_samp_factor=1;
return retval;
}
static int setDecompDefaults(struct jpeg_decompress_struct *dinfo,
int pixelFormat)
{
int retval=0;
switch(pixelFormat)
{
case TJPF_GRAY:
dinfo->out_color_space=JCS_GRAYSCALE; break;
#if JCS_EXTENSIONS==1
case TJPF_RGB:
dinfo->out_color_space=JCS_EXT_RGB; break;
case TJPF_BGR:
dinfo->out_color_space=JCS_EXT_BGR; break;
case TJPF_RGBX:
dinfo->out_color_space=JCS_EXT_RGBX; break;
case TJPF_BGRX:
dinfo->out_color_space=JCS_EXT_BGRX; break;
case TJPF_XRGB:
dinfo->out_color_space=JCS_EXT_XRGB; break;
case TJPF_XBGR:
dinfo->out_color_space=JCS_EXT_XBGR; break;
#if JCS_ALPHA_EXTENSIONS==1
case TJPF_RGBA:
dinfo->out_color_space=JCS_EXT_RGBA; break;
case TJPF_BGRA:
dinfo->out_color_space=JCS_EXT_BGRA; break;
case TJPF_ARGB:
dinfo->out_color_space=JCS_EXT_ARGB; break;
case TJPF_ABGR:
dinfo->out_color_space=JCS_EXT_ABGR; break;
#endif
#else
case TJPF_RGB:
case TJPF_BGR:
case TJPF_RGBX:
case TJPF_BGRX:
case TJPF_XRGB:
case TJPF_XBGR:
case TJPF_RGBA:
case TJPF_BGRA:
case TJPF_ARGB:
case TJPF_ABGR:
dinfo->out_color_space=JCS_RGB; break;
#endif
default:
_throw("Unsupported pixel format");
}
bailout:
return retval;
}
static int getSubsamp(j_decompress_ptr dinfo)
{
int retval=-1, i, k;
for(i=0; i<NUMSUBOPT; i++)
{
if(dinfo->num_components==pixelsize[i])
{
if(dinfo->comp_info[0].h_samp_factor==tjMCUWidth[i]/8
&& dinfo->comp_info[0].v_samp_factor==tjMCUHeight[i]/8)
{
int match=0;
for(k=1; k<dinfo->num_components; k++)
{
if(dinfo->comp_info[k].h_samp_factor==1
&& dinfo->comp_info[k].v_samp_factor==1)
match++;
}
if(match==dinfo->num_components-1)
{
retval=i; break;
}
}
}
}
return retval;
}
#ifndef JCS_EXTENSIONS
/* Conversion functions to emulate the colorspace extensions. This allows the
TurboJPEG wrapper to be used with libjpeg */
#define TORGB(PS, ROFFSET, GOFFSET, BOFFSET) { \
int rowPad=pitch-width*PS; \
while(height--) \
{ \
unsigned char *endOfRow=src+width*PS; \
while(src<endOfRow) \
{ \
dst[RGB_RED]=src[ROFFSET]; \
dst[RGB_GREEN]=src[GOFFSET]; \
dst[RGB_BLUE]=src[BOFFSET]; \
dst+=RGB_PIXELSIZE; src+=PS; \
} \
src+=rowPad; \
} \
}
static unsigned char *toRGB(unsigned char *src, int width, int pitch,
int height, int pixelFormat, unsigned char *dst)
{
unsigned char *retval=src;
switch(pixelFormat)
{
case TJPF_RGB:
#if RGB_RED!=0 || RGB_GREEN!=1 || RGB_BLUE!=2 || RGB_PIXELSIZE!=3
retval=dst; TORGB(3, 0, 1, 2);
#endif
break;
case TJPF_BGR:
#if RGB_RED!=2 || RGB_GREEN!=1 || RGB_BLUE!=0 || RGB_PIXELSIZE!=3
retval=dst; TORGB(3, 2, 1, 0);
#endif
break;
case TJPF_RGBX:
case TJPF_RGBA:
#if RGB_RED!=0 || RGB_GREEN!=1 || RGB_BLUE!=2 || RGB_PIXELSIZE!=4
retval=dst; TORGB(4, 0, 1, 2);
#endif
break;
case TJPF_BGRX:
case TJPF_BGRA:
#if RGB_RED!=2 || RGB_GREEN!=1 || RGB_BLUE!=0 || RGB_PIXELSIZE!=4
retval=dst; TORGB(4, 2, 1, 0);
#endif
break;
case TJPF_XRGB:
case TJPF_ARGB:
#if RGB_RED!=1 || RGB_GREEN!=2 || RGB_BLUE!=3 || RGB_PIXELSIZE!=4
retval=dst; TORGB(4, 1, 2, 3);
#endif
break;
case TJPF_XBGR:
case TJPF_ABGR:
#if RGB_RED!=3 || RGB_GREEN!=2 || RGB_BLUE!=1 || RGB_PIXELSIZE!=4
retval=dst; TORGB(4, 3, 2, 1);
#endif
break;
}
return retval;
}
Add TurboVNC encoding support. TurboVNC is a variant of TightVNC that uses the same client/server protocol (RFB version 3.8t), and thus it is fully cross-compatible with TightVNC and TigerVNC (with one exception, which is noted below.) Both the TightVNC and TurboVNC encoders analyze each rectangle, pick out regions of solid color to send separately, and send the remaining subrectangles using mono, indexed color, JPEG, or raw encoding, depending on the number of colors in the subrectangle. However, TurboVNC uses a fundamentally different selection algorithm to determine the appropriate subencoding to use for each subrectangle. Thus, while it sends a protocol stream that can be decoded by any TightVNC-compatible viewer, the mix of subencoding types in this protocol stream will be different from those generated by a TightVNC server. The research that led to TurboVNC is described in the following report: http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf. In summary: 20 RFB captures, representing "common" 2D and 3D application workloads (the 3D workloads were run using VirtualGL), were studied using the TightVNC encoder in isolation. Some of the analysis features in the TightVNC encoder, such as smoothness detection, were found to generate a lot of CPU usage with little or no benefit in compression, so those features were disabled. JPEG encoding was accelerated using libjpeg-turbo (which achieves a 2-4x speedup over plain libjpeg on modern x86 or ARM processors.) Finally, the "palette threshold" (minimum number of colors that the subrectangle must have before it is compressed using JPEG or raw) was adjusted to account for the fact that JPEG encoding is now quite a bit faster (meaning that we can now use it more without a CPU penalty.) TurboVNC has additional optimizations, such as the ability to count colors and encode JPEG images directly from the framebuffer without first translating the pixels into RGB. The TurboVNC encoder compares quite favorably in terms of compression ratio with TightVNC and generally encodes a great deal faster (often an order of magnitude or more.) The version of the TurboVNC encoder included in this patch is roughly equivalent to the one found in version 0.6 of the Unix TurboVNC Server, with a few minor patches integrated from TurboVNC 1.1. TurboVNC 1.0 added multi-threading capabilities, which can be added in later if desired (at the expense of making libvncserver depend on libpthread.) Because TurboVNC uses a fundamentally different mix of subencodings than TightVNC, because it uses the identical protocol (and thus a viewer really has no idea whether it's talking to a TightVNC or TurboVNC server), and because it doesn't support rfbTightPng (and in fact conflicts with it-- see below), the TurboVNC and TightVNC encoders cannot be enabled simultaneously. Compatibility: In *most* cases, a TurboVNC-enabled viewer is fully compatible with a TightVNC server, and vice versa. TurboVNC supports pseudo-encodings for specifying a fine-grained (1-100) quality scale and specifying chrominance subsampling. If a TurboVNC viewer sends those to a TightVNC server, then the TightVNC server ignores them, so the TurboVNC viewer also sends the quality on a 0-9 scale that the TightVNC server can understand. Similarly, the TurboVNC server checks first for fine-grained quality and subsampling pseudo-encodings from the viewer, and failing to receive those, it then checks for the TightVNC 0-9 quality pseudo-encoding. There is one case in which the two systems are not compatible, and that is when a TightVNC or TigerVNC viewer requests compression level 0 without JPEG from a TurboVNC server. For performance reasons, this causes the TurboVNC server to send images directly to the viewer, bypassing Zlib. When the TurboVNC server does this, it also sets bits 7-4 in the compression control byte to rfbTightNoZlib (0x0A), which is unfortunately the same value as rfbTightPng. Older TightVNC viewers that don't handle PNG will assume that the stream is uncompressed but still encapsulated in a Zlib structure, whereas newer PNG-supporting TightVNC viewers will assume that the stream is PNG. In either case, the viewer will probably crash. Since most VNC viewers don't expose compression level 0 in the GUI, this is a relatively rare situation. Description of changes: configure.ac -- Added support for libjpeg-turbo. If passed an argument of --with-turbovnc, configure will now run (or, if cross-compiling, just link) a test program that determines whether the libjpeg library being used is libjpeg-turbo. libjpeg-turbo must be used when building the TurboVNC encoder, because the TurboVNC encoder relies on the libjpeg-turbo colorspace extensions in order to compress images directly out of the framebuffer (which may be, for instance, BGRA rather than RGB.) libjpeg-turbo can optionally be used with the TightVNC encoder as well, but the speedup will only be marginal (the report linked above explains why in more detail, but basically it's because of Amdahl's Law. The TightVNC encoder was designed with the assumption that JPEG had a very high CPU cost, and thus JPEG is used only sparingly.) -- Added a new configure variable, JPEG_LDFLAGS. This is necessitated by the fact that libjpeg-turbo often distributes libjpeg.a and libjpeg.so in /opt/libjpeg-turbo/lib32 or /opt/libjpeg-turbo/lib64, and many people prefer to statically link with it. Thus, more flexibility is needed than is provided by --with-jpeg. If JPEG_LDFLAGS is specified, then it overrides the changes to LDFLAGS enacted by --with-jpeg (but --with-jpeg is still used to set the include path.) The addition of JPEG_LDFLAGS necessitated replacing AC_CHECK_LIB with AC_LINK_IFELSE (because AC_CHECK_LIB automatically sets LIBS to -ljpeg, which is not what we want if we're, for instance, linking statically with libjpeg-turbo.) -- configure does not check for PNG support if TurboVNC encoding is enabled. This prevents the rfbSendRectEncodingTightPng() function from being compiled in, since the TurboVNC encoder doesn't (and can't) support it. common/turbojpeg.c, common/turbojpeg.h -- TurboJPEG is a simple API used to compress and decompress JPEG images in memory. It was originally implemented because it was desirable to use different types of underlying technologies to compress JPEG on different platforms (mediaLib on SPARC, Quicktime on PPC Macs, Intel Performance Primitives, etc.) These days, however, libjpeg-turbo is the only underlying technology used by TurboVNC, so TurboJPEG's purpose is largely just code simplicity and flexibility. Thus, since there is no real need for libvncserver to use any technology other than libjpeg-turbo for compressing JPEG, the TurboJPEG wrapper for libjpeg-turbo has been included in-tree so that libvncserver can be directly linked with libjpeg-turbo. This is convenient because many modern Linux distros (Fedora, Ubuntu, etc.) now ship libjpeg-turbo as their default libjpeg library. libvncserver/rfbserver.c -- Added logic to check for the TurboVNC fine-grained quality level and subsampling encodings and to map Tight (0-9) quality levels to appropriate fine-grained quality level and subsampling values if communicating with a TightVNC/TigerVNC viewer. libvncserver/turbo.c -- TurboVNC encoder (compiled instead of libvncserver/tight.c) rfb/rfb.h -- Added support for the TurboVNC subsampling level rfb/rfbproto.h -- Added constants for the TurboVNC fine quality level and subsampling encodings as well as the rfbTightNoZlib constant and notes on its usage.
13 years ago
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
#define FROMRGB(PS, ROFFSET, GOFFSET, BOFFSET, SETALPHA) { \
int rowPad=pitch-width*PS; \
while(height--) \
{ \
unsigned char *endOfRow=dst+width*PS; \
while(dst<endOfRow) \
{ \
dst[ROFFSET]=src[RGB_RED]; \
dst[GOFFSET]=src[RGB_GREEN]; \
dst[BOFFSET]=src[RGB_BLUE]; \
SETALPHA \
dst+=PS; src+=RGB_PIXELSIZE; \
} \
dst+=rowPad; \
} \
}
Add TurboVNC encoding support. TurboVNC is a variant of TightVNC that uses the same client/server protocol (RFB version 3.8t), and thus it is fully cross-compatible with TightVNC and TigerVNC (with one exception, which is noted below.) Both the TightVNC and TurboVNC encoders analyze each rectangle, pick out regions of solid color to send separately, and send the remaining subrectangles using mono, indexed color, JPEG, or raw encoding, depending on the number of colors in the subrectangle. However, TurboVNC uses a fundamentally different selection algorithm to determine the appropriate subencoding to use for each subrectangle. Thus, while it sends a protocol stream that can be decoded by any TightVNC-compatible viewer, the mix of subencoding types in this protocol stream will be different from those generated by a TightVNC server. The research that led to TurboVNC is described in the following report: http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf. In summary: 20 RFB captures, representing "common" 2D and 3D application workloads (the 3D workloads were run using VirtualGL), were studied using the TightVNC encoder in isolation. Some of the analysis features in the TightVNC encoder, such as smoothness detection, were found to generate a lot of CPU usage with little or no benefit in compression, so those features were disabled. JPEG encoding was accelerated using libjpeg-turbo (which achieves a 2-4x speedup over plain libjpeg on modern x86 or ARM processors.) Finally, the "palette threshold" (minimum number of colors that the subrectangle must have before it is compressed using JPEG or raw) was adjusted to account for the fact that JPEG encoding is now quite a bit faster (meaning that we can now use it more without a CPU penalty.) TurboVNC has additional optimizations, such as the ability to count colors and encode JPEG images directly from the framebuffer without first translating the pixels into RGB. The TurboVNC encoder compares quite favorably in terms of compression ratio with TightVNC and generally encodes a great deal faster (often an order of magnitude or more.) The version of the TurboVNC encoder included in this patch is roughly equivalent to the one found in version 0.6 of the Unix TurboVNC Server, with a few minor patches integrated from TurboVNC 1.1. TurboVNC 1.0 added multi-threading capabilities, which can be added in later if desired (at the expense of making libvncserver depend on libpthread.) Because TurboVNC uses a fundamentally different mix of subencodings than TightVNC, because it uses the identical protocol (and thus a viewer really has no idea whether it's talking to a TightVNC or TurboVNC server), and because it doesn't support rfbTightPng (and in fact conflicts with it-- see below), the TurboVNC and TightVNC encoders cannot be enabled simultaneously. Compatibility: In *most* cases, a TurboVNC-enabled viewer is fully compatible with a TightVNC server, and vice versa. TurboVNC supports pseudo-encodings for specifying a fine-grained (1-100) quality scale and specifying chrominance subsampling. If a TurboVNC viewer sends those to a TightVNC server, then the TightVNC server ignores them, so the TurboVNC viewer also sends the quality on a 0-9 scale that the TightVNC server can understand. Similarly, the TurboVNC server checks first for fine-grained quality and subsampling pseudo-encodings from the viewer, and failing to receive those, it then checks for the TightVNC 0-9 quality pseudo-encoding. There is one case in which the two systems are not compatible, and that is when a TightVNC or TigerVNC viewer requests compression level 0 without JPEG from a TurboVNC server. For performance reasons, this causes the TurboVNC server to send images directly to the viewer, bypassing Zlib. When the TurboVNC server does this, it also sets bits 7-4 in the compression control byte to rfbTightNoZlib (0x0A), which is unfortunately the same value as rfbTightPng. Older TightVNC viewers that don't handle PNG will assume that the stream is uncompressed but still encapsulated in a Zlib structure, whereas newer PNG-supporting TightVNC viewers will assume that the stream is PNG. In either case, the viewer will probably crash. Since most VNC viewers don't expose compression level 0 in the GUI, this is a relatively rare situation. Description of changes: configure.ac -- Added support for libjpeg-turbo. If passed an argument of --with-turbovnc, configure will now run (or, if cross-compiling, just link) a test program that determines whether the libjpeg library being used is libjpeg-turbo. libjpeg-turbo must be used when building the TurboVNC encoder, because the TurboVNC encoder relies on the libjpeg-turbo colorspace extensions in order to compress images directly out of the framebuffer (which may be, for instance, BGRA rather than RGB.) libjpeg-turbo can optionally be used with the TightVNC encoder as well, but the speedup will only be marginal (the report linked above explains why in more detail, but basically it's because of Amdahl's Law. The TightVNC encoder was designed with the assumption that JPEG had a very high CPU cost, and thus JPEG is used only sparingly.) -- Added a new configure variable, JPEG_LDFLAGS. This is necessitated by the fact that libjpeg-turbo often distributes libjpeg.a and libjpeg.so in /opt/libjpeg-turbo/lib32 or /opt/libjpeg-turbo/lib64, and many people prefer to statically link with it. Thus, more flexibility is needed than is provided by --with-jpeg. If JPEG_LDFLAGS is specified, then it overrides the changes to LDFLAGS enacted by --with-jpeg (but --with-jpeg is still used to set the include path.) The addition of JPEG_LDFLAGS necessitated replacing AC_CHECK_LIB with AC_LINK_IFELSE (because AC_CHECK_LIB automatically sets LIBS to -ljpeg, which is not what we want if we're, for instance, linking statically with libjpeg-turbo.) -- configure does not check for PNG support if TurboVNC encoding is enabled. This prevents the rfbSendRectEncodingTightPng() function from being compiled in, since the TurboVNC encoder doesn't (and can't) support it. common/turbojpeg.c, common/turbojpeg.h -- TurboJPEG is a simple API used to compress and decompress JPEG images in memory. It was originally implemented because it was desirable to use different types of underlying technologies to compress JPEG on different platforms (mediaLib on SPARC, Quicktime on PPC Macs, Intel Performance Primitives, etc.) These days, however, libjpeg-turbo is the only underlying technology used by TurboVNC, so TurboJPEG's purpose is largely just code simplicity and flexibility. Thus, since there is no real need for libvncserver to use any technology other than libjpeg-turbo for compressing JPEG, the TurboJPEG wrapper for libjpeg-turbo has been included in-tree so that libvncserver can be directly linked with libjpeg-turbo. This is convenient because many modern Linux distros (Fedora, Ubuntu, etc.) now ship libjpeg-turbo as their default libjpeg library. libvncserver/rfbserver.c -- Added logic to check for the TurboVNC fine-grained quality level and subsampling encodings and to map Tight (0-9) quality levels to appropriate fine-grained quality level and subsampling values if communicating with a TightVNC/TigerVNC viewer. libvncserver/turbo.c -- TurboVNC encoder (compiled instead of libvncserver/tight.c) rfb/rfb.h -- Added support for the TurboVNC subsampling level rfb/rfbproto.h -- Added constants for the TurboVNC fine quality level and subsampling encodings as well as the rfbTightNoZlib constant and notes on its usage.
13 years ago
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
static void fromRGB(unsigned char *src, unsigned char *dst, int width,
int pitch, int height, int pixelFormat)
{
switch(pixelFormat)
{
case TJPF_RGB:
#if RGB_RED!=0 || RGB_GREEN!=1 || RGB_BLUE!=2 || RGB_PIXELSIZE!=3
FROMRGB(3, 0, 1, 2,);
#endif
break;
case TJPF_BGR:
#if RGB_RED!=2 || RGB_GREEN!=1 || RGB_BLUE!=0 || RGB_PIXELSIZE!=3
FROMRGB(3, 2, 1, 0,);
#endif
break;
case TJPF_RGBX:
#if RGB_RED!=0 || RGB_GREEN!=1 || RGB_BLUE!=2 || RGB_PIXELSIZE!=4
FROMRGB(4, 0, 1, 2,);
#endif
break;
case TJPF_RGBA:
#if RGB_RED!=0 || RGB_GREEN!=1 || RGB_BLUE!=2 || RGB_PIXELSIZE!=4
FROMRGB(4, 0, 1, 2, dst[3]=0xFF;);
#endif
break;
case TJPF_BGRX:
#if RGB_RED!=2 || RGB_GREEN!=1 || RGB_BLUE!=0 || RGB_PIXELSIZE!=4
FROMRGB(4, 2, 1, 0,);
#endif
break;
case TJPF_BGRA:
#if RGB_RED!=2 || RGB_GREEN!=1 || RGB_BLUE!=0 || RGB_PIXELSIZE!=4
FROMRGB(4, 2, 1, 0, dst[3]=0xFF;); return;
#endif
break;
case TJPF_XRGB:
#if RGB_RED!=1 || RGB_GREEN!=2 || RGB_BLUE!=3 || RGB_PIXELSIZE!=4
FROMRGB(4, 1, 2, 3,); return;
#endif
break;
case TJPF_ARGB:
#if RGB_RED!=1 || RGB_GREEN!=2 || RGB_BLUE!=3 || RGB_PIXELSIZE!=4
FROMRGB(4, 1, 2, 3, dst[0]=0xFF;); return;
#endif
break;
case TJPF_XBGR:
#if RGB_RED!=3 || RGB_GREEN!=2 || RGB_BLUE!=1 || RGB_PIXELSIZE!=4
FROMRGB(4, 3, 2, 1,); return;
#endif
break;
case TJPF_ABGR:
#if RGB_RED!=3 || RGB_GREEN!=2 || RGB_BLUE!=1 || RGB_PIXELSIZE!=4
FROMRGB(4, 3, 2, 1, dst[0]=0xFF;); return;
#endif
break;
}
}
Add TurboVNC encoding support. TurboVNC is a variant of TightVNC that uses the same client/server protocol (RFB version 3.8t), and thus it is fully cross-compatible with TightVNC and TigerVNC (with one exception, which is noted below.) Both the TightVNC and TurboVNC encoders analyze each rectangle, pick out regions of solid color to send separately, and send the remaining subrectangles using mono, indexed color, JPEG, or raw encoding, depending on the number of colors in the subrectangle. However, TurboVNC uses a fundamentally different selection algorithm to determine the appropriate subencoding to use for each subrectangle. Thus, while it sends a protocol stream that can be decoded by any TightVNC-compatible viewer, the mix of subencoding types in this protocol stream will be different from those generated by a TightVNC server. The research that led to TurboVNC is described in the following report: http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf. In summary: 20 RFB captures, representing "common" 2D and 3D application workloads (the 3D workloads were run using VirtualGL), were studied using the TightVNC encoder in isolation. Some of the analysis features in the TightVNC encoder, such as smoothness detection, were found to generate a lot of CPU usage with little or no benefit in compression, so those features were disabled. JPEG encoding was accelerated using libjpeg-turbo (which achieves a 2-4x speedup over plain libjpeg on modern x86 or ARM processors.) Finally, the "palette threshold" (minimum number of colors that the subrectangle must have before it is compressed using JPEG or raw) was adjusted to account for the fact that JPEG encoding is now quite a bit faster (meaning that we can now use it more without a CPU penalty.) TurboVNC has additional optimizations, such as the ability to count colors and encode JPEG images directly from the framebuffer without first translating the pixels into RGB. The TurboVNC encoder compares quite favorably in terms of compression ratio with TightVNC and generally encodes a great deal faster (often an order of magnitude or more.) The version of the TurboVNC encoder included in this patch is roughly equivalent to the one found in version 0.6 of the Unix TurboVNC Server, with a few minor patches integrated from TurboVNC 1.1. TurboVNC 1.0 added multi-threading capabilities, which can be added in later if desired (at the expense of making libvncserver depend on libpthread.) Because TurboVNC uses a fundamentally different mix of subencodings than TightVNC, because it uses the identical protocol (and thus a viewer really has no idea whether it's talking to a TightVNC or TurboVNC server), and because it doesn't support rfbTightPng (and in fact conflicts with it-- see below), the TurboVNC and TightVNC encoders cannot be enabled simultaneously. Compatibility: In *most* cases, a TurboVNC-enabled viewer is fully compatible with a TightVNC server, and vice versa. TurboVNC supports pseudo-encodings for specifying a fine-grained (1-100) quality scale and specifying chrominance subsampling. If a TurboVNC viewer sends those to a TightVNC server, then the TightVNC server ignores them, so the TurboVNC viewer also sends the quality on a 0-9 scale that the TightVNC server can understand. Similarly, the TurboVNC server checks first for fine-grained quality and subsampling pseudo-encodings from the viewer, and failing to receive those, it then checks for the TightVNC 0-9 quality pseudo-encoding. There is one case in which the two systems are not compatible, and that is when a TightVNC or TigerVNC viewer requests compression level 0 without JPEG from a TurboVNC server. For performance reasons, this causes the TurboVNC server to send images directly to the viewer, bypassing Zlib. When the TurboVNC server does this, it also sets bits 7-4 in the compression control byte to rfbTightNoZlib (0x0A), which is unfortunately the same value as rfbTightPng. Older TightVNC viewers that don't handle PNG will assume that the stream is uncompressed but still encapsulated in a Zlib structure, whereas newer PNG-supporting TightVNC viewers will assume that the stream is PNG. In either case, the viewer will probably crash. Since most VNC viewers don't expose compression level 0 in the GUI, this is a relatively rare situation. Description of changes: configure.ac -- Added support for libjpeg-turbo. If passed an argument of --with-turbovnc, configure will now run (or, if cross-compiling, just link) a test program that determines whether the libjpeg library being used is libjpeg-turbo. libjpeg-turbo must be used when building the TurboVNC encoder, because the TurboVNC encoder relies on the libjpeg-turbo colorspace extensions in order to compress images directly out of the framebuffer (which may be, for instance, BGRA rather than RGB.) libjpeg-turbo can optionally be used with the TightVNC encoder as well, but the speedup will only be marginal (the report linked above explains why in more detail, but basically it's because of Amdahl's Law. The TightVNC encoder was designed with the assumption that JPEG had a very high CPU cost, and thus JPEG is used only sparingly.) -- Added a new configure variable, JPEG_LDFLAGS. This is necessitated by the fact that libjpeg-turbo often distributes libjpeg.a and libjpeg.so in /opt/libjpeg-turbo/lib32 or /opt/libjpeg-turbo/lib64, and many people prefer to statically link with it. Thus, more flexibility is needed than is provided by --with-jpeg. If JPEG_LDFLAGS is specified, then it overrides the changes to LDFLAGS enacted by --with-jpeg (but --with-jpeg is still used to set the include path.) The addition of JPEG_LDFLAGS necessitated replacing AC_CHECK_LIB with AC_LINK_IFELSE (because AC_CHECK_LIB automatically sets LIBS to -ljpeg, which is not what we want if we're, for instance, linking statically with libjpeg-turbo.) -- configure does not check for PNG support if TurboVNC encoding is enabled. This prevents the rfbSendRectEncodingTightPng() function from being compiled in, since the TurboVNC encoder doesn't (and can't) support it. common/turbojpeg.c, common/turbojpeg.h -- TurboJPEG is a simple API used to compress and decompress JPEG images in memory. It was originally implemented because it was desirable to use different types of underlying technologies to compress JPEG on different platforms (mediaLib on SPARC, Quicktime on PPC Macs, Intel Performance Primitives, etc.) These days, however, libjpeg-turbo is the only underlying technology used by TurboVNC, so TurboJPEG's purpose is largely just code simplicity and flexibility. Thus, since there is no real need for libvncserver to use any technology other than libjpeg-turbo for compressing JPEG, the TurboJPEG wrapper for libjpeg-turbo has been included in-tree so that libvncserver can be directly linked with libjpeg-turbo. This is convenient because many modern Linux distros (Fedora, Ubuntu, etc.) now ship libjpeg-turbo as their default libjpeg library. libvncserver/rfbserver.c -- Added logic to check for the TurboVNC fine-grained quality level and subsampling encodings and to map Tight (0-9) quality levels to appropriate fine-grained quality level and subsampling values if communicating with a TightVNC/TigerVNC viewer. libvncserver/turbo.c -- TurboVNC encoder (compiled instead of libvncserver/tight.c) rfb/rfb.h -- Added support for the TurboVNC subsampling level rfb/rfbproto.h -- Added constants for the TurboVNC fine quality level and subsampling encodings as well as the rfbTightNoZlib constant and notes on its usage.
13 years ago
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
#endif
Add TurboVNC encoding support. TurboVNC is a variant of TightVNC that uses the same client/server protocol (RFB version 3.8t), and thus it is fully cross-compatible with TightVNC and TigerVNC (with one exception, which is noted below.) Both the TightVNC and TurboVNC encoders analyze each rectangle, pick out regions of solid color to send separately, and send the remaining subrectangles using mono, indexed color, JPEG, or raw encoding, depending on the number of colors in the subrectangle. However, TurboVNC uses a fundamentally different selection algorithm to determine the appropriate subencoding to use for each subrectangle. Thus, while it sends a protocol stream that can be decoded by any TightVNC-compatible viewer, the mix of subencoding types in this protocol stream will be different from those generated by a TightVNC server. The research that led to TurboVNC is described in the following report: http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf. In summary: 20 RFB captures, representing "common" 2D and 3D application workloads (the 3D workloads were run using VirtualGL), were studied using the TightVNC encoder in isolation. Some of the analysis features in the TightVNC encoder, such as smoothness detection, were found to generate a lot of CPU usage with little or no benefit in compression, so those features were disabled. JPEG encoding was accelerated using libjpeg-turbo (which achieves a 2-4x speedup over plain libjpeg on modern x86 or ARM processors.) Finally, the "palette threshold" (minimum number of colors that the subrectangle must have before it is compressed using JPEG or raw) was adjusted to account for the fact that JPEG encoding is now quite a bit faster (meaning that we can now use it more without a CPU penalty.) TurboVNC has additional optimizations, such as the ability to count colors and encode JPEG images directly from the framebuffer without first translating the pixels into RGB. The TurboVNC encoder compares quite favorably in terms of compression ratio with TightVNC and generally encodes a great deal faster (often an order of magnitude or more.) The version of the TurboVNC encoder included in this patch is roughly equivalent to the one found in version 0.6 of the Unix TurboVNC Server, with a few minor patches integrated from TurboVNC 1.1. TurboVNC 1.0 added multi-threading capabilities, which can be added in later if desired (at the expense of making libvncserver depend on libpthread.) Because TurboVNC uses a fundamentally different mix of subencodings than TightVNC, because it uses the identical protocol (and thus a viewer really has no idea whether it's talking to a TightVNC or TurboVNC server), and because it doesn't support rfbTightPng (and in fact conflicts with it-- see below), the TurboVNC and TightVNC encoders cannot be enabled simultaneously. Compatibility: In *most* cases, a TurboVNC-enabled viewer is fully compatible with a TightVNC server, and vice versa. TurboVNC supports pseudo-encodings for specifying a fine-grained (1-100) quality scale and specifying chrominance subsampling. If a TurboVNC viewer sends those to a TightVNC server, then the TightVNC server ignores them, so the TurboVNC viewer also sends the quality on a 0-9 scale that the TightVNC server can understand. Similarly, the TurboVNC server checks first for fine-grained quality and subsampling pseudo-encodings from the viewer, and failing to receive those, it then checks for the TightVNC 0-9 quality pseudo-encoding. There is one case in which the two systems are not compatible, and that is when a TightVNC or TigerVNC viewer requests compression level 0 without JPEG from a TurboVNC server. For performance reasons, this causes the TurboVNC server to send images directly to the viewer, bypassing Zlib. When the TurboVNC server does this, it also sets bits 7-4 in the compression control byte to rfbTightNoZlib (0x0A), which is unfortunately the same value as rfbTightPng. Older TightVNC viewers that don't handle PNG will assume that the stream is uncompressed but still encapsulated in a Zlib structure, whereas newer PNG-supporting TightVNC viewers will assume that the stream is PNG. In either case, the viewer will probably crash. Since most VNC viewers don't expose compression level 0 in the GUI, this is a relatively rare situation. Description of changes: configure.ac -- Added support for libjpeg-turbo. If passed an argument of --with-turbovnc, configure will now run (or, if cross-compiling, just link) a test program that determines whether the libjpeg library being used is libjpeg-turbo. libjpeg-turbo must be used when building the TurboVNC encoder, because the TurboVNC encoder relies on the libjpeg-turbo colorspace extensions in order to compress images directly out of the framebuffer (which may be, for instance, BGRA rather than RGB.) libjpeg-turbo can optionally be used with the TightVNC encoder as well, but the speedup will only be marginal (the report linked above explains why in more detail, but basically it's because of Amdahl's Law. The TightVNC encoder was designed with the assumption that JPEG had a very high CPU cost, and thus JPEG is used only sparingly.) -- Added a new configure variable, JPEG_LDFLAGS. This is necessitated by the fact that libjpeg-turbo often distributes libjpeg.a and libjpeg.so in /opt/libjpeg-turbo/lib32 or /opt/libjpeg-turbo/lib64, and many people prefer to statically link with it. Thus, more flexibility is needed than is provided by --with-jpeg. If JPEG_LDFLAGS is specified, then it overrides the changes to LDFLAGS enacted by --with-jpeg (but --with-jpeg is still used to set the include path.) The addition of JPEG_LDFLAGS necessitated replacing AC_CHECK_LIB with AC_LINK_IFELSE (because AC_CHECK_LIB automatically sets LIBS to -ljpeg, which is not what we want if we're, for instance, linking statically with libjpeg-turbo.) -- configure does not check for PNG support if TurboVNC encoding is enabled. This prevents the rfbSendRectEncodingTightPng() function from being compiled in, since the TurboVNC encoder doesn't (and can't) support it. common/turbojpeg.c, common/turbojpeg.h -- TurboJPEG is a simple API used to compress and decompress JPEG images in memory. It was originally implemented because it was desirable to use different types of underlying technologies to compress JPEG on different platforms (mediaLib on SPARC, Quicktime on PPC Macs, Intel Performance Primitives, etc.) These days, however, libjpeg-turbo is the only underlying technology used by TurboVNC, so TurboJPEG's purpose is largely just code simplicity and flexibility. Thus, since there is no real need for libvncserver to use any technology other than libjpeg-turbo for compressing JPEG, the TurboJPEG wrapper for libjpeg-turbo has been included in-tree so that libvncserver can be directly linked with libjpeg-turbo. This is convenient because many modern Linux distros (Fedora, Ubuntu, etc.) now ship libjpeg-turbo as their default libjpeg library. libvncserver/rfbserver.c -- Added logic to check for the TurboVNC fine-grained quality level and subsampling encodings and to map Tight (0-9) quality levels to appropriate fine-grained quality level and subsampling values if communicating with a TightVNC/TigerVNC viewer. libvncserver/turbo.c -- TurboVNC encoder (compiled instead of libvncserver/tight.c) rfb/rfb.h -- Added support for the TurboVNC subsampling level rfb/rfbproto.h -- Added constants for the TurboVNC fine quality level and subsampling encodings as well as the rfbTightNoZlib constant and notes on its usage.
13 years ago
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
/* General API functions */
DLLEXPORT char* DLLCALL tjGetErrorStr(void)
{
return errStr;
}
DLLEXPORT int DLLCALL tjDestroy(tjhandle handle)
{
getinstance(handle);
if(setjmp(this->jerr.setjmp_buffer)) return -1;
if(this->init&COMPRESS) jpeg_destroy_compress(cinfo);
if(this->init&DECOMPRESS) jpeg_destroy_decompress(dinfo);
free(this);
return 0;
}
/* Compressor */
static boolean empty_output_buffer(j_compress_ptr cinfo)
Add TurboVNC encoding support. TurboVNC is a variant of TightVNC that uses the same client/server protocol (RFB version 3.8t), and thus it is fully cross-compatible with TightVNC and TigerVNC (with one exception, which is noted below.) Both the TightVNC and TurboVNC encoders analyze each rectangle, pick out regions of solid color to send separately, and send the remaining subrectangles using mono, indexed color, JPEG, or raw encoding, depending on the number of colors in the subrectangle. However, TurboVNC uses a fundamentally different selection algorithm to determine the appropriate subencoding to use for each subrectangle. Thus, while it sends a protocol stream that can be decoded by any TightVNC-compatible viewer, the mix of subencoding types in this protocol stream will be different from those generated by a TightVNC server. The research that led to TurboVNC is described in the following report: http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf. In summary: 20 RFB captures, representing "common" 2D and 3D application workloads (the 3D workloads were run using VirtualGL), were studied using the TightVNC encoder in isolation. Some of the analysis features in the TightVNC encoder, such as smoothness detection, were found to generate a lot of CPU usage with little or no benefit in compression, so those features were disabled. JPEG encoding was accelerated using libjpeg-turbo (which achieves a 2-4x speedup over plain libjpeg on modern x86 or ARM processors.) Finally, the "palette threshold" (minimum number of colors that the subrectangle must have before it is compressed using JPEG or raw) was adjusted to account for the fact that JPEG encoding is now quite a bit faster (meaning that we can now use it more without a CPU penalty.) TurboVNC has additional optimizations, such as the ability to count colors and encode JPEG images directly from the framebuffer without first translating the pixels into RGB. The TurboVNC encoder compares quite favorably in terms of compression ratio with TightVNC and generally encodes a great deal faster (often an order of magnitude or more.) The version of the TurboVNC encoder included in this patch is roughly equivalent to the one found in version 0.6 of the Unix TurboVNC Server, with a few minor patches integrated from TurboVNC 1.1. TurboVNC 1.0 added multi-threading capabilities, which can be added in later if desired (at the expense of making libvncserver depend on libpthread.) Because TurboVNC uses a fundamentally different mix of subencodings than TightVNC, because it uses the identical protocol (and thus a viewer really has no idea whether it's talking to a TightVNC or TurboVNC server), and because it doesn't support rfbTightPng (and in fact conflicts with it-- see below), the TurboVNC and TightVNC encoders cannot be enabled simultaneously. Compatibility: In *most* cases, a TurboVNC-enabled viewer is fully compatible with a TightVNC server, and vice versa. TurboVNC supports pseudo-encodings for specifying a fine-grained (1-100) quality scale and specifying chrominance subsampling. If a TurboVNC viewer sends those to a TightVNC server, then the TightVNC server ignores them, so the TurboVNC viewer also sends the quality on a 0-9 scale that the TightVNC server can understand. Similarly, the TurboVNC server checks first for fine-grained quality and subsampling pseudo-encodings from the viewer, and failing to receive those, it then checks for the TightVNC 0-9 quality pseudo-encoding. There is one case in which the two systems are not compatible, and that is when a TightVNC or TigerVNC viewer requests compression level 0 without JPEG from a TurboVNC server. For performance reasons, this causes the TurboVNC server to send images directly to the viewer, bypassing Zlib. When the TurboVNC server does this, it also sets bits 7-4 in the compression control byte to rfbTightNoZlib (0x0A), which is unfortunately the same value as rfbTightPng. Older TightVNC viewers that don't handle PNG will assume that the stream is uncompressed but still encapsulated in a Zlib structure, whereas newer PNG-supporting TightVNC viewers will assume that the stream is PNG. In either case, the viewer will probably crash. Since most VNC viewers don't expose compression level 0 in the GUI, this is a relatively rare situation. Description of changes: configure.ac -- Added support for libjpeg-turbo. If passed an argument of --with-turbovnc, configure will now run (or, if cross-compiling, just link) a test program that determines whether the libjpeg library being used is libjpeg-turbo. libjpeg-turbo must be used when building the TurboVNC encoder, because the TurboVNC encoder relies on the libjpeg-turbo colorspace extensions in order to compress images directly out of the framebuffer (which may be, for instance, BGRA rather than RGB.) libjpeg-turbo can optionally be used with the TightVNC encoder as well, but the speedup will only be marginal (the report linked above explains why in more detail, but basically it's because of Amdahl's Law. The TightVNC encoder was designed with the assumption that JPEG had a very high CPU cost, and thus JPEG is used only sparingly.) -- Added a new configure variable, JPEG_LDFLAGS. This is necessitated by the fact that libjpeg-turbo often distributes libjpeg.a and libjpeg.so in /opt/libjpeg-turbo/lib32 or /opt/libjpeg-turbo/lib64, and many people prefer to statically link with it. Thus, more flexibility is needed than is provided by --with-jpeg. If JPEG_LDFLAGS is specified, then it overrides the changes to LDFLAGS enacted by --with-jpeg (but --with-jpeg is still used to set the include path.) The addition of JPEG_LDFLAGS necessitated replacing AC_CHECK_LIB with AC_LINK_IFELSE (because AC_CHECK_LIB automatically sets LIBS to -ljpeg, which is not what we want if we're, for instance, linking statically with libjpeg-turbo.) -- configure does not check for PNG support if TurboVNC encoding is enabled. This prevents the rfbSendRectEncodingTightPng() function from being compiled in, since the TurboVNC encoder doesn't (and can't) support it. common/turbojpeg.c, common/turbojpeg.h -- TurboJPEG is a simple API used to compress and decompress JPEG images in memory. It was originally implemented because it was desirable to use different types of underlying technologies to compress JPEG on different platforms (mediaLib on SPARC, Quicktime on PPC Macs, Intel Performance Primitives, etc.) These days, however, libjpeg-turbo is the only underlying technology used by TurboVNC, so TurboJPEG's purpose is largely just code simplicity and flexibility. Thus, since there is no real need for libvncserver to use any technology other than libjpeg-turbo for compressing JPEG, the TurboJPEG wrapper for libjpeg-turbo has been included in-tree so that libvncserver can be directly linked with libjpeg-turbo. This is convenient because many modern Linux distros (Fedora, Ubuntu, etc.) now ship libjpeg-turbo as their default libjpeg library. libvncserver/rfbserver.c -- Added logic to check for the TurboVNC fine-grained quality level and subsampling encodings and to map Tight (0-9) quality levels to appropriate fine-grained quality level and subsampling values if communicating with a TightVNC/TigerVNC viewer. libvncserver/turbo.c -- TurboVNC encoder (compiled instead of libvncserver/tight.c) rfb/rfb.h -- Added support for the TurboVNC subsampling level rfb/rfbproto.h -- Added constants for the TurboVNC fine quality level and subsampling encodings as well as the rfbTightNoZlib constant and notes on its usage.
13 years ago
{
ERREXIT(cinfo, JERR_BUFFER_SIZE);
return TRUE;
}
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
static void dst_noop(j_compress_ptr cinfo)
Add TurboVNC encoding support. TurboVNC is a variant of TightVNC that uses the same client/server protocol (RFB version 3.8t), and thus it is fully cross-compatible with TightVNC and TigerVNC (with one exception, which is noted below.) Both the TightVNC and TurboVNC encoders analyze each rectangle, pick out regions of solid color to send separately, and send the remaining subrectangles using mono, indexed color, JPEG, or raw encoding, depending on the number of colors in the subrectangle. However, TurboVNC uses a fundamentally different selection algorithm to determine the appropriate subencoding to use for each subrectangle. Thus, while it sends a protocol stream that can be decoded by any TightVNC-compatible viewer, the mix of subencoding types in this protocol stream will be different from those generated by a TightVNC server. The research that led to TurboVNC is described in the following report: http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf. In summary: 20 RFB captures, representing "common" 2D and 3D application workloads (the 3D workloads were run using VirtualGL), were studied using the TightVNC encoder in isolation. Some of the analysis features in the TightVNC encoder, such as smoothness detection, were found to generate a lot of CPU usage with little or no benefit in compression, so those features were disabled. JPEG encoding was accelerated using libjpeg-turbo (which achieves a 2-4x speedup over plain libjpeg on modern x86 or ARM processors.) Finally, the "palette threshold" (minimum number of colors that the subrectangle must have before it is compressed using JPEG or raw) was adjusted to account for the fact that JPEG encoding is now quite a bit faster (meaning that we can now use it more without a CPU penalty.) TurboVNC has additional optimizations, such as the ability to count colors and encode JPEG images directly from the framebuffer without first translating the pixels into RGB. The TurboVNC encoder compares quite favorably in terms of compression ratio with TightVNC and generally encodes a great deal faster (often an order of magnitude or more.) The version of the TurboVNC encoder included in this patch is roughly equivalent to the one found in version 0.6 of the Unix TurboVNC Server, with a few minor patches integrated from TurboVNC 1.1. TurboVNC 1.0 added multi-threading capabilities, which can be added in later if desired (at the expense of making libvncserver depend on libpthread.) Because TurboVNC uses a fundamentally different mix of subencodings than TightVNC, because it uses the identical protocol (and thus a viewer really has no idea whether it's talking to a TightVNC or TurboVNC server), and because it doesn't support rfbTightPng (and in fact conflicts with it-- see below), the TurboVNC and TightVNC encoders cannot be enabled simultaneously. Compatibility: In *most* cases, a TurboVNC-enabled viewer is fully compatible with a TightVNC server, and vice versa. TurboVNC supports pseudo-encodings for specifying a fine-grained (1-100) quality scale and specifying chrominance subsampling. If a TurboVNC viewer sends those to a TightVNC server, then the TightVNC server ignores them, so the TurboVNC viewer also sends the quality on a 0-9 scale that the TightVNC server can understand. Similarly, the TurboVNC server checks first for fine-grained quality and subsampling pseudo-encodings from the viewer, and failing to receive those, it then checks for the TightVNC 0-9 quality pseudo-encoding. There is one case in which the two systems are not compatible, and that is when a TightVNC or TigerVNC viewer requests compression level 0 without JPEG from a TurboVNC server. For performance reasons, this causes the TurboVNC server to send images directly to the viewer, bypassing Zlib. When the TurboVNC server does this, it also sets bits 7-4 in the compression control byte to rfbTightNoZlib (0x0A), which is unfortunately the same value as rfbTightPng. Older TightVNC viewers that don't handle PNG will assume that the stream is uncompressed but still encapsulated in a Zlib structure, whereas newer PNG-supporting TightVNC viewers will assume that the stream is PNG. In either case, the viewer will probably crash. Since most VNC viewers don't expose compression level 0 in the GUI, this is a relatively rare situation. Description of changes: configure.ac -- Added support for libjpeg-turbo. If passed an argument of --with-turbovnc, configure will now run (or, if cross-compiling, just link) a test program that determines whether the libjpeg library being used is libjpeg-turbo. libjpeg-turbo must be used when building the TurboVNC encoder, because the TurboVNC encoder relies on the libjpeg-turbo colorspace extensions in order to compress images directly out of the framebuffer (which may be, for instance, BGRA rather than RGB.) libjpeg-turbo can optionally be used with the TightVNC encoder as well, but the speedup will only be marginal (the report linked above explains why in more detail, but basically it's because of Amdahl's Law. The TightVNC encoder was designed with the assumption that JPEG had a very high CPU cost, and thus JPEG is used only sparingly.) -- Added a new configure variable, JPEG_LDFLAGS. This is necessitated by the fact that libjpeg-turbo often distributes libjpeg.a and libjpeg.so in /opt/libjpeg-turbo/lib32 or /opt/libjpeg-turbo/lib64, and many people prefer to statically link with it. Thus, more flexibility is needed than is provided by --with-jpeg. If JPEG_LDFLAGS is specified, then it overrides the changes to LDFLAGS enacted by --with-jpeg (but --with-jpeg is still used to set the include path.) The addition of JPEG_LDFLAGS necessitated replacing AC_CHECK_LIB with AC_LINK_IFELSE (because AC_CHECK_LIB automatically sets LIBS to -ljpeg, which is not what we want if we're, for instance, linking statically with libjpeg-turbo.) -- configure does not check for PNG support if TurboVNC encoding is enabled. This prevents the rfbSendRectEncodingTightPng() function from being compiled in, since the TurboVNC encoder doesn't (and can't) support it. common/turbojpeg.c, common/turbojpeg.h -- TurboJPEG is a simple API used to compress and decompress JPEG images in memory. It was originally implemented because it was desirable to use different types of underlying technologies to compress JPEG on different platforms (mediaLib on SPARC, Quicktime on PPC Macs, Intel Performance Primitives, etc.) These days, however, libjpeg-turbo is the only underlying technology used by TurboVNC, so TurboJPEG's purpose is largely just code simplicity and flexibility. Thus, since there is no real need for libvncserver to use any technology other than libjpeg-turbo for compressing JPEG, the TurboJPEG wrapper for libjpeg-turbo has been included in-tree so that libvncserver can be directly linked with libjpeg-turbo. This is convenient because many modern Linux distros (Fedora, Ubuntu, etc.) now ship libjpeg-turbo as their default libjpeg library. libvncserver/rfbserver.c -- Added logic to check for the TurboVNC fine-grained quality level and subsampling encodings and to map Tight (0-9) quality levels to appropriate fine-grained quality level and subsampling values if communicating with a TightVNC/TigerVNC viewer. libvncserver/turbo.c -- TurboVNC encoder (compiled instead of libvncserver/tight.c) rfb/rfb.h -- Added support for the TurboVNC subsampling level rfb/rfbproto.h -- Added constants for the TurboVNC fine quality level and subsampling encodings as well as the rfbTightNoZlib constant and notes on its usage.
13 years ago
{
}
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
static tjhandle _tjInitCompress(tjinstance *this)
{
/* This is also straight out of example.c */
this->cinfo.err=jpeg_std_error(&this->jerr.pub);
this->jerr.pub.error_exit=my_error_exit;
this->jerr.pub.output_message=my_output_message;
if(setjmp(this->jerr.setjmp_buffer))
{
/* If we get here, the JPEG code has signaled an error. */
if(this) free(this); return NULL;
}
jpeg_create_compress(&this->cinfo);
this->cinfo.dest=&this->jdst;
this->jdst.init_destination=dst_noop;
this->jdst.empty_output_buffer=empty_output_buffer;
this->jdst.term_destination=dst_noop;
this->init|=COMPRESS;
return (tjhandle)this;
}
Add TurboVNC encoding support. TurboVNC is a variant of TightVNC that uses the same client/server protocol (RFB version 3.8t), and thus it is fully cross-compatible with TightVNC and TigerVNC (with one exception, which is noted below.) Both the TightVNC and TurboVNC encoders analyze each rectangle, pick out regions of solid color to send separately, and send the remaining subrectangles using mono, indexed color, JPEG, or raw encoding, depending on the number of colors in the subrectangle. However, TurboVNC uses a fundamentally different selection algorithm to determine the appropriate subencoding to use for each subrectangle. Thus, while it sends a protocol stream that can be decoded by any TightVNC-compatible viewer, the mix of subencoding types in this protocol stream will be different from those generated by a TightVNC server. The research that led to TurboVNC is described in the following report: http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf. In summary: 20 RFB captures, representing "common" 2D and 3D application workloads (the 3D workloads were run using VirtualGL), were studied using the TightVNC encoder in isolation. Some of the analysis features in the TightVNC encoder, such as smoothness detection, were found to generate a lot of CPU usage with little or no benefit in compression, so those features were disabled. JPEG encoding was accelerated using libjpeg-turbo (which achieves a 2-4x speedup over plain libjpeg on modern x86 or ARM processors.) Finally, the "palette threshold" (minimum number of colors that the subrectangle must have before it is compressed using JPEG or raw) was adjusted to account for the fact that JPEG encoding is now quite a bit faster (meaning that we can now use it more without a CPU penalty.) TurboVNC has additional optimizations, such as the ability to count colors and encode JPEG images directly from the framebuffer without first translating the pixels into RGB. The TurboVNC encoder compares quite favorably in terms of compression ratio with TightVNC and generally encodes a great deal faster (often an order of magnitude or more.) The version of the TurboVNC encoder included in this patch is roughly equivalent to the one found in version 0.6 of the Unix TurboVNC Server, with a few minor patches integrated from TurboVNC 1.1. TurboVNC 1.0 added multi-threading capabilities, which can be added in later if desired (at the expense of making libvncserver depend on libpthread.) Because TurboVNC uses a fundamentally different mix of subencodings than TightVNC, because it uses the identical protocol (and thus a viewer really has no idea whether it's talking to a TightVNC or TurboVNC server), and because it doesn't support rfbTightPng (and in fact conflicts with it-- see below), the TurboVNC and TightVNC encoders cannot be enabled simultaneously. Compatibility: In *most* cases, a TurboVNC-enabled viewer is fully compatible with a TightVNC server, and vice versa. TurboVNC supports pseudo-encodings for specifying a fine-grained (1-100) quality scale and specifying chrominance subsampling. If a TurboVNC viewer sends those to a TightVNC server, then the TightVNC server ignores them, so the TurboVNC viewer also sends the quality on a 0-9 scale that the TightVNC server can understand. Similarly, the TurboVNC server checks first for fine-grained quality and subsampling pseudo-encodings from the viewer, and failing to receive those, it then checks for the TightVNC 0-9 quality pseudo-encoding. There is one case in which the two systems are not compatible, and that is when a TightVNC or TigerVNC viewer requests compression level 0 without JPEG from a TurboVNC server. For performance reasons, this causes the TurboVNC server to send images directly to the viewer, bypassing Zlib. When the TurboVNC server does this, it also sets bits 7-4 in the compression control byte to rfbTightNoZlib (0x0A), which is unfortunately the same value as rfbTightPng. Older TightVNC viewers that don't handle PNG will assume that the stream is uncompressed but still encapsulated in a Zlib structure, whereas newer PNG-supporting TightVNC viewers will assume that the stream is PNG. In either case, the viewer will probably crash. Since most VNC viewers don't expose compression level 0 in the GUI, this is a relatively rare situation. Description of changes: configure.ac -- Added support for libjpeg-turbo. If passed an argument of --with-turbovnc, configure will now run (or, if cross-compiling, just link) a test program that determines whether the libjpeg library being used is libjpeg-turbo. libjpeg-turbo must be used when building the TurboVNC encoder, because the TurboVNC encoder relies on the libjpeg-turbo colorspace extensions in order to compress images directly out of the framebuffer (which may be, for instance, BGRA rather than RGB.) libjpeg-turbo can optionally be used with the TightVNC encoder as well, but the speedup will only be marginal (the report linked above explains why in more detail, but basically it's because of Amdahl's Law. The TightVNC encoder was designed with the assumption that JPEG had a very high CPU cost, and thus JPEG is used only sparingly.) -- Added a new configure variable, JPEG_LDFLAGS. This is necessitated by the fact that libjpeg-turbo often distributes libjpeg.a and libjpeg.so in /opt/libjpeg-turbo/lib32 or /opt/libjpeg-turbo/lib64, and many people prefer to statically link with it. Thus, more flexibility is needed than is provided by --with-jpeg. If JPEG_LDFLAGS is specified, then it overrides the changes to LDFLAGS enacted by --with-jpeg (but --with-jpeg is still used to set the include path.) The addition of JPEG_LDFLAGS necessitated replacing AC_CHECK_LIB with AC_LINK_IFELSE (because AC_CHECK_LIB automatically sets LIBS to -ljpeg, which is not what we want if we're, for instance, linking statically with libjpeg-turbo.) -- configure does not check for PNG support if TurboVNC encoding is enabled. This prevents the rfbSendRectEncodingTightPng() function from being compiled in, since the TurboVNC encoder doesn't (and can't) support it. common/turbojpeg.c, common/turbojpeg.h -- TurboJPEG is a simple API used to compress and decompress JPEG images in memory. It was originally implemented because it was desirable to use different types of underlying technologies to compress JPEG on different platforms (mediaLib on SPARC, Quicktime on PPC Macs, Intel Performance Primitives, etc.) These days, however, libjpeg-turbo is the only underlying technology used by TurboVNC, so TurboJPEG's purpose is largely just code simplicity and flexibility. Thus, since there is no real need for libvncserver to use any technology other than libjpeg-turbo for compressing JPEG, the TurboJPEG wrapper for libjpeg-turbo has been included in-tree so that libvncserver can be directly linked with libjpeg-turbo. This is convenient because many modern Linux distros (Fedora, Ubuntu, etc.) now ship libjpeg-turbo as their default libjpeg library. libvncserver/rfbserver.c -- Added logic to check for the TurboVNC fine-grained quality level and subsampling encodings and to map Tight (0-9) quality levels to appropriate fine-grained quality level and subsampling values if communicating with a TightVNC/TigerVNC viewer. libvncserver/turbo.c -- TurboVNC encoder (compiled instead of libvncserver/tight.c) rfb/rfb.h -- Added support for the TurboVNC subsampling level rfb/rfbproto.h -- Added constants for the TurboVNC fine quality level and subsampling encodings as well as the rfbTightNoZlib constant and notes on its usage.
13 years ago
DLLEXPORT tjhandle DLLCALL tjInitCompress(void)
{
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
tjinstance *this=NULL;
if((this=(tjinstance *)malloc(sizeof(tjinstance)))==NULL)
{
snprintf(errStr, JMSG_LENGTH_MAX,
"tjInitCompress(): Memory allocation failure");
return NULL;
Add TurboVNC encoding support. TurboVNC is a variant of TightVNC that uses the same client/server protocol (RFB version 3.8t), and thus it is fully cross-compatible with TightVNC and TigerVNC (with one exception, which is noted below.) Both the TightVNC and TurboVNC encoders analyze each rectangle, pick out regions of solid color to send separately, and send the remaining subrectangles using mono, indexed color, JPEG, or raw encoding, depending on the number of colors in the subrectangle. However, TurboVNC uses a fundamentally different selection algorithm to determine the appropriate subencoding to use for each subrectangle. Thus, while it sends a protocol stream that can be decoded by any TightVNC-compatible viewer, the mix of subencoding types in this protocol stream will be different from those generated by a TightVNC server. The research that led to TurboVNC is described in the following report: http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf. In summary: 20 RFB captures, representing "common" 2D and 3D application workloads (the 3D workloads were run using VirtualGL), were studied using the TightVNC encoder in isolation. Some of the analysis features in the TightVNC encoder, such as smoothness detection, were found to generate a lot of CPU usage with little or no benefit in compression, so those features were disabled. JPEG encoding was accelerated using libjpeg-turbo (which achieves a 2-4x speedup over plain libjpeg on modern x86 or ARM processors.) Finally, the "palette threshold" (minimum number of colors that the subrectangle must have before it is compressed using JPEG or raw) was adjusted to account for the fact that JPEG encoding is now quite a bit faster (meaning that we can now use it more without a CPU penalty.) TurboVNC has additional optimizations, such as the ability to count colors and encode JPEG images directly from the framebuffer without first translating the pixels into RGB. The TurboVNC encoder compares quite favorably in terms of compression ratio with TightVNC and generally encodes a great deal faster (often an order of magnitude or more.) The version of the TurboVNC encoder included in this patch is roughly equivalent to the one found in version 0.6 of the Unix TurboVNC Server, with a few minor patches integrated from TurboVNC 1.1. TurboVNC 1.0 added multi-threading capabilities, which can be added in later if desired (at the expense of making libvncserver depend on libpthread.) Because TurboVNC uses a fundamentally different mix of subencodings than TightVNC, because it uses the identical protocol (and thus a viewer really has no idea whether it's talking to a TightVNC or TurboVNC server), and because it doesn't support rfbTightPng (and in fact conflicts with it-- see below), the TurboVNC and TightVNC encoders cannot be enabled simultaneously. Compatibility: In *most* cases, a TurboVNC-enabled viewer is fully compatible with a TightVNC server, and vice versa. TurboVNC supports pseudo-encodings for specifying a fine-grained (1-100) quality scale and specifying chrominance subsampling. If a TurboVNC viewer sends those to a TightVNC server, then the TightVNC server ignores them, so the TurboVNC viewer also sends the quality on a 0-9 scale that the TightVNC server can understand. Similarly, the TurboVNC server checks first for fine-grained quality and subsampling pseudo-encodings from the viewer, and failing to receive those, it then checks for the TightVNC 0-9 quality pseudo-encoding. There is one case in which the two systems are not compatible, and that is when a TightVNC or TigerVNC viewer requests compression level 0 without JPEG from a TurboVNC server. For performance reasons, this causes the TurboVNC server to send images directly to the viewer, bypassing Zlib. When the TurboVNC server does this, it also sets bits 7-4 in the compression control byte to rfbTightNoZlib (0x0A), which is unfortunately the same value as rfbTightPng. Older TightVNC viewers that don't handle PNG will assume that the stream is uncompressed but still encapsulated in a Zlib structure, whereas newer PNG-supporting TightVNC viewers will assume that the stream is PNG. In either case, the viewer will probably crash. Since most VNC viewers don't expose compression level 0 in the GUI, this is a relatively rare situation. Description of changes: configure.ac -- Added support for libjpeg-turbo. If passed an argument of --with-turbovnc, configure will now run (or, if cross-compiling, just link) a test program that determines whether the libjpeg library being used is libjpeg-turbo. libjpeg-turbo must be used when building the TurboVNC encoder, because the TurboVNC encoder relies on the libjpeg-turbo colorspace extensions in order to compress images directly out of the framebuffer (which may be, for instance, BGRA rather than RGB.) libjpeg-turbo can optionally be used with the TightVNC encoder as well, but the speedup will only be marginal (the report linked above explains why in more detail, but basically it's because of Amdahl's Law. The TightVNC encoder was designed with the assumption that JPEG had a very high CPU cost, and thus JPEG is used only sparingly.) -- Added a new configure variable, JPEG_LDFLAGS. This is necessitated by the fact that libjpeg-turbo often distributes libjpeg.a and libjpeg.so in /opt/libjpeg-turbo/lib32 or /opt/libjpeg-turbo/lib64, and many people prefer to statically link with it. Thus, more flexibility is needed than is provided by --with-jpeg. If JPEG_LDFLAGS is specified, then it overrides the changes to LDFLAGS enacted by --with-jpeg (but --with-jpeg is still used to set the include path.) The addition of JPEG_LDFLAGS necessitated replacing AC_CHECK_LIB with AC_LINK_IFELSE (because AC_CHECK_LIB automatically sets LIBS to -ljpeg, which is not what we want if we're, for instance, linking statically with libjpeg-turbo.) -- configure does not check for PNG support if TurboVNC encoding is enabled. This prevents the rfbSendRectEncodingTightPng() function from being compiled in, since the TurboVNC encoder doesn't (and can't) support it. common/turbojpeg.c, common/turbojpeg.h -- TurboJPEG is a simple API used to compress and decompress JPEG images in memory. It was originally implemented because it was desirable to use different types of underlying technologies to compress JPEG on different platforms (mediaLib on SPARC, Quicktime on PPC Macs, Intel Performance Primitives, etc.) These days, however, libjpeg-turbo is the only underlying technology used by TurboVNC, so TurboJPEG's purpose is largely just code simplicity and flexibility. Thus, since there is no real need for libvncserver to use any technology other than libjpeg-turbo for compressing JPEG, the TurboJPEG wrapper for libjpeg-turbo has been included in-tree so that libvncserver can be directly linked with libjpeg-turbo. This is convenient because many modern Linux distros (Fedora, Ubuntu, etc.) now ship libjpeg-turbo as their default libjpeg library. libvncserver/rfbserver.c -- Added logic to check for the TurboVNC fine-grained quality level and subsampling encodings and to map Tight (0-9) quality levels to appropriate fine-grained quality level and subsampling values if communicating with a TightVNC/TigerVNC viewer. libvncserver/turbo.c -- TurboVNC encoder (compiled instead of libvncserver/tight.c) rfb/rfb.h -- Added support for the TurboVNC subsampling level rfb/rfbproto.h -- Added constants for the TurboVNC fine quality level and subsampling encodings as well as the rfbTightNoZlib constant and notes on its usage.
13 years ago
}
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
MEMZERO(this, sizeof(tjinstance));
return _tjInitCompress(this);
}
Add TurboVNC encoding support. TurboVNC is a variant of TightVNC that uses the same client/server protocol (RFB version 3.8t), and thus it is fully cross-compatible with TightVNC and TigerVNC (with one exception, which is noted below.) Both the TightVNC and TurboVNC encoders analyze each rectangle, pick out regions of solid color to send separately, and send the remaining subrectangles using mono, indexed color, JPEG, or raw encoding, depending on the number of colors in the subrectangle. However, TurboVNC uses a fundamentally different selection algorithm to determine the appropriate subencoding to use for each subrectangle. Thus, while it sends a protocol stream that can be decoded by any TightVNC-compatible viewer, the mix of subencoding types in this protocol stream will be different from those generated by a TightVNC server. The research that led to TurboVNC is described in the following report: http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf. In summary: 20 RFB captures, representing "common" 2D and 3D application workloads (the 3D workloads were run using VirtualGL), were studied using the TightVNC encoder in isolation. Some of the analysis features in the TightVNC encoder, such as smoothness detection, were found to generate a lot of CPU usage with little or no benefit in compression, so those features were disabled. JPEG encoding was accelerated using libjpeg-turbo (which achieves a 2-4x speedup over plain libjpeg on modern x86 or ARM processors.) Finally, the "palette threshold" (minimum number of colors that the subrectangle must have before it is compressed using JPEG or raw) was adjusted to account for the fact that JPEG encoding is now quite a bit faster (meaning that we can now use it more without a CPU penalty.) TurboVNC has additional optimizations, such as the ability to count colors and encode JPEG images directly from the framebuffer without first translating the pixels into RGB. The TurboVNC encoder compares quite favorably in terms of compression ratio with TightVNC and generally encodes a great deal faster (often an order of magnitude or more.) The version of the TurboVNC encoder included in this patch is roughly equivalent to the one found in version 0.6 of the Unix TurboVNC Server, with a few minor patches integrated from TurboVNC 1.1. TurboVNC 1.0 added multi-threading capabilities, which can be added in later if desired (at the expense of making libvncserver depend on libpthread.) Because TurboVNC uses a fundamentally different mix of subencodings than TightVNC, because it uses the identical protocol (and thus a viewer really has no idea whether it's talking to a TightVNC or TurboVNC server), and because it doesn't support rfbTightPng (and in fact conflicts with it-- see below), the TurboVNC and TightVNC encoders cannot be enabled simultaneously. Compatibility: In *most* cases, a TurboVNC-enabled viewer is fully compatible with a TightVNC server, and vice versa. TurboVNC supports pseudo-encodings for specifying a fine-grained (1-100) quality scale and specifying chrominance subsampling. If a TurboVNC viewer sends those to a TightVNC server, then the TightVNC server ignores them, so the TurboVNC viewer also sends the quality on a 0-9 scale that the TightVNC server can understand. Similarly, the TurboVNC server checks first for fine-grained quality and subsampling pseudo-encodings from the viewer, and failing to receive those, it then checks for the TightVNC 0-9 quality pseudo-encoding. There is one case in which the two systems are not compatible, and that is when a TightVNC or TigerVNC viewer requests compression level 0 without JPEG from a TurboVNC server. For performance reasons, this causes the TurboVNC server to send images directly to the viewer, bypassing Zlib. When the TurboVNC server does this, it also sets bits 7-4 in the compression control byte to rfbTightNoZlib (0x0A), which is unfortunately the same value as rfbTightPng. Older TightVNC viewers that don't handle PNG will assume that the stream is uncompressed but still encapsulated in a Zlib structure, whereas newer PNG-supporting TightVNC viewers will assume that the stream is PNG. In either case, the viewer will probably crash. Since most VNC viewers don't expose compression level 0 in the GUI, this is a relatively rare situation. Description of changes: configure.ac -- Added support for libjpeg-turbo. If passed an argument of --with-turbovnc, configure will now run (or, if cross-compiling, just link) a test program that determines whether the libjpeg library being used is libjpeg-turbo. libjpeg-turbo must be used when building the TurboVNC encoder, because the TurboVNC encoder relies on the libjpeg-turbo colorspace extensions in order to compress images directly out of the framebuffer (which may be, for instance, BGRA rather than RGB.) libjpeg-turbo can optionally be used with the TightVNC encoder as well, but the speedup will only be marginal (the report linked above explains why in more detail, but basically it's because of Amdahl's Law. The TightVNC encoder was designed with the assumption that JPEG had a very high CPU cost, and thus JPEG is used only sparingly.) -- Added a new configure variable, JPEG_LDFLAGS. This is necessitated by the fact that libjpeg-turbo often distributes libjpeg.a and libjpeg.so in /opt/libjpeg-turbo/lib32 or /opt/libjpeg-turbo/lib64, and many people prefer to statically link with it. Thus, more flexibility is needed than is provided by --with-jpeg. If JPEG_LDFLAGS is specified, then it overrides the changes to LDFLAGS enacted by --with-jpeg (but --with-jpeg is still used to set the include path.) The addition of JPEG_LDFLAGS necessitated replacing AC_CHECK_LIB with AC_LINK_IFELSE (because AC_CHECK_LIB automatically sets LIBS to -ljpeg, which is not what we want if we're, for instance, linking statically with libjpeg-turbo.) -- configure does not check for PNG support if TurboVNC encoding is enabled. This prevents the rfbSendRectEncodingTightPng() function from being compiled in, since the TurboVNC encoder doesn't (and can't) support it. common/turbojpeg.c, common/turbojpeg.h -- TurboJPEG is a simple API used to compress and decompress JPEG images in memory. It was originally implemented because it was desirable to use different types of underlying technologies to compress JPEG on different platforms (mediaLib on SPARC, Quicktime on PPC Macs, Intel Performance Primitives, etc.) These days, however, libjpeg-turbo is the only underlying technology used by TurboVNC, so TurboJPEG's purpose is largely just code simplicity and flexibility. Thus, since there is no real need for libvncserver to use any technology other than libjpeg-turbo for compressing JPEG, the TurboJPEG wrapper for libjpeg-turbo has been included in-tree so that libvncserver can be directly linked with libjpeg-turbo. This is convenient because many modern Linux distros (Fedora, Ubuntu, etc.) now ship libjpeg-turbo as their default libjpeg library. libvncserver/rfbserver.c -- Added logic to check for the TurboVNC fine-grained quality level and subsampling encodings and to map Tight (0-9) quality levels to appropriate fine-grained quality level and subsampling values if communicating with a TightVNC/TigerVNC viewer. libvncserver/turbo.c -- TurboVNC encoder (compiled instead of libvncserver/tight.c) rfb/rfb.h -- Added support for the TurboVNC subsampling level rfb/rfbproto.h -- Added constants for the TurboVNC fine quality level and subsampling encodings as well as the rfbTightNoZlib constant and notes on its usage.
13 years ago
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
DLLEXPORT unsigned long DLLCALL tjBufSize(int width, int height,
int jpegSubsamp)
{
unsigned long retval=0; int mcuw, mcuh, chromasf;
if(width<1 || height<1 || jpegSubsamp<0 || jpegSubsamp>=NUMSUBOPT)
_throw("tjBufSize(): Invalid argument");
Add TurboVNC encoding support. TurboVNC is a variant of TightVNC that uses the same client/server protocol (RFB version 3.8t), and thus it is fully cross-compatible with TightVNC and TigerVNC (with one exception, which is noted below.) Both the TightVNC and TurboVNC encoders analyze each rectangle, pick out regions of solid color to send separately, and send the remaining subrectangles using mono, indexed color, JPEG, or raw encoding, depending on the number of colors in the subrectangle. However, TurboVNC uses a fundamentally different selection algorithm to determine the appropriate subencoding to use for each subrectangle. Thus, while it sends a protocol stream that can be decoded by any TightVNC-compatible viewer, the mix of subencoding types in this protocol stream will be different from those generated by a TightVNC server. The research that led to TurboVNC is described in the following report: http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf. In summary: 20 RFB captures, representing "common" 2D and 3D application workloads (the 3D workloads were run using VirtualGL), were studied using the TightVNC encoder in isolation. Some of the analysis features in the TightVNC encoder, such as smoothness detection, were found to generate a lot of CPU usage with little or no benefit in compression, so those features were disabled. JPEG encoding was accelerated using libjpeg-turbo (which achieves a 2-4x speedup over plain libjpeg on modern x86 or ARM processors.) Finally, the "palette threshold" (minimum number of colors that the subrectangle must have before it is compressed using JPEG or raw) was adjusted to account for the fact that JPEG encoding is now quite a bit faster (meaning that we can now use it more without a CPU penalty.) TurboVNC has additional optimizations, such as the ability to count colors and encode JPEG images directly from the framebuffer without first translating the pixels into RGB. The TurboVNC encoder compares quite favorably in terms of compression ratio with TightVNC and generally encodes a great deal faster (often an order of magnitude or more.) The version of the TurboVNC encoder included in this patch is roughly equivalent to the one found in version 0.6 of the Unix TurboVNC Server, with a few minor patches integrated from TurboVNC 1.1. TurboVNC 1.0 added multi-threading capabilities, which can be added in later if desired (at the expense of making libvncserver depend on libpthread.) Because TurboVNC uses a fundamentally different mix of subencodings than TightVNC, because it uses the identical protocol (and thus a viewer really has no idea whether it's talking to a TightVNC or TurboVNC server), and because it doesn't support rfbTightPng (and in fact conflicts with it-- see below), the TurboVNC and TightVNC encoders cannot be enabled simultaneously. Compatibility: In *most* cases, a TurboVNC-enabled viewer is fully compatible with a TightVNC server, and vice versa. TurboVNC supports pseudo-encodings for specifying a fine-grained (1-100) quality scale and specifying chrominance subsampling. If a TurboVNC viewer sends those to a TightVNC server, then the TightVNC server ignores them, so the TurboVNC viewer also sends the quality on a 0-9 scale that the TightVNC server can understand. Similarly, the TurboVNC server checks first for fine-grained quality and subsampling pseudo-encodings from the viewer, and failing to receive those, it then checks for the TightVNC 0-9 quality pseudo-encoding. There is one case in which the two systems are not compatible, and that is when a TightVNC or TigerVNC viewer requests compression level 0 without JPEG from a TurboVNC server. For performance reasons, this causes the TurboVNC server to send images directly to the viewer, bypassing Zlib. When the TurboVNC server does this, it also sets bits 7-4 in the compression control byte to rfbTightNoZlib (0x0A), which is unfortunately the same value as rfbTightPng. Older TightVNC viewers that don't handle PNG will assume that the stream is uncompressed but still encapsulated in a Zlib structure, whereas newer PNG-supporting TightVNC viewers will assume that the stream is PNG. In either case, the viewer will probably crash. Since most VNC viewers don't expose compression level 0 in the GUI, this is a relatively rare situation. Description of changes: configure.ac -- Added support for libjpeg-turbo. If passed an argument of --with-turbovnc, configure will now run (or, if cross-compiling, just link) a test program that determines whether the libjpeg library being used is libjpeg-turbo. libjpeg-turbo must be used when building the TurboVNC encoder, because the TurboVNC encoder relies on the libjpeg-turbo colorspace extensions in order to compress images directly out of the framebuffer (which may be, for instance, BGRA rather than RGB.) libjpeg-turbo can optionally be used with the TightVNC encoder as well, but the speedup will only be marginal (the report linked above explains why in more detail, but basically it's because of Amdahl's Law. The TightVNC encoder was designed with the assumption that JPEG had a very high CPU cost, and thus JPEG is used only sparingly.) -- Added a new configure variable, JPEG_LDFLAGS. This is necessitated by the fact that libjpeg-turbo often distributes libjpeg.a and libjpeg.so in /opt/libjpeg-turbo/lib32 or /opt/libjpeg-turbo/lib64, and many people prefer to statically link with it. Thus, more flexibility is needed than is provided by --with-jpeg. If JPEG_LDFLAGS is specified, then it overrides the changes to LDFLAGS enacted by --with-jpeg (but --with-jpeg is still used to set the include path.) The addition of JPEG_LDFLAGS necessitated replacing AC_CHECK_LIB with AC_LINK_IFELSE (because AC_CHECK_LIB automatically sets LIBS to -ljpeg, which is not what we want if we're, for instance, linking statically with libjpeg-turbo.) -- configure does not check for PNG support if TurboVNC encoding is enabled. This prevents the rfbSendRectEncodingTightPng() function from being compiled in, since the TurboVNC encoder doesn't (and can't) support it. common/turbojpeg.c, common/turbojpeg.h -- TurboJPEG is a simple API used to compress and decompress JPEG images in memory. It was originally implemented because it was desirable to use different types of underlying technologies to compress JPEG on different platforms (mediaLib on SPARC, Quicktime on PPC Macs, Intel Performance Primitives, etc.) These days, however, libjpeg-turbo is the only underlying technology used by TurboVNC, so TurboJPEG's purpose is largely just code simplicity and flexibility. Thus, since there is no real need for libvncserver to use any technology other than libjpeg-turbo for compressing JPEG, the TurboJPEG wrapper for libjpeg-turbo has been included in-tree so that libvncserver can be directly linked with libjpeg-turbo. This is convenient because many modern Linux distros (Fedora, Ubuntu, etc.) now ship libjpeg-turbo as their default libjpeg library. libvncserver/rfbserver.c -- Added logic to check for the TurboVNC fine-grained quality level and subsampling encodings and to map Tight (0-9) quality levels to appropriate fine-grained quality level and subsampling values if communicating with a TightVNC/TigerVNC viewer. libvncserver/turbo.c -- TurboVNC encoder (compiled instead of libvncserver/tight.c) rfb/rfb.h -- Added support for the TurboVNC subsampling level rfb/rfbproto.h -- Added constants for the TurboVNC fine quality level and subsampling encodings as well as the rfbTightNoZlib constant and notes on its usage.
13 years ago
/*
* This allows for rare corner cases in which a JPEG image can actually be
* larger than the uncompressed input (we wouldn't mention it if it hadn't
* happened before.)
*/
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
mcuw=tjMCUWidth[jpegSubsamp];
mcuh=tjMCUHeight[jpegSubsamp];
chromasf=jpegSubsamp==TJSAMP_GRAY? 0: 4*64/(mcuw*mcuh);
retval=PAD(width, mcuw) * PAD(height, mcuh) * (2 + chromasf) + 2048;
bailout:
return retval;
Add TurboVNC encoding support. TurboVNC is a variant of TightVNC that uses the same client/server protocol (RFB version 3.8t), and thus it is fully cross-compatible with TightVNC and TigerVNC (with one exception, which is noted below.) Both the TightVNC and TurboVNC encoders analyze each rectangle, pick out regions of solid color to send separately, and send the remaining subrectangles using mono, indexed color, JPEG, or raw encoding, depending on the number of colors in the subrectangle. However, TurboVNC uses a fundamentally different selection algorithm to determine the appropriate subencoding to use for each subrectangle. Thus, while it sends a protocol stream that can be decoded by any TightVNC-compatible viewer, the mix of subencoding types in this protocol stream will be different from those generated by a TightVNC server. The research that led to TurboVNC is described in the following report: http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf. In summary: 20 RFB captures, representing "common" 2D and 3D application workloads (the 3D workloads were run using VirtualGL), were studied using the TightVNC encoder in isolation. Some of the analysis features in the TightVNC encoder, such as smoothness detection, were found to generate a lot of CPU usage with little or no benefit in compression, so those features were disabled. JPEG encoding was accelerated using libjpeg-turbo (which achieves a 2-4x speedup over plain libjpeg on modern x86 or ARM processors.) Finally, the "palette threshold" (minimum number of colors that the subrectangle must have before it is compressed using JPEG or raw) was adjusted to account for the fact that JPEG encoding is now quite a bit faster (meaning that we can now use it more without a CPU penalty.) TurboVNC has additional optimizations, such as the ability to count colors and encode JPEG images directly from the framebuffer without first translating the pixels into RGB. The TurboVNC encoder compares quite favorably in terms of compression ratio with TightVNC and generally encodes a great deal faster (often an order of magnitude or more.) The version of the TurboVNC encoder included in this patch is roughly equivalent to the one found in version 0.6 of the Unix TurboVNC Server, with a few minor patches integrated from TurboVNC 1.1. TurboVNC 1.0 added multi-threading capabilities, which can be added in later if desired (at the expense of making libvncserver depend on libpthread.) Because TurboVNC uses a fundamentally different mix of subencodings than TightVNC, because it uses the identical protocol (and thus a viewer really has no idea whether it's talking to a TightVNC or TurboVNC server), and because it doesn't support rfbTightPng (and in fact conflicts with it-- see below), the TurboVNC and TightVNC encoders cannot be enabled simultaneously. Compatibility: In *most* cases, a TurboVNC-enabled viewer is fully compatible with a TightVNC server, and vice versa. TurboVNC supports pseudo-encodings for specifying a fine-grained (1-100) quality scale and specifying chrominance subsampling. If a TurboVNC viewer sends those to a TightVNC server, then the TightVNC server ignores them, so the TurboVNC viewer also sends the quality on a 0-9 scale that the TightVNC server can understand. Similarly, the TurboVNC server checks first for fine-grained quality and subsampling pseudo-encodings from the viewer, and failing to receive those, it then checks for the TightVNC 0-9 quality pseudo-encoding. There is one case in which the two systems are not compatible, and that is when a TightVNC or TigerVNC viewer requests compression level 0 without JPEG from a TurboVNC server. For performance reasons, this causes the TurboVNC server to send images directly to the viewer, bypassing Zlib. When the TurboVNC server does this, it also sets bits 7-4 in the compression control byte to rfbTightNoZlib (0x0A), which is unfortunately the same value as rfbTightPng. Older TightVNC viewers that don't handle PNG will assume that the stream is uncompressed but still encapsulated in a Zlib structure, whereas newer PNG-supporting TightVNC viewers will assume that the stream is PNG. In either case, the viewer will probably crash. Since most VNC viewers don't expose compression level 0 in the GUI, this is a relatively rare situation. Description of changes: configure.ac -- Added support for libjpeg-turbo. If passed an argument of --with-turbovnc, configure will now run (or, if cross-compiling, just link) a test program that determines whether the libjpeg library being used is libjpeg-turbo. libjpeg-turbo must be used when building the TurboVNC encoder, because the TurboVNC encoder relies on the libjpeg-turbo colorspace extensions in order to compress images directly out of the framebuffer (which may be, for instance, BGRA rather than RGB.) libjpeg-turbo can optionally be used with the TightVNC encoder as well, but the speedup will only be marginal (the report linked above explains why in more detail, but basically it's because of Amdahl's Law. The TightVNC encoder was designed with the assumption that JPEG had a very high CPU cost, and thus JPEG is used only sparingly.) -- Added a new configure variable, JPEG_LDFLAGS. This is necessitated by the fact that libjpeg-turbo often distributes libjpeg.a and libjpeg.so in /opt/libjpeg-turbo/lib32 or /opt/libjpeg-turbo/lib64, and many people prefer to statically link with it. Thus, more flexibility is needed than is provided by --with-jpeg. If JPEG_LDFLAGS is specified, then it overrides the changes to LDFLAGS enacted by --with-jpeg (but --with-jpeg is still used to set the include path.) The addition of JPEG_LDFLAGS necessitated replacing AC_CHECK_LIB with AC_LINK_IFELSE (because AC_CHECK_LIB automatically sets LIBS to -ljpeg, which is not what we want if we're, for instance, linking statically with libjpeg-turbo.) -- configure does not check for PNG support if TurboVNC encoding is enabled. This prevents the rfbSendRectEncodingTightPng() function from being compiled in, since the TurboVNC encoder doesn't (and can't) support it. common/turbojpeg.c, common/turbojpeg.h -- TurboJPEG is a simple API used to compress and decompress JPEG images in memory. It was originally implemented because it was desirable to use different types of underlying technologies to compress JPEG on different platforms (mediaLib on SPARC, Quicktime on PPC Macs, Intel Performance Primitives, etc.) These days, however, libjpeg-turbo is the only underlying technology used by TurboVNC, so TurboJPEG's purpose is largely just code simplicity and flexibility. Thus, since there is no real need for libvncserver to use any technology other than libjpeg-turbo for compressing JPEG, the TurboJPEG wrapper for libjpeg-turbo has been included in-tree so that libvncserver can be directly linked with libjpeg-turbo. This is convenient because many modern Linux distros (Fedora, Ubuntu, etc.) now ship libjpeg-turbo as their default libjpeg library. libvncserver/rfbserver.c -- Added logic to check for the TurboVNC fine-grained quality level and subsampling encodings and to map Tight (0-9) quality levels to appropriate fine-grained quality level and subsampling values if communicating with a TightVNC/TigerVNC viewer. libvncserver/turbo.c -- TurboVNC encoder (compiled instead of libvncserver/tight.c) rfb/rfb.h -- Added support for the TurboVNC subsampling level rfb/rfbproto.h -- Added constants for the TurboVNC fine quality level and subsampling encodings as well as the rfbTightNoZlib constant and notes on its usage.
13 years ago
}
DLLEXPORT unsigned long DLLCALL TJBUFSIZE(int width, int height)
{
unsigned long retval=0;
if(width<1 || height<1)
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
_throw("TJBUFSIZE(): Invalid argument");
Add TurboVNC encoding support. TurboVNC is a variant of TightVNC that uses the same client/server protocol (RFB version 3.8t), and thus it is fully cross-compatible with TightVNC and TigerVNC (with one exception, which is noted below.) Both the TightVNC and TurboVNC encoders analyze each rectangle, pick out regions of solid color to send separately, and send the remaining subrectangles using mono, indexed color, JPEG, or raw encoding, depending on the number of colors in the subrectangle. However, TurboVNC uses a fundamentally different selection algorithm to determine the appropriate subencoding to use for each subrectangle. Thus, while it sends a protocol stream that can be decoded by any TightVNC-compatible viewer, the mix of subencoding types in this protocol stream will be different from those generated by a TightVNC server. The research that led to TurboVNC is described in the following report: http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf. In summary: 20 RFB captures, representing "common" 2D and 3D application workloads (the 3D workloads were run using VirtualGL), were studied using the TightVNC encoder in isolation. Some of the analysis features in the TightVNC encoder, such as smoothness detection, were found to generate a lot of CPU usage with little or no benefit in compression, so those features were disabled. JPEG encoding was accelerated using libjpeg-turbo (which achieves a 2-4x speedup over plain libjpeg on modern x86 or ARM processors.) Finally, the "palette threshold" (minimum number of colors that the subrectangle must have before it is compressed using JPEG or raw) was adjusted to account for the fact that JPEG encoding is now quite a bit faster (meaning that we can now use it more without a CPU penalty.) TurboVNC has additional optimizations, such as the ability to count colors and encode JPEG images directly from the framebuffer without first translating the pixels into RGB. The TurboVNC encoder compares quite favorably in terms of compression ratio with TightVNC and generally encodes a great deal faster (often an order of magnitude or more.) The version of the TurboVNC encoder included in this patch is roughly equivalent to the one found in version 0.6 of the Unix TurboVNC Server, with a few minor patches integrated from TurboVNC 1.1. TurboVNC 1.0 added multi-threading capabilities, which can be added in later if desired (at the expense of making libvncserver depend on libpthread.) Because TurboVNC uses a fundamentally different mix of subencodings than TightVNC, because it uses the identical protocol (and thus a viewer really has no idea whether it's talking to a TightVNC or TurboVNC server), and because it doesn't support rfbTightPng (and in fact conflicts with it-- see below), the TurboVNC and TightVNC encoders cannot be enabled simultaneously. Compatibility: In *most* cases, a TurboVNC-enabled viewer is fully compatible with a TightVNC server, and vice versa. TurboVNC supports pseudo-encodings for specifying a fine-grained (1-100) quality scale and specifying chrominance subsampling. If a TurboVNC viewer sends those to a TightVNC server, then the TightVNC server ignores them, so the TurboVNC viewer also sends the quality on a 0-9 scale that the TightVNC server can understand. Similarly, the TurboVNC server checks first for fine-grained quality and subsampling pseudo-encodings from the viewer, and failing to receive those, it then checks for the TightVNC 0-9 quality pseudo-encoding. There is one case in which the two systems are not compatible, and that is when a TightVNC or TigerVNC viewer requests compression level 0 without JPEG from a TurboVNC server. For performance reasons, this causes the TurboVNC server to send images directly to the viewer, bypassing Zlib. When the TurboVNC server does this, it also sets bits 7-4 in the compression control byte to rfbTightNoZlib (0x0A), which is unfortunately the same value as rfbTightPng. Older TightVNC viewers that don't handle PNG will assume that the stream is uncompressed but still encapsulated in a Zlib structure, whereas newer PNG-supporting TightVNC viewers will assume that the stream is PNG. In either case, the viewer will probably crash. Since most VNC viewers don't expose compression level 0 in the GUI, this is a relatively rare situation. Description of changes: configure.ac -- Added support for libjpeg-turbo. If passed an argument of --with-turbovnc, configure will now run (or, if cross-compiling, just link) a test program that determines whether the libjpeg library being used is libjpeg-turbo. libjpeg-turbo must be used when building the TurboVNC encoder, because the TurboVNC encoder relies on the libjpeg-turbo colorspace extensions in order to compress images directly out of the framebuffer (which may be, for instance, BGRA rather than RGB.) libjpeg-turbo can optionally be used with the TightVNC encoder as well, but the speedup will only be marginal (the report linked above explains why in more detail, but basically it's because of Amdahl's Law. The TightVNC encoder was designed with the assumption that JPEG had a very high CPU cost, and thus JPEG is used only sparingly.) -- Added a new configure variable, JPEG_LDFLAGS. This is necessitated by the fact that libjpeg-turbo often distributes libjpeg.a and libjpeg.so in /opt/libjpeg-turbo/lib32 or /opt/libjpeg-turbo/lib64, and many people prefer to statically link with it. Thus, more flexibility is needed than is provided by --with-jpeg. If JPEG_LDFLAGS is specified, then it overrides the changes to LDFLAGS enacted by --with-jpeg (but --with-jpeg is still used to set the include path.) The addition of JPEG_LDFLAGS necessitated replacing AC_CHECK_LIB with AC_LINK_IFELSE (because AC_CHECK_LIB automatically sets LIBS to -ljpeg, which is not what we want if we're, for instance, linking statically with libjpeg-turbo.) -- configure does not check for PNG support if TurboVNC encoding is enabled. This prevents the rfbSendRectEncodingTightPng() function from being compiled in, since the TurboVNC encoder doesn't (and can't) support it. common/turbojpeg.c, common/turbojpeg.h -- TurboJPEG is a simple API used to compress and decompress JPEG images in memory. It was originally implemented because it was desirable to use different types of underlying technologies to compress JPEG on different platforms (mediaLib on SPARC, Quicktime on PPC Macs, Intel Performance Primitives, etc.) These days, however, libjpeg-turbo is the only underlying technology used by TurboVNC, so TurboJPEG's purpose is largely just code simplicity and flexibility. Thus, since there is no real need for libvncserver to use any technology other than libjpeg-turbo for compressing JPEG, the TurboJPEG wrapper for libjpeg-turbo has been included in-tree so that libvncserver can be directly linked with libjpeg-turbo. This is convenient because many modern Linux distros (Fedora, Ubuntu, etc.) now ship libjpeg-turbo as their default libjpeg library. libvncserver/rfbserver.c -- Added logic to check for the TurboVNC fine-grained quality level and subsampling encodings and to map Tight (0-9) quality levels to appropriate fine-grained quality level and subsampling values if communicating with a TightVNC/TigerVNC viewer. libvncserver/turbo.c -- TurboVNC encoder (compiled instead of libvncserver/tight.c) rfb/rfb.h -- Added support for the TurboVNC subsampling level rfb/rfbproto.h -- Added constants for the TurboVNC fine quality level and subsampling encodings as well as the rfbTightNoZlib constant and notes on its usage.
13 years ago
/*
* This allows for rare corner cases in which a JPEG image can actually be
* larger than the uncompressed input (we wouldn't mention it if it hadn't
* happened before.)
*/
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
retval=PAD(width, 16) * PAD(height, 16) * 6 + 2048;
Add TurboVNC encoding support. TurboVNC is a variant of TightVNC that uses the same client/server protocol (RFB version 3.8t), and thus it is fully cross-compatible with TightVNC and TigerVNC (with one exception, which is noted below.) Both the TightVNC and TurboVNC encoders analyze each rectangle, pick out regions of solid color to send separately, and send the remaining subrectangles using mono, indexed color, JPEG, or raw encoding, depending on the number of colors in the subrectangle. However, TurboVNC uses a fundamentally different selection algorithm to determine the appropriate subencoding to use for each subrectangle. Thus, while it sends a protocol stream that can be decoded by any TightVNC-compatible viewer, the mix of subencoding types in this protocol stream will be different from those generated by a TightVNC server. The research that led to TurboVNC is described in the following report: http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf. In summary: 20 RFB captures, representing "common" 2D and 3D application workloads (the 3D workloads were run using VirtualGL), were studied using the TightVNC encoder in isolation. Some of the analysis features in the TightVNC encoder, such as smoothness detection, were found to generate a lot of CPU usage with little or no benefit in compression, so those features were disabled. JPEG encoding was accelerated using libjpeg-turbo (which achieves a 2-4x speedup over plain libjpeg on modern x86 or ARM processors.) Finally, the "palette threshold" (minimum number of colors that the subrectangle must have before it is compressed using JPEG or raw) was adjusted to account for the fact that JPEG encoding is now quite a bit faster (meaning that we can now use it more without a CPU penalty.) TurboVNC has additional optimizations, such as the ability to count colors and encode JPEG images directly from the framebuffer without first translating the pixels into RGB. The TurboVNC encoder compares quite favorably in terms of compression ratio with TightVNC and generally encodes a great deal faster (often an order of magnitude or more.) The version of the TurboVNC encoder included in this patch is roughly equivalent to the one found in version 0.6 of the Unix TurboVNC Server, with a few minor patches integrated from TurboVNC 1.1. TurboVNC 1.0 added multi-threading capabilities, which can be added in later if desired (at the expense of making libvncserver depend on libpthread.) Because TurboVNC uses a fundamentally different mix of subencodings than TightVNC, because it uses the identical protocol (and thus a viewer really has no idea whether it's talking to a TightVNC or TurboVNC server), and because it doesn't support rfbTightPng (and in fact conflicts with it-- see below), the TurboVNC and TightVNC encoders cannot be enabled simultaneously. Compatibility: In *most* cases, a TurboVNC-enabled viewer is fully compatible with a TightVNC server, and vice versa. TurboVNC supports pseudo-encodings for specifying a fine-grained (1-100) quality scale and specifying chrominance subsampling. If a TurboVNC viewer sends those to a TightVNC server, then the TightVNC server ignores them, so the TurboVNC viewer also sends the quality on a 0-9 scale that the TightVNC server can understand. Similarly, the TurboVNC server checks first for fine-grained quality and subsampling pseudo-encodings from the viewer, and failing to receive those, it then checks for the TightVNC 0-9 quality pseudo-encoding. There is one case in which the two systems are not compatible, and that is when a TightVNC or TigerVNC viewer requests compression level 0 without JPEG from a TurboVNC server. For performance reasons, this causes the TurboVNC server to send images directly to the viewer, bypassing Zlib. When the TurboVNC server does this, it also sets bits 7-4 in the compression control byte to rfbTightNoZlib (0x0A), which is unfortunately the same value as rfbTightPng. Older TightVNC viewers that don't handle PNG will assume that the stream is uncompressed but still encapsulated in a Zlib structure, whereas newer PNG-supporting TightVNC viewers will assume that the stream is PNG. In either case, the viewer will probably crash. Since most VNC viewers don't expose compression level 0 in the GUI, this is a relatively rare situation. Description of changes: configure.ac -- Added support for libjpeg-turbo. If passed an argument of --with-turbovnc, configure will now run (or, if cross-compiling, just link) a test program that determines whether the libjpeg library being used is libjpeg-turbo. libjpeg-turbo must be used when building the TurboVNC encoder, because the TurboVNC encoder relies on the libjpeg-turbo colorspace extensions in order to compress images directly out of the framebuffer (which may be, for instance, BGRA rather than RGB.) libjpeg-turbo can optionally be used with the TightVNC encoder as well, but the speedup will only be marginal (the report linked above explains why in more detail, but basically it's because of Amdahl's Law. The TightVNC encoder was designed with the assumption that JPEG had a very high CPU cost, and thus JPEG is used only sparingly.) -- Added a new configure variable, JPEG_LDFLAGS. This is necessitated by the fact that libjpeg-turbo often distributes libjpeg.a and libjpeg.so in /opt/libjpeg-turbo/lib32 or /opt/libjpeg-turbo/lib64, and many people prefer to statically link with it. Thus, more flexibility is needed than is provided by --with-jpeg. If JPEG_LDFLAGS is specified, then it overrides the changes to LDFLAGS enacted by --with-jpeg (but --with-jpeg is still used to set the include path.) The addition of JPEG_LDFLAGS necessitated replacing AC_CHECK_LIB with AC_LINK_IFELSE (because AC_CHECK_LIB automatically sets LIBS to -ljpeg, which is not what we want if we're, for instance, linking statically with libjpeg-turbo.) -- configure does not check for PNG support if TurboVNC encoding is enabled. This prevents the rfbSendRectEncodingTightPng() function from being compiled in, since the TurboVNC encoder doesn't (and can't) support it. common/turbojpeg.c, common/turbojpeg.h -- TurboJPEG is a simple API used to compress and decompress JPEG images in memory. It was originally implemented because it was desirable to use different types of underlying technologies to compress JPEG on different platforms (mediaLib on SPARC, Quicktime on PPC Macs, Intel Performance Primitives, etc.) These days, however, libjpeg-turbo is the only underlying technology used by TurboVNC, so TurboJPEG's purpose is largely just code simplicity and flexibility. Thus, since there is no real need for libvncserver to use any technology other than libjpeg-turbo for compressing JPEG, the TurboJPEG wrapper for libjpeg-turbo has been included in-tree so that libvncserver can be directly linked with libjpeg-turbo. This is convenient because many modern Linux distros (Fedora, Ubuntu, etc.) now ship libjpeg-turbo as their default libjpeg library. libvncserver/rfbserver.c -- Added logic to check for the TurboVNC fine-grained quality level and subsampling encodings and to map Tight (0-9) quality levels to appropriate fine-grained quality level and subsampling values if communicating with a TightVNC/TigerVNC viewer. libvncserver/turbo.c -- TurboVNC encoder (compiled instead of libvncserver/tight.c) rfb/rfb.h -- Added support for the TurboVNC subsampling level rfb/rfbproto.h -- Added constants for the TurboVNC fine quality level and subsampling encodings as well as the rfbTightNoZlib constant and notes on its usage.
13 years ago
bailout:
return retval;
}
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
DLLEXPORT int DLLCALL tjCompress2(tjhandle handle, unsigned char *srcBuf,
int width, int pitch, int height, int pixelFormat, unsigned char **jpegBuf,
unsigned long *jpegSize, int jpegSubsamp, int jpegQual, int flags)
Add TurboVNC encoding support. TurboVNC is a variant of TightVNC that uses the same client/server protocol (RFB version 3.8t), and thus it is fully cross-compatible with TightVNC and TigerVNC (with one exception, which is noted below.) Both the TightVNC and TurboVNC encoders analyze each rectangle, pick out regions of solid color to send separately, and send the remaining subrectangles using mono, indexed color, JPEG, or raw encoding, depending on the number of colors in the subrectangle. However, TurboVNC uses a fundamentally different selection algorithm to determine the appropriate subencoding to use for each subrectangle. Thus, while it sends a protocol stream that can be decoded by any TightVNC-compatible viewer, the mix of subencoding types in this protocol stream will be different from those generated by a TightVNC server. The research that led to TurboVNC is described in the following report: http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf. In summary: 20 RFB captures, representing "common" 2D and 3D application workloads (the 3D workloads were run using VirtualGL), were studied using the TightVNC encoder in isolation. Some of the analysis features in the TightVNC encoder, such as smoothness detection, were found to generate a lot of CPU usage with little or no benefit in compression, so those features were disabled. JPEG encoding was accelerated using libjpeg-turbo (which achieves a 2-4x speedup over plain libjpeg on modern x86 or ARM processors.) Finally, the "palette threshold" (minimum number of colors that the subrectangle must have before it is compressed using JPEG or raw) was adjusted to account for the fact that JPEG encoding is now quite a bit faster (meaning that we can now use it more without a CPU penalty.) TurboVNC has additional optimizations, such as the ability to count colors and encode JPEG images directly from the framebuffer without first translating the pixels into RGB. The TurboVNC encoder compares quite favorably in terms of compression ratio with TightVNC and generally encodes a great deal faster (often an order of magnitude or more.) The version of the TurboVNC encoder included in this patch is roughly equivalent to the one found in version 0.6 of the Unix TurboVNC Server, with a few minor patches integrated from TurboVNC 1.1. TurboVNC 1.0 added multi-threading capabilities, which can be added in later if desired (at the expense of making libvncserver depend on libpthread.) Because TurboVNC uses a fundamentally different mix of subencodings than TightVNC, because it uses the identical protocol (and thus a viewer really has no idea whether it's talking to a TightVNC or TurboVNC server), and because it doesn't support rfbTightPng (and in fact conflicts with it-- see below), the TurboVNC and TightVNC encoders cannot be enabled simultaneously. Compatibility: In *most* cases, a TurboVNC-enabled viewer is fully compatible with a TightVNC server, and vice versa. TurboVNC supports pseudo-encodings for specifying a fine-grained (1-100) quality scale and specifying chrominance subsampling. If a TurboVNC viewer sends those to a TightVNC server, then the TightVNC server ignores them, so the TurboVNC viewer also sends the quality on a 0-9 scale that the TightVNC server can understand. Similarly, the TurboVNC server checks first for fine-grained quality and subsampling pseudo-encodings from the viewer, and failing to receive those, it then checks for the TightVNC 0-9 quality pseudo-encoding. There is one case in which the two systems are not compatible, and that is when a TightVNC or TigerVNC viewer requests compression level 0 without JPEG from a TurboVNC server. For performance reasons, this causes the TurboVNC server to send images directly to the viewer, bypassing Zlib. When the TurboVNC server does this, it also sets bits 7-4 in the compression control byte to rfbTightNoZlib (0x0A), which is unfortunately the same value as rfbTightPng. Older TightVNC viewers that don't handle PNG will assume that the stream is uncompressed but still encapsulated in a Zlib structure, whereas newer PNG-supporting TightVNC viewers will assume that the stream is PNG. In either case, the viewer will probably crash. Since most VNC viewers don't expose compression level 0 in the GUI, this is a relatively rare situation. Description of changes: configure.ac -- Added support for libjpeg-turbo. If passed an argument of --with-turbovnc, configure will now run (or, if cross-compiling, just link) a test program that determines whether the libjpeg library being used is libjpeg-turbo. libjpeg-turbo must be used when building the TurboVNC encoder, because the TurboVNC encoder relies on the libjpeg-turbo colorspace extensions in order to compress images directly out of the framebuffer (which may be, for instance, BGRA rather than RGB.) libjpeg-turbo can optionally be used with the TightVNC encoder as well, but the speedup will only be marginal (the report linked above explains why in more detail, but basically it's because of Amdahl's Law. The TightVNC encoder was designed with the assumption that JPEG had a very high CPU cost, and thus JPEG is used only sparingly.) -- Added a new configure variable, JPEG_LDFLAGS. This is necessitated by the fact that libjpeg-turbo often distributes libjpeg.a and libjpeg.so in /opt/libjpeg-turbo/lib32 or /opt/libjpeg-turbo/lib64, and many people prefer to statically link with it. Thus, more flexibility is needed than is provided by --with-jpeg. If JPEG_LDFLAGS is specified, then it overrides the changes to LDFLAGS enacted by --with-jpeg (but --with-jpeg is still used to set the include path.) The addition of JPEG_LDFLAGS necessitated replacing AC_CHECK_LIB with AC_LINK_IFELSE (because AC_CHECK_LIB automatically sets LIBS to -ljpeg, which is not what we want if we're, for instance, linking statically with libjpeg-turbo.) -- configure does not check for PNG support if TurboVNC encoding is enabled. This prevents the rfbSendRectEncodingTightPng() function from being compiled in, since the TurboVNC encoder doesn't (and can't) support it. common/turbojpeg.c, common/turbojpeg.h -- TurboJPEG is a simple API used to compress and decompress JPEG images in memory. It was originally implemented because it was desirable to use different types of underlying technologies to compress JPEG on different platforms (mediaLib on SPARC, Quicktime on PPC Macs, Intel Performance Primitives, etc.) These days, however, libjpeg-turbo is the only underlying technology used by TurboVNC, so TurboJPEG's purpose is largely just code simplicity and flexibility. Thus, since there is no real need for libvncserver to use any technology other than libjpeg-turbo for compressing JPEG, the TurboJPEG wrapper for libjpeg-turbo has been included in-tree so that libvncserver can be directly linked with libjpeg-turbo. This is convenient because many modern Linux distros (Fedora, Ubuntu, etc.) now ship libjpeg-turbo as their default libjpeg library. libvncserver/rfbserver.c -- Added logic to check for the TurboVNC fine-grained quality level and subsampling encodings and to map Tight (0-9) quality levels to appropriate fine-grained quality level and subsampling values if communicating with a TightVNC/TigerVNC viewer. libvncserver/turbo.c -- TurboVNC encoder (compiled instead of libvncserver/tight.c) rfb/rfb.h -- Added support for the TurboVNC subsampling level rfb/rfbproto.h -- Added constants for the TurboVNC fine quality level and subsampling encodings as well as the rfbTightNoZlib constant and notes on its usage.
13 years ago
{
int i, retval=0; JSAMPROW *row_pointer=NULL;
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
#ifndef JCS_EXTENSIONS
unsigned char *rgbBuf=NULL;
Add TurboVNC encoding support. TurboVNC is a variant of TightVNC that uses the same client/server protocol (RFB version 3.8t), and thus it is fully cross-compatible with TightVNC and TigerVNC (with one exception, which is noted below.) Both the TightVNC and TurboVNC encoders analyze each rectangle, pick out regions of solid color to send separately, and send the remaining subrectangles using mono, indexed color, JPEG, or raw encoding, depending on the number of colors in the subrectangle. However, TurboVNC uses a fundamentally different selection algorithm to determine the appropriate subencoding to use for each subrectangle. Thus, while it sends a protocol stream that can be decoded by any TightVNC-compatible viewer, the mix of subencoding types in this protocol stream will be different from those generated by a TightVNC server. The research that led to TurboVNC is described in the following report: http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf. In summary: 20 RFB captures, representing "common" 2D and 3D application workloads (the 3D workloads were run using VirtualGL), were studied using the TightVNC encoder in isolation. Some of the analysis features in the TightVNC encoder, such as smoothness detection, were found to generate a lot of CPU usage with little or no benefit in compression, so those features were disabled. JPEG encoding was accelerated using libjpeg-turbo (which achieves a 2-4x speedup over plain libjpeg on modern x86 or ARM processors.) Finally, the "palette threshold" (minimum number of colors that the subrectangle must have before it is compressed using JPEG or raw) was adjusted to account for the fact that JPEG encoding is now quite a bit faster (meaning that we can now use it more without a CPU penalty.) TurboVNC has additional optimizations, such as the ability to count colors and encode JPEG images directly from the framebuffer without first translating the pixels into RGB. The TurboVNC encoder compares quite favorably in terms of compression ratio with TightVNC and generally encodes a great deal faster (often an order of magnitude or more.) The version of the TurboVNC encoder included in this patch is roughly equivalent to the one found in version 0.6 of the Unix TurboVNC Server, with a few minor patches integrated from TurboVNC 1.1. TurboVNC 1.0 added multi-threading capabilities, which can be added in later if desired (at the expense of making libvncserver depend on libpthread.) Because TurboVNC uses a fundamentally different mix of subencodings than TightVNC, because it uses the identical protocol (and thus a viewer really has no idea whether it's talking to a TightVNC or TurboVNC server), and because it doesn't support rfbTightPng (and in fact conflicts with it-- see below), the TurboVNC and TightVNC encoders cannot be enabled simultaneously. Compatibility: In *most* cases, a TurboVNC-enabled viewer is fully compatible with a TightVNC server, and vice versa. TurboVNC supports pseudo-encodings for specifying a fine-grained (1-100) quality scale and specifying chrominance subsampling. If a TurboVNC viewer sends those to a TightVNC server, then the TightVNC server ignores them, so the TurboVNC viewer also sends the quality on a 0-9 scale that the TightVNC server can understand. Similarly, the TurboVNC server checks first for fine-grained quality and subsampling pseudo-encodings from the viewer, and failing to receive those, it then checks for the TightVNC 0-9 quality pseudo-encoding. There is one case in which the two systems are not compatible, and that is when a TightVNC or TigerVNC viewer requests compression level 0 without JPEG from a TurboVNC server. For performance reasons, this causes the TurboVNC server to send images directly to the viewer, bypassing Zlib. When the TurboVNC server does this, it also sets bits 7-4 in the compression control byte to rfbTightNoZlib (0x0A), which is unfortunately the same value as rfbTightPng. Older TightVNC viewers that don't handle PNG will assume that the stream is uncompressed but still encapsulated in a Zlib structure, whereas newer PNG-supporting TightVNC viewers will assume that the stream is PNG. In either case, the viewer will probably crash. Since most VNC viewers don't expose compression level 0 in the GUI, this is a relatively rare situation. Description of changes: configure.ac -- Added support for libjpeg-turbo. If passed an argument of --with-turbovnc, configure will now run (or, if cross-compiling, just link) a test program that determines whether the libjpeg library being used is libjpeg-turbo. libjpeg-turbo must be used when building the TurboVNC encoder, because the TurboVNC encoder relies on the libjpeg-turbo colorspace extensions in order to compress images directly out of the framebuffer (which may be, for instance, BGRA rather than RGB.) libjpeg-turbo can optionally be used with the TightVNC encoder as well, but the speedup will only be marginal (the report linked above explains why in more detail, but basically it's because of Amdahl's Law. The TightVNC encoder was designed with the assumption that JPEG had a very high CPU cost, and thus JPEG is used only sparingly.) -- Added a new configure variable, JPEG_LDFLAGS. This is necessitated by the fact that libjpeg-turbo often distributes libjpeg.a and libjpeg.so in /opt/libjpeg-turbo/lib32 or /opt/libjpeg-turbo/lib64, and many people prefer to statically link with it. Thus, more flexibility is needed than is provided by --with-jpeg. If JPEG_LDFLAGS is specified, then it overrides the changes to LDFLAGS enacted by --with-jpeg (but --with-jpeg is still used to set the include path.) The addition of JPEG_LDFLAGS necessitated replacing AC_CHECK_LIB with AC_LINK_IFELSE (because AC_CHECK_LIB automatically sets LIBS to -ljpeg, which is not what we want if we're, for instance, linking statically with libjpeg-turbo.) -- configure does not check for PNG support if TurboVNC encoding is enabled. This prevents the rfbSendRectEncodingTightPng() function from being compiled in, since the TurboVNC encoder doesn't (and can't) support it. common/turbojpeg.c, common/turbojpeg.h -- TurboJPEG is a simple API used to compress and decompress JPEG images in memory. It was originally implemented because it was desirable to use different types of underlying technologies to compress JPEG on different platforms (mediaLib on SPARC, Quicktime on PPC Macs, Intel Performance Primitives, etc.) These days, however, libjpeg-turbo is the only underlying technology used by TurboVNC, so TurboJPEG's purpose is largely just code simplicity and flexibility. Thus, since there is no real need for libvncserver to use any technology other than libjpeg-turbo for compressing JPEG, the TurboJPEG wrapper for libjpeg-turbo has been included in-tree so that libvncserver can be directly linked with libjpeg-turbo. This is convenient because many modern Linux distros (Fedora, Ubuntu, etc.) now ship libjpeg-turbo as their default libjpeg library. libvncserver/rfbserver.c -- Added logic to check for the TurboVNC fine-grained quality level and subsampling encodings and to map Tight (0-9) quality levels to appropriate fine-grained quality level and subsampling values if communicating with a TightVNC/TigerVNC viewer. libvncserver/turbo.c -- TurboVNC encoder (compiled instead of libvncserver/tight.c) rfb/rfb.h -- Added support for the TurboVNC subsampling level rfb/rfbproto.h -- Added constants for the TurboVNC fine quality level and subsampling encodings as well as the rfbTightNoZlib constant and notes on its usage.
13 years ago
#endif
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
getinstance(handle)
if((this->init&COMPRESS)==0)
_throw("tjCompress2(): Instance has not been initialized for compression");
Add TurboVNC encoding support. TurboVNC is a variant of TightVNC that uses the same client/server protocol (RFB version 3.8t), and thus it is fully cross-compatible with TightVNC and TigerVNC (with one exception, which is noted below.) Both the TightVNC and TurboVNC encoders analyze each rectangle, pick out regions of solid color to send separately, and send the remaining subrectangles using mono, indexed color, JPEG, or raw encoding, depending on the number of colors in the subrectangle. However, TurboVNC uses a fundamentally different selection algorithm to determine the appropriate subencoding to use for each subrectangle. Thus, while it sends a protocol stream that can be decoded by any TightVNC-compatible viewer, the mix of subencoding types in this protocol stream will be different from those generated by a TightVNC server. The research that led to TurboVNC is described in the following report: http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf. In summary: 20 RFB captures, representing "common" 2D and 3D application workloads (the 3D workloads were run using VirtualGL), were studied using the TightVNC encoder in isolation. Some of the analysis features in the TightVNC encoder, such as smoothness detection, were found to generate a lot of CPU usage with little or no benefit in compression, so those features were disabled. JPEG encoding was accelerated using libjpeg-turbo (which achieves a 2-4x speedup over plain libjpeg on modern x86 or ARM processors.) Finally, the "palette threshold" (minimum number of colors that the subrectangle must have before it is compressed using JPEG or raw) was adjusted to account for the fact that JPEG encoding is now quite a bit faster (meaning that we can now use it more without a CPU penalty.) TurboVNC has additional optimizations, such as the ability to count colors and encode JPEG images directly from the framebuffer without first translating the pixels into RGB. The TurboVNC encoder compares quite favorably in terms of compression ratio with TightVNC and generally encodes a great deal faster (often an order of magnitude or more.) The version of the TurboVNC encoder included in this patch is roughly equivalent to the one found in version 0.6 of the Unix TurboVNC Server, with a few minor patches integrated from TurboVNC 1.1. TurboVNC 1.0 added multi-threading capabilities, which can be added in later if desired (at the expense of making libvncserver depend on libpthread.) Because TurboVNC uses a fundamentally different mix of subencodings than TightVNC, because it uses the identical protocol (and thus a viewer really has no idea whether it's talking to a TightVNC or TurboVNC server), and because it doesn't support rfbTightPng (and in fact conflicts with it-- see below), the TurboVNC and TightVNC encoders cannot be enabled simultaneously. Compatibility: In *most* cases, a TurboVNC-enabled viewer is fully compatible with a TightVNC server, and vice versa. TurboVNC supports pseudo-encodings for specifying a fine-grained (1-100) quality scale and specifying chrominance subsampling. If a TurboVNC viewer sends those to a TightVNC server, then the TightVNC server ignores them, so the TurboVNC viewer also sends the quality on a 0-9 scale that the TightVNC server can understand. Similarly, the TurboVNC server checks first for fine-grained quality and subsampling pseudo-encodings from the viewer, and failing to receive those, it then checks for the TightVNC 0-9 quality pseudo-encoding. There is one case in which the two systems are not compatible, and that is when a TightVNC or TigerVNC viewer requests compression level 0 without JPEG from a TurboVNC server. For performance reasons, this causes the TurboVNC server to send images directly to the viewer, bypassing Zlib. When the TurboVNC server does this, it also sets bits 7-4 in the compression control byte to rfbTightNoZlib (0x0A), which is unfortunately the same value as rfbTightPng. Older TightVNC viewers that don't handle PNG will assume that the stream is uncompressed but still encapsulated in a Zlib structure, whereas newer PNG-supporting TightVNC viewers will assume that the stream is PNG. In either case, the viewer will probably crash. Since most VNC viewers don't expose compression level 0 in the GUI, this is a relatively rare situation. Description of changes: configure.ac -- Added support for libjpeg-turbo. If passed an argument of --with-turbovnc, configure will now run (or, if cross-compiling, just link) a test program that determines whether the libjpeg library being used is libjpeg-turbo. libjpeg-turbo must be used when building the TurboVNC encoder, because the TurboVNC encoder relies on the libjpeg-turbo colorspace extensions in order to compress images directly out of the framebuffer (which may be, for instance, BGRA rather than RGB.) libjpeg-turbo can optionally be used with the TightVNC encoder as well, but the speedup will only be marginal (the report linked above explains why in more detail, but basically it's because of Amdahl's Law. The TightVNC encoder was designed with the assumption that JPEG had a very high CPU cost, and thus JPEG is used only sparingly.) -- Added a new configure variable, JPEG_LDFLAGS. This is necessitated by the fact that libjpeg-turbo often distributes libjpeg.a and libjpeg.so in /opt/libjpeg-turbo/lib32 or /opt/libjpeg-turbo/lib64, and many people prefer to statically link with it. Thus, more flexibility is needed than is provided by --with-jpeg. If JPEG_LDFLAGS is specified, then it overrides the changes to LDFLAGS enacted by --with-jpeg (but --with-jpeg is still used to set the include path.) The addition of JPEG_LDFLAGS necessitated replacing AC_CHECK_LIB with AC_LINK_IFELSE (because AC_CHECK_LIB automatically sets LIBS to -ljpeg, which is not what we want if we're, for instance, linking statically with libjpeg-turbo.) -- configure does not check for PNG support if TurboVNC encoding is enabled. This prevents the rfbSendRectEncodingTightPng() function from being compiled in, since the TurboVNC encoder doesn't (and can't) support it. common/turbojpeg.c, common/turbojpeg.h -- TurboJPEG is a simple API used to compress and decompress JPEG images in memory. It was originally implemented because it was desirable to use different types of underlying technologies to compress JPEG on different platforms (mediaLib on SPARC, Quicktime on PPC Macs, Intel Performance Primitives, etc.) These days, however, libjpeg-turbo is the only underlying technology used by TurboVNC, so TurboJPEG's purpose is largely just code simplicity and flexibility. Thus, since there is no real need for libvncserver to use any technology other than libjpeg-turbo for compressing JPEG, the TurboJPEG wrapper for libjpeg-turbo has been included in-tree so that libvncserver can be directly linked with libjpeg-turbo. This is convenient because many modern Linux distros (Fedora, Ubuntu, etc.) now ship libjpeg-turbo as their default libjpeg library. libvncserver/rfbserver.c -- Added logic to check for the TurboVNC fine-grained quality level and subsampling encodings and to map Tight (0-9) quality levels to appropriate fine-grained quality level and subsampling values if communicating with a TightVNC/TigerVNC viewer. libvncserver/turbo.c -- TurboVNC encoder (compiled instead of libvncserver/tight.c) rfb/rfb.h -- Added support for the TurboVNC subsampling level rfb/rfbproto.h -- Added constants for the TurboVNC fine quality level and subsampling encodings as well as the rfbTightNoZlib constant and notes on its usage.
13 years ago
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
if(srcBuf==NULL || width<=0 || pitch<0 || height<=0 || pixelFormat<0
|| pixelFormat>=TJ_NUMPF || jpegBuf==NULL || jpegSize==NULL
|| jpegSubsamp<0 || jpegSubsamp>=NUMSUBOPT || jpegQual<0 || jpegQual>100)
_throw("tjCompress2(): Invalid argument");
if(setjmp(this->jerr.setjmp_buffer))
{
/* If we get here, the JPEG code has signaled an error. */
Add TurboVNC encoding support. TurboVNC is a variant of TightVNC that uses the same client/server protocol (RFB version 3.8t), and thus it is fully cross-compatible with TightVNC and TigerVNC (with one exception, which is noted below.) Both the TightVNC and TurboVNC encoders analyze each rectangle, pick out regions of solid color to send separately, and send the remaining subrectangles using mono, indexed color, JPEG, or raw encoding, depending on the number of colors in the subrectangle. However, TurboVNC uses a fundamentally different selection algorithm to determine the appropriate subencoding to use for each subrectangle. Thus, while it sends a protocol stream that can be decoded by any TightVNC-compatible viewer, the mix of subencoding types in this protocol stream will be different from those generated by a TightVNC server. The research that led to TurboVNC is described in the following report: http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf. In summary: 20 RFB captures, representing "common" 2D and 3D application workloads (the 3D workloads were run using VirtualGL), were studied using the TightVNC encoder in isolation. Some of the analysis features in the TightVNC encoder, such as smoothness detection, were found to generate a lot of CPU usage with little or no benefit in compression, so those features were disabled. JPEG encoding was accelerated using libjpeg-turbo (which achieves a 2-4x speedup over plain libjpeg on modern x86 or ARM processors.) Finally, the "palette threshold" (minimum number of colors that the subrectangle must have before it is compressed using JPEG or raw) was adjusted to account for the fact that JPEG encoding is now quite a bit faster (meaning that we can now use it more without a CPU penalty.) TurboVNC has additional optimizations, such as the ability to count colors and encode JPEG images directly from the framebuffer without first translating the pixels into RGB. The TurboVNC encoder compares quite favorably in terms of compression ratio with TightVNC and generally encodes a great deal faster (often an order of magnitude or more.) The version of the TurboVNC encoder included in this patch is roughly equivalent to the one found in version 0.6 of the Unix TurboVNC Server, with a few minor patches integrated from TurboVNC 1.1. TurboVNC 1.0 added multi-threading capabilities, which can be added in later if desired (at the expense of making libvncserver depend on libpthread.) Because TurboVNC uses a fundamentally different mix of subencodings than TightVNC, because it uses the identical protocol (and thus a viewer really has no idea whether it's talking to a TightVNC or TurboVNC server), and because it doesn't support rfbTightPng (and in fact conflicts with it-- see below), the TurboVNC and TightVNC encoders cannot be enabled simultaneously. Compatibility: In *most* cases, a TurboVNC-enabled viewer is fully compatible with a TightVNC server, and vice versa. TurboVNC supports pseudo-encodings for specifying a fine-grained (1-100) quality scale and specifying chrominance subsampling. If a TurboVNC viewer sends those to a TightVNC server, then the TightVNC server ignores them, so the TurboVNC viewer also sends the quality on a 0-9 scale that the TightVNC server can understand. Similarly, the TurboVNC server checks first for fine-grained quality and subsampling pseudo-encodings from the viewer, and failing to receive those, it then checks for the TightVNC 0-9 quality pseudo-encoding. There is one case in which the two systems are not compatible, and that is when a TightVNC or TigerVNC viewer requests compression level 0 without JPEG from a TurboVNC server. For performance reasons, this causes the TurboVNC server to send images directly to the viewer, bypassing Zlib. When the TurboVNC server does this, it also sets bits 7-4 in the compression control byte to rfbTightNoZlib (0x0A), which is unfortunately the same value as rfbTightPng. Older TightVNC viewers that don't handle PNG will assume that the stream is uncompressed but still encapsulated in a Zlib structure, whereas newer PNG-supporting TightVNC viewers will assume that the stream is PNG. In either case, the viewer will probably crash. Since most VNC viewers don't expose compression level 0 in the GUI, this is a relatively rare situation. Description of changes: configure.ac -- Added support for libjpeg-turbo. If passed an argument of --with-turbovnc, configure will now run (or, if cross-compiling, just link) a test program that determines whether the libjpeg library being used is libjpeg-turbo. libjpeg-turbo must be used when building the TurboVNC encoder, because the TurboVNC encoder relies on the libjpeg-turbo colorspace extensions in order to compress images directly out of the framebuffer (which may be, for instance, BGRA rather than RGB.) libjpeg-turbo can optionally be used with the TightVNC encoder as well, but the speedup will only be marginal (the report linked above explains why in more detail, but basically it's because of Amdahl's Law. The TightVNC encoder was designed with the assumption that JPEG had a very high CPU cost, and thus JPEG is used only sparingly.) -- Added a new configure variable, JPEG_LDFLAGS. This is necessitated by the fact that libjpeg-turbo often distributes libjpeg.a and libjpeg.so in /opt/libjpeg-turbo/lib32 or /opt/libjpeg-turbo/lib64, and many people prefer to statically link with it. Thus, more flexibility is needed than is provided by --with-jpeg. If JPEG_LDFLAGS is specified, then it overrides the changes to LDFLAGS enacted by --with-jpeg (but --with-jpeg is still used to set the include path.) The addition of JPEG_LDFLAGS necessitated replacing AC_CHECK_LIB with AC_LINK_IFELSE (because AC_CHECK_LIB automatically sets LIBS to -ljpeg, which is not what we want if we're, for instance, linking statically with libjpeg-turbo.) -- configure does not check for PNG support if TurboVNC encoding is enabled. This prevents the rfbSendRectEncodingTightPng() function from being compiled in, since the TurboVNC encoder doesn't (and can't) support it. common/turbojpeg.c, common/turbojpeg.h -- TurboJPEG is a simple API used to compress and decompress JPEG images in memory. It was originally implemented because it was desirable to use different types of underlying technologies to compress JPEG on different platforms (mediaLib on SPARC, Quicktime on PPC Macs, Intel Performance Primitives, etc.) These days, however, libjpeg-turbo is the only underlying technology used by TurboVNC, so TurboJPEG's purpose is largely just code simplicity and flexibility. Thus, since there is no real need for libvncserver to use any technology other than libjpeg-turbo for compressing JPEG, the TurboJPEG wrapper for libjpeg-turbo has been included in-tree so that libvncserver can be directly linked with libjpeg-turbo. This is convenient because many modern Linux distros (Fedora, Ubuntu, etc.) now ship libjpeg-turbo as their default libjpeg library. libvncserver/rfbserver.c -- Added logic to check for the TurboVNC fine-grained quality level and subsampling encodings and to map Tight (0-9) quality levels to appropriate fine-grained quality level and subsampling values if communicating with a TightVNC/TigerVNC viewer. libvncserver/turbo.c -- TurboVNC encoder (compiled instead of libvncserver/tight.c) rfb/rfb.h -- Added support for the TurboVNC subsampling level rfb/rfbproto.h -- Added constants for the TurboVNC fine quality level and subsampling encodings as well as the rfbTightNoZlib constant and notes on its usage.
13 years ago
retval=-1;
goto bailout;
}
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
if(pitch==0) pitch=width*tjPixelSize[pixelFormat];
Add TurboVNC encoding support. TurboVNC is a variant of TightVNC that uses the same client/server protocol (RFB version 3.8t), and thus it is fully cross-compatible with TightVNC and TigerVNC (with one exception, which is noted below.) Both the TightVNC and TurboVNC encoders analyze each rectangle, pick out regions of solid color to send separately, and send the remaining subrectangles using mono, indexed color, JPEG, or raw encoding, depending on the number of colors in the subrectangle. However, TurboVNC uses a fundamentally different selection algorithm to determine the appropriate subencoding to use for each subrectangle. Thus, while it sends a protocol stream that can be decoded by any TightVNC-compatible viewer, the mix of subencoding types in this protocol stream will be different from those generated by a TightVNC server. The research that led to TurboVNC is described in the following report: http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf. In summary: 20 RFB captures, representing "common" 2D and 3D application workloads (the 3D workloads were run using VirtualGL), were studied using the TightVNC encoder in isolation. Some of the analysis features in the TightVNC encoder, such as smoothness detection, were found to generate a lot of CPU usage with little or no benefit in compression, so those features were disabled. JPEG encoding was accelerated using libjpeg-turbo (which achieves a 2-4x speedup over plain libjpeg on modern x86 or ARM processors.) Finally, the "palette threshold" (minimum number of colors that the subrectangle must have before it is compressed using JPEG or raw) was adjusted to account for the fact that JPEG encoding is now quite a bit faster (meaning that we can now use it more without a CPU penalty.) TurboVNC has additional optimizations, such as the ability to count colors and encode JPEG images directly from the framebuffer without first translating the pixels into RGB. The TurboVNC encoder compares quite favorably in terms of compression ratio with TightVNC and generally encodes a great deal faster (often an order of magnitude or more.) The version of the TurboVNC encoder included in this patch is roughly equivalent to the one found in version 0.6 of the Unix TurboVNC Server, with a few minor patches integrated from TurboVNC 1.1. TurboVNC 1.0 added multi-threading capabilities, which can be added in later if desired (at the expense of making libvncserver depend on libpthread.) Because TurboVNC uses a fundamentally different mix of subencodings than TightVNC, because it uses the identical protocol (and thus a viewer really has no idea whether it's talking to a TightVNC or TurboVNC server), and because it doesn't support rfbTightPng (and in fact conflicts with it-- see below), the TurboVNC and TightVNC encoders cannot be enabled simultaneously. Compatibility: In *most* cases, a TurboVNC-enabled viewer is fully compatible with a TightVNC server, and vice versa. TurboVNC supports pseudo-encodings for specifying a fine-grained (1-100) quality scale and specifying chrominance subsampling. If a TurboVNC viewer sends those to a TightVNC server, then the TightVNC server ignores them, so the TurboVNC viewer also sends the quality on a 0-9 scale that the TightVNC server can understand. Similarly, the TurboVNC server checks first for fine-grained quality and subsampling pseudo-encodings from the viewer, and failing to receive those, it then checks for the TightVNC 0-9 quality pseudo-encoding. There is one case in which the two systems are not compatible, and that is when a TightVNC or TigerVNC viewer requests compression level 0 without JPEG from a TurboVNC server. For performance reasons, this causes the TurboVNC server to send images directly to the viewer, bypassing Zlib. When the TurboVNC server does this, it also sets bits 7-4 in the compression control byte to rfbTightNoZlib (0x0A), which is unfortunately the same value as rfbTightPng. Older TightVNC viewers that don't handle PNG will assume that the stream is uncompressed but still encapsulated in a Zlib structure, whereas newer PNG-supporting TightVNC viewers will assume that the stream is PNG. In either case, the viewer will probably crash. Since most VNC viewers don't expose compression level 0 in the GUI, this is a relatively rare situation. Description of changes: configure.ac -- Added support for libjpeg-turbo. If passed an argument of --with-turbovnc, configure will now run (or, if cross-compiling, just link) a test program that determines whether the libjpeg library being used is libjpeg-turbo. libjpeg-turbo must be used when building the TurboVNC encoder, because the TurboVNC encoder relies on the libjpeg-turbo colorspace extensions in order to compress images directly out of the framebuffer (which may be, for instance, BGRA rather than RGB.) libjpeg-turbo can optionally be used with the TightVNC encoder as well, but the speedup will only be marginal (the report linked above explains why in more detail, but basically it's because of Amdahl's Law. The TightVNC encoder was designed with the assumption that JPEG had a very high CPU cost, and thus JPEG is used only sparingly.) -- Added a new configure variable, JPEG_LDFLAGS. This is necessitated by the fact that libjpeg-turbo often distributes libjpeg.a and libjpeg.so in /opt/libjpeg-turbo/lib32 or /opt/libjpeg-turbo/lib64, and many people prefer to statically link with it. Thus, more flexibility is needed than is provided by --with-jpeg. If JPEG_LDFLAGS is specified, then it overrides the changes to LDFLAGS enacted by --with-jpeg (but --with-jpeg is still used to set the include path.) The addition of JPEG_LDFLAGS necessitated replacing AC_CHECK_LIB with AC_LINK_IFELSE (because AC_CHECK_LIB automatically sets LIBS to -ljpeg, which is not what we want if we're, for instance, linking statically with libjpeg-turbo.) -- configure does not check for PNG support if TurboVNC encoding is enabled. This prevents the rfbSendRectEncodingTightPng() function from being compiled in, since the TurboVNC encoder doesn't (and can't) support it. common/turbojpeg.c, common/turbojpeg.h -- TurboJPEG is a simple API used to compress and decompress JPEG images in memory. It was originally implemented because it was desirable to use different types of underlying technologies to compress JPEG on different platforms (mediaLib on SPARC, Quicktime on PPC Macs, Intel Performance Primitives, etc.) These days, however, libjpeg-turbo is the only underlying technology used by TurboVNC, so TurboJPEG's purpose is largely just code simplicity and flexibility. Thus, since there is no real need for libvncserver to use any technology other than libjpeg-turbo for compressing JPEG, the TurboJPEG wrapper for libjpeg-turbo has been included in-tree so that libvncserver can be directly linked with libjpeg-turbo. This is convenient because many modern Linux distros (Fedora, Ubuntu, etc.) now ship libjpeg-turbo as their default libjpeg library. libvncserver/rfbserver.c -- Added logic to check for the TurboVNC fine-grained quality level and subsampling encodings and to map Tight (0-9) quality levels to appropriate fine-grained quality level and subsampling values if communicating with a TightVNC/TigerVNC viewer. libvncserver/turbo.c -- TurboVNC encoder (compiled instead of libvncserver/tight.c) rfb/rfb.h -- Added support for the TurboVNC subsampling level rfb/rfbproto.h -- Added constants for the TurboVNC fine quality level and subsampling encodings as well as the rfbTightNoZlib constant and notes on its usage.
13 years ago
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
#ifndef JCS_EXTENSIONS
if(pixelFormat!=TJPF_GRAY)
{
rgbBuf=(unsigned char *)malloc(width*height*RGB_PIXELSIZE);
if(!rgbBuf) _throw("tjCompress2(): Memory allocation failure");
srcBuf=toRGB(srcBuf, width, pitch, height, pixelFormat, rgbBuf);
pitch=width*RGB_PIXELSIZE;
}
#endif
cinfo->image_width=width;
cinfo->image_height=height;
if(flags&TJFLAG_FORCEMMX) putenv("JSIMD_FORCEMMX=1");
else if(flags&TJFLAG_FORCESSE) putenv("JSIMD_FORCESSE=1");
else if(flags&TJFLAG_FORCESSE2) putenv("JSIMD_FORCESSE2=1");
Add TurboVNC encoding support. TurboVNC is a variant of TightVNC that uses the same client/server protocol (RFB version 3.8t), and thus it is fully cross-compatible with TightVNC and TigerVNC (with one exception, which is noted below.) Both the TightVNC and TurboVNC encoders analyze each rectangle, pick out regions of solid color to send separately, and send the remaining subrectangles using mono, indexed color, JPEG, or raw encoding, depending on the number of colors in the subrectangle. However, TurboVNC uses a fundamentally different selection algorithm to determine the appropriate subencoding to use for each subrectangle. Thus, while it sends a protocol stream that can be decoded by any TightVNC-compatible viewer, the mix of subencoding types in this protocol stream will be different from those generated by a TightVNC server. The research that led to TurboVNC is described in the following report: http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf. In summary: 20 RFB captures, representing "common" 2D and 3D application workloads (the 3D workloads were run using VirtualGL), were studied using the TightVNC encoder in isolation. Some of the analysis features in the TightVNC encoder, such as smoothness detection, were found to generate a lot of CPU usage with little or no benefit in compression, so those features were disabled. JPEG encoding was accelerated using libjpeg-turbo (which achieves a 2-4x speedup over plain libjpeg on modern x86 or ARM processors.) Finally, the "palette threshold" (minimum number of colors that the subrectangle must have before it is compressed using JPEG or raw) was adjusted to account for the fact that JPEG encoding is now quite a bit faster (meaning that we can now use it more without a CPU penalty.) TurboVNC has additional optimizations, such as the ability to count colors and encode JPEG images directly from the framebuffer without first translating the pixels into RGB. The TurboVNC encoder compares quite favorably in terms of compression ratio with TightVNC and generally encodes a great deal faster (often an order of magnitude or more.) The version of the TurboVNC encoder included in this patch is roughly equivalent to the one found in version 0.6 of the Unix TurboVNC Server, with a few minor patches integrated from TurboVNC 1.1. TurboVNC 1.0 added multi-threading capabilities, which can be added in later if desired (at the expense of making libvncserver depend on libpthread.) Because TurboVNC uses a fundamentally different mix of subencodings than TightVNC, because it uses the identical protocol (and thus a viewer really has no idea whether it's talking to a TightVNC or TurboVNC server), and because it doesn't support rfbTightPng (and in fact conflicts with it-- see below), the TurboVNC and TightVNC encoders cannot be enabled simultaneously. Compatibility: In *most* cases, a TurboVNC-enabled viewer is fully compatible with a TightVNC server, and vice versa. TurboVNC supports pseudo-encodings for specifying a fine-grained (1-100) quality scale and specifying chrominance subsampling. If a TurboVNC viewer sends those to a TightVNC server, then the TightVNC server ignores them, so the TurboVNC viewer also sends the quality on a 0-9 scale that the TightVNC server can understand. Similarly, the TurboVNC server checks first for fine-grained quality and subsampling pseudo-encodings from the viewer, and failing to receive those, it then checks for the TightVNC 0-9 quality pseudo-encoding. There is one case in which the two systems are not compatible, and that is when a TightVNC or TigerVNC viewer requests compression level 0 without JPEG from a TurboVNC server. For performance reasons, this causes the TurboVNC server to send images directly to the viewer, bypassing Zlib. When the TurboVNC server does this, it also sets bits 7-4 in the compression control byte to rfbTightNoZlib (0x0A), which is unfortunately the same value as rfbTightPng. Older TightVNC viewers that don't handle PNG will assume that the stream is uncompressed but still encapsulated in a Zlib structure, whereas newer PNG-supporting TightVNC viewers will assume that the stream is PNG. In either case, the viewer will probably crash. Since most VNC viewers don't expose compression level 0 in the GUI, this is a relatively rare situation. Description of changes: configure.ac -- Added support for libjpeg-turbo. If passed an argument of --with-turbovnc, configure will now run (or, if cross-compiling, just link) a test program that determines whether the libjpeg library being used is libjpeg-turbo. libjpeg-turbo must be used when building the TurboVNC encoder, because the TurboVNC encoder relies on the libjpeg-turbo colorspace extensions in order to compress images directly out of the framebuffer (which may be, for instance, BGRA rather than RGB.) libjpeg-turbo can optionally be used with the TightVNC encoder as well, but the speedup will only be marginal (the report linked above explains why in more detail, but basically it's because of Amdahl's Law. The TightVNC encoder was designed with the assumption that JPEG had a very high CPU cost, and thus JPEG is used only sparingly.) -- Added a new configure variable, JPEG_LDFLAGS. This is necessitated by the fact that libjpeg-turbo often distributes libjpeg.a and libjpeg.so in /opt/libjpeg-turbo/lib32 or /opt/libjpeg-turbo/lib64, and many people prefer to statically link with it. Thus, more flexibility is needed than is provided by --with-jpeg. If JPEG_LDFLAGS is specified, then it overrides the changes to LDFLAGS enacted by --with-jpeg (but --with-jpeg is still used to set the include path.) The addition of JPEG_LDFLAGS necessitated replacing AC_CHECK_LIB with AC_LINK_IFELSE (because AC_CHECK_LIB automatically sets LIBS to -ljpeg, which is not what we want if we're, for instance, linking statically with libjpeg-turbo.) -- configure does not check for PNG support if TurboVNC encoding is enabled. This prevents the rfbSendRectEncodingTightPng() function from being compiled in, since the TurboVNC encoder doesn't (and can't) support it. common/turbojpeg.c, common/turbojpeg.h -- TurboJPEG is a simple API used to compress and decompress JPEG images in memory. It was originally implemented because it was desirable to use different types of underlying technologies to compress JPEG on different platforms (mediaLib on SPARC, Quicktime on PPC Macs, Intel Performance Primitives, etc.) These days, however, libjpeg-turbo is the only underlying technology used by TurboVNC, so TurboJPEG's purpose is largely just code simplicity and flexibility. Thus, since there is no real need for libvncserver to use any technology other than libjpeg-turbo for compressing JPEG, the TurboJPEG wrapper for libjpeg-turbo has been included in-tree so that libvncserver can be directly linked with libjpeg-turbo. This is convenient because many modern Linux distros (Fedora, Ubuntu, etc.) now ship libjpeg-turbo as their default libjpeg library. libvncserver/rfbserver.c -- Added logic to check for the TurboVNC fine-grained quality level and subsampling encodings and to map Tight (0-9) quality levels to appropriate fine-grained quality level and subsampling values if communicating with a TightVNC/TigerVNC viewer. libvncserver/turbo.c -- TurboVNC encoder (compiled instead of libvncserver/tight.c) rfb/rfb.h -- Added support for the TurboVNC subsampling level rfb/rfbproto.h -- Added constants for the TurboVNC fine quality level and subsampling encodings as well as the rfbTightNoZlib constant and notes on its usage.
13 years ago
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
if(setCompDefaults(cinfo, pixelFormat, jpegSubsamp, jpegQual)==-1)
return -1;
Add TurboVNC encoding support. TurboVNC is a variant of TightVNC that uses the same client/server protocol (RFB version 3.8t), and thus it is fully cross-compatible with TightVNC and TigerVNC (with one exception, which is noted below.) Both the TightVNC and TurboVNC encoders analyze each rectangle, pick out regions of solid color to send separately, and send the remaining subrectangles using mono, indexed color, JPEG, or raw encoding, depending on the number of colors in the subrectangle. However, TurboVNC uses a fundamentally different selection algorithm to determine the appropriate subencoding to use for each subrectangle. Thus, while it sends a protocol stream that can be decoded by any TightVNC-compatible viewer, the mix of subencoding types in this protocol stream will be different from those generated by a TightVNC server. The research that led to TurboVNC is described in the following report: http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf. In summary: 20 RFB captures, representing "common" 2D and 3D application workloads (the 3D workloads were run using VirtualGL), were studied using the TightVNC encoder in isolation. Some of the analysis features in the TightVNC encoder, such as smoothness detection, were found to generate a lot of CPU usage with little or no benefit in compression, so those features were disabled. JPEG encoding was accelerated using libjpeg-turbo (which achieves a 2-4x speedup over plain libjpeg on modern x86 or ARM processors.) Finally, the "palette threshold" (minimum number of colors that the subrectangle must have before it is compressed using JPEG or raw) was adjusted to account for the fact that JPEG encoding is now quite a bit faster (meaning that we can now use it more without a CPU penalty.) TurboVNC has additional optimizations, such as the ability to count colors and encode JPEG images directly from the framebuffer without first translating the pixels into RGB. The TurboVNC encoder compares quite favorably in terms of compression ratio with TightVNC and generally encodes a great deal faster (often an order of magnitude or more.) The version of the TurboVNC encoder included in this patch is roughly equivalent to the one found in version 0.6 of the Unix TurboVNC Server, with a few minor patches integrated from TurboVNC 1.1. TurboVNC 1.0 added multi-threading capabilities, which can be added in later if desired (at the expense of making libvncserver depend on libpthread.) Because TurboVNC uses a fundamentally different mix of subencodings than TightVNC, because it uses the identical protocol (and thus a viewer really has no idea whether it's talking to a TightVNC or TurboVNC server), and because it doesn't support rfbTightPng (and in fact conflicts with it-- see below), the TurboVNC and TightVNC encoders cannot be enabled simultaneously. Compatibility: In *most* cases, a TurboVNC-enabled viewer is fully compatible with a TightVNC server, and vice versa. TurboVNC supports pseudo-encodings for specifying a fine-grained (1-100) quality scale and specifying chrominance subsampling. If a TurboVNC viewer sends those to a TightVNC server, then the TightVNC server ignores them, so the TurboVNC viewer also sends the quality on a 0-9 scale that the TightVNC server can understand. Similarly, the TurboVNC server checks first for fine-grained quality and subsampling pseudo-encodings from the viewer, and failing to receive those, it then checks for the TightVNC 0-9 quality pseudo-encoding. There is one case in which the two systems are not compatible, and that is when a TightVNC or TigerVNC viewer requests compression level 0 without JPEG from a TurboVNC server. For performance reasons, this causes the TurboVNC server to send images directly to the viewer, bypassing Zlib. When the TurboVNC server does this, it also sets bits 7-4 in the compression control byte to rfbTightNoZlib (0x0A), which is unfortunately the same value as rfbTightPng. Older TightVNC viewers that don't handle PNG will assume that the stream is uncompressed but still encapsulated in a Zlib structure, whereas newer PNG-supporting TightVNC viewers will assume that the stream is PNG. In either case, the viewer will probably crash. Since most VNC viewers don't expose compression level 0 in the GUI, this is a relatively rare situation. Description of changes: configure.ac -- Added support for libjpeg-turbo. If passed an argument of --with-turbovnc, configure will now run (or, if cross-compiling, just link) a test program that determines whether the libjpeg library being used is libjpeg-turbo. libjpeg-turbo must be used when building the TurboVNC encoder, because the TurboVNC encoder relies on the libjpeg-turbo colorspace extensions in order to compress images directly out of the framebuffer (which may be, for instance, BGRA rather than RGB.) libjpeg-turbo can optionally be used with the TightVNC encoder as well, but the speedup will only be marginal (the report linked above explains why in more detail, but basically it's because of Amdahl's Law. The TightVNC encoder was designed with the assumption that JPEG had a very high CPU cost, and thus JPEG is used only sparingly.) -- Added a new configure variable, JPEG_LDFLAGS. This is necessitated by the fact that libjpeg-turbo often distributes libjpeg.a and libjpeg.so in /opt/libjpeg-turbo/lib32 or /opt/libjpeg-turbo/lib64, and many people prefer to statically link with it. Thus, more flexibility is needed than is provided by --with-jpeg. If JPEG_LDFLAGS is specified, then it overrides the changes to LDFLAGS enacted by --with-jpeg (but --with-jpeg is still used to set the include path.) The addition of JPEG_LDFLAGS necessitated replacing AC_CHECK_LIB with AC_LINK_IFELSE (because AC_CHECK_LIB automatically sets LIBS to -ljpeg, which is not what we want if we're, for instance, linking statically with libjpeg-turbo.) -- configure does not check for PNG support if TurboVNC encoding is enabled. This prevents the rfbSendRectEncodingTightPng() function from being compiled in, since the TurboVNC encoder doesn't (and can't) support it. common/turbojpeg.c, common/turbojpeg.h -- TurboJPEG is a simple API used to compress and decompress JPEG images in memory. It was originally implemented because it was desirable to use different types of underlying technologies to compress JPEG on different platforms (mediaLib on SPARC, Quicktime on PPC Macs, Intel Performance Primitives, etc.) These days, however, libjpeg-turbo is the only underlying technology used by TurboVNC, so TurboJPEG's purpose is largely just code simplicity and flexibility. Thus, since there is no real need for libvncserver to use any technology other than libjpeg-turbo for compressing JPEG, the TurboJPEG wrapper for libjpeg-turbo has been included in-tree so that libvncserver can be directly linked with libjpeg-turbo. This is convenient because many modern Linux distros (Fedora, Ubuntu, etc.) now ship libjpeg-turbo as their default libjpeg library. libvncserver/rfbserver.c -- Added logic to check for the TurboVNC fine-grained quality level and subsampling encodings and to map Tight (0-9) quality levels to appropriate fine-grained quality level and subsampling values if communicating with a TightVNC/TigerVNC viewer. libvncserver/turbo.c -- TurboVNC encoder (compiled instead of libvncserver/tight.c) rfb/rfb.h -- Added support for the TurboVNC subsampling level rfb/rfbproto.h -- Added constants for the TurboVNC fine quality level and subsampling encodings as well as the rfbTightNoZlib constant and notes on its usage.
13 years ago
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
this->jdst.next_output_byte=*jpegBuf;
this->jdst.free_in_buffer=tjBufSize(width, height, jpegSubsamp);
Add TurboVNC encoding support. TurboVNC is a variant of TightVNC that uses the same client/server protocol (RFB version 3.8t), and thus it is fully cross-compatible with TightVNC and TigerVNC (with one exception, which is noted below.) Both the TightVNC and TurboVNC encoders analyze each rectangle, pick out regions of solid color to send separately, and send the remaining subrectangles using mono, indexed color, JPEG, or raw encoding, depending on the number of colors in the subrectangle. However, TurboVNC uses a fundamentally different selection algorithm to determine the appropriate subencoding to use for each subrectangle. Thus, while it sends a protocol stream that can be decoded by any TightVNC-compatible viewer, the mix of subencoding types in this protocol stream will be different from those generated by a TightVNC server. The research that led to TurboVNC is described in the following report: http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf. In summary: 20 RFB captures, representing "common" 2D and 3D application workloads (the 3D workloads were run using VirtualGL), were studied using the TightVNC encoder in isolation. Some of the analysis features in the TightVNC encoder, such as smoothness detection, were found to generate a lot of CPU usage with little or no benefit in compression, so those features were disabled. JPEG encoding was accelerated using libjpeg-turbo (which achieves a 2-4x speedup over plain libjpeg on modern x86 or ARM processors.) Finally, the "palette threshold" (minimum number of colors that the subrectangle must have before it is compressed using JPEG or raw) was adjusted to account for the fact that JPEG encoding is now quite a bit faster (meaning that we can now use it more without a CPU penalty.) TurboVNC has additional optimizations, such as the ability to count colors and encode JPEG images directly from the framebuffer without first translating the pixels into RGB. The TurboVNC encoder compares quite favorably in terms of compression ratio with TightVNC and generally encodes a great deal faster (often an order of magnitude or more.) The version of the TurboVNC encoder included in this patch is roughly equivalent to the one found in version 0.6 of the Unix TurboVNC Server, with a few minor patches integrated from TurboVNC 1.1. TurboVNC 1.0 added multi-threading capabilities, which can be added in later if desired (at the expense of making libvncserver depend on libpthread.) Because TurboVNC uses a fundamentally different mix of subencodings than TightVNC, because it uses the identical protocol (and thus a viewer really has no idea whether it's talking to a TightVNC or TurboVNC server), and because it doesn't support rfbTightPng (and in fact conflicts with it-- see below), the TurboVNC and TightVNC encoders cannot be enabled simultaneously. Compatibility: In *most* cases, a TurboVNC-enabled viewer is fully compatible with a TightVNC server, and vice versa. TurboVNC supports pseudo-encodings for specifying a fine-grained (1-100) quality scale and specifying chrominance subsampling. If a TurboVNC viewer sends those to a TightVNC server, then the TightVNC server ignores them, so the TurboVNC viewer also sends the quality on a 0-9 scale that the TightVNC server can understand. Similarly, the TurboVNC server checks first for fine-grained quality and subsampling pseudo-encodings from the viewer, and failing to receive those, it then checks for the TightVNC 0-9 quality pseudo-encoding. There is one case in which the two systems are not compatible, and that is when a TightVNC or TigerVNC viewer requests compression level 0 without JPEG from a TurboVNC server. For performance reasons, this causes the TurboVNC server to send images directly to the viewer, bypassing Zlib. When the TurboVNC server does this, it also sets bits 7-4 in the compression control byte to rfbTightNoZlib (0x0A), which is unfortunately the same value as rfbTightPng. Older TightVNC viewers that don't handle PNG will assume that the stream is uncompressed but still encapsulated in a Zlib structure, whereas newer PNG-supporting TightVNC viewers will assume that the stream is PNG. In either case, the viewer will probably crash. Since most VNC viewers don't expose compression level 0 in the GUI, this is a relatively rare situation. Description of changes: configure.ac -- Added support for libjpeg-turbo. If passed an argument of --with-turbovnc, configure will now run (or, if cross-compiling, just link) a test program that determines whether the libjpeg library being used is libjpeg-turbo. libjpeg-turbo must be used when building the TurboVNC encoder, because the TurboVNC encoder relies on the libjpeg-turbo colorspace extensions in order to compress images directly out of the framebuffer (which may be, for instance, BGRA rather than RGB.) libjpeg-turbo can optionally be used with the TightVNC encoder as well, but the speedup will only be marginal (the report linked above explains why in more detail, but basically it's because of Amdahl's Law. The TightVNC encoder was designed with the assumption that JPEG had a very high CPU cost, and thus JPEG is used only sparingly.) -- Added a new configure variable, JPEG_LDFLAGS. This is necessitated by the fact that libjpeg-turbo often distributes libjpeg.a and libjpeg.so in /opt/libjpeg-turbo/lib32 or /opt/libjpeg-turbo/lib64, and many people prefer to statically link with it. Thus, more flexibility is needed than is provided by --with-jpeg. If JPEG_LDFLAGS is specified, then it overrides the changes to LDFLAGS enacted by --with-jpeg (but --with-jpeg is still used to set the include path.) The addition of JPEG_LDFLAGS necessitated replacing AC_CHECK_LIB with AC_LINK_IFELSE (because AC_CHECK_LIB automatically sets LIBS to -ljpeg, which is not what we want if we're, for instance, linking statically with libjpeg-turbo.) -- configure does not check for PNG support if TurboVNC encoding is enabled. This prevents the rfbSendRectEncodingTightPng() function from being compiled in, since the TurboVNC encoder doesn't (and can't) support it. common/turbojpeg.c, common/turbojpeg.h -- TurboJPEG is a simple API used to compress and decompress JPEG images in memory. It was originally implemented because it was desirable to use different types of underlying technologies to compress JPEG on different platforms (mediaLib on SPARC, Quicktime on PPC Macs, Intel Performance Primitives, etc.) These days, however, libjpeg-turbo is the only underlying technology used by TurboVNC, so TurboJPEG's purpose is largely just code simplicity and flexibility. Thus, since there is no real need for libvncserver to use any technology other than libjpeg-turbo for compressing JPEG, the TurboJPEG wrapper for libjpeg-turbo has been included in-tree so that libvncserver can be directly linked with libjpeg-turbo. This is convenient because many modern Linux distros (Fedora, Ubuntu, etc.) now ship libjpeg-turbo as their default libjpeg library. libvncserver/rfbserver.c -- Added logic to check for the TurboVNC fine-grained quality level and subsampling encodings and to map Tight (0-9) quality levels to appropriate fine-grained quality level and subsampling values if communicating with a TightVNC/TigerVNC viewer. libvncserver/turbo.c -- TurboVNC encoder (compiled instead of libvncserver/tight.c) rfb/rfb.h -- Added support for the TurboVNC subsampling level rfb/rfbproto.h -- Added constants for the TurboVNC fine quality level and subsampling encodings as well as the rfbTightNoZlib constant and notes on its usage.
13 years ago
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
jpeg_start_compress(cinfo, TRUE);
Add TurboVNC encoding support. TurboVNC is a variant of TightVNC that uses the same client/server protocol (RFB version 3.8t), and thus it is fully cross-compatible with TightVNC and TigerVNC (with one exception, which is noted below.) Both the TightVNC and TurboVNC encoders analyze each rectangle, pick out regions of solid color to send separately, and send the remaining subrectangles using mono, indexed color, JPEG, or raw encoding, depending on the number of colors in the subrectangle. However, TurboVNC uses a fundamentally different selection algorithm to determine the appropriate subencoding to use for each subrectangle. Thus, while it sends a protocol stream that can be decoded by any TightVNC-compatible viewer, the mix of subencoding types in this protocol stream will be different from those generated by a TightVNC server. The research that led to TurboVNC is described in the following report: http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf. In summary: 20 RFB captures, representing "common" 2D and 3D application workloads (the 3D workloads were run using VirtualGL), were studied using the TightVNC encoder in isolation. Some of the analysis features in the TightVNC encoder, such as smoothness detection, were found to generate a lot of CPU usage with little or no benefit in compression, so those features were disabled. JPEG encoding was accelerated using libjpeg-turbo (which achieves a 2-4x speedup over plain libjpeg on modern x86 or ARM processors.) Finally, the "palette threshold" (minimum number of colors that the subrectangle must have before it is compressed using JPEG or raw) was adjusted to account for the fact that JPEG encoding is now quite a bit faster (meaning that we can now use it more without a CPU penalty.) TurboVNC has additional optimizations, such as the ability to count colors and encode JPEG images directly from the framebuffer without first translating the pixels into RGB. The TurboVNC encoder compares quite favorably in terms of compression ratio with TightVNC and generally encodes a great deal faster (often an order of magnitude or more.) The version of the TurboVNC encoder included in this patch is roughly equivalent to the one found in version 0.6 of the Unix TurboVNC Server, with a few minor patches integrated from TurboVNC 1.1. TurboVNC 1.0 added multi-threading capabilities, which can be added in later if desired (at the expense of making libvncserver depend on libpthread.) Because TurboVNC uses a fundamentally different mix of subencodings than TightVNC, because it uses the identical protocol (and thus a viewer really has no idea whether it's talking to a TightVNC or TurboVNC server), and because it doesn't support rfbTightPng (and in fact conflicts with it-- see below), the TurboVNC and TightVNC encoders cannot be enabled simultaneously. Compatibility: In *most* cases, a TurboVNC-enabled viewer is fully compatible with a TightVNC server, and vice versa. TurboVNC supports pseudo-encodings for specifying a fine-grained (1-100) quality scale and specifying chrominance subsampling. If a TurboVNC viewer sends those to a TightVNC server, then the TightVNC server ignores them, so the TurboVNC viewer also sends the quality on a 0-9 scale that the TightVNC server can understand. Similarly, the TurboVNC server checks first for fine-grained quality and subsampling pseudo-encodings from the viewer, and failing to receive those, it then checks for the TightVNC 0-9 quality pseudo-encoding. There is one case in which the two systems are not compatible, and that is when a TightVNC or TigerVNC viewer requests compression level 0 without JPEG from a TurboVNC server. For performance reasons, this causes the TurboVNC server to send images directly to the viewer, bypassing Zlib. When the TurboVNC server does this, it also sets bits 7-4 in the compression control byte to rfbTightNoZlib (0x0A), which is unfortunately the same value as rfbTightPng. Older TightVNC viewers that don't handle PNG will assume that the stream is uncompressed but still encapsulated in a Zlib structure, whereas newer PNG-supporting TightVNC viewers will assume that the stream is PNG. In either case, the viewer will probably crash. Since most VNC viewers don't expose compression level 0 in the GUI, this is a relatively rare situation. Description of changes: configure.ac -- Added support for libjpeg-turbo. If passed an argument of --with-turbovnc, configure will now run (or, if cross-compiling, just link) a test program that determines whether the libjpeg library being used is libjpeg-turbo. libjpeg-turbo must be used when building the TurboVNC encoder, because the TurboVNC encoder relies on the libjpeg-turbo colorspace extensions in order to compress images directly out of the framebuffer (which may be, for instance, BGRA rather than RGB.) libjpeg-turbo can optionally be used with the TightVNC encoder as well, but the speedup will only be marginal (the report linked above explains why in more detail, but basically it's because of Amdahl's Law. The TightVNC encoder was designed with the assumption that JPEG had a very high CPU cost, and thus JPEG is used only sparingly.) -- Added a new configure variable, JPEG_LDFLAGS. This is necessitated by the fact that libjpeg-turbo often distributes libjpeg.a and libjpeg.so in /opt/libjpeg-turbo/lib32 or /opt/libjpeg-turbo/lib64, and many people prefer to statically link with it. Thus, more flexibility is needed than is provided by --with-jpeg. If JPEG_LDFLAGS is specified, then it overrides the changes to LDFLAGS enacted by --with-jpeg (but --with-jpeg is still used to set the include path.) The addition of JPEG_LDFLAGS necessitated replacing AC_CHECK_LIB with AC_LINK_IFELSE (because AC_CHECK_LIB automatically sets LIBS to -ljpeg, which is not what we want if we're, for instance, linking statically with libjpeg-turbo.) -- configure does not check for PNG support if TurboVNC encoding is enabled. This prevents the rfbSendRectEncodingTightPng() function from being compiled in, since the TurboVNC encoder doesn't (and can't) support it. common/turbojpeg.c, common/turbojpeg.h -- TurboJPEG is a simple API used to compress and decompress JPEG images in memory. It was originally implemented because it was desirable to use different types of underlying technologies to compress JPEG on different platforms (mediaLib on SPARC, Quicktime on PPC Macs, Intel Performance Primitives, etc.) These days, however, libjpeg-turbo is the only underlying technology used by TurboVNC, so TurboJPEG's purpose is largely just code simplicity and flexibility. Thus, since there is no real need for libvncserver to use any technology other than libjpeg-turbo for compressing JPEG, the TurboJPEG wrapper for libjpeg-turbo has been included in-tree so that libvncserver can be directly linked with libjpeg-turbo. This is convenient because many modern Linux distros (Fedora, Ubuntu, etc.) now ship libjpeg-turbo as their default libjpeg library. libvncserver/rfbserver.c -- Added logic to check for the TurboVNC fine-grained quality level and subsampling encodings and to map Tight (0-9) quality levels to appropriate fine-grained quality level and subsampling values if communicating with a TightVNC/TigerVNC viewer. libvncserver/turbo.c -- TurboVNC encoder (compiled instead of libvncserver/tight.c) rfb/rfb.h -- Added support for the TurboVNC subsampling level rfb/rfbproto.h -- Added constants for the TurboVNC fine quality level and subsampling encodings as well as the rfbTightNoZlib constant and notes on its usage.
13 years ago
if((row_pointer=(JSAMPROW *)malloc(sizeof(JSAMPROW)*height))==NULL)
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
_throw("tjCompress2(): Memory allocation failure");
Add TurboVNC encoding support. TurboVNC is a variant of TightVNC that uses the same client/server protocol (RFB version 3.8t), and thus it is fully cross-compatible with TightVNC and TigerVNC (with one exception, which is noted below.) Both the TightVNC and TurboVNC encoders analyze each rectangle, pick out regions of solid color to send separately, and send the remaining subrectangles using mono, indexed color, JPEG, or raw encoding, depending on the number of colors in the subrectangle. However, TurboVNC uses a fundamentally different selection algorithm to determine the appropriate subencoding to use for each subrectangle. Thus, while it sends a protocol stream that can be decoded by any TightVNC-compatible viewer, the mix of subencoding types in this protocol stream will be different from those generated by a TightVNC server. The research that led to TurboVNC is described in the following report: http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf. In summary: 20 RFB captures, representing "common" 2D and 3D application workloads (the 3D workloads were run using VirtualGL), were studied using the TightVNC encoder in isolation. Some of the analysis features in the TightVNC encoder, such as smoothness detection, were found to generate a lot of CPU usage with little or no benefit in compression, so those features were disabled. JPEG encoding was accelerated using libjpeg-turbo (which achieves a 2-4x speedup over plain libjpeg on modern x86 or ARM processors.) Finally, the "palette threshold" (minimum number of colors that the subrectangle must have before it is compressed using JPEG or raw) was adjusted to account for the fact that JPEG encoding is now quite a bit faster (meaning that we can now use it more without a CPU penalty.) TurboVNC has additional optimizations, such as the ability to count colors and encode JPEG images directly from the framebuffer without first translating the pixels into RGB. The TurboVNC encoder compares quite favorably in terms of compression ratio with TightVNC and generally encodes a great deal faster (often an order of magnitude or more.) The version of the TurboVNC encoder included in this patch is roughly equivalent to the one found in version 0.6 of the Unix TurboVNC Server, with a few minor patches integrated from TurboVNC 1.1. TurboVNC 1.0 added multi-threading capabilities, which can be added in later if desired (at the expense of making libvncserver depend on libpthread.) Because TurboVNC uses a fundamentally different mix of subencodings than TightVNC, because it uses the identical protocol (and thus a viewer really has no idea whether it's talking to a TightVNC or TurboVNC server), and because it doesn't support rfbTightPng (and in fact conflicts with it-- see below), the TurboVNC and TightVNC encoders cannot be enabled simultaneously. Compatibility: In *most* cases, a TurboVNC-enabled viewer is fully compatible with a TightVNC server, and vice versa. TurboVNC supports pseudo-encodings for specifying a fine-grained (1-100) quality scale and specifying chrominance subsampling. If a TurboVNC viewer sends those to a TightVNC server, then the TightVNC server ignores them, so the TurboVNC viewer also sends the quality on a 0-9 scale that the TightVNC server can understand. Similarly, the TurboVNC server checks first for fine-grained quality and subsampling pseudo-encodings from the viewer, and failing to receive those, it then checks for the TightVNC 0-9 quality pseudo-encoding. There is one case in which the two systems are not compatible, and that is when a TightVNC or TigerVNC viewer requests compression level 0 without JPEG from a TurboVNC server. For performance reasons, this causes the TurboVNC server to send images directly to the viewer, bypassing Zlib. When the TurboVNC server does this, it also sets bits 7-4 in the compression control byte to rfbTightNoZlib (0x0A), which is unfortunately the same value as rfbTightPng. Older TightVNC viewers that don't handle PNG will assume that the stream is uncompressed but still encapsulated in a Zlib structure, whereas newer PNG-supporting TightVNC viewers will assume that the stream is PNG. In either case, the viewer will probably crash. Since most VNC viewers don't expose compression level 0 in the GUI, this is a relatively rare situation. Description of changes: configure.ac -- Added support for libjpeg-turbo. If passed an argument of --with-turbovnc, configure will now run (or, if cross-compiling, just link) a test program that determines whether the libjpeg library being used is libjpeg-turbo. libjpeg-turbo must be used when building the TurboVNC encoder, because the TurboVNC encoder relies on the libjpeg-turbo colorspace extensions in order to compress images directly out of the framebuffer (which may be, for instance, BGRA rather than RGB.) libjpeg-turbo can optionally be used with the TightVNC encoder as well, but the speedup will only be marginal (the report linked above explains why in more detail, but basically it's because of Amdahl's Law. The TightVNC encoder was designed with the assumption that JPEG had a very high CPU cost, and thus JPEG is used only sparingly.) -- Added a new configure variable, JPEG_LDFLAGS. This is necessitated by the fact that libjpeg-turbo often distributes libjpeg.a and libjpeg.so in /opt/libjpeg-turbo/lib32 or /opt/libjpeg-turbo/lib64, and many people prefer to statically link with it. Thus, more flexibility is needed than is provided by --with-jpeg. If JPEG_LDFLAGS is specified, then it overrides the changes to LDFLAGS enacted by --with-jpeg (but --with-jpeg is still used to set the include path.) The addition of JPEG_LDFLAGS necessitated replacing AC_CHECK_LIB with AC_LINK_IFELSE (because AC_CHECK_LIB automatically sets LIBS to -ljpeg, which is not what we want if we're, for instance, linking statically with libjpeg-turbo.) -- configure does not check for PNG support if TurboVNC encoding is enabled. This prevents the rfbSendRectEncodingTightPng() function from being compiled in, since the TurboVNC encoder doesn't (and can't) support it. common/turbojpeg.c, common/turbojpeg.h -- TurboJPEG is a simple API used to compress and decompress JPEG images in memory. It was originally implemented because it was desirable to use different types of underlying technologies to compress JPEG on different platforms (mediaLib on SPARC, Quicktime on PPC Macs, Intel Performance Primitives, etc.) These days, however, libjpeg-turbo is the only underlying technology used by TurboVNC, so TurboJPEG's purpose is largely just code simplicity and flexibility. Thus, since there is no real need for libvncserver to use any technology other than libjpeg-turbo for compressing JPEG, the TurboJPEG wrapper for libjpeg-turbo has been included in-tree so that libvncserver can be directly linked with libjpeg-turbo. This is convenient because many modern Linux distros (Fedora, Ubuntu, etc.) now ship libjpeg-turbo as their default libjpeg library. libvncserver/rfbserver.c -- Added logic to check for the TurboVNC fine-grained quality level and subsampling encodings and to map Tight (0-9) quality levels to appropriate fine-grained quality level and subsampling values if communicating with a TightVNC/TigerVNC viewer. libvncserver/turbo.c -- TurboVNC encoder (compiled instead of libvncserver/tight.c) rfb/rfb.h -- Added support for the TurboVNC subsampling level rfb/rfbproto.h -- Added constants for the TurboVNC fine quality level and subsampling encodings as well as the rfbTightNoZlib constant and notes on its usage.
13 years ago
for(i=0; i<height; i++)
{
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
if(flags&TJFLAG_BOTTOMUP) row_pointer[i]=&srcBuf[(height-i-1)*pitch];
else row_pointer[i]=&srcBuf[i*pitch];
Add TurboVNC encoding support. TurboVNC is a variant of TightVNC that uses the same client/server protocol (RFB version 3.8t), and thus it is fully cross-compatible with TightVNC and TigerVNC (with one exception, which is noted below.) Both the TightVNC and TurboVNC encoders analyze each rectangle, pick out regions of solid color to send separately, and send the remaining subrectangles using mono, indexed color, JPEG, or raw encoding, depending on the number of colors in the subrectangle. However, TurboVNC uses a fundamentally different selection algorithm to determine the appropriate subencoding to use for each subrectangle. Thus, while it sends a protocol stream that can be decoded by any TightVNC-compatible viewer, the mix of subencoding types in this protocol stream will be different from those generated by a TightVNC server. The research that led to TurboVNC is described in the following report: http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf. In summary: 20 RFB captures, representing "common" 2D and 3D application workloads (the 3D workloads were run using VirtualGL), were studied using the TightVNC encoder in isolation. Some of the analysis features in the TightVNC encoder, such as smoothness detection, were found to generate a lot of CPU usage with little or no benefit in compression, so those features were disabled. JPEG encoding was accelerated using libjpeg-turbo (which achieves a 2-4x speedup over plain libjpeg on modern x86 or ARM processors.) Finally, the "palette threshold" (minimum number of colors that the subrectangle must have before it is compressed using JPEG or raw) was adjusted to account for the fact that JPEG encoding is now quite a bit faster (meaning that we can now use it more without a CPU penalty.) TurboVNC has additional optimizations, such as the ability to count colors and encode JPEG images directly from the framebuffer without first translating the pixels into RGB. The TurboVNC encoder compares quite favorably in terms of compression ratio with TightVNC and generally encodes a great deal faster (often an order of magnitude or more.) The version of the TurboVNC encoder included in this patch is roughly equivalent to the one found in version 0.6 of the Unix TurboVNC Server, with a few minor patches integrated from TurboVNC 1.1. TurboVNC 1.0 added multi-threading capabilities, which can be added in later if desired (at the expense of making libvncserver depend on libpthread.) Because TurboVNC uses a fundamentally different mix of subencodings than TightVNC, because it uses the identical protocol (and thus a viewer really has no idea whether it's talking to a TightVNC or TurboVNC server), and because it doesn't support rfbTightPng (and in fact conflicts with it-- see below), the TurboVNC and TightVNC encoders cannot be enabled simultaneously. Compatibility: In *most* cases, a TurboVNC-enabled viewer is fully compatible with a TightVNC server, and vice versa. TurboVNC supports pseudo-encodings for specifying a fine-grained (1-100) quality scale and specifying chrominance subsampling. If a TurboVNC viewer sends those to a TightVNC server, then the TightVNC server ignores them, so the TurboVNC viewer also sends the quality on a 0-9 scale that the TightVNC server can understand. Similarly, the TurboVNC server checks first for fine-grained quality and subsampling pseudo-encodings from the viewer, and failing to receive those, it then checks for the TightVNC 0-9 quality pseudo-encoding. There is one case in which the two systems are not compatible, and that is when a TightVNC or TigerVNC viewer requests compression level 0 without JPEG from a TurboVNC server. For performance reasons, this causes the TurboVNC server to send images directly to the viewer, bypassing Zlib. When the TurboVNC server does this, it also sets bits 7-4 in the compression control byte to rfbTightNoZlib (0x0A), which is unfortunately the same value as rfbTightPng. Older TightVNC viewers that don't handle PNG will assume that the stream is uncompressed but still encapsulated in a Zlib structure, whereas newer PNG-supporting TightVNC viewers will assume that the stream is PNG. In either case, the viewer will probably crash. Since most VNC viewers don't expose compression level 0 in the GUI, this is a relatively rare situation. Description of changes: configure.ac -- Added support for libjpeg-turbo. If passed an argument of --with-turbovnc, configure will now run (or, if cross-compiling, just link) a test program that determines whether the libjpeg library being used is libjpeg-turbo. libjpeg-turbo must be used when building the TurboVNC encoder, because the TurboVNC encoder relies on the libjpeg-turbo colorspace extensions in order to compress images directly out of the framebuffer (which may be, for instance, BGRA rather than RGB.) libjpeg-turbo can optionally be used with the TightVNC encoder as well, but the speedup will only be marginal (the report linked above explains why in more detail, but basically it's because of Amdahl's Law. The TightVNC encoder was designed with the assumption that JPEG had a very high CPU cost, and thus JPEG is used only sparingly.) -- Added a new configure variable, JPEG_LDFLAGS. This is necessitated by the fact that libjpeg-turbo often distributes libjpeg.a and libjpeg.so in /opt/libjpeg-turbo/lib32 or /opt/libjpeg-turbo/lib64, and many people prefer to statically link with it. Thus, more flexibility is needed than is provided by --with-jpeg. If JPEG_LDFLAGS is specified, then it overrides the changes to LDFLAGS enacted by --with-jpeg (but --with-jpeg is still used to set the include path.) The addition of JPEG_LDFLAGS necessitated replacing AC_CHECK_LIB with AC_LINK_IFELSE (because AC_CHECK_LIB automatically sets LIBS to -ljpeg, which is not what we want if we're, for instance, linking statically with libjpeg-turbo.) -- configure does not check for PNG support if TurboVNC encoding is enabled. This prevents the rfbSendRectEncodingTightPng() function from being compiled in, since the TurboVNC encoder doesn't (and can't) support it. common/turbojpeg.c, common/turbojpeg.h -- TurboJPEG is a simple API used to compress and decompress JPEG images in memory. It was originally implemented because it was desirable to use different types of underlying technologies to compress JPEG on different platforms (mediaLib on SPARC, Quicktime on PPC Macs, Intel Performance Primitives, etc.) These days, however, libjpeg-turbo is the only underlying technology used by TurboVNC, so TurboJPEG's purpose is largely just code simplicity and flexibility. Thus, since there is no real need for libvncserver to use any technology other than libjpeg-turbo for compressing JPEG, the TurboJPEG wrapper for libjpeg-turbo has been included in-tree so that libvncserver can be directly linked with libjpeg-turbo. This is convenient because many modern Linux distros (Fedora, Ubuntu, etc.) now ship libjpeg-turbo as their default libjpeg library. libvncserver/rfbserver.c -- Added logic to check for the TurboVNC fine-grained quality level and subsampling encodings and to map Tight (0-9) quality levels to appropriate fine-grained quality level and subsampling values if communicating with a TightVNC/TigerVNC viewer. libvncserver/turbo.c -- TurboVNC encoder (compiled instead of libvncserver/tight.c) rfb/rfb.h -- Added support for the TurboVNC subsampling level rfb/rfbproto.h -- Added constants for the TurboVNC fine quality level and subsampling encodings as well as the rfbTightNoZlib constant and notes on its usage.
13 years ago
}
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
while(cinfo->next_scanline<cinfo->image_height)
Add TurboVNC encoding support. TurboVNC is a variant of TightVNC that uses the same client/server protocol (RFB version 3.8t), and thus it is fully cross-compatible with TightVNC and TigerVNC (with one exception, which is noted below.) Both the TightVNC and TurboVNC encoders analyze each rectangle, pick out regions of solid color to send separately, and send the remaining subrectangles using mono, indexed color, JPEG, or raw encoding, depending on the number of colors in the subrectangle. However, TurboVNC uses a fundamentally different selection algorithm to determine the appropriate subencoding to use for each subrectangle. Thus, while it sends a protocol stream that can be decoded by any TightVNC-compatible viewer, the mix of subencoding types in this protocol stream will be different from those generated by a TightVNC server. The research that led to TurboVNC is described in the following report: http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf. In summary: 20 RFB captures, representing "common" 2D and 3D application workloads (the 3D workloads were run using VirtualGL), were studied using the TightVNC encoder in isolation. Some of the analysis features in the TightVNC encoder, such as smoothness detection, were found to generate a lot of CPU usage with little or no benefit in compression, so those features were disabled. JPEG encoding was accelerated using libjpeg-turbo (which achieves a 2-4x speedup over plain libjpeg on modern x86 or ARM processors.) Finally, the "palette threshold" (minimum number of colors that the subrectangle must have before it is compressed using JPEG or raw) was adjusted to account for the fact that JPEG encoding is now quite a bit faster (meaning that we can now use it more without a CPU penalty.) TurboVNC has additional optimizations, such as the ability to count colors and encode JPEG images directly from the framebuffer without first translating the pixels into RGB. The TurboVNC encoder compares quite favorably in terms of compression ratio with TightVNC and generally encodes a great deal faster (often an order of magnitude or more.) The version of the TurboVNC encoder included in this patch is roughly equivalent to the one found in version 0.6 of the Unix TurboVNC Server, with a few minor patches integrated from TurboVNC 1.1. TurboVNC 1.0 added multi-threading capabilities, which can be added in later if desired (at the expense of making libvncserver depend on libpthread.) Because TurboVNC uses a fundamentally different mix of subencodings than TightVNC, because it uses the identical protocol (and thus a viewer really has no idea whether it's talking to a TightVNC or TurboVNC server), and because it doesn't support rfbTightPng (and in fact conflicts with it-- see below), the TurboVNC and TightVNC encoders cannot be enabled simultaneously. Compatibility: In *most* cases, a TurboVNC-enabled viewer is fully compatible with a TightVNC server, and vice versa. TurboVNC supports pseudo-encodings for specifying a fine-grained (1-100) quality scale and specifying chrominance subsampling. If a TurboVNC viewer sends those to a TightVNC server, then the TightVNC server ignores them, so the TurboVNC viewer also sends the quality on a 0-9 scale that the TightVNC server can understand. Similarly, the TurboVNC server checks first for fine-grained quality and subsampling pseudo-encodings from the viewer, and failing to receive those, it then checks for the TightVNC 0-9 quality pseudo-encoding. There is one case in which the two systems are not compatible, and that is when a TightVNC or TigerVNC viewer requests compression level 0 without JPEG from a TurboVNC server. For performance reasons, this causes the TurboVNC server to send images directly to the viewer, bypassing Zlib. When the TurboVNC server does this, it also sets bits 7-4 in the compression control byte to rfbTightNoZlib (0x0A), which is unfortunately the same value as rfbTightPng. Older TightVNC viewers that don't handle PNG will assume that the stream is uncompressed but still encapsulated in a Zlib structure, whereas newer PNG-supporting TightVNC viewers will assume that the stream is PNG. In either case, the viewer will probably crash. Since most VNC viewers don't expose compression level 0 in the GUI, this is a relatively rare situation. Description of changes: configure.ac -- Added support for libjpeg-turbo. If passed an argument of --with-turbovnc, configure will now run (or, if cross-compiling, just link) a test program that determines whether the libjpeg library being used is libjpeg-turbo. libjpeg-turbo must be used when building the TurboVNC encoder, because the TurboVNC encoder relies on the libjpeg-turbo colorspace extensions in order to compress images directly out of the framebuffer (which may be, for instance, BGRA rather than RGB.) libjpeg-turbo can optionally be used with the TightVNC encoder as well, but the speedup will only be marginal (the report linked above explains why in more detail, but basically it's because of Amdahl's Law. The TightVNC encoder was designed with the assumption that JPEG had a very high CPU cost, and thus JPEG is used only sparingly.) -- Added a new configure variable, JPEG_LDFLAGS. This is necessitated by the fact that libjpeg-turbo often distributes libjpeg.a and libjpeg.so in /opt/libjpeg-turbo/lib32 or /opt/libjpeg-turbo/lib64, and many people prefer to statically link with it. Thus, more flexibility is needed than is provided by --with-jpeg. If JPEG_LDFLAGS is specified, then it overrides the changes to LDFLAGS enacted by --with-jpeg (but --with-jpeg is still used to set the include path.) The addition of JPEG_LDFLAGS necessitated replacing AC_CHECK_LIB with AC_LINK_IFELSE (because AC_CHECK_LIB automatically sets LIBS to -ljpeg, which is not what we want if we're, for instance, linking statically with libjpeg-turbo.) -- configure does not check for PNG support if TurboVNC encoding is enabled. This prevents the rfbSendRectEncodingTightPng() function from being compiled in, since the TurboVNC encoder doesn't (and can't) support it. common/turbojpeg.c, common/turbojpeg.h -- TurboJPEG is a simple API used to compress and decompress JPEG images in memory. It was originally implemented because it was desirable to use different types of underlying technologies to compress JPEG on different platforms (mediaLib on SPARC, Quicktime on PPC Macs, Intel Performance Primitives, etc.) These days, however, libjpeg-turbo is the only underlying technology used by TurboVNC, so TurboJPEG's purpose is largely just code simplicity and flexibility. Thus, since there is no real need for libvncserver to use any technology other than libjpeg-turbo for compressing JPEG, the TurboJPEG wrapper for libjpeg-turbo has been included in-tree so that libvncserver can be directly linked with libjpeg-turbo. This is convenient because many modern Linux distros (Fedora, Ubuntu, etc.) now ship libjpeg-turbo as their default libjpeg library. libvncserver/rfbserver.c -- Added logic to check for the TurboVNC fine-grained quality level and subsampling encodings and to map Tight (0-9) quality levels to appropriate fine-grained quality level and subsampling values if communicating with a TightVNC/TigerVNC viewer. libvncserver/turbo.c -- TurboVNC encoder (compiled instead of libvncserver/tight.c) rfb/rfb.h -- Added support for the TurboVNC subsampling level rfb/rfbproto.h -- Added constants for the TurboVNC fine quality level and subsampling encodings as well as the rfbTightNoZlib constant and notes on its usage.
13 years ago
{
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
jpeg_write_scanlines(cinfo, &row_pointer[cinfo->next_scanline],
cinfo->image_height-cinfo->next_scanline);
Add TurboVNC encoding support. TurboVNC is a variant of TightVNC that uses the same client/server protocol (RFB version 3.8t), and thus it is fully cross-compatible with TightVNC and TigerVNC (with one exception, which is noted below.) Both the TightVNC and TurboVNC encoders analyze each rectangle, pick out regions of solid color to send separately, and send the remaining subrectangles using mono, indexed color, JPEG, or raw encoding, depending on the number of colors in the subrectangle. However, TurboVNC uses a fundamentally different selection algorithm to determine the appropriate subencoding to use for each subrectangle. Thus, while it sends a protocol stream that can be decoded by any TightVNC-compatible viewer, the mix of subencoding types in this protocol stream will be different from those generated by a TightVNC server. The research that led to TurboVNC is described in the following report: http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf. In summary: 20 RFB captures, representing "common" 2D and 3D application workloads (the 3D workloads were run using VirtualGL), were studied using the TightVNC encoder in isolation. Some of the analysis features in the TightVNC encoder, such as smoothness detection, were found to generate a lot of CPU usage with little or no benefit in compression, so those features were disabled. JPEG encoding was accelerated using libjpeg-turbo (which achieves a 2-4x speedup over plain libjpeg on modern x86 or ARM processors.) Finally, the "palette threshold" (minimum number of colors that the subrectangle must have before it is compressed using JPEG or raw) was adjusted to account for the fact that JPEG encoding is now quite a bit faster (meaning that we can now use it more without a CPU penalty.) TurboVNC has additional optimizations, such as the ability to count colors and encode JPEG images directly from the framebuffer without first translating the pixels into RGB. The TurboVNC encoder compares quite favorably in terms of compression ratio with TightVNC and generally encodes a great deal faster (often an order of magnitude or more.) The version of the TurboVNC encoder included in this patch is roughly equivalent to the one found in version 0.6 of the Unix TurboVNC Server, with a few minor patches integrated from TurboVNC 1.1. TurboVNC 1.0 added multi-threading capabilities, which can be added in later if desired (at the expense of making libvncserver depend on libpthread.) Because TurboVNC uses a fundamentally different mix of subencodings than TightVNC, because it uses the identical protocol (and thus a viewer really has no idea whether it's talking to a TightVNC or TurboVNC server), and because it doesn't support rfbTightPng (and in fact conflicts with it-- see below), the TurboVNC and TightVNC encoders cannot be enabled simultaneously. Compatibility: In *most* cases, a TurboVNC-enabled viewer is fully compatible with a TightVNC server, and vice versa. TurboVNC supports pseudo-encodings for specifying a fine-grained (1-100) quality scale and specifying chrominance subsampling. If a TurboVNC viewer sends those to a TightVNC server, then the TightVNC server ignores them, so the TurboVNC viewer also sends the quality on a 0-9 scale that the TightVNC server can understand. Similarly, the TurboVNC server checks first for fine-grained quality and subsampling pseudo-encodings from the viewer, and failing to receive those, it then checks for the TightVNC 0-9 quality pseudo-encoding. There is one case in which the two systems are not compatible, and that is when a TightVNC or TigerVNC viewer requests compression level 0 without JPEG from a TurboVNC server. For performance reasons, this causes the TurboVNC server to send images directly to the viewer, bypassing Zlib. When the TurboVNC server does this, it also sets bits 7-4 in the compression control byte to rfbTightNoZlib (0x0A), which is unfortunately the same value as rfbTightPng. Older TightVNC viewers that don't handle PNG will assume that the stream is uncompressed but still encapsulated in a Zlib structure, whereas newer PNG-supporting TightVNC viewers will assume that the stream is PNG. In either case, the viewer will probably crash. Since most VNC viewers don't expose compression level 0 in the GUI, this is a relatively rare situation. Description of changes: configure.ac -- Added support for libjpeg-turbo. If passed an argument of --with-turbovnc, configure will now run (or, if cross-compiling, just link) a test program that determines whether the libjpeg library being used is libjpeg-turbo. libjpeg-turbo must be used when building the TurboVNC encoder, because the TurboVNC encoder relies on the libjpeg-turbo colorspace extensions in order to compress images directly out of the framebuffer (which may be, for instance, BGRA rather than RGB.) libjpeg-turbo can optionally be used with the TightVNC encoder as well, but the speedup will only be marginal (the report linked above explains why in more detail, but basically it's because of Amdahl's Law. The TightVNC encoder was designed with the assumption that JPEG had a very high CPU cost, and thus JPEG is used only sparingly.) -- Added a new configure variable, JPEG_LDFLAGS. This is necessitated by the fact that libjpeg-turbo often distributes libjpeg.a and libjpeg.so in /opt/libjpeg-turbo/lib32 or /opt/libjpeg-turbo/lib64, and many people prefer to statically link with it. Thus, more flexibility is needed than is provided by --with-jpeg. If JPEG_LDFLAGS is specified, then it overrides the changes to LDFLAGS enacted by --with-jpeg (but --with-jpeg is still used to set the include path.) The addition of JPEG_LDFLAGS necessitated replacing AC_CHECK_LIB with AC_LINK_IFELSE (because AC_CHECK_LIB automatically sets LIBS to -ljpeg, which is not what we want if we're, for instance, linking statically with libjpeg-turbo.) -- configure does not check for PNG support if TurboVNC encoding is enabled. This prevents the rfbSendRectEncodingTightPng() function from being compiled in, since the TurboVNC encoder doesn't (and can't) support it. common/turbojpeg.c, common/turbojpeg.h -- TurboJPEG is a simple API used to compress and decompress JPEG images in memory. It was originally implemented because it was desirable to use different types of underlying technologies to compress JPEG on different platforms (mediaLib on SPARC, Quicktime on PPC Macs, Intel Performance Primitives, etc.) These days, however, libjpeg-turbo is the only underlying technology used by TurboVNC, so TurboJPEG's purpose is largely just code simplicity and flexibility. Thus, since there is no real need for libvncserver to use any technology other than libjpeg-turbo for compressing JPEG, the TurboJPEG wrapper for libjpeg-turbo has been included in-tree so that libvncserver can be directly linked with libjpeg-turbo. This is convenient because many modern Linux distros (Fedora, Ubuntu, etc.) now ship libjpeg-turbo as their default libjpeg library. libvncserver/rfbserver.c -- Added logic to check for the TurboVNC fine-grained quality level and subsampling encodings and to map Tight (0-9) quality levels to appropriate fine-grained quality level and subsampling values if communicating with a TightVNC/TigerVNC viewer. libvncserver/turbo.c -- TurboVNC encoder (compiled instead of libvncserver/tight.c) rfb/rfb.h -- Added support for the TurboVNC subsampling level rfb/rfbproto.h -- Added constants for the TurboVNC fine quality level and subsampling encodings as well as the rfbTightNoZlib constant and notes on its usage.
13 years ago
}
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
jpeg_finish_compress(cinfo);
*jpegSize=tjBufSize(width, height, jpegSubsamp)
-(unsigned long)(this->jdst.free_in_buffer);
Add TurboVNC encoding support. TurboVNC is a variant of TightVNC that uses the same client/server protocol (RFB version 3.8t), and thus it is fully cross-compatible with TightVNC and TigerVNC (with one exception, which is noted below.) Both the TightVNC and TurboVNC encoders analyze each rectangle, pick out regions of solid color to send separately, and send the remaining subrectangles using mono, indexed color, JPEG, or raw encoding, depending on the number of colors in the subrectangle. However, TurboVNC uses a fundamentally different selection algorithm to determine the appropriate subencoding to use for each subrectangle. Thus, while it sends a protocol stream that can be decoded by any TightVNC-compatible viewer, the mix of subencoding types in this protocol stream will be different from those generated by a TightVNC server. The research that led to TurboVNC is described in the following report: http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf. In summary: 20 RFB captures, representing "common" 2D and 3D application workloads (the 3D workloads were run using VirtualGL), were studied using the TightVNC encoder in isolation. Some of the analysis features in the TightVNC encoder, such as smoothness detection, were found to generate a lot of CPU usage with little or no benefit in compression, so those features were disabled. JPEG encoding was accelerated using libjpeg-turbo (which achieves a 2-4x speedup over plain libjpeg on modern x86 or ARM processors.) Finally, the "palette threshold" (minimum number of colors that the subrectangle must have before it is compressed using JPEG or raw) was adjusted to account for the fact that JPEG encoding is now quite a bit faster (meaning that we can now use it more without a CPU penalty.) TurboVNC has additional optimizations, such as the ability to count colors and encode JPEG images directly from the framebuffer without first translating the pixels into RGB. The TurboVNC encoder compares quite favorably in terms of compression ratio with TightVNC and generally encodes a great deal faster (often an order of magnitude or more.) The version of the TurboVNC encoder included in this patch is roughly equivalent to the one found in version 0.6 of the Unix TurboVNC Server, with a few minor patches integrated from TurboVNC 1.1. TurboVNC 1.0 added multi-threading capabilities, which can be added in later if desired (at the expense of making libvncserver depend on libpthread.) Because TurboVNC uses a fundamentally different mix of subencodings than TightVNC, because it uses the identical protocol (and thus a viewer really has no idea whether it's talking to a TightVNC or TurboVNC server), and because it doesn't support rfbTightPng (and in fact conflicts with it-- see below), the TurboVNC and TightVNC encoders cannot be enabled simultaneously. Compatibility: In *most* cases, a TurboVNC-enabled viewer is fully compatible with a TightVNC server, and vice versa. TurboVNC supports pseudo-encodings for specifying a fine-grained (1-100) quality scale and specifying chrominance subsampling. If a TurboVNC viewer sends those to a TightVNC server, then the TightVNC server ignores them, so the TurboVNC viewer also sends the quality on a 0-9 scale that the TightVNC server can understand. Similarly, the TurboVNC server checks first for fine-grained quality and subsampling pseudo-encodings from the viewer, and failing to receive those, it then checks for the TightVNC 0-9 quality pseudo-encoding. There is one case in which the two systems are not compatible, and that is when a TightVNC or TigerVNC viewer requests compression level 0 without JPEG from a TurboVNC server. For performance reasons, this causes the TurboVNC server to send images directly to the viewer, bypassing Zlib. When the TurboVNC server does this, it also sets bits 7-4 in the compression control byte to rfbTightNoZlib (0x0A), which is unfortunately the same value as rfbTightPng. Older TightVNC viewers that don't handle PNG will assume that the stream is uncompressed but still encapsulated in a Zlib structure, whereas newer PNG-supporting TightVNC viewers will assume that the stream is PNG. In either case, the viewer will probably crash. Since most VNC viewers don't expose compression level 0 in the GUI, this is a relatively rare situation. Description of changes: configure.ac -- Added support for libjpeg-turbo. If passed an argument of --with-turbovnc, configure will now run (or, if cross-compiling, just link) a test program that determines whether the libjpeg library being used is libjpeg-turbo. libjpeg-turbo must be used when building the TurboVNC encoder, because the TurboVNC encoder relies on the libjpeg-turbo colorspace extensions in order to compress images directly out of the framebuffer (which may be, for instance, BGRA rather than RGB.) libjpeg-turbo can optionally be used with the TightVNC encoder as well, but the speedup will only be marginal (the report linked above explains why in more detail, but basically it's because of Amdahl's Law. The TightVNC encoder was designed with the assumption that JPEG had a very high CPU cost, and thus JPEG is used only sparingly.) -- Added a new configure variable, JPEG_LDFLAGS. This is necessitated by the fact that libjpeg-turbo often distributes libjpeg.a and libjpeg.so in /opt/libjpeg-turbo/lib32 or /opt/libjpeg-turbo/lib64, and many people prefer to statically link with it. Thus, more flexibility is needed than is provided by --with-jpeg. If JPEG_LDFLAGS is specified, then it overrides the changes to LDFLAGS enacted by --with-jpeg (but --with-jpeg is still used to set the include path.) The addition of JPEG_LDFLAGS necessitated replacing AC_CHECK_LIB with AC_LINK_IFELSE (because AC_CHECK_LIB automatically sets LIBS to -ljpeg, which is not what we want if we're, for instance, linking statically with libjpeg-turbo.) -- configure does not check for PNG support if TurboVNC encoding is enabled. This prevents the rfbSendRectEncodingTightPng() function from being compiled in, since the TurboVNC encoder doesn't (and can't) support it. common/turbojpeg.c, common/turbojpeg.h -- TurboJPEG is a simple API used to compress and decompress JPEG images in memory. It was originally implemented because it was desirable to use different types of underlying technologies to compress JPEG on different platforms (mediaLib on SPARC, Quicktime on PPC Macs, Intel Performance Primitives, etc.) These days, however, libjpeg-turbo is the only underlying technology used by TurboVNC, so TurboJPEG's purpose is largely just code simplicity and flexibility. Thus, since there is no real need for libvncserver to use any technology other than libjpeg-turbo for compressing JPEG, the TurboJPEG wrapper for libjpeg-turbo has been included in-tree so that libvncserver can be directly linked with libjpeg-turbo. This is convenient because many modern Linux distros (Fedora, Ubuntu, etc.) now ship libjpeg-turbo as their default libjpeg library. libvncserver/rfbserver.c -- Added logic to check for the TurboVNC fine-grained quality level and subsampling encodings and to map Tight (0-9) quality levels to appropriate fine-grained quality level and subsampling values if communicating with a TightVNC/TigerVNC viewer. libvncserver/turbo.c -- TurboVNC encoder (compiled instead of libvncserver/tight.c) rfb/rfb.h -- Added support for the TurboVNC subsampling level rfb/rfbproto.h -- Added constants for the TurboVNC fine quality level and subsampling encodings as well as the rfbTightNoZlib constant and notes on its usage.
13 years ago
bailout:
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
if(cinfo->global_state>CSTATE_START) jpeg_abort_compress(cinfo);
#ifndef JCS_EXTENSIONS
if(rgbBuf) free(rgbBuf);
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
#endif
Add TurboVNC encoding support. TurboVNC is a variant of TightVNC that uses the same client/server protocol (RFB version 3.8t), and thus it is fully cross-compatible with TightVNC and TigerVNC (with one exception, which is noted below.) Both the TightVNC and TurboVNC encoders analyze each rectangle, pick out regions of solid color to send separately, and send the remaining subrectangles using mono, indexed color, JPEG, or raw encoding, depending on the number of colors in the subrectangle. However, TurboVNC uses a fundamentally different selection algorithm to determine the appropriate subencoding to use for each subrectangle. Thus, while it sends a protocol stream that can be decoded by any TightVNC-compatible viewer, the mix of subencoding types in this protocol stream will be different from those generated by a TightVNC server. The research that led to TurboVNC is described in the following report: http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf. In summary: 20 RFB captures, representing "common" 2D and 3D application workloads (the 3D workloads were run using VirtualGL), were studied using the TightVNC encoder in isolation. Some of the analysis features in the TightVNC encoder, such as smoothness detection, were found to generate a lot of CPU usage with little or no benefit in compression, so those features were disabled. JPEG encoding was accelerated using libjpeg-turbo (which achieves a 2-4x speedup over plain libjpeg on modern x86 or ARM processors.) Finally, the "palette threshold" (minimum number of colors that the subrectangle must have before it is compressed using JPEG or raw) was adjusted to account for the fact that JPEG encoding is now quite a bit faster (meaning that we can now use it more without a CPU penalty.) TurboVNC has additional optimizations, such as the ability to count colors and encode JPEG images directly from the framebuffer without first translating the pixels into RGB. The TurboVNC encoder compares quite favorably in terms of compression ratio with TightVNC and generally encodes a great deal faster (often an order of magnitude or more.) The version of the TurboVNC encoder included in this patch is roughly equivalent to the one found in version 0.6 of the Unix TurboVNC Server, with a few minor patches integrated from TurboVNC 1.1. TurboVNC 1.0 added multi-threading capabilities, which can be added in later if desired (at the expense of making libvncserver depend on libpthread.) Because TurboVNC uses a fundamentally different mix of subencodings than TightVNC, because it uses the identical protocol (and thus a viewer really has no idea whether it's talking to a TightVNC or TurboVNC server), and because it doesn't support rfbTightPng (and in fact conflicts with it-- see below), the TurboVNC and TightVNC encoders cannot be enabled simultaneously. Compatibility: In *most* cases, a TurboVNC-enabled viewer is fully compatible with a TightVNC server, and vice versa. TurboVNC supports pseudo-encodings for specifying a fine-grained (1-100) quality scale and specifying chrominance subsampling. If a TurboVNC viewer sends those to a TightVNC server, then the TightVNC server ignores them, so the TurboVNC viewer also sends the quality on a 0-9 scale that the TightVNC server can understand. Similarly, the TurboVNC server checks first for fine-grained quality and subsampling pseudo-encodings from the viewer, and failing to receive those, it then checks for the TightVNC 0-9 quality pseudo-encoding. There is one case in which the two systems are not compatible, and that is when a TightVNC or TigerVNC viewer requests compression level 0 without JPEG from a TurboVNC server. For performance reasons, this causes the TurboVNC server to send images directly to the viewer, bypassing Zlib. When the TurboVNC server does this, it also sets bits 7-4 in the compression control byte to rfbTightNoZlib (0x0A), which is unfortunately the same value as rfbTightPng. Older TightVNC viewers that don't handle PNG will assume that the stream is uncompressed but still encapsulated in a Zlib structure, whereas newer PNG-supporting TightVNC viewers will assume that the stream is PNG. In either case, the viewer will probably crash. Since most VNC viewers don't expose compression level 0 in the GUI, this is a relatively rare situation. Description of changes: configure.ac -- Added support for libjpeg-turbo. If passed an argument of --with-turbovnc, configure will now run (or, if cross-compiling, just link) a test program that determines whether the libjpeg library being used is libjpeg-turbo. libjpeg-turbo must be used when building the TurboVNC encoder, because the TurboVNC encoder relies on the libjpeg-turbo colorspace extensions in order to compress images directly out of the framebuffer (which may be, for instance, BGRA rather than RGB.) libjpeg-turbo can optionally be used with the TightVNC encoder as well, but the speedup will only be marginal (the report linked above explains why in more detail, but basically it's because of Amdahl's Law. The TightVNC encoder was designed with the assumption that JPEG had a very high CPU cost, and thus JPEG is used only sparingly.) -- Added a new configure variable, JPEG_LDFLAGS. This is necessitated by the fact that libjpeg-turbo often distributes libjpeg.a and libjpeg.so in /opt/libjpeg-turbo/lib32 or /opt/libjpeg-turbo/lib64, and many people prefer to statically link with it. Thus, more flexibility is needed than is provided by --with-jpeg. If JPEG_LDFLAGS is specified, then it overrides the changes to LDFLAGS enacted by --with-jpeg (but --with-jpeg is still used to set the include path.) The addition of JPEG_LDFLAGS necessitated replacing AC_CHECK_LIB with AC_LINK_IFELSE (because AC_CHECK_LIB automatically sets LIBS to -ljpeg, which is not what we want if we're, for instance, linking statically with libjpeg-turbo.) -- configure does not check for PNG support if TurboVNC encoding is enabled. This prevents the rfbSendRectEncodingTightPng() function from being compiled in, since the TurboVNC encoder doesn't (and can't) support it. common/turbojpeg.c, common/turbojpeg.h -- TurboJPEG is a simple API used to compress and decompress JPEG images in memory. It was originally implemented because it was desirable to use different types of underlying technologies to compress JPEG on different platforms (mediaLib on SPARC, Quicktime on PPC Macs, Intel Performance Primitives, etc.) These days, however, libjpeg-turbo is the only underlying technology used by TurboVNC, so TurboJPEG's purpose is largely just code simplicity and flexibility. Thus, since there is no real need for libvncserver to use any technology other than libjpeg-turbo for compressing JPEG, the TurboJPEG wrapper for libjpeg-turbo has been included in-tree so that libvncserver can be directly linked with libjpeg-turbo. This is convenient because many modern Linux distros (Fedora, Ubuntu, etc.) now ship libjpeg-turbo as their default libjpeg library. libvncserver/rfbserver.c -- Added logic to check for the TurboVNC fine-grained quality level and subsampling encodings and to map Tight (0-9) quality levels to appropriate fine-grained quality level and subsampling values if communicating with a TightVNC/TigerVNC viewer. libvncserver/turbo.c -- TurboVNC encoder (compiled instead of libvncserver/tight.c) rfb/rfb.h -- Added support for the TurboVNC subsampling level rfb/rfbproto.h -- Added constants for the TurboVNC fine quality level and subsampling encodings as well as the rfbTightNoZlib constant and notes on its usage.
13 years ago
if(row_pointer) free(row_pointer);
return retval;
}
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
DLLEXPORT int DLLCALL tjCompress(tjhandle handle, unsigned char *srcBuf,
int width, int pitch, int height, int pixelSize, unsigned char *jpegBuf,
unsigned long *jpegSize, int jpegSubsamp, int jpegQual, int flags)
{
int retval=0; unsigned long size;
retval=tjCompress2(handle, srcBuf, width, pitch, height,
getPixelFormat(pixelSize, flags), &jpegBuf, &size, jpegSubsamp, jpegQual,
flags);
*jpegSize=size;
return retval;
}
Add TurboVNC encoding support. TurboVNC is a variant of TightVNC that uses the same client/server protocol (RFB version 3.8t), and thus it is fully cross-compatible with TightVNC and TigerVNC (with one exception, which is noted below.) Both the TightVNC and TurboVNC encoders analyze each rectangle, pick out regions of solid color to send separately, and send the remaining subrectangles using mono, indexed color, JPEG, or raw encoding, depending on the number of colors in the subrectangle. However, TurboVNC uses a fundamentally different selection algorithm to determine the appropriate subencoding to use for each subrectangle. Thus, while it sends a protocol stream that can be decoded by any TightVNC-compatible viewer, the mix of subencoding types in this protocol stream will be different from those generated by a TightVNC server. The research that led to TurboVNC is described in the following report: http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf. In summary: 20 RFB captures, representing "common" 2D and 3D application workloads (the 3D workloads were run using VirtualGL), were studied using the TightVNC encoder in isolation. Some of the analysis features in the TightVNC encoder, such as smoothness detection, were found to generate a lot of CPU usage with little or no benefit in compression, so those features were disabled. JPEG encoding was accelerated using libjpeg-turbo (which achieves a 2-4x speedup over plain libjpeg on modern x86 or ARM processors.) Finally, the "palette threshold" (minimum number of colors that the subrectangle must have before it is compressed using JPEG or raw) was adjusted to account for the fact that JPEG encoding is now quite a bit faster (meaning that we can now use it more without a CPU penalty.) TurboVNC has additional optimizations, such as the ability to count colors and encode JPEG images directly from the framebuffer without first translating the pixels into RGB. The TurboVNC encoder compares quite favorably in terms of compression ratio with TightVNC and generally encodes a great deal faster (often an order of magnitude or more.) The version of the TurboVNC encoder included in this patch is roughly equivalent to the one found in version 0.6 of the Unix TurboVNC Server, with a few minor patches integrated from TurboVNC 1.1. TurboVNC 1.0 added multi-threading capabilities, which can be added in later if desired (at the expense of making libvncserver depend on libpthread.) Because TurboVNC uses a fundamentally different mix of subencodings than TightVNC, because it uses the identical protocol (and thus a viewer really has no idea whether it's talking to a TightVNC or TurboVNC server), and because it doesn't support rfbTightPng (and in fact conflicts with it-- see below), the TurboVNC and TightVNC encoders cannot be enabled simultaneously. Compatibility: In *most* cases, a TurboVNC-enabled viewer is fully compatible with a TightVNC server, and vice versa. TurboVNC supports pseudo-encodings for specifying a fine-grained (1-100) quality scale and specifying chrominance subsampling. If a TurboVNC viewer sends those to a TightVNC server, then the TightVNC server ignores them, so the TurboVNC viewer also sends the quality on a 0-9 scale that the TightVNC server can understand. Similarly, the TurboVNC server checks first for fine-grained quality and subsampling pseudo-encodings from the viewer, and failing to receive those, it then checks for the TightVNC 0-9 quality pseudo-encoding. There is one case in which the two systems are not compatible, and that is when a TightVNC or TigerVNC viewer requests compression level 0 without JPEG from a TurboVNC server. For performance reasons, this causes the TurboVNC server to send images directly to the viewer, bypassing Zlib. When the TurboVNC server does this, it also sets bits 7-4 in the compression control byte to rfbTightNoZlib (0x0A), which is unfortunately the same value as rfbTightPng. Older TightVNC viewers that don't handle PNG will assume that the stream is uncompressed but still encapsulated in a Zlib structure, whereas newer PNG-supporting TightVNC viewers will assume that the stream is PNG. In either case, the viewer will probably crash. Since most VNC viewers don't expose compression level 0 in the GUI, this is a relatively rare situation. Description of changes: configure.ac -- Added support for libjpeg-turbo. If passed an argument of --with-turbovnc, configure will now run (or, if cross-compiling, just link) a test program that determines whether the libjpeg library being used is libjpeg-turbo. libjpeg-turbo must be used when building the TurboVNC encoder, because the TurboVNC encoder relies on the libjpeg-turbo colorspace extensions in order to compress images directly out of the framebuffer (which may be, for instance, BGRA rather than RGB.) libjpeg-turbo can optionally be used with the TightVNC encoder as well, but the speedup will only be marginal (the report linked above explains why in more detail, but basically it's because of Amdahl's Law. The TightVNC encoder was designed with the assumption that JPEG had a very high CPU cost, and thus JPEG is used only sparingly.) -- Added a new configure variable, JPEG_LDFLAGS. This is necessitated by the fact that libjpeg-turbo often distributes libjpeg.a and libjpeg.so in /opt/libjpeg-turbo/lib32 or /opt/libjpeg-turbo/lib64, and many people prefer to statically link with it. Thus, more flexibility is needed than is provided by --with-jpeg. If JPEG_LDFLAGS is specified, then it overrides the changes to LDFLAGS enacted by --with-jpeg (but --with-jpeg is still used to set the include path.) The addition of JPEG_LDFLAGS necessitated replacing AC_CHECK_LIB with AC_LINK_IFELSE (because AC_CHECK_LIB automatically sets LIBS to -ljpeg, which is not what we want if we're, for instance, linking statically with libjpeg-turbo.) -- configure does not check for PNG support if TurboVNC encoding is enabled. This prevents the rfbSendRectEncodingTightPng() function from being compiled in, since the TurboVNC encoder doesn't (and can't) support it. common/turbojpeg.c, common/turbojpeg.h -- TurboJPEG is a simple API used to compress and decompress JPEG images in memory. It was originally implemented because it was desirable to use different types of underlying technologies to compress JPEG on different platforms (mediaLib on SPARC, Quicktime on PPC Macs, Intel Performance Primitives, etc.) These days, however, libjpeg-turbo is the only underlying technology used by TurboVNC, so TurboJPEG's purpose is largely just code simplicity and flexibility. Thus, since there is no real need for libvncserver to use any technology other than libjpeg-turbo for compressing JPEG, the TurboJPEG wrapper for libjpeg-turbo has been included in-tree so that libvncserver can be directly linked with libjpeg-turbo. This is convenient because many modern Linux distros (Fedora, Ubuntu, etc.) now ship libjpeg-turbo as their default libjpeg library. libvncserver/rfbserver.c -- Added logic to check for the TurboVNC fine-grained quality level and subsampling encodings and to map Tight (0-9) quality levels to appropriate fine-grained quality level and subsampling values if communicating with a TightVNC/TigerVNC viewer. libvncserver/turbo.c -- TurboVNC encoder (compiled instead of libvncserver/tight.c) rfb/rfb.h -- Added support for the TurboVNC subsampling level rfb/rfbproto.h -- Added constants for the TurboVNC fine quality level and subsampling encodings as well as the rfbTightNoZlib constant and notes on its usage.
13 years ago
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
/* Decompressor */
Add TurboVNC encoding support. TurboVNC is a variant of TightVNC that uses the same client/server protocol (RFB version 3.8t), and thus it is fully cross-compatible with TightVNC and TigerVNC (with one exception, which is noted below.) Both the TightVNC and TurboVNC encoders analyze each rectangle, pick out regions of solid color to send separately, and send the remaining subrectangles using mono, indexed color, JPEG, or raw encoding, depending on the number of colors in the subrectangle. However, TurboVNC uses a fundamentally different selection algorithm to determine the appropriate subencoding to use for each subrectangle. Thus, while it sends a protocol stream that can be decoded by any TightVNC-compatible viewer, the mix of subencoding types in this protocol stream will be different from those generated by a TightVNC server. The research that led to TurboVNC is described in the following report: http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf. In summary: 20 RFB captures, representing "common" 2D and 3D application workloads (the 3D workloads were run using VirtualGL), were studied using the TightVNC encoder in isolation. Some of the analysis features in the TightVNC encoder, such as smoothness detection, were found to generate a lot of CPU usage with little or no benefit in compression, so those features were disabled. JPEG encoding was accelerated using libjpeg-turbo (which achieves a 2-4x speedup over plain libjpeg on modern x86 or ARM processors.) Finally, the "palette threshold" (minimum number of colors that the subrectangle must have before it is compressed using JPEG or raw) was adjusted to account for the fact that JPEG encoding is now quite a bit faster (meaning that we can now use it more without a CPU penalty.) TurboVNC has additional optimizations, such as the ability to count colors and encode JPEG images directly from the framebuffer without first translating the pixels into RGB. The TurboVNC encoder compares quite favorably in terms of compression ratio with TightVNC and generally encodes a great deal faster (often an order of magnitude or more.) The version of the TurboVNC encoder included in this patch is roughly equivalent to the one found in version 0.6 of the Unix TurboVNC Server, with a few minor patches integrated from TurboVNC 1.1. TurboVNC 1.0 added multi-threading capabilities, which can be added in later if desired (at the expense of making libvncserver depend on libpthread.) Because TurboVNC uses a fundamentally different mix of subencodings than TightVNC, because it uses the identical protocol (and thus a viewer really has no idea whether it's talking to a TightVNC or TurboVNC server), and because it doesn't support rfbTightPng (and in fact conflicts with it-- see below), the TurboVNC and TightVNC encoders cannot be enabled simultaneously. Compatibility: In *most* cases, a TurboVNC-enabled viewer is fully compatible with a TightVNC server, and vice versa. TurboVNC supports pseudo-encodings for specifying a fine-grained (1-100) quality scale and specifying chrominance subsampling. If a TurboVNC viewer sends those to a TightVNC server, then the TightVNC server ignores them, so the TurboVNC viewer also sends the quality on a 0-9 scale that the TightVNC server can understand. Similarly, the TurboVNC server checks first for fine-grained quality and subsampling pseudo-encodings from the viewer, and failing to receive those, it then checks for the TightVNC 0-9 quality pseudo-encoding. There is one case in which the two systems are not compatible, and that is when a TightVNC or TigerVNC viewer requests compression level 0 without JPEG from a TurboVNC server. For performance reasons, this causes the TurboVNC server to send images directly to the viewer, bypassing Zlib. When the TurboVNC server does this, it also sets bits 7-4 in the compression control byte to rfbTightNoZlib (0x0A), which is unfortunately the same value as rfbTightPng. Older TightVNC viewers that don't handle PNG will assume that the stream is uncompressed but still encapsulated in a Zlib structure, whereas newer PNG-supporting TightVNC viewers will assume that the stream is PNG. In either case, the viewer will probably crash. Since most VNC viewers don't expose compression level 0 in the GUI, this is a relatively rare situation. Description of changes: configure.ac -- Added support for libjpeg-turbo. If passed an argument of --with-turbovnc, configure will now run (or, if cross-compiling, just link) a test program that determines whether the libjpeg library being used is libjpeg-turbo. libjpeg-turbo must be used when building the TurboVNC encoder, because the TurboVNC encoder relies on the libjpeg-turbo colorspace extensions in order to compress images directly out of the framebuffer (which may be, for instance, BGRA rather than RGB.) libjpeg-turbo can optionally be used with the TightVNC encoder as well, but the speedup will only be marginal (the report linked above explains why in more detail, but basically it's because of Amdahl's Law. The TightVNC encoder was designed with the assumption that JPEG had a very high CPU cost, and thus JPEG is used only sparingly.) -- Added a new configure variable, JPEG_LDFLAGS. This is necessitated by the fact that libjpeg-turbo often distributes libjpeg.a and libjpeg.so in /opt/libjpeg-turbo/lib32 or /opt/libjpeg-turbo/lib64, and many people prefer to statically link with it. Thus, more flexibility is needed than is provided by --with-jpeg. If JPEG_LDFLAGS is specified, then it overrides the changes to LDFLAGS enacted by --with-jpeg (but --with-jpeg is still used to set the include path.) The addition of JPEG_LDFLAGS necessitated replacing AC_CHECK_LIB with AC_LINK_IFELSE (because AC_CHECK_LIB automatically sets LIBS to -ljpeg, which is not what we want if we're, for instance, linking statically with libjpeg-turbo.) -- configure does not check for PNG support if TurboVNC encoding is enabled. This prevents the rfbSendRectEncodingTightPng() function from being compiled in, since the TurboVNC encoder doesn't (and can't) support it. common/turbojpeg.c, common/turbojpeg.h -- TurboJPEG is a simple API used to compress and decompress JPEG images in memory. It was originally implemented because it was desirable to use different types of underlying technologies to compress JPEG on different platforms (mediaLib on SPARC, Quicktime on PPC Macs, Intel Performance Primitives, etc.) These days, however, libjpeg-turbo is the only underlying technology used by TurboVNC, so TurboJPEG's purpose is largely just code simplicity and flexibility. Thus, since there is no real need for libvncserver to use any technology other than libjpeg-turbo for compressing JPEG, the TurboJPEG wrapper for libjpeg-turbo has been included in-tree so that libvncserver can be directly linked with libjpeg-turbo. This is convenient because many modern Linux distros (Fedora, Ubuntu, etc.) now ship libjpeg-turbo as their default libjpeg library. libvncserver/rfbserver.c -- Added logic to check for the TurboVNC fine-grained quality level and subsampling encodings and to map Tight (0-9) quality levels to appropriate fine-grained quality level and subsampling values if communicating with a TightVNC/TigerVNC viewer. libvncserver/turbo.c -- TurboVNC encoder (compiled instead of libvncserver/tight.c) rfb/rfb.h -- Added support for the TurboVNC subsampling level rfb/rfbproto.h -- Added constants for the TurboVNC fine quality level and subsampling encodings as well as the rfbTightNoZlib constant and notes on its usage.
13 years ago
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
static boolean fill_input_buffer(j_decompress_ptr dinfo)
Add TurboVNC encoding support. TurboVNC is a variant of TightVNC that uses the same client/server protocol (RFB version 3.8t), and thus it is fully cross-compatible with TightVNC and TigerVNC (with one exception, which is noted below.) Both the TightVNC and TurboVNC encoders analyze each rectangle, pick out regions of solid color to send separately, and send the remaining subrectangles using mono, indexed color, JPEG, or raw encoding, depending on the number of colors in the subrectangle. However, TurboVNC uses a fundamentally different selection algorithm to determine the appropriate subencoding to use for each subrectangle. Thus, while it sends a protocol stream that can be decoded by any TightVNC-compatible viewer, the mix of subencoding types in this protocol stream will be different from those generated by a TightVNC server. The research that led to TurboVNC is described in the following report: http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf. In summary: 20 RFB captures, representing "common" 2D and 3D application workloads (the 3D workloads were run using VirtualGL), were studied using the TightVNC encoder in isolation. Some of the analysis features in the TightVNC encoder, such as smoothness detection, were found to generate a lot of CPU usage with little or no benefit in compression, so those features were disabled. JPEG encoding was accelerated using libjpeg-turbo (which achieves a 2-4x speedup over plain libjpeg on modern x86 or ARM processors.) Finally, the "palette threshold" (minimum number of colors that the subrectangle must have before it is compressed using JPEG or raw) was adjusted to account for the fact that JPEG encoding is now quite a bit faster (meaning that we can now use it more without a CPU penalty.) TurboVNC has additional optimizations, such as the ability to count colors and encode JPEG images directly from the framebuffer without first translating the pixels into RGB. The TurboVNC encoder compares quite favorably in terms of compression ratio with TightVNC and generally encodes a great deal faster (often an order of magnitude or more.) The version of the TurboVNC encoder included in this patch is roughly equivalent to the one found in version 0.6 of the Unix TurboVNC Server, with a few minor patches integrated from TurboVNC 1.1. TurboVNC 1.0 added multi-threading capabilities, which can be added in later if desired (at the expense of making libvncserver depend on libpthread.) Because TurboVNC uses a fundamentally different mix of subencodings than TightVNC, because it uses the identical protocol (and thus a viewer really has no idea whether it's talking to a TightVNC or TurboVNC server), and because it doesn't support rfbTightPng (and in fact conflicts with it-- see below), the TurboVNC and TightVNC encoders cannot be enabled simultaneously. Compatibility: In *most* cases, a TurboVNC-enabled viewer is fully compatible with a TightVNC server, and vice versa. TurboVNC supports pseudo-encodings for specifying a fine-grained (1-100) quality scale and specifying chrominance subsampling. If a TurboVNC viewer sends those to a TightVNC server, then the TightVNC server ignores them, so the TurboVNC viewer also sends the quality on a 0-9 scale that the TightVNC server can understand. Similarly, the TurboVNC server checks first for fine-grained quality and subsampling pseudo-encodings from the viewer, and failing to receive those, it then checks for the TightVNC 0-9 quality pseudo-encoding. There is one case in which the two systems are not compatible, and that is when a TightVNC or TigerVNC viewer requests compression level 0 without JPEG from a TurboVNC server. For performance reasons, this causes the TurboVNC server to send images directly to the viewer, bypassing Zlib. When the TurboVNC server does this, it also sets bits 7-4 in the compression control byte to rfbTightNoZlib (0x0A), which is unfortunately the same value as rfbTightPng. Older TightVNC viewers that don't handle PNG will assume that the stream is uncompressed but still encapsulated in a Zlib structure, whereas newer PNG-supporting TightVNC viewers will assume that the stream is PNG. In either case, the viewer will probably crash. Since most VNC viewers don't expose compression level 0 in the GUI, this is a relatively rare situation. Description of changes: configure.ac -- Added support for libjpeg-turbo. If passed an argument of --with-turbovnc, configure will now run (or, if cross-compiling, just link) a test program that determines whether the libjpeg library being used is libjpeg-turbo. libjpeg-turbo must be used when building the TurboVNC encoder, because the TurboVNC encoder relies on the libjpeg-turbo colorspace extensions in order to compress images directly out of the framebuffer (which may be, for instance, BGRA rather than RGB.) libjpeg-turbo can optionally be used with the TightVNC encoder as well, but the speedup will only be marginal (the report linked above explains why in more detail, but basically it's because of Amdahl's Law. The TightVNC encoder was designed with the assumption that JPEG had a very high CPU cost, and thus JPEG is used only sparingly.) -- Added a new configure variable, JPEG_LDFLAGS. This is necessitated by the fact that libjpeg-turbo often distributes libjpeg.a and libjpeg.so in /opt/libjpeg-turbo/lib32 or /opt/libjpeg-turbo/lib64, and many people prefer to statically link with it. Thus, more flexibility is needed than is provided by --with-jpeg. If JPEG_LDFLAGS is specified, then it overrides the changes to LDFLAGS enacted by --with-jpeg (but --with-jpeg is still used to set the include path.) The addition of JPEG_LDFLAGS necessitated replacing AC_CHECK_LIB with AC_LINK_IFELSE (because AC_CHECK_LIB automatically sets LIBS to -ljpeg, which is not what we want if we're, for instance, linking statically with libjpeg-turbo.) -- configure does not check for PNG support if TurboVNC encoding is enabled. This prevents the rfbSendRectEncodingTightPng() function from being compiled in, since the TurboVNC encoder doesn't (and can't) support it. common/turbojpeg.c, common/turbojpeg.h -- TurboJPEG is a simple API used to compress and decompress JPEG images in memory. It was originally implemented because it was desirable to use different types of underlying technologies to compress JPEG on different platforms (mediaLib on SPARC, Quicktime on PPC Macs, Intel Performance Primitives, etc.) These days, however, libjpeg-turbo is the only underlying technology used by TurboVNC, so TurboJPEG's purpose is largely just code simplicity and flexibility. Thus, since there is no real need for libvncserver to use any technology other than libjpeg-turbo for compressing JPEG, the TurboJPEG wrapper for libjpeg-turbo has been included in-tree so that libvncserver can be directly linked with libjpeg-turbo. This is convenient because many modern Linux distros (Fedora, Ubuntu, etc.) now ship libjpeg-turbo as their default libjpeg library. libvncserver/rfbserver.c -- Added logic to check for the TurboVNC fine-grained quality level and subsampling encodings and to map Tight (0-9) quality levels to appropriate fine-grained quality level and subsampling values if communicating with a TightVNC/TigerVNC viewer. libvncserver/turbo.c -- TurboVNC encoder (compiled instead of libvncserver/tight.c) rfb/rfb.h -- Added support for the TurboVNC subsampling level rfb/rfbproto.h -- Added constants for the TurboVNC fine quality level and subsampling encodings as well as the rfbTightNoZlib constant and notes on its usage.
13 years ago
{
ERREXIT(dinfo, JERR_BUFFER_SIZE);
return TRUE;
}
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
static void skip_input_data(j_decompress_ptr dinfo, long num_bytes)
Add TurboVNC encoding support. TurboVNC is a variant of TightVNC that uses the same client/server protocol (RFB version 3.8t), and thus it is fully cross-compatible with TightVNC and TigerVNC (with one exception, which is noted below.) Both the TightVNC and TurboVNC encoders analyze each rectangle, pick out regions of solid color to send separately, and send the remaining subrectangles using mono, indexed color, JPEG, or raw encoding, depending on the number of colors in the subrectangle. However, TurboVNC uses a fundamentally different selection algorithm to determine the appropriate subencoding to use for each subrectangle. Thus, while it sends a protocol stream that can be decoded by any TightVNC-compatible viewer, the mix of subencoding types in this protocol stream will be different from those generated by a TightVNC server. The research that led to TurboVNC is described in the following report: http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf. In summary: 20 RFB captures, representing "common" 2D and 3D application workloads (the 3D workloads were run using VirtualGL), were studied using the TightVNC encoder in isolation. Some of the analysis features in the TightVNC encoder, such as smoothness detection, were found to generate a lot of CPU usage with little or no benefit in compression, so those features were disabled. JPEG encoding was accelerated using libjpeg-turbo (which achieves a 2-4x speedup over plain libjpeg on modern x86 or ARM processors.) Finally, the "palette threshold" (minimum number of colors that the subrectangle must have before it is compressed using JPEG or raw) was adjusted to account for the fact that JPEG encoding is now quite a bit faster (meaning that we can now use it more without a CPU penalty.) TurboVNC has additional optimizations, such as the ability to count colors and encode JPEG images directly from the framebuffer without first translating the pixels into RGB. The TurboVNC encoder compares quite favorably in terms of compression ratio with TightVNC and generally encodes a great deal faster (often an order of magnitude or more.) The version of the TurboVNC encoder included in this patch is roughly equivalent to the one found in version 0.6 of the Unix TurboVNC Server, with a few minor patches integrated from TurboVNC 1.1. TurboVNC 1.0 added multi-threading capabilities, which can be added in later if desired (at the expense of making libvncserver depend on libpthread.) Because TurboVNC uses a fundamentally different mix of subencodings than TightVNC, because it uses the identical protocol (and thus a viewer really has no idea whether it's talking to a TightVNC or TurboVNC server), and because it doesn't support rfbTightPng (and in fact conflicts with it-- see below), the TurboVNC and TightVNC encoders cannot be enabled simultaneously. Compatibility: In *most* cases, a TurboVNC-enabled viewer is fully compatible with a TightVNC server, and vice versa. TurboVNC supports pseudo-encodings for specifying a fine-grained (1-100) quality scale and specifying chrominance subsampling. If a TurboVNC viewer sends those to a TightVNC server, then the TightVNC server ignores them, so the TurboVNC viewer also sends the quality on a 0-9 scale that the TightVNC server can understand. Similarly, the TurboVNC server checks first for fine-grained quality and subsampling pseudo-encodings from the viewer, and failing to receive those, it then checks for the TightVNC 0-9 quality pseudo-encoding. There is one case in which the two systems are not compatible, and that is when a TightVNC or TigerVNC viewer requests compression level 0 without JPEG from a TurboVNC server. For performance reasons, this causes the TurboVNC server to send images directly to the viewer, bypassing Zlib. When the TurboVNC server does this, it also sets bits 7-4 in the compression control byte to rfbTightNoZlib (0x0A), which is unfortunately the same value as rfbTightPng. Older TightVNC viewers that don't handle PNG will assume that the stream is uncompressed but still encapsulated in a Zlib structure, whereas newer PNG-supporting TightVNC viewers will assume that the stream is PNG. In either case, the viewer will probably crash. Since most VNC viewers don't expose compression level 0 in the GUI, this is a relatively rare situation. Description of changes: configure.ac -- Added support for libjpeg-turbo. If passed an argument of --with-turbovnc, configure will now run (or, if cross-compiling, just link) a test program that determines whether the libjpeg library being used is libjpeg-turbo. libjpeg-turbo must be used when building the TurboVNC encoder, because the TurboVNC encoder relies on the libjpeg-turbo colorspace extensions in order to compress images directly out of the framebuffer (which may be, for instance, BGRA rather than RGB.) libjpeg-turbo can optionally be used with the TightVNC encoder as well, but the speedup will only be marginal (the report linked above explains why in more detail, but basically it's because of Amdahl's Law. The TightVNC encoder was designed with the assumption that JPEG had a very high CPU cost, and thus JPEG is used only sparingly.) -- Added a new configure variable, JPEG_LDFLAGS. This is necessitated by the fact that libjpeg-turbo often distributes libjpeg.a and libjpeg.so in /opt/libjpeg-turbo/lib32 or /opt/libjpeg-turbo/lib64, and many people prefer to statically link with it. Thus, more flexibility is needed than is provided by --with-jpeg. If JPEG_LDFLAGS is specified, then it overrides the changes to LDFLAGS enacted by --with-jpeg (but --with-jpeg is still used to set the include path.) The addition of JPEG_LDFLAGS necessitated replacing AC_CHECK_LIB with AC_LINK_IFELSE (because AC_CHECK_LIB automatically sets LIBS to -ljpeg, which is not what we want if we're, for instance, linking statically with libjpeg-turbo.) -- configure does not check for PNG support if TurboVNC encoding is enabled. This prevents the rfbSendRectEncodingTightPng() function from being compiled in, since the TurboVNC encoder doesn't (and can't) support it. common/turbojpeg.c, common/turbojpeg.h -- TurboJPEG is a simple API used to compress and decompress JPEG images in memory. It was originally implemented because it was desirable to use different types of underlying technologies to compress JPEG on different platforms (mediaLib on SPARC, Quicktime on PPC Macs, Intel Performance Primitives, etc.) These days, however, libjpeg-turbo is the only underlying technology used by TurboVNC, so TurboJPEG's purpose is largely just code simplicity and flexibility. Thus, since there is no real need for libvncserver to use any technology other than libjpeg-turbo for compressing JPEG, the TurboJPEG wrapper for libjpeg-turbo has been included in-tree so that libvncserver can be directly linked with libjpeg-turbo. This is convenient because many modern Linux distros (Fedora, Ubuntu, etc.) now ship libjpeg-turbo as their default libjpeg library. libvncserver/rfbserver.c -- Added logic to check for the TurboVNC fine-grained quality level and subsampling encodings and to map Tight (0-9) quality levels to appropriate fine-grained quality level and subsampling values if communicating with a TightVNC/TigerVNC viewer. libvncserver/turbo.c -- TurboVNC encoder (compiled instead of libvncserver/tight.c) rfb/rfb.h -- Added support for the TurboVNC subsampling level rfb/rfbproto.h -- Added constants for the TurboVNC fine quality level and subsampling encodings as well as the rfbTightNoZlib constant and notes on its usage.
13 years ago
{
dinfo->src->next_input_byte += (size_t) num_bytes;
dinfo->src->bytes_in_buffer -= (size_t) num_bytes;
}
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
static void src_noop(j_decompress_ptr dinfo)
Add TurboVNC encoding support. TurboVNC is a variant of TightVNC that uses the same client/server protocol (RFB version 3.8t), and thus it is fully cross-compatible with TightVNC and TigerVNC (with one exception, which is noted below.) Both the TightVNC and TurboVNC encoders analyze each rectangle, pick out regions of solid color to send separately, and send the remaining subrectangles using mono, indexed color, JPEG, or raw encoding, depending on the number of colors in the subrectangle. However, TurboVNC uses a fundamentally different selection algorithm to determine the appropriate subencoding to use for each subrectangle. Thus, while it sends a protocol stream that can be decoded by any TightVNC-compatible viewer, the mix of subencoding types in this protocol stream will be different from those generated by a TightVNC server. The research that led to TurboVNC is described in the following report: http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf. In summary: 20 RFB captures, representing "common" 2D and 3D application workloads (the 3D workloads were run using VirtualGL), were studied using the TightVNC encoder in isolation. Some of the analysis features in the TightVNC encoder, such as smoothness detection, were found to generate a lot of CPU usage with little or no benefit in compression, so those features were disabled. JPEG encoding was accelerated using libjpeg-turbo (which achieves a 2-4x speedup over plain libjpeg on modern x86 or ARM processors.) Finally, the "palette threshold" (minimum number of colors that the subrectangle must have before it is compressed using JPEG or raw) was adjusted to account for the fact that JPEG encoding is now quite a bit faster (meaning that we can now use it more without a CPU penalty.) TurboVNC has additional optimizations, such as the ability to count colors and encode JPEG images directly from the framebuffer without first translating the pixels into RGB. The TurboVNC encoder compares quite favorably in terms of compression ratio with TightVNC and generally encodes a great deal faster (often an order of magnitude or more.) The version of the TurboVNC encoder included in this patch is roughly equivalent to the one found in version 0.6 of the Unix TurboVNC Server, with a few minor patches integrated from TurboVNC 1.1. TurboVNC 1.0 added multi-threading capabilities, which can be added in later if desired (at the expense of making libvncserver depend on libpthread.) Because TurboVNC uses a fundamentally different mix of subencodings than TightVNC, because it uses the identical protocol (and thus a viewer really has no idea whether it's talking to a TightVNC or TurboVNC server), and because it doesn't support rfbTightPng (and in fact conflicts with it-- see below), the TurboVNC and TightVNC encoders cannot be enabled simultaneously. Compatibility: In *most* cases, a TurboVNC-enabled viewer is fully compatible with a TightVNC server, and vice versa. TurboVNC supports pseudo-encodings for specifying a fine-grained (1-100) quality scale and specifying chrominance subsampling. If a TurboVNC viewer sends those to a TightVNC server, then the TightVNC server ignores them, so the TurboVNC viewer also sends the quality on a 0-9 scale that the TightVNC server can understand. Similarly, the TurboVNC server checks first for fine-grained quality and subsampling pseudo-encodings from the viewer, and failing to receive those, it then checks for the TightVNC 0-9 quality pseudo-encoding. There is one case in which the two systems are not compatible, and that is when a TightVNC or TigerVNC viewer requests compression level 0 without JPEG from a TurboVNC server. For performance reasons, this causes the TurboVNC server to send images directly to the viewer, bypassing Zlib. When the TurboVNC server does this, it also sets bits 7-4 in the compression control byte to rfbTightNoZlib (0x0A), which is unfortunately the same value as rfbTightPng. Older TightVNC viewers that don't handle PNG will assume that the stream is uncompressed but still encapsulated in a Zlib structure, whereas newer PNG-supporting TightVNC viewers will assume that the stream is PNG. In either case, the viewer will probably crash. Since most VNC viewers don't expose compression level 0 in the GUI, this is a relatively rare situation. Description of changes: configure.ac -- Added support for libjpeg-turbo. If passed an argument of --with-turbovnc, configure will now run (or, if cross-compiling, just link) a test program that determines whether the libjpeg library being used is libjpeg-turbo. libjpeg-turbo must be used when building the TurboVNC encoder, because the TurboVNC encoder relies on the libjpeg-turbo colorspace extensions in order to compress images directly out of the framebuffer (which may be, for instance, BGRA rather than RGB.) libjpeg-turbo can optionally be used with the TightVNC encoder as well, but the speedup will only be marginal (the report linked above explains why in more detail, but basically it's because of Amdahl's Law. The TightVNC encoder was designed with the assumption that JPEG had a very high CPU cost, and thus JPEG is used only sparingly.) -- Added a new configure variable, JPEG_LDFLAGS. This is necessitated by the fact that libjpeg-turbo often distributes libjpeg.a and libjpeg.so in /opt/libjpeg-turbo/lib32 or /opt/libjpeg-turbo/lib64, and many people prefer to statically link with it. Thus, more flexibility is needed than is provided by --with-jpeg. If JPEG_LDFLAGS is specified, then it overrides the changes to LDFLAGS enacted by --with-jpeg (but --with-jpeg is still used to set the include path.) The addition of JPEG_LDFLAGS necessitated replacing AC_CHECK_LIB with AC_LINK_IFELSE (because AC_CHECK_LIB automatically sets LIBS to -ljpeg, which is not what we want if we're, for instance, linking statically with libjpeg-turbo.) -- configure does not check for PNG support if TurboVNC encoding is enabled. This prevents the rfbSendRectEncodingTightPng() function from being compiled in, since the TurboVNC encoder doesn't (and can't) support it. common/turbojpeg.c, common/turbojpeg.h -- TurboJPEG is a simple API used to compress and decompress JPEG images in memory. It was originally implemented because it was desirable to use different types of underlying technologies to compress JPEG on different platforms (mediaLib on SPARC, Quicktime on PPC Macs, Intel Performance Primitives, etc.) These days, however, libjpeg-turbo is the only underlying technology used by TurboVNC, so TurboJPEG's purpose is largely just code simplicity and flexibility. Thus, since there is no real need for libvncserver to use any technology other than libjpeg-turbo for compressing JPEG, the TurboJPEG wrapper for libjpeg-turbo has been included in-tree so that libvncserver can be directly linked with libjpeg-turbo. This is convenient because many modern Linux distros (Fedora, Ubuntu, etc.) now ship libjpeg-turbo as their default libjpeg library. libvncserver/rfbserver.c -- Added logic to check for the TurboVNC fine-grained quality level and subsampling encodings and to map Tight (0-9) quality levels to appropriate fine-grained quality level and subsampling values if communicating with a TightVNC/TigerVNC viewer. libvncserver/turbo.c -- TurboVNC encoder (compiled instead of libvncserver/tight.c) rfb/rfb.h -- Added support for the TurboVNC subsampling level rfb/rfbproto.h -- Added constants for the TurboVNC fine quality level and subsampling encodings as well as the rfbTightNoZlib constant and notes on its usage.
13 years ago
{
}
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
static tjhandle _tjInitDecompress(tjinstance *this)
Add TurboVNC encoding support. TurboVNC is a variant of TightVNC that uses the same client/server protocol (RFB version 3.8t), and thus it is fully cross-compatible with TightVNC and TigerVNC (with one exception, which is noted below.) Both the TightVNC and TurboVNC encoders analyze each rectangle, pick out regions of solid color to send separately, and send the remaining subrectangles using mono, indexed color, JPEG, or raw encoding, depending on the number of colors in the subrectangle. However, TurboVNC uses a fundamentally different selection algorithm to determine the appropriate subencoding to use for each subrectangle. Thus, while it sends a protocol stream that can be decoded by any TightVNC-compatible viewer, the mix of subencoding types in this protocol stream will be different from those generated by a TightVNC server. The research that led to TurboVNC is described in the following report: http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf. In summary: 20 RFB captures, representing "common" 2D and 3D application workloads (the 3D workloads were run using VirtualGL), were studied using the TightVNC encoder in isolation. Some of the analysis features in the TightVNC encoder, such as smoothness detection, were found to generate a lot of CPU usage with little or no benefit in compression, so those features were disabled. JPEG encoding was accelerated using libjpeg-turbo (which achieves a 2-4x speedup over plain libjpeg on modern x86 or ARM processors.) Finally, the "palette threshold" (minimum number of colors that the subrectangle must have before it is compressed using JPEG or raw) was adjusted to account for the fact that JPEG encoding is now quite a bit faster (meaning that we can now use it more without a CPU penalty.) TurboVNC has additional optimizations, such as the ability to count colors and encode JPEG images directly from the framebuffer without first translating the pixels into RGB. The TurboVNC encoder compares quite favorably in terms of compression ratio with TightVNC and generally encodes a great deal faster (often an order of magnitude or more.) The version of the TurboVNC encoder included in this patch is roughly equivalent to the one found in version 0.6 of the Unix TurboVNC Server, with a few minor patches integrated from TurboVNC 1.1. TurboVNC 1.0 added multi-threading capabilities, which can be added in later if desired (at the expense of making libvncserver depend on libpthread.) Because TurboVNC uses a fundamentally different mix of subencodings than TightVNC, because it uses the identical protocol (and thus a viewer really has no idea whether it's talking to a TightVNC or TurboVNC server), and because it doesn't support rfbTightPng (and in fact conflicts with it-- see below), the TurboVNC and TightVNC encoders cannot be enabled simultaneously. Compatibility: In *most* cases, a TurboVNC-enabled viewer is fully compatible with a TightVNC server, and vice versa. TurboVNC supports pseudo-encodings for specifying a fine-grained (1-100) quality scale and specifying chrominance subsampling. If a TurboVNC viewer sends those to a TightVNC server, then the TightVNC server ignores them, so the TurboVNC viewer also sends the quality on a 0-9 scale that the TightVNC server can understand. Similarly, the TurboVNC server checks first for fine-grained quality and subsampling pseudo-encodings from the viewer, and failing to receive those, it then checks for the TightVNC 0-9 quality pseudo-encoding. There is one case in which the two systems are not compatible, and that is when a TightVNC or TigerVNC viewer requests compression level 0 without JPEG from a TurboVNC server. For performance reasons, this causes the TurboVNC server to send images directly to the viewer, bypassing Zlib. When the TurboVNC server does this, it also sets bits 7-4 in the compression control byte to rfbTightNoZlib (0x0A), which is unfortunately the same value as rfbTightPng. Older TightVNC viewers that don't handle PNG will assume that the stream is uncompressed but still encapsulated in a Zlib structure, whereas newer PNG-supporting TightVNC viewers will assume that the stream is PNG. In either case, the viewer will probably crash. Since most VNC viewers don't expose compression level 0 in the GUI, this is a relatively rare situation. Description of changes: configure.ac -- Added support for libjpeg-turbo. If passed an argument of --with-turbovnc, configure will now run (or, if cross-compiling, just link) a test program that determines whether the libjpeg library being used is libjpeg-turbo. libjpeg-turbo must be used when building the TurboVNC encoder, because the TurboVNC encoder relies on the libjpeg-turbo colorspace extensions in order to compress images directly out of the framebuffer (which may be, for instance, BGRA rather than RGB.) libjpeg-turbo can optionally be used with the TightVNC encoder as well, but the speedup will only be marginal (the report linked above explains why in more detail, but basically it's because of Amdahl's Law. The TightVNC encoder was designed with the assumption that JPEG had a very high CPU cost, and thus JPEG is used only sparingly.) -- Added a new configure variable, JPEG_LDFLAGS. This is necessitated by the fact that libjpeg-turbo often distributes libjpeg.a and libjpeg.so in /opt/libjpeg-turbo/lib32 or /opt/libjpeg-turbo/lib64, and many people prefer to statically link with it. Thus, more flexibility is needed than is provided by --with-jpeg. If JPEG_LDFLAGS is specified, then it overrides the changes to LDFLAGS enacted by --with-jpeg (but --with-jpeg is still used to set the include path.) The addition of JPEG_LDFLAGS necessitated replacing AC_CHECK_LIB with AC_LINK_IFELSE (because AC_CHECK_LIB automatically sets LIBS to -ljpeg, which is not what we want if we're, for instance, linking statically with libjpeg-turbo.) -- configure does not check for PNG support if TurboVNC encoding is enabled. This prevents the rfbSendRectEncodingTightPng() function from being compiled in, since the TurboVNC encoder doesn't (and can't) support it. common/turbojpeg.c, common/turbojpeg.h -- TurboJPEG is a simple API used to compress and decompress JPEG images in memory. It was originally implemented because it was desirable to use different types of underlying technologies to compress JPEG on different platforms (mediaLib on SPARC, Quicktime on PPC Macs, Intel Performance Primitives, etc.) These days, however, libjpeg-turbo is the only underlying technology used by TurboVNC, so TurboJPEG's purpose is largely just code simplicity and flexibility. Thus, since there is no real need for libvncserver to use any technology other than libjpeg-turbo for compressing JPEG, the TurboJPEG wrapper for libjpeg-turbo has been included in-tree so that libvncserver can be directly linked with libjpeg-turbo. This is convenient because many modern Linux distros (Fedora, Ubuntu, etc.) now ship libjpeg-turbo as their default libjpeg library. libvncserver/rfbserver.c -- Added logic to check for the TurboVNC fine-grained quality level and subsampling encodings and to map Tight (0-9) quality levels to appropriate fine-grained quality level and subsampling values if communicating with a TightVNC/TigerVNC viewer. libvncserver/turbo.c -- TurboVNC encoder (compiled instead of libvncserver/tight.c) rfb/rfb.h -- Added support for the TurboVNC subsampling level rfb/rfbproto.h -- Added constants for the TurboVNC fine quality level and subsampling encodings as well as the rfbTightNoZlib constant and notes on its usage.
13 years ago
{
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
/* This is also straight out of example.c */
this->dinfo.err=jpeg_std_error(&this->jerr.pub);
this->jerr.pub.error_exit=my_error_exit;
this->jerr.pub.output_message=my_output_message;
if(setjmp(this->jerr.setjmp_buffer))
{
/* If we get here, the JPEG code has signaled an error. */
if(this) free(this); return NULL;
Add TurboVNC encoding support. TurboVNC is a variant of TightVNC that uses the same client/server protocol (RFB version 3.8t), and thus it is fully cross-compatible with TightVNC and TigerVNC (with one exception, which is noted below.) Both the TightVNC and TurboVNC encoders analyze each rectangle, pick out regions of solid color to send separately, and send the remaining subrectangles using mono, indexed color, JPEG, or raw encoding, depending on the number of colors in the subrectangle. However, TurboVNC uses a fundamentally different selection algorithm to determine the appropriate subencoding to use for each subrectangle. Thus, while it sends a protocol stream that can be decoded by any TightVNC-compatible viewer, the mix of subencoding types in this protocol stream will be different from those generated by a TightVNC server. The research that led to TurboVNC is described in the following report: http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf. In summary: 20 RFB captures, representing "common" 2D and 3D application workloads (the 3D workloads were run using VirtualGL), were studied using the TightVNC encoder in isolation. Some of the analysis features in the TightVNC encoder, such as smoothness detection, were found to generate a lot of CPU usage with little or no benefit in compression, so those features were disabled. JPEG encoding was accelerated using libjpeg-turbo (which achieves a 2-4x speedup over plain libjpeg on modern x86 or ARM processors.) Finally, the "palette threshold" (minimum number of colors that the subrectangle must have before it is compressed using JPEG or raw) was adjusted to account for the fact that JPEG encoding is now quite a bit faster (meaning that we can now use it more without a CPU penalty.) TurboVNC has additional optimizations, such as the ability to count colors and encode JPEG images directly from the framebuffer without first translating the pixels into RGB. The TurboVNC encoder compares quite favorably in terms of compression ratio with TightVNC and generally encodes a great deal faster (often an order of magnitude or more.) The version of the TurboVNC encoder included in this patch is roughly equivalent to the one found in version 0.6 of the Unix TurboVNC Server, with a few minor patches integrated from TurboVNC 1.1. TurboVNC 1.0 added multi-threading capabilities, which can be added in later if desired (at the expense of making libvncserver depend on libpthread.) Because TurboVNC uses a fundamentally different mix of subencodings than TightVNC, because it uses the identical protocol (and thus a viewer really has no idea whether it's talking to a TightVNC or TurboVNC server), and because it doesn't support rfbTightPng (and in fact conflicts with it-- see below), the TurboVNC and TightVNC encoders cannot be enabled simultaneously. Compatibility: In *most* cases, a TurboVNC-enabled viewer is fully compatible with a TightVNC server, and vice versa. TurboVNC supports pseudo-encodings for specifying a fine-grained (1-100) quality scale and specifying chrominance subsampling. If a TurboVNC viewer sends those to a TightVNC server, then the TightVNC server ignores them, so the TurboVNC viewer also sends the quality on a 0-9 scale that the TightVNC server can understand. Similarly, the TurboVNC server checks first for fine-grained quality and subsampling pseudo-encodings from the viewer, and failing to receive those, it then checks for the TightVNC 0-9 quality pseudo-encoding. There is one case in which the two systems are not compatible, and that is when a TightVNC or TigerVNC viewer requests compression level 0 without JPEG from a TurboVNC server. For performance reasons, this causes the TurboVNC server to send images directly to the viewer, bypassing Zlib. When the TurboVNC server does this, it also sets bits 7-4 in the compression control byte to rfbTightNoZlib (0x0A), which is unfortunately the same value as rfbTightPng. Older TightVNC viewers that don't handle PNG will assume that the stream is uncompressed but still encapsulated in a Zlib structure, whereas newer PNG-supporting TightVNC viewers will assume that the stream is PNG. In either case, the viewer will probably crash. Since most VNC viewers don't expose compression level 0 in the GUI, this is a relatively rare situation. Description of changes: configure.ac -- Added support for libjpeg-turbo. If passed an argument of --with-turbovnc, configure will now run (or, if cross-compiling, just link) a test program that determines whether the libjpeg library being used is libjpeg-turbo. libjpeg-turbo must be used when building the TurboVNC encoder, because the TurboVNC encoder relies on the libjpeg-turbo colorspace extensions in order to compress images directly out of the framebuffer (which may be, for instance, BGRA rather than RGB.) libjpeg-turbo can optionally be used with the TightVNC encoder as well, but the speedup will only be marginal (the report linked above explains why in more detail, but basically it's because of Amdahl's Law. The TightVNC encoder was designed with the assumption that JPEG had a very high CPU cost, and thus JPEG is used only sparingly.) -- Added a new configure variable, JPEG_LDFLAGS. This is necessitated by the fact that libjpeg-turbo often distributes libjpeg.a and libjpeg.so in /opt/libjpeg-turbo/lib32 or /opt/libjpeg-turbo/lib64, and many people prefer to statically link with it. Thus, more flexibility is needed than is provided by --with-jpeg. If JPEG_LDFLAGS is specified, then it overrides the changes to LDFLAGS enacted by --with-jpeg (but --with-jpeg is still used to set the include path.) The addition of JPEG_LDFLAGS necessitated replacing AC_CHECK_LIB with AC_LINK_IFELSE (because AC_CHECK_LIB automatically sets LIBS to -ljpeg, which is not what we want if we're, for instance, linking statically with libjpeg-turbo.) -- configure does not check for PNG support if TurboVNC encoding is enabled. This prevents the rfbSendRectEncodingTightPng() function from being compiled in, since the TurboVNC encoder doesn't (and can't) support it. common/turbojpeg.c, common/turbojpeg.h -- TurboJPEG is a simple API used to compress and decompress JPEG images in memory. It was originally implemented because it was desirable to use different types of underlying technologies to compress JPEG on different platforms (mediaLib on SPARC, Quicktime on PPC Macs, Intel Performance Primitives, etc.) These days, however, libjpeg-turbo is the only underlying technology used by TurboVNC, so TurboJPEG's purpose is largely just code simplicity and flexibility. Thus, since there is no real need for libvncserver to use any technology other than libjpeg-turbo for compressing JPEG, the TurboJPEG wrapper for libjpeg-turbo has been included in-tree so that libvncserver can be directly linked with libjpeg-turbo. This is convenient because many modern Linux distros (Fedora, Ubuntu, etc.) now ship libjpeg-turbo as their default libjpeg library. libvncserver/rfbserver.c -- Added logic to check for the TurboVNC fine-grained quality level and subsampling encodings and to map Tight (0-9) quality levels to appropriate fine-grained quality level and subsampling values if communicating with a TightVNC/TigerVNC viewer. libvncserver/turbo.c -- TurboVNC encoder (compiled instead of libvncserver/tight.c) rfb/rfb.h -- Added support for the TurboVNC subsampling level rfb/rfbproto.h -- Added constants for the TurboVNC fine quality level and subsampling encodings as well as the rfbTightNoZlib constant and notes on its usage.
13 years ago
}
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
jpeg_create_decompress(&this->dinfo);
this->dinfo.src=&this->jsrc;
this->jsrc.init_source=src_noop;
this->jsrc.fill_input_buffer=fill_input_buffer;
this->jsrc.skip_input_data=skip_input_data;
this->jsrc.resync_to_restart=jpeg_resync_to_restart;
this->jsrc.term_source=src_noop;
this->init|=DECOMPRESS;
return (tjhandle)this;
}
Add TurboVNC encoding support. TurboVNC is a variant of TightVNC that uses the same client/server protocol (RFB version 3.8t), and thus it is fully cross-compatible with TightVNC and TigerVNC (with one exception, which is noted below.) Both the TightVNC and TurboVNC encoders analyze each rectangle, pick out regions of solid color to send separately, and send the remaining subrectangles using mono, indexed color, JPEG, or raw encoding, depending on the number of colors in the subrectangle. However, TurboVNC uses a fundamentally different selection algorithm to determine the appropriate subencoding to use for each subrectangle. Thus, while it sends a protocol stream that can be decoded by any TightVNC-compatible viewer, the mix of subencoding types in this protocol stream will be different from those generated by a TightVNC server. The research that led to TurboVNC is described in the following report: http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf. In summary: 20 RFB captures, representing "common" 2D and 3D application workloads (the 3D workloads were run using VirtualGL), were studied using the TightVNC encoder in isolation. Some of the analysis features in the TightVNC encoder, such as smoothness detection, were found to generate a lot of CPU usage with little or no benefit in compression, so those features were disabled. JPEG encoding was accelerated using libjpeg-turbo (which achieves a 2-4x speedup over plain libjpeg on modern x86 or ARM processors.) Finally, the "palette threshold" (minimum number of colors that the subrectangle must have before it is compressed using JPEG or raw) was adjusted to account for the fact that JPEG encoding is now quite a bit faster (meaning that we can now use it more without a CPU penalty.) TurboVNC has additional optimizations, such as the ability to count colors and encode JPEG images directly from the framebuffer without first translating the pixels into RGB. The TurboVNC encoder compares quite favorably in terms of compression ratio with TightVNC and generally encodes a great deal faster (often an order of magnitude or more.) The version of the TurboVNC encoder included in this patch is roughly equivalent to the one found in version 0.6 of the Unix TurboVNC Server, with a few minor patches integrated from TurboVNC 1.1. TurboVNC 1.0 added multi-threading capabilities, which can be added in later if desired (at the expense of making libvncserver depend on libpthread.) Because TurboVNC uses a fundamentally different mix of subencodings than TightVNC, because it uses the identical protocol (and thus a viewer really has no idea whether it's talking to a TightVNC or TurboVNC server), and because it doesn't support rfbTightPng (and in fact conflicts with it-- see below), the TurboVNC and TightVNC encoders cannot be enabled simultaneously. Compatibility: In *most* cases, a TurboVNC-enabled viewer is fully compatible with a TightVNC server, and vice versa. TurboVNC supports pseudo-encodings for specifying a fine-grained (1-100) quality scale and specifying chrominance subsampling. If a TurboVNC viewer sends those to a TightVNC server, then the TightVNC server ignores them, so the TurboVNC viewer also sends the quality on a 0-9 scale that the TightVNC server can understand. Similarly, the TurboVNC server checks first for fine-grained quality and subsampling pseudo-encodings from the viewer, and failing to receive those, it then checks for the TightVNC 0-9 quality pseudo-encoding. There is one case in which the two systems are not compatible, and that is when a TightVNC or TigerVNC viewer requests compression level 0 without JPEG from a TurboVNC server. For performance reasons, this causes the TurboVNC server to send images directly to the viewer, bypassing Zlib. When the TurboVNC server does this, it also sets bits 7-4 in the compression control byte to rfbTightNoZlib (0x0A), which is unfortunately the same value as rfbTightPng. Older TightVNC viewers that don't handle PNG will assume that the stream is uncompressed but still encapsulated in a Zlib structure, whereas newer PNG-supporting TightVNC viewers will assume that the stream is PNG. In either case, the viewer will probably crash. Since most VNC viewers don't expose compression level 0 in the GUI, this is a relatively rare situation. Description of changes: configure.ac -- Added support for libjpeg-turbo. If passed an argument of --with-turbovnc, configure will now run (or, if cross-compiling, just link) a test program that determines whether the libjpeg library being used is libjpeg-turbo. libjpeg-turbo must be used when building the TurboVNC encoder, because the TurboVNC encoder relies on the libjpeg-turbo colorspace extensions in order to compress images directly out of the framebuffer (which may be, for instance, BGRA rather than RGB.) libjpeg-turbo can optionally be used with the TightVNC encoder as well, but the speedup will only be marginal (the report linked above explains why in more detail, but basically it's because of Amdahl's Law. The TightVNC encoder was designed with the assumption that JPEG had a very high CPU cost, and thus JPEG is used only sparingly.) -- Added a new configure variable, JPEG_LDFLAGS. This is necessitated by the fact that libjpeg-turbo often distributes libjpeg.a and libjpeg.so in /opt/libjpeg-turbo/lib32 or /opt/libjpeg-turbo/lib64, and many people prefer to statically link with it. Thus, more flexibility is needed than is provided by --with-jpeg. If JPEG_LDFLAGS is specified, then it overrides the changes to LDFLAGS enacted by --with-jpeg (but --with-jpeg is still used to set the include path.) The addition of JPEG_LDFLAGS necessitated replacing AC_CHECK_LIB with AC_LINK_IFELSE (because AC_CHECK_LIB automatically sets LIBS to -ljpeg, which is not what we want if we're, for instance, linking statically with libjpeg-turbo.) -- configure does not check for PNG support if TurboVNC encoding is enabled. This prevents the rfbSendRectEncodingTightPng() function from being compiled in, since the TurboVNC encoder doesn't (and can't) support it. common/turbojpeg.c, common/turbojpeg.h -- TurboJPEG is a simple API used to compress and decompress JPEG images in memory. It was originally implemented because it was desirable to use different types of underlying technologies to compress JPEG on different platforms (mediaLib on SPARC, Quicktime on PPC Macs, Intel Performance Primitives, etc.) These days, however, libjpeg-turbo is the only underlying technology used by TurboVNC, so TurboJPEG's purpose is largely just code simplicity and flexibility. Thus, since there is no real need for libvncserver to use any technology other than libjpeg-turbo for compressing JPEG, the TurboJPEG wrapper for libjpeg-turbo has been included in-tree so that libvncserver can be directly linked with libjpeg-turbo. This is convenient because many modern Linux distros (Fedora, Ubuntu, etc.) now ship libjpeg-turbo as their default libjpeg library. libvncserver/rfbserver.c -- Added logic to check for the TurboVNC fine-grained quality level and subsampling encodings and to map Tight (0-9) quality levels to appropriate fine-grained quality level and subsampling values if communicating with a TightVNC/TigerVNC viewer. libvncserver/turbo.c -- TurboVNC encoder (compiled instead of libvncserver/tight.c) rfb/rfb.h -- Added support for the TurboVNC subsampling level rfb/rfbproto.h -- Added constants for the TurboVNC fine quality level and subsampling encodings as well as the rfbTightNoZlib constant and notes on its usage.
13 years ago
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
DLLEXPORT tjhandle DLLCALL tjInitDecompress(void)
{
tjinstance *this;
if((this=(tjinstance *)malloc(sizeof(tjinstance)))==NULL)
{
snprintf(errStr, JMSG_LENGTH_MAX,
"tjInitDecompress(): Memory allocation failure");
return NULL;
}
MEMZERO(this, sizeof(tjinstance));
return _tjInitDecompress(this);
Add TurboVNC encoding support. TurboVNC is a variant of TightVNC that uses the same client/server protocol (RFB version 3.8t), and thus it is fully cross-compatible with TightVNC and TigerVNC (with one exception, which is noted below.) Both the TightVNC and TurboVNC encoders analyze each rectangle, pick out regions of solid color to send separately, and send the remaining subrectangles using mono, indexed color, JPEG, or raw encoding, depending on the number of colors in the subrectangle. However, TurboVNC uses a fundamentally different selection algorithm to determine the appropriate subencoding to use for each subrectangle. Thus, while it sends a protocol stream that can be decoded by any TightVNC-compatible viewer, the mix of subencoding types in this protocol stream will be different from those generated by a TightVNC server. The research that led to TurboVNC is described in the following report: http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf. In summary: 20 RFB captures, representing "common" 2D and 3D application workloads (the 3D workloads were run using VirtualGL), were studied using the TightVNC encoder in isolation. Some of the analysis features in the TightVNC encoder, such as smoothness detection, were found to generate a lot of CPU usage with little or no benefit in compression, so those features were disabled. JPEG encoding was accelerated using libjpeg-turbo (which achieves a 2-4x speedup over plain libjpeg on modern x86 or ARM processors.) Finally, the "palette threshold" (minimum number of colors that the subrectangle must have before it is compressed using JPEG or raw) was adjusted to account for the fact that JPEG encoding is now quite a bit faster (meaning that we can now use it more without a CPU penalty.) TurboVNC has additional optimizations, such as the ability to count colors and encode JPEG images directly from the framebuffer without first translating the pixels into RGB. The TurboVNC encoder compares quite favorably in terms of compression ratio with TightVNC and generally encodes a great deal faster (often an order of magnitude or more.) The version of the TurboVNC encoder included in this patch is roughly equivalent to the one found in version 0.6 of the Unix TurboVNC Server, with a few minor patches integrated from TurboVNC 1.1. TurboVNC 1.0 added multi-threading capabilities, which can be added in later if desired (at the expense of making libvncserver depend on libpthread.) Because TurboVNC uses a fundamentally different mix of subencodings than TightVNC, because it uses the identical protocol (and thus a viewer really has no idea whether it's talking to a TightVNC or TurboVNC server), and because it doesn't support rfbTightPng (and in fact conflicts with it-- see below), the TurboVNC and TightVNC encoders cannot be enabled simultaneously. Compatibility: In *most* cases, a TurboVNC-enabled viewer is fully compatible with a TightVNC server, and vice versa. TurboVNC supports pseudo-encodings for specifying a fine-grained (1-100) quality scale and specifying chrominance subsampling. If a TurboVNC viewer sends those to a TightVNC server, then the TightVNC server ignores them, so the TurboVNC viewer also sends the quality on a 0-9 scale that the TightVNC server can understand. Similarly, the TurboVNC server checks first for fine-grained quality and subsampling pseudo-encodings from the viewer, and failing to receive those, it then checks for the TightVNC 0-9 quality pseudo-encoding. There is one case in which the two systems are not compatible, and that is when a TightVNC or TigerVNC viewer requests compression level 0 without JPEG from a TurboVNC server. For performance reasons, this causes the TurboVNC server to send images directly to the viewer, bypassing Zlib. When the TurboVNC server does this, it also sets bits 7-4 in the compression control byte to rfbTightNoZlib (0x0A), which is unfortunately the same value as rfbTightPng. Older TightVNC viewers that don't handle PNG will assume that the stream is uncompressed but still encapsulated in a Zlib structure, whereas newer PNG-supporting TightVNC viewers will assume that the stream is PNG. In either case, the viewer will probably crash. Since most VNC viewers don't expose compression level 0 in the GUI, this is a relatively rare situation. Description of changes: configure.ac -- Added support for libjpeg-turbo. If passed an argument of --with-turbovnc, configure will now run (or, if cross-compiling, just link) a test program that determines whether the libjpeg library being used is libjpeg-turbo. libjpeg-turbo must be used when building the TurboVNC encoder, because the TurboVNC encoder relies on the libjpeg-turbo colorspace extensions in order to compress images directly out of the framebuffer (which may be, for instance, BGRA rather than RGB.) libjpeg-turbo can optionally be used with the TightVNC encoder as well, but the speedup will only be marginal (the report linked above explains why in more detail, but basically it's because of Amdahl's Law. The TightVNC encoder was designed with the assumption that JPEG had a very high CPU cost, and thus JPEG is used only sparingly.) -- Added a new configure variable, JPEG_LDFLAGS. This is necessitated by the fact that libjpeg-turbo often distributes libjpeg.a and libjpeg.so in /opt/libjpeg-turbo/lib32 or /opt/libjpeg-turbo/lib64, and many people prefer to statically link with it. Thus, more flexibility is needed than is provided by --with-jpeg. If JPEG_LDFLAGS is specified, then it overrides the changes to LDFLAGS enacted by --with-jpeg (but --with-jpeg is still used to set the include path.) The addition of JPEG_LDFLAGS necessitated replacing AC_CHECK_LIB with AC_LINK_IFELSE (because AC_CHECK_LIB automatically sets LIBS to -ljpeg, which is not what we want if we're, for instance, linking statically with libjpeg-turbo.) -- configure does not check for PNG support if TurboVNC encoding is enabled. This prevents the rfbSendRectEncodingTightPng() function from being compiled in, since the TurboVNC encoder doesn't (and can't) support it. common/turbojpeg.c, common/turbojpeg.h -- TurboJPEG is a simple API used to compress and decompress JPEG images in memory. It was originally implemented because it was desirable to use different types of underlying technologies to compress JPEG on different platforms (mediaLib on SPARC, Quicktime on PPC Macs, Intel Performance Primitives, etc.) These days, however, libjpeg-turbo is the only underlying technology used by TurboVNC, so TurboJPEG's purpose is largely just code simplicity and flexibility. Thus, since there is no real need for libvncserver to use any technology other than libjpeg-turbo for compressing JPEG, the TurboJPEG wrapper for libjpeg-turbo has been included in-tree so that libvncserver can be directly linked with libjpeg-turbo. This is convenient because many modern Linux distros (Fedora, Ubuntu, etc.) now ship libjpeg-turbo as their default libjpeg library. libvncserver/rfbserver.c -- Added logic to check for the TurboVNC fine-grained quality level and subsampling encodings and to map Tight (0-9) quality levels to appropriate fine-grained quality level and subsampling values if communicating with a TightVNC/TigerVNC viewer. libvncserver/turbo.c -- TurboVNC encoder (compiled instead of libvncserver/tight.c) rfb/rfb.h -- Added support for the TurboVNC subsampling level rfb/rfbproto.h -- Added constants for the TurboVNC fine quality level and subsampling encodings as well as the rfbTightNoZlib constant and notes on its usage.
13 years ago
}
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
DLLEXPORT int DLLCALL tjDecompressHeader2(tjhandle handle,
unsigned char *jpegBuf, unsigned long jpegSize, int *width, int *height,
int *jpegSubsamp)
Add TurboVNC encoding support. TurboVNC is a variant of TightVNC that uses the same client/server protocol (RFB version 3.8t), and thus it is fully cross-compatible with TightVNC and TigerVNC (with one exception, which is noted below.) Both the TightVNC and TurboVNC encoders analyze each rectangle, pick out regions of solid color to send separately, and send the remaining subrectangles using mono, indexed color, JPEG, or raw encoding, depending on the number of colors in the subrectangle. However, TurboVNC uses a fundamentally different selection algorithm to determine the appropriate subencoding to use for each subrectangle. Thus, while it sends a protocol stream that can be decoded by any TightVNC-compatible viewer, the mix of subencoding types in this protocol stream will be different from those generated by a TightVNC server. The research that led to TurboVNC is described in the following report: http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf. In summary: 20 RFB captures, representing "common" 2D and 3D application workloads (the 3D workloads were run using VirtualGL), were studied using the TightVNC encoder in isolation. Some of the analysis features in the TightVNC encoder, such as smoothness detection, were found to generate a lot of CPU usage with little or no benefit in compression, so those features were disabled. JPEG encoding was accelerated using libjpeg-turbo (which achieves a 2-4x speedup over plain libjpeg on modern x86 or ARM processors.) Finally, the "palette threshold" (minimum number of colors that the subrectangle must have before it is compressed using JPEG or raw) was adjusted to account for the fact that JPEG encoding is now quite a bit faster (meaning that we can now use it more without a CPU penalty.) TurboVNC has additional optimizations, such as the ability to count colors and encode JPEG images directly from the framebuffer without first translating the pixels into RGB. The TurboVNC encoder compares quite favorably in terms of compression ratio with TightVNC and generally encodes a great deal faster (often an order of magnitude or more.) The version of the TurboVNC encoder included in this patch is roughly equivalent to the one found in version 0.6 of the Unix TurboVNC Server, with a few minor patches integrated from TurboVNC 1.1. TurboVNC 1.0 added multi-threading capabilities, which can be added in later if desired (at the expense of making libvncserver depend on libpthread.) Because TurboVNC uses a fundamentally different mix of subencodings than TightVNC, because it uses the identical protocol (and thus a viewer really has no idea whether it's talking to a TightVNC or TurboVNC server), and because it doesn't support rfbTightPng (and in fact conflicts with it-- see below), the TurboVNC and TightVNC encoders cannot be enabled simultaneously. Compatibility: In *most* cases, a TurboVNC-enabled viewer is fully compatible with a TightVNC server, and vice versa. TurboVNC supports pseudo-encodings for specifying a fine-grained (1-100) quality scale and specifying chrominance subsampling. If a TurboVNC viewer sends those to a TightVNC server, then the TightVNC server ignores them, so the TurboVNC viewer also sends the quality on a 0-9 scale that the TightVNC server can understand. Similarly, the TurboVNC server checks first for fine-grained quality and subsampling pseudo-encodings from the viewer, and failing to receive those, it then checks for the TightVNC 0-9 quality pseudo-encoding. There is one case in which the two systems are not compatible, and that is when a TightVNC or TigerVNC viewer requests compression level 0 without JPEG from a TurboVNC server. For performance reasons, this causes the TurboVNC server to send images directly to the viewer, bypassing Zlib. When the TurboVNC server does this, it also sets bits 7-4 in the compression control byte to rfbTightNoZlib (0x0A), which is unfortunately the same value as rfbTightPng. Older TightVNC viewers that don't handle PNG will assume that the stream is uncompressed but still encapsulated in a Zlib structure, whereas newer PNG-supporting TightVNC viewers will assume that the stream is PNG. In either case, the viewer will probably crash. Since most VNC viewers don't expose compression level 0 in the GUI, this is a relatively rare situation. Description of changes: configure.ac -- Added support for libjpeg-turbo. If passed an argument of --with-turbovnc, configure will now run (or, if cross-compiling, just link) a test program that determines whether the libjpeg library being used is libjpeg-turbo. libjpeg-turbo must be used when building the TurboVNC encoder, because the TurboVNC encoder relies on the libjpeg-turbo colorspace extensions in order to compress images directly out of the framebuffer (which may be, for instance, BGRA rather than RGB.) libjpeg-turbo can optionally be used with the TightVNC encoder as well, but the speedup will only be marginal (the report linked above explains why in more detail, but basically it's because of Amdahl's Law. The TightVNC encoder was designed with the assumption that JPEG had a very high CPU cost, and thus JPEG is used only sparingly.) -- Added a new configure variable, JPEG_LDFLAGS. This is necessitated by the fact that libjpeg-turbo often distributes libjpeg.a and libjpeg.so in /opt/libjpeg-turbo/lib32 or /opt/libjpeg-turbo/lib64, and many people prefer to statically link with it. Thus, more flexibility is needed than is provided by --with-jpeg. If JPEG_LDFLAGS is specified, then it overrides the changes to LDFLAGS enacted by --with-jpeg (but --with-jpeg is still used to set the include path.) The addition of JPEG_LDFLAGS necessitated replacing AC_CHECK_LIB with AC_LINK_IFELSE (because AC_CHECK_LIB automatically sets LIBS to -ljpeg, which is not what we want if we're, for instance, linking statically with libjpeg-turbo.) -- configure does not check for PNG support if TurboVNC encoding is enabled. This prevents the rfbSendRectEncodingTightPng() function from being compiled in, since the TurboVNC encoder doesn't (and can't) support it. common/turbojpeg.c, common/turbojpeg.h -- TurboJPEG is a simple API used to compress and decompress JPEG images in memory. It was originally implemented because it was desirable to use different types of underlying technologies to compress JPEG on different platforms (mediaLib on SPARC, Quicktime on PPC Macs, Intel Performance Primitives, etc.) These days, however, libjpeg-turbo is the only underlying technology used by TurboVNC, so TurboJPEG's purpose is largely just code simplicity and flexibility. Thus, since there is no real need for libvncserver to use any technology other than libjpeg-turbo for compressing JPEG, the TurboJPEG wrapper for libjpeg-turbo has been included in-tree so that libvncserver can be directly linked with libjpeg-turbo. This is convenient because many modern Linux distros (Fedora, Ubuntu, etc.) now ship libjpeg-turbo as their default libjpeg library. libvncserver/rfbserver.c -- Added logic to check for the TurboVNC fine-grained quality level and subsampling encodings and to map Tight (0-9) quality levels to appropriate fine-grained quality level and subsampling values if communicating with a TightVNC/TigerVNC viewer. libvncserver/turbo.c -- TurboVNC encoder (compiled instead of libvncserver/tight.c) rfb/rfb.h -- Added support for the TurboVNC subsampling level rfb/rfbproto.h -- Added constants for the TurboVNC fine quality level and subsampling encodings as well as the rfbTightNoZlib constant and notes on its usage.
13 years ago
{
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
int retval=0;
Add TurboVNC encoding support. TurboVNC is a variant of TightVNC that uses the same client/server protocol (RFB version 3.8t), and thus it is fully cross-compatible with TightVNC and TigerVNC (with one exception, which is noted below.) Both the TightVNC and TurboVNC encoders analyze each rectangle, pick out regions of solid color to send separately, and send the remaining subrectangles using mono, indexed color, JPEG, or raw encoding, depending on the number of colors in the subrectangle. However, TurboVNC uses a fundamentally different selection algorithm to determine the appropriate subencoding to use for each subrectangle. Thus, while it sends a protocol stream that can be decoded by any TightVNC-compatible viewer, the mix of subencoding types in this protocol stream will be different from those generated by a TightVNC server. The research that led to TurboVNC is described in the following report: http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf. In summary: 20 RFB captures, representing "common" 2D and 3D application workloads (the 3D workloads were run using VirtualGL), were studied using the TightVNC encoder in isolation. Some of the analysis features in the TightVNC encoder, such as smoothness detection, were found to generate a lot of CPU usage with little or no benefit in compression, so those features were disabled. JPEG encoding was accelerated using libjpeg-turbo (which achieves a 2-4x speedup over plain libjpeg on modern x86 or ARM processors.) Finally, the "palette threshold" (minimum number of colors that the subrectangle must have before it is compressed using JPEG or raw) was adjusted to account for the fact that JPEG encoding is now quite a bit faster (meaning that we can now use it more without a CPU penalty.) TurboVNC has additional optimizations, such as the ability to count colors and encode JPEG images directly from the framebuffer without first translating the pixels into RGB. The TurboVNC encoder compares quite favorably in terms of compression ratio with TightVNC and generally encodes a great deal faster (often an order of magnitude or more.) The version of the TurboVNC encoder included in this patch is roughly equivalent to the one found in version 0.6 of the Unix TurboVNC Server, with a few minor patches integrated from TurboVNC 1.1. TurboVNC 1.0 added multi-threading capabilities, which can be added in later if desired (at the expense of making libvncserver depend on libpthread.) Because TurboVNC uses a fundamentally different mix of subencodings than TightVNC, because it uses the identical protocol (and thus a viewer really has no idea whether it's talking to a TightVNC or TurboVNC server), and because it doesn't support rfbTightPng (and in fact conflicts with it-- see below), the TurboVNC and TightVNC encoders cannot be enabled simultaneously. Compatibility: In *most* cases, a TurboVNC-enabled viewer is fully compatible with a TightVNC server, and vice versa. TurboVNC supports pseudo-encodings for specifying a fine-grained (1-100) quality scale and specifying chrominance subsampling. If a TurboVNC viewer sends those to a TightVNC server, then the TightVNC server ignores them, so the TurboVNC viewer also sends the quality on a 0-9 scale that the TightVNC server can understand. Similarly, the TurboVNC server checks first for fine-grained quality and subsampling pseudo-encodings from the viewer, and failing to receive those, it then checks for the TightVNC 0-9 quality pseudo-encoding. There is one case in which the two systems are not compatible, and that is when a TightVNC or TigerVNC viewer requests compression level 0 without JPEG from a TurboVNC server. For performance reasons, this causes the TurboVNC server to send images directly to the viewer, bypassing Zlib. When the TurboVNC server does this, it also sets bits 7-4 in the compression control byte to rfbTightNoZlib (0x0A), which is unfortunately the same value as rfbTightPng. Older TightVNC viewers that don't handle PNG will assume that the stream is uncompressed but still encapsulated in a Zlib structure, whereas newer PNG-supporting TightVNC viewers will assume that the stream is PNG. In either case, the viewer will probably crash. Since most VNC viewers don't expose compression level 0 in the GUI, this is a relatively rare situation. Description of changes: configure.ac -- Added support for libjpeg-turbo. If passed an argument of --with-turbovnc, configure will now run (or, if cross-compiling, just link) a test program that determines whether the libjpeg library being used is libjpeg-turbo. libjpeg-turbo must be used when building the TurboVNC encoder, because the TurboVNC encoder relies on the libjpeg-turbo colorspace extensions in order to compress images directly out of the framebuffer (which may be, for instance, BGRA rather than RGB.) libjpeg-turbo can optionally be used with the TightVNC encoder as well, but the speedup will only be marginal (the report linked above explains why in more detail, but basically it's because of Amdahl's Law. The TightVNC encoder was designed with the assumption that JPEG had a very high CPU cost, and thus JPEG is used only sparingly.) -- Added a new configure variable, JPEG_LDFLAGS. This is necessitated by the fact that libjpeg-turbo often distributes libjpeg.a and libjpeg.so in /opt/libjpeg-turbo/lib32 or /opt/libjpeg-turbo/lib64, and many people prefer to statically link with it. Thus, more flexibility is needed than is provided by --with-jpeg. If JPEG_LDFLAGS is specified, then it overrides the changes to LDFLAGS enacted by --with-jpeg (but --with-jpeg is still used to set the include path.) The addition of JPEG_LDFLAGS necessitated replacing AC_CHECK_LIB with AC_LINK_IFELSE (because AC_CHECK_LIB automatically sets LIBS to -ljpeg, which is not what we want if we're, for instance, linking statically with libjpeg-turbo.) -- configure does not check for PNG support if TurboVNC encoding is enabled. This prevents the rfbSendRectEncodingTightPng() function from being compiled in, since the TurboVNC encoder doesn't (and can't) support it. common/turbojpeg.c, common/turbojpeg.h -- TurboJPEG is a simple API used to compress and decompress JPEG images in memory. It was originally implemented because it was desirable to use different types of underlying technologies to compress JPEG on different platforms (mediaLib on SPARC, Quicktime on PPC Macs, Intel Performance Primitives, etc.) These days, however, libjpeg-turbo is the only underlying technology used by TurboVNC, so TurboJPEG's purpose is largely just code simplicity and flexibility. Thus, since there is no real need for libvncserver to use any technology other than libjpeg-turbo for compressing JPEG, the TurboJPEG wrapper for libjpeg-turbo has been included in-tree so that libvncserver can be directly linked with libjpeg-turbo. This is convenient because many modern Linux distros (Fedora, Ubuntu, etc.) now ship libjpeg-turbo as their default libjpeg library. libvncserver/rfbserver.c -- Added logic to check for the TurboVNC fine-grained quality level and subsampling encodings and to map Tight (0-9) quality levels to appropriate fine-grained quality level and subsampling values if communicating with a TightVNC/TigerVNC viewer. libvncserver/turbo.c -- TurboVNC encoder (compiled instead of libvncserver/tight.c) rfb/rfb.h -- Added support for the TurboVNC subsampling level rfb/rfbproto.h -- Added constants for the TurboVNC fine quality level and subsampling encodings as well as the rfbTightNoZlib constant and notes on its usage.
13 years ago
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
getinstance(handle);
if((this->init&DECOMPRESS)==0)
_throw("tjDecompressHeader2(): Instance has not been initialized for decompression");
Add TurboVNC encoding support. TurboVNC is a variant of TightVNC that uses the same client/server protocol (RFB version 3.8t), and thus it is fully cross-compatible with TightVNC and TigerVNC (with one exception, which is noted below.) Both the TightVNC and TurboVNC encoders analyze each rectangle, pick out regions of solid color to send separately, and send the remaining subrectangles using mono, indexed color, JPEG, or raw encoding, depending on the number of colors in the subrectangle. However, TurboVNC uses a fundamentally different selection algorithm to determine the appropriate subencoding to use for each subrectangle. Thus, while it sends a protocol stream that can be decoded by any TightVNC-compatible viewer, the mix of subencoding types in this protocol stream will be different from those generated by a TightVNC server. The research that led to TurboVNC is described in the following report: http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf. In summary: 20 RFB captures, representing "common" 2D and 3D application workloads (the 3D workloads were run using VirtualGL), were studied using the TightVNC encoder in isolation. Some of the analysis features in the TightVNC encoder, such as smoothness detection, were found to generate a lot of CPU usage with little or no benefit in compression, so those features were disabled. JPEG encoding was accelerated using libjpeg-turbo (which achieves a 2-4x speedup over plain libjpeg on modern x86 or ARM processors.) Finally, the "palette threshold" (minimum number of colors that the subrectangle must have before it is compressed using JPEG or raw) was adjusted to account for the fact that JPEG encoding is now quite a bit faster (meaning that we can now use it more without a CPU penalty.) TurboVNC has additional optimizations, such as the ability to count colors and encode JPEG images directly from the framebuffer without first translating the pixels into RGB. The TurboVNC encoder compares quite favorably in terms of compression ratio with TightVNC and generally encodes a great deal faster (often an order of magnitude or more.) The version of the TurboVNC encoder included in this patch is roughly equivalent to the one found in version 0.6 of the Unix TurboVNC Server, with a few minor patches integrated from TurboVNC 1.1. TurboVNC 1.0 added multi-threading capabilities, which can be added in later if desired (at the expense of making libvncserver depend on libpthread.) Because TurboVNC uses a fundamentally different mix of subencodings than TightVNC, because it uses the identical protocol (and thus a viewer really has no idea whether it's talking to a TightVNC or TurboVNC server), and because it doesn't support rfbTightPng (and in fact conflicts with it-- see below), the TurboVNC and TightVNC encoders cannot be enabled simultaneously. Compatibility: In *most* cases, a TurboVNC-enabled viewer is fully compatible with a TightVNC server, and vice versa. TurboVNC supports pseudo-encodings for specifying a fine-grained (1-100) quality scale and specifying chrominance subsampling. If a TurboVNC viewer sends those to a TightVNC server, then the TightVNC server ignores them, so the TurboVNC viewer also sends the quality on a 0-9 scale that the TightVNC server can understand. Similarly, the TurboVNC server checks first for fine-grained quality and subsampling pseudo-encodings from the viewer, and failing to receive those, it then checks for the TightVNC 0-9 quality pseudo-encoding. There is one case in which the two systems are not compatible, and that is when a TightVNC or TigerVNC viewer requests compression level 0 without JPEG from a TurboVNC server. For performance reasons, this causes the TurboVNC server to send images directly to the viewer, bypassing Zlib. When the TurboVNC server does this, it also sets bits 7-4 in the compression control byte to rfbTightNoZlib (0x0A), which is unfortunately the same value as rfbTightPng. Older TightVNC viewers that don't handle PNG will assume that the stream is uncompressed but still encapsulated in a Zlib structure, whereas newer PNG-supporting TightVNC viewers will assume that the stream is PNG. In either case, the viewer will probably crash. Since most VNC viewers don't expose compression level 0 in the GUI, this is a relatively rare situation. Description of changes: configure.ac -- Added support for libjpeg-turbo. If passed an argument of --with-turbovnc, configure will now run (or, if cross-compiling, just link) a test program that determines whether the libjpeg library being used is libjpeg-turbo. libjpeg-turbo must be used when building the TurboVNC encoder, because the TurboVNC encoder relies on the libjpeg-turbo colorspace extensions in order to compress images directly out of the framebuffer (which may be, for instance, BGRA rather than RGB.) libjpeg-turbo can optionally be used with the TightVNC encoder as well, but the speedup will only be marginal (the report linked above explains why in more detail, but basically it's because of Amdahl's Law. The TightVNC encoder was designed with the assumption that JPEG had a very high CPU cost, and thus JPEG is used only sparingly.) -- Added a new configure variable, JPEG_LDFLAGS. This is necessitated by the fact that libjpeg-turbo often distributes libjpeg.a and libjpeg.so in /opt/libjpeg-turbo/lib32 or /opt/libjpeg-turbo/lib64, and many people prefer to statically link with it. Thus, more flexibility is needed than is provided by --with-jpeg. If JPEG_LDFLAGS is specified, then it overrides the changes to LDFLAGS enacted by --with-jpeg (but --with-jpeg is still used to set the include path.) The addition of JPEG_LDFLAGS necessitated replacing AC_CHECK_LIB with AC_LINK_IFELSE (because AC_CHECK_LIB automatically sets LIBS to -ljpeg, which is not what we want if we're, for instance, linking statically with libjpeg-turbo.) -- configure does not check for PNG support if TurboVNC encoding is enabled. This prevents the rfbSendRectEncodingTightPng() function from being compiled in, since the TurboVNC encoder doesn't (and can't) support it. common/turbojpeg.c, common/turbojpeg.h -- TurboJPEG is a simple API used to compress and decompress JPEG images in memory. It was originally implemented because it was desirable to use different types of underlying technologies to compress JPEG on different platforms (mediaLib on SPARC, Quicktime on PPC Macs, Intel Performance Primitives, etc.) These days, however, libjpeg-turbo is the only underlying technology used by TurboVNC, so TurboJPEG's purpose is largely just code simplicity and flexibility. Thus, since there is no real need for libvncserver to use any technology other than libjpeg-turbo for compressing JPEG, the TurboJPEG wrapper for libjpeg-turbo has been included in-tree so that libvncserver can be directly linked with libjpeg-turbo. This is convenient because many modern Linux distros (Fedora, Ubuntu, etc.) now ship libjpeg-turbo as their default libjpeg library. libvncserver/rfbserver.c -- Added logic to check for the TurboVNC fine-grained quality level and subsampling encodings and to map Tight (0-9) quality levels to appropriate fine-grained quality level and subsampling values if communicating with a TightVNC/TigerVNC viewer. libvncserver/turbo.c -- TurboVNC encoder (compiled instead of libvncserver/tight.c) rfb/rfb.h -- Added support for the TurboVNC subsampling level rfb/rfbproto.h -- Added constants for the TurboVNC fine quality level and subsampling encodings as well as the rfbTightNoZlib constant and notes on its usage.
13 years ago
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
if(jpegBuf==NULL || jpegSize<=0 || width==NULL || height==NULL
|| jpegSubsamp==NULL)
_throw("tjDecompressHeader2(): Invalid argument");
Add TurboVNC encoding support. TurboVNC is a variant of TightVNC that uses the same client/server protocol (RFB version 3.8t), and thus it is fully cross-compatible with TightVNC and TigerVNC (with one exception, which is noted below.) Both the TightVNC and TurboVNC encoders analyze each rectangle, pick out regions of solid color to send separately, and send the remaining subrectangles using mono, indexed color, JPEG, or raw encoding, depending on the number of colors in the subrectangle. However, TurboVNC uses a fundamentally different selection algorithm to determine the appropriate subencoding to use for each subrectangle. Thus, while it sends a protocol stream that can be decoded by any TightVNC-compatible viewer, the mix of subencoding types in this protocol stream will be different from those generated by a TightVNC server. The research that led to TurboVNC is described in the following report: http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf. In summary: 20 RFB captures, representing "common" 2D and 3D application workloads (the 3D workloads were run using VirtualGL), were studied using the TightVNC encoder in isolation. Some of the analysis features in the TightVNC encoder, such as smoothness detection, were found to generate a lot of CPU usage with little or no benefit in compression, so those features were disabled. JPEG encoding was accelerated using libjpeg-turbo (which achieves a 2-4x speedup over plain libjpeg on modern x86 or ARM processors.) Finally, the "palette threshold" (minimum number of colors that the subrectangle must have before it is compressed using JPEG or raw) was adjusted to account for the fact that JPEG encoding is now quite a bit faster (meaning that we can now use it more without a CPU penalty.) TurboVNC has additional optimizations, such as the ability to count colors and encode JPEG images directly from the framebuffer without first translating the pixels into RGB. The TurboVNC encoder compares quite favorably in terms of compression ratio with TightVNC and generally encodes a great deal faster (often an order of magnitude or more.) The version of the TurboVNC encoder included in this patch is roughly equivalent to the one found in version 0.6 of the Unix TurboVNC Server, with a few minor patches integrated from TurboVNC 1.1. TurboVNC 1.0 added multi-threading capabilities, which can be added in later if desired (at the expense of making libvncserver depend on libpthread.) Because TurboVNC uses a fundamentally different mix of subencodings than TightVNC, because it uses the identical protocol (and thus a viewer really has no idea whether it's talking to a TightVNC or TurboVNC server), and because it doesn't support rfbTightPng (and in fact conflicts with it-- see below), the TurboVNC and TightVNC encoders cannot be enabled simultaneously. Compatibility: In *most* cases, a TurboVNC-enabled viewer is fully compatible with a TightVNC server, and vice versa. TurboVNC supports pseudo-encodings for specifying a fine-grained (1-100) quality scale and specifying chrominance subsampling. If a TurboVNC viewer sends those to a TightVNC server, then the TightVNC server ignores them, so the TurboVNC viewer also sends the quality on a 0-9 scale that the TightVNC server can understand. Similarly, the TurboVNC server checks first for fine-grained quality and subsampling pseudo-encodings from the viewer, and failing to receive those, it then checks for the TightVNC 0-9 quality pseudo-encoding. There is one case in which the two systems are not compatible, and that is when a TightVNC or TigerVNC viewer requests compression level 0 without JPEG from a TurboVNC server. For performance reasons, this causes the TurboVNC server to send images directly to the viewer, bypassing Zlib. When the TurboVNC server does this, it also sets bits 7-4 in the compression control byte to rfbTightNoZlib (0x0A), which is unfortunately the same value as rfbTightPng. Older TightVNC viewers that don't handle PNG will assume that the stream is uncompressed but still encapsulated in a Zlib structure, whereas newer PNG-supporting TightVNC viewers will assume that the stream is PNG. In either case, the viewer will probably crash. Since most VNC viewers don't expose compression level 0 in the GUI, this is a relatively rare situation. Description of changes: configure.ac -- Added support for libjpeg-turbo. If passed an argument of --with-turbovnc, configure will now run (or, if cross-compiling, just link) a test program that determines whether the libjpeg library being used is libjpeg-turbo. libjpeg-turbo must be used when building the TurboVNC encoder, because the TurboVNC encoder relies on the libjpeg-turbo colorspace extensions in order to compress images directly out of the framebuffer (which may be, for instance, BGRA rather than RGB.) libjpeg-turbo can optionally be used with the TightVNC encoder as well, but the speedup will only be marginal (the report linked above explains why in more detail, but basically it's because of Amdahl's Law. The TightVNC encoder was designed with the assumption that JPEG had a very high CPU cost, and thus JPEG is used only sparingly.) -- Added a new configure variable, JPEG_LDFLAGS. This is necessitated by the fact that libjpeg-turbo often distributes libjpeg.a and libjpeg.so in /opt/libjpeg-turbo/lib32 or /opt/libjpeg-turbo/lib64, and many people prefer to statically link with it. Thus, more flexibility is needed than is provided by --with-jpeg. If JPEG_LDFLAGS is specified, then it overrides the changes to LDFLAGS enacted by --with-jpeg (but --with-jpeg is still used to set the include path.) The addition of JPEG_LDFLAGS necessitated replacing AC_CHECK_LIB with AC_LINK_IFELSE (because AC_CHECK_LIB automatically sets LIBS to -ljpeg, which is not what we want if we're, for instance, linking statically with libjpeg-turbo.) -- configure does not check for PNG support if TurboVNC encoding is enabled. This prevents the rfbSendRectEncodingTightPng() function from being compiled in, since the TurboVNC encoder doesn't (and can't) support it. common/turbojpeg.c, common/turbojpeg.h -- TurboJPEG is a simple API used to compress and decompress JPEG images in memory. It was originally implemented because it was desirable to use different types of underlying technologies to compress JPEG on different platforms (mediaLib on SPARC, Quicktime on PPC Macs, Intel Performance Primitives, etc.) These days, however, libjpeg-turbo is the only underlying technology used by TurboVNC, so TurboJPEG's purpose is largely just code simplicity and flexibility. Thus, since there is no real need for libvncserver to use any technology other than libjpeg-turbo for compressing JPEG, the TurboJPEG wrapper for libjpeg-turbo has been included in-tree so that libvncserver can be directly linked with libjpeg-turbo. This is convenient because many modern Linux distros (Fedora, Ubuntu, etc.) now ship libjpeg-turbo as their default libjpeg library. libvncserver/rfbserver.c -- Added logic to check for the TurboVNC fine-grained quality level and subsampling encodings and to map Tight (0-9) quality levels to appropriate fine-grained quality level and subsampling values if communicating with a TightVNC/TigerVNC viewer. libvncserver/turbo.c -- TurboVNC encoder (compiled instead of libvncserver/tight.c) rfb/rfb.h -- Added support for the TurboVNC subsampling level rfb/rfbproto.h -- Added constants for the TurboVNC fine quality level and subsampling encodings as well as the rfbTightNoZlib constant and notes on its usage.
13 years ago
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
if(setjmp(this->jerr.setjmp_buffer))
{
/* If we get here, the JPEG code has signaled an error. */
Add TurboVNC encoding support. TurboVNC is a variant of TightVNC that uses the same client/server protocol (RFB version 3.8t), and thus it is fully cross-compatible with TightVNC and TigerVNC (with one exception, which is noted below.) Both the TightVNC and TurboVNC encoders analyze each rectangle, pick out regions of solid color to send separately, and send the remaining subrectangles using mono, indexed color, JPEG, or raw encoding, depending on the number of colors in the subrectangle. However, TurboVNC uses a fundamentally different selection algorithm to determine the appropriate subencoding to use for each subrectangle. Thus, while it sends a protocol stream that can be decoded by any TightVNC-compatible viewer, the mix of subencoding types in this protocol stream will be different from those generated by a TightVNC server. The research that led to TurboVNC is described in the following report: http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf. In summary: 20 RFB captures, representing "common" 2D and 3D application workloads (the 3D workloads were run using VirtualGL), were studied using the TightVNC encoder in isolation. Some of the analysis features in the TightVNC encoder, such as smoothness detection, were found to generate a lot of CPU usage with little or no benefit in compression, so those features were disabled. JPEG encoding was accelerated using libjpeg-turbo (which achieves a 2-4x speedup over plain libjpeg on modern x86 or ARM processors.) Finally, the "palette threshold" (minimum number of colors that the subrectangle must have before it is compressed using JPEG or raw) was adjusted to account for the fact that JPEG encoding is now quite a bit faster (meaning that we can now use it more without a CPU penalty.) TurboVNC has additional optimizations, such as the ability to count colors and encode JPEG images directly from the framebuffer without first translating the pixels into RGB. The TurboVNC encoder compares quite favorably in terms of compression ratio with TightVNC and generally encodes a great deal faster (often an order of magnitude or more.) The version of the TurboVNC encoder included in this patch is roughly equivalent to the one found in version 0.6 of the Unix TurboVNC Server, with a few minor patches integrated from TurboVNC 1.1. TurboVNC 1.0 added multi-threading capabilities, which can be added in later if desired (at the expense of making libvncserver depend on libpthread.) Because TurboVNC uses a fundamentally different mix of subencodings than TightVNC, because it uses the identical protocol (and thus a viewer really has no idea whether it's talking to a TightVNC or TurboVNC server), and because it doesn't support rfbTightPng (and in fact conflicts with it-- see below), the TurboVNC and TightVNC encoders cannot be enabled simultaneously. Compatibility: In *most* cases, a TurboVNC-enabled viewer is fully compatible with a TightVNC server, and vice versa. TurboVNC supports pseudo-encodings for specifying a fine-grained (1-100) quality scale and specifying chrominance subsampling. If a TurboVNC viewer sends those to a TightVNC server, then the TightVNC server ignores them, so the TurboVNC viewer also sends the quality on a 0-9 scale that the TightVNC server can understand. Similarly, the TurboVNC server checks first for fine-grained quality and subsampling pseudo-encodings from the viewer, and failing to receive those, it then checks for the TightVNC 0-9 quality pseudo-encoding. There is one case in which the two systems are not compatible, and that is when a TightVNC or TigerVNC viewer requests compression level 0 without JPEG from a TurboVNC server. For performance reasons, this causes the TurboVNC server to send images directly to the viewer, bypassing Zlib. When the TurboVNC server does this, it also sets bits 7-4 in the compression control byte to rfbTightNoZlib (0x0A), which is unfortunately the same value as rfbTightPng. Older TightVNC viewers that don't handle PNG will assume that the stream is uncompressed but still encapsulated in a Zlib structure, whereas newer PNG-supporting TightVNC viewers will assume that the stream is PNG. In either case, the viewer will probably crash. Since most VNC viewers don't expose compression level 0 in the GUI, this is a relatively rare situation. Description of changes: configure.ac -- Added support for libjpeg-turbo. If passed an argument of --with-turbovnc, configure will now run (or, if cross-compiling, just link) a test program that determines whether the libjpeg library being used is libjpeg-turbo. libjpeg-turbo must be used when building the TurboVNC encoder, because the TurboVNC encoder relies on the libjpeg-turbo colorspace extensions in order to compress images directly out of the framebuffer (which may be, for instance, BGRA rather than RGB.) libjpeg-turbo can optionally be used with the TightVNC encoder as well, but the speedup will only be marginal (the report linked above explains why in more detail, but basically it's because of Amdahl's Law. The TightVNC encoder was designed with the assumption that JPEG had a very high CPU cost, and thus JPEG is used only sparingly.) -- Added a new configure variable, JPEG_LDFLAGS. This is necessitated by the fact that libjpeg-turbo often distributes libjpeg.a and libjpeg.so in /opt/libjpeg-turbo/lib32 or /opt/libjpeg-turbo/lib64, and many people prefer to statically link with it. Thus, more flexibility is needed than is provided by --with-jpeg. If JPEG_LDFLAGS is specified, then it overrides the changes to LDFLAGS enacted by --with-jpeg (but --with-jpeg is still used to set the include path.) The addition of JPEG_LDFLAGS necessitated replacing AC_CHECK_LIB with AC_LINK_IFELSE (because AC_CHECK_LIB automatically sets LIBS to -ljpeg, which is not what we want if we're, for instance, linking statically with libjpeg-turbo.) -- configure does not check for PNG support if TurboVNC encoding is enabled. This prevents the rfbSendRectEncodingTightPng() function from being compiled in, since the TurboVNC encoder doesn't (and can't) support it. common/turbojpeg.c, common/turbojpeg.h -- TurboJPEG is a simple API used to compress and decompress JPEG images in memory. It was originally implemented because it was desirable to use different types of underlying technologies to compress JPEG on different platforms (mediaLib on SPARC, Quicktime on PPC Macs, Intel Performance Primitives, etc.) These days, however, libjpeg-turbo is the only underlying technology used by TurboVNC, so TurboJPEG's purpose is largely just code simplicity and flexibility. Thus, since there is no real need for libvncserver to use any technology other than libjpeg-turbo for compressing JPEG, the TurboJPEG wrapper for libjpeg-turbo has been included in-tree so that libvncserver can be directly linked with libjpeg-turbo. This is convenient because many modern Linux distros (Fedora, Ubuntu, etc.) now ship libjpeg-turbo as their default libjpeg library. libvncserver/rfbserver.c -- Added logic to check for the TurboVNC fine-grained quality level and subsampling encodings and to map Tight (0-9) quality levels to appropriate fine-grained quality level and subsampling values if communicating with a TightVNC/TigerVNC viewer. libvncserver/turbo.c -- TurboVNC encoder (compiled instead of libvncserver/tight.c) rfb/rfb.h -- Added support for the TurboVNC subsampling level rfb/rfbproto.h -- Added constants for the TurboVNC fine quality level and subsampling encodings as well as the rfbTightNoZlib constant and notes on its usage.
13 years ago
return -1;
}
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
this->jsrc.bytes_in_buffer=jpegSize;
this->jsrc.next_input_byte=jpegBuf;
jpeg_read_header(dinfo, TRUE);
Add TurboVNC encoding support. TurboVNC is a variant of TightVNC that uses the same client/server protocol (RFB version 3.8t), and thus it is fully cross-compatible with TightVNC and TigerVNC (with one exception, which is noted below.) Both the TightVNC and TurboVNC encoders analyze each rectangle, pick out regions of solid color to send separately, and send the remaining subrectangles using mono, indexed color, JPEG, or raw encoding, depending on the number of colors in the subrectangle. However, TurboVNC uses a fundamentally different selection algorithm to determine the appropriate subencoding to use for each subrectangle. Thus, while it sends a protocol stream that can be decoded by any TightVNC-compatible viewer, the mix of subencoding types in this protocol stream will be different from those generated by a TightVNC server. The research that led to TurboVNC is described in the following report: http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf. In summary: 20 RFB captures, representing "common" 2D and 3D application workloads (the 3D workloads were run using VirtualGL), were studied using the TightVNC encoder in isolation. Some of the analysis features in the TightVNC encoder, such as smoothness detection, were found to generate a lot of CPU usage with little or no benefit in compression, so those features were disabled. JPEG encoding was accelerated using libjpeg-turbo (which achieves a 2-4x speedup over plain libjpeg on modern x86 or ARM processors.) Finally, the "palette threshold" (minimum number of colors that the subrectangle must have before it is compressed using JPEG or raw) was adjusted to account for the fact that JPEG encoding is now quite a bit faster (meaning that we can now use it more without a CPU penalty.) TurboVNC has additional optimizations, such as the ability to count colors and encode JPEG images directly from the framebuffer without first translating the pixels into RGB. The TurboVNC encoder compares quite favorably in terms of compression ratio with TightVNC and generally encodes a great deal faster (often an order of magnitude or more.) The version of the TurboVNC encoder included in this patch is roughly equivalent to the one found in version 0.6 of the Unix TurboVNC Server, with a few minor patches integrated from TurboVNC 1.1. TurboVNC 1.0 added multi-threading capabilities, which can be added in later if desired (at the expense of making libvncserver depend on libpthread.) Because TurboVNC uses a fundamentally different mix of subencodings than TightVNC, because it uses the identical protocol (and thus a viewer really has no idea whether it's talking to a TightVNC or TurboVNC server), and because it doesn't support rfbTightPng (and in fact conflicts with it-- see below), the TurboVNC and TightVNC encoders cannot be enabled simultaneously. Compatibility: In *most* cases, a TurboVNC-enabled viewer is fully compatible with a TightVNC server, and vice versa. TurboVNC supports pseudo-encodings for specifying a fine-grained (1-100) quality scale and specifying chrominance subsampling. If a TurboVNC viewer sends those to a TightVNC server, then the TightVNC server ignores them, so the TurboVNC viewer also sends the quality on a 0-9 scale that the TightVNC server can understand. Similarly, the TurboVNC server checks first for fine-grained quality and subsampling pseudo-encodings from the viewer, and failing to receive those, it then checks for the TightVNC 0-9 quality pseudo-encoding. There is one case in which the two systems are not compatible, and that is when a TightVNC or TigerVNC viewer requests compression level 0 without JPEG from a TurboVNC server. For performance reasons, this causes the TurboVNC server to send images directly to the viewer, bypassing Zlib. When the TurboVNC server does this, it also sets bits 7-4 in the compression control byte to rfbTightNoZlib (0x0A), which is unfortunately the same value as rfbTightPng. Older TightVNC viewers that don't handle PNG will assume that the stream is uncompressed but still encapsulated in a Zlib structure, whereas newer PNG-supporting TightVNC viewers will assume that the stream is PNG. In either case, the viewer will probably crash. Since most VNC viewers don't expose compression level 0 in the GUI, this is a relatively rare situation. Description of changes: configure.ac -- Added support for libjpeg-turbo. If passed an argument of --with-turbovnc, configure will now run (or, if cross-compiling, just link) a test program that determines whether the libjpeg library being used is libjpeg-turbo. libjpeg-turbo must be used when building the TurboVNC encoder, because the TurboVNC encoder relies on the libjpeg-turbo colorspace extensions in order to compress images directly out of the framebuffer (which may be, for instance, BGRA rather than RGB.) libjpeg-turbo can optionally be used with the TightVNC encoder as well, but the speedup will only be marginal (the report linked above explains why in more detail, but basically it's because of Amdahl's Law. The TightVNC encoder was designed with the assumption that JPEG had a very high CPU cost, and thus JPEG is used only sparingly.) -- Added a new configure variable, JPEG_LDFLAGS. This is necessitated by the fact that libjpeg-turbo often distributes libjpeg.a and libjpeg.so in /opt/libjpeg-turbo/lib32 or /opt/libjpeg-turbo/lib64, and many people prefer to statically link with it. Thus, more flexibility is needed than is provided by --with-jpeg. If JPEG_LDFLAGS is specified, then it overrides the changes to LDFLAGS enacted by --with-jpeg (but --with-jpeg is still used to set the include path.) The addition of JPEG_LDFLAGS necessitated replacing AC_CHECK_LIB with AC_LINK_IFELSE (because AC_CHECK_LIB automatically sets LIBS to -ljpeg, which is not what we want if we're, for instance, linking statically with libjpeg-turbo.) -- configure does not check for PNG support if TurboVNC encoding is enabled. This prevents the rfbSendRectEncodingTightPng() function from being compiled in, since the TurboVNC encoder doesn't (and can't) support it. common/turbojpeg.c, common/turbojpeg.h -- TurboJPEG is a simple API used to compress and decompress JPEG images in memory. It was originally implemented because it was desirable to use different types of underlying technologies to compress JPEG on different platforms (mediaLib on SPARC, Quicktime on PPC Macs, Intel Performance Primitives, etc.) These days, however, libjpeg-turbo is the only underlying technology used by TurboVNC, so TurboJPEG's purpose is largely just code simplicity and flexibility. Thus, since there is no real need for libvncserver to use any technology other than libjpeg-turbo for compressing JPEG, the TurboJPEG wrapper for libjpeg-turbo has been included in-tree so that libvncserver can be directly linked with libjpeg-turbo. This is convenient because many modern Linux distros (Fedora, Ubuntu, etc.) now ship libjpeg-turbo as their default libjpeg library. libvncserver/rfbserver.c -- Added logic to check for the TurboVNC fine-grained quality level and subsampling encodings and to map Tight (0-9) quality levels to appropriate fine-grained quality level and subsampling values if communicating with a TightVNC/TigerVNC viewer. libvncserver/turbo.c -- TurboVNC encoder (compiled instead of libvncserver/tight.c) rfb/rfb.h -- Added support for the TurboVNC subsampling level rfb/rfbproto.h -- Added constants for the TurboVNC fine quality level and subsampling encodings as well as the rfbTightNoZlib constant and notes on its usage.
13 years ago
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
*width=dinfo->image_width;
*height=dinfo->image_height;
*jpegSubsamp=getSubsamp(dinfo);
Add TurboVNC encoding support. TurboVNC is a variant of TightVNC that uses the same client/server protocol (RFB version 3.8t), and thus it is fully cross-compatible with TightVNC and TigerVNC (with one exception, which is noted below.) Both the TightVNC and TurboVNC encoders analyze each rectangle, pick out regions of solid color to send separately, and send the remaining subrectangles using mono, indexed color, JPEG, or raw encoding, depending on the number of colors in the subrectangle. However, TurboVNC uses a fundamentally different selection algorithm to determine the appropriate subencoding to use for each subrectangle. Thus, while it sends a protocol stream that can be decoded by any TightVNC-compatible viewer, the mix of subencoding types in this protocol stream will be different from those generated by a TightVNC server. The research that led to TurboVNC is described in the following report: http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf. In summary: 20 RFB captures, representing "common" 2D and 3D application workloads (the 3D workloads were run using VirtualGL), were studied using the TightVNC encoder in isolation. Some of the analysis features in the TightVNC encoder, such as smoothness detection, were found to generate a lot of CPU usage with little or no benefit in compression, so those features were disabled. JPEG encoding was accelerated using libjpeg-turbo (which achieves a 2-4x speedup over plain libjpeg on modern x86 or ARM processors.) Finally, the "palette threshold" (minimum number of colors that the subrectangle must have before it is compressed using JPEG or raw) was adjusted to account for the fact that JPEG encoding is now quite a bit faster (meaning that we can now use it more without a CPU penalty.) TurboVNC has additional optimizations, such as the ability to count colors and encode JPEG images directly from the framebuffer without first translating the pixels into RGB. The TurboVNC encoder compares quite favorably in terms of compression ratio with TightVNC and generally encodes a great deal faster (often an order of magnitude or more.) The version of the TurboVNC encoder included in this patch is roughly equivalent to the one found in version 0.6 of the Unix TurboVNC Server, with a few minor patches integrated from TurboVNC 1.1. TurboVNC 1.0 added multi-threading capabilities, which can be added in later if desired (at the expense of making libvncserver depend on libpthread.) Because TurboVNC uses a fundamentally different mix of subencodings than TightVNC, because it uses the identical protocol (and thus a viewer really has no idea whether it's talking to a TightVNC or TurboVNC server), and because it doesn't support rfbTightPng (and in fact conflicts with it-- see below), the TurboVNC and TightVNC encoders cannot be enabled simultaneously. Compatibility: In *most* cases, a TurboVNC-enabled viewer is fully compatible with a TightVNC server, and vice versa. TurboVNC supports pseudo-encodings for specifying a fine-grained (1-100) quality scale and specifying chrominance subsampling. If a TurboVNC viewer sends those to a TightVNC server, then the TightVNC server ignores them, so the TurboVNC viewer also sends the quality on a 0-9 scale that the TightVNC server can understand. Similarly, the TurboVNC server checks first for fine-grained quality and subsampling pseudo-encodings from the viewer, and failing to receive those, it then checks for the TightVNC 0-9 quality pseudo-encoding. There is one case in which the two systems are not compatible, and that is when a TightVNC or TigerVNC viewer requests compression level 0 without JPEG from a TurboVNC server. For performance reasons, this causes the TurboVNC server to send images directly to the viewer, bypassing Zlib. When the TurboVNC server does this, it also sets bits 7-4 in the compression control byte to rfbTightNoZlib (0x0A), which is unfortunately the same value as rfbTightPng. Older TightVNC viewers that don't handle PNG will assume that the stream is uncompressed but still encapsulated in a Zlib structure, whereas newer PNG-supporting TightVNC viewers will assume that the stream is PNG. In either case, the viewer will probably crash. Since most VNC viewers don't expose compression level 0 in the GUI, this is a relatively rare situation. Description of changes: configure.ac -- Added support for libjpeg-turbo. If passed an argument of --with-turbovnc, configure will now run (or, if cross-compiling, just link) a test program that determines whether the libjpeg library being used is libjpeg-turbo. libjpeg-turbo must be used when building the TurboVNC encoder, because the TurboVNC encoder relies on the libjpeg-turbo colorspace extensions in order to compress images directly out of the framebuffer (which may be, for instance, BGRA rather than RGB.) libjpeg-turbo can optionally be used with the TightVNC encoder as well, but the speedup will only be marginal (the report linked above explains why in more detail, but basically it's because of Amdahl's Law. The TightVNC encoder was designed with the assumption that JPEG had a very high CPU cost, and thus JPEG is used only sparingly.) -- Added a new configure variable, JPEG_LDFLAGS. This is necessitated by the fact that libjpeg-turbo often distributes libjpeg.a and libjpeg.so in /opt/libjpeg-turbo/lib32 or /opt/libjpeg-turbo/lib64, and many people prefer to statically link with it. Thus, more flexibility is needed than is provided by --with-jpeg. If JPEG_LDFLAGS is specified, then it overrides the changes to LDFLAGS enacted by --with-jpeg (but --with-jpeg is still used to set the include path.) The addition of JPEG_LDFLAGS necessitated replacing AC_CHECK_LIB with AC_LINK_IFELSE (because AC_CHECK_LIB automatically sets LIBS to -ljpeg, which is not what we want if we're, for instance, linking statically with libjpeg-turbo.) -- configure does not check for PNG support if TurboVNC encoding is enabled. This prevents the rfbSendRectEncodingTightPng() function from being compiled in, since the TurboVNC encoder doesn't (and can't) support it. common/turbojpeg.c, common/turbojpeg.h -- TurboJPEG is a simple API used to compress and decompress JPEG images in memory. It was originally implemented because it was desirable to use different types of underlying technologies to compress JPEG on different platforms (mediaLib on SPARC, Quicktime on PPC Macs, Intel Performance Primitives, etc.) These days, however, libjpeg-turbo is the only underlying technology used by TurboVNC, so TurboJPEG's purpose is largely just code simplicity and flexibility. Thus, since there is no real need for libvncserver to use any technology other than libjpeg-turbo for compressing JPEG, the TurboJPEG wrapper for libjpeg-turbo has been included in-tree so that libvncserver can be directly linked with libjpeg-turbo. This is convenient because many modern Linux distros (Fedora, Ubuntu, etc.) now ship libjpeg-turbo as their default libjpeg library. libvncserver/rfbserver.c -- Added logic to check for the TurboVNC fine-grained quality level and subsampling encodings and to map Tight (0-9) quality levels to appropriate fine-grained quality level and subsampling values if communicating with a TightVNC/TigerVNC viewer. libvncserver/turbo.c -- TurboVNC encoder (compiled instead of libvncserver/tight.c) rfb/rfb.h -- Added support for the TurboVNC subsampling level rfb/rfbproto.h -- Added constants for the TurboVNC fine quality level and subsampling encodings as well as the rfbTightNoZlib constant and notes on its usage.
13 years ago
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
jpeg_abort_decompress(dinfo);
Add TurboVNC encoding support. TurboVNC is a variant of TightVNC that uses the same client/server protocol (RFB version 3.8t), and thus it is fully cross-compatible with TightVNC and TigerVNC (with one exception, which is noted below.) Both the TightVNC and TurboVNC encoders analyze each rectangle, pick out regions of solid color to send separately, and send the remaining subrectangles using mono, indexed color, JPEG, or raw encoding, depending on the number of colors in the subrectangle. However, TurboVNC uses a fundamentally different selection algorithm to determine the appropriate subencoding to use for each subrectangle. Thus, while it sends a protocol stream that can be decoded by any TightVNC-compatible viewer, the mix of subencoding types in this protocol stream will be different from those generated by a TightVNC server. The research that led to TurboVNC is described in the following report: http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf. In summary: 20 RFB captures, representing "common" 2D and 3D application workloads (the 3D workloads were run using VirtualGL), were studied using the TightVNC encoder in isolation. Some of the analysis features in the TightVNC encoder, such as smoothness detection, were found to generate a lot of CPU usage with little or no benefit in compression, so those features were disabled. JPEG encoding was accelerated using libjpeg-turbo (which achieves a 2-4x speedup over plain libjpeg on modern x86 or ARM processors.) Finally, the "palette threshold" (minimum number of colors that the subrectangle must have before it is compressed using JPEG or raw) was adjusted to account for the fact that JPEG encoding is now quite a bit faster (meaning that we can now use it more without a CPU penalty.) TurboVNC has additional optimizations, such as the ability to count colors and encode JPEG images directly from the framebuffer without first translating the pixels into RGB. The TurboVNC encoder compares quite favorably in terms of compression ratio with TightVNC and generally encodes a great deal faster (often an order of magnitude or more.) The version of the TurboVNC encoder included in this patch is roughly equivalent to the one found in version 0.6 of the Unix TurboVNC Server, with a few minor patches integrated from TurboVNC 1.1. TurboVNC 1.0 added multi-threading capabilities, which can be added in later if desired (at the expense of making libvncserver depend on libpthread.) Because TurboVNC uses a fundamentally different mix of subencodings than TightVNC, because it uses the identical protocol (and thus a viewer really has no idea whether it's talking to a TightVNC or TurboVNC server), and because it doesn't support rfbTightPng (and in fact conflicts with it-- see below), the TurboVNC and TightVNC encoders cannot be enabled simultaneously. Compatibility: In *most* cases, a TurboVNC-enabled viewer is fully compatible with a TightVNC server, and vice versa. TurboVNC supports pseudo-encodings for specifying a fine-grained (1-100) quality scale and specifying chrominance subsampling. If a TurboVNC viewer sends those to a TightVNC server, then the TightVNC server ignores them, so the TurboVNC viewer also sends the quality on a 0-9 scale that the TightVNC server can understand. Similarly, the TurboVNC server checks first for fine-grained quality and subsampling pseudo-encodings from the viewer, and failing to receive those, it then checks for the TightVNC 0-9 quality pseudo-encoding. There is one case in which the two systems are not compatible, and that is when a TightVNC or TigerVNC viewer requests compression level 0 without JPEG from a TurboVNC server. For performance reasons, this causes the TurboVNC server to send images directly to the viewer, bypassing Zlib. When the TurboVNC server does this, it also sets bits 7-4 in the compression control byte to rfbTightNoZlib (0x0A), which is unfortunately the same value as rfbTightPng. Older TightVNC viewers that don't handle PNG will assume that the stream is uncompressed but still encapsulated in a Zlib structure, whereas newer PNG-supporting TightVNC viewers will assume that the stream is PNG. In either case, the viewer will probably crash. Since most VNC viewers don't expose compression level 0 in the GUI, this is a relatively rare situation. Description of changes: configure.ac -- Added support for libjpeg-turbo. If passed an argument of --with-turbovnc, configure will now run (or, if cross-compiling, just link) a test program that determines whether the libjpeg library being used is libjpeg-turbo. libjpeg-turbo must be used when building the TurboVNC encoder, because the TurboVNC encoder relies on the libjpeg-turbo colorspace extensions in order to compress images directly out of the framebuffer (which may be, for instance, BGRA rather than RGB.) libjpeg-turbo can optionally be used with the TightVNC encoder as well, but the speedup will only be marginal (the report linked above explains why in more detail, but basically it's because of Amdahl's Law. The TightVNC encoder was designed with the assumption that JPEG had a very high CPU cost, and thus JPEG is used only sparingly.) -- Added a new configure variable, JPEG_LDFLAGS. This is necessitated by the fact that libjpeg-turbo often distributes libjpeg.a and libjpeg.so in /opt/libjpeg-turbo/lib32 or /opt/libjpeg-turbo/lib64, and many people prefer to statically link with it. Thus, more flexibility is needed than is provided by --with-jpeg. If JPEG_LDFLAGS is specified, then it overrides the changes to LDFLAGS enacted by --with-jpeg (but --with-jpeg is still used to set the include path.) The addition of JPEG_LDFLAGS necessitated replacing AC_CHECK_LIB with AC_LINK_IFELSE (because AC_CHECK_LIB automatically sets LIBS to -ljpeg, which is not what we want if we're, for instance, linking statically with libjpeg-turbo.) -- configure does not check for PNG support if TurboVNC encoding is enabled. This prevents the rfbSendRectEncodingTightPng() function from being compiled in, since the TurboVNC encoder doesn't (and can't) support it. common/turbojpeg.c, common/turbojpeg.h -- TurboJPEG is a simple API used to compress and decompress JPEG images in memory. It was originally implemented because it was desirable to use different types of underlying technologies to compress JPEG on different platforms (mediaLib on SPARC, Quicktime on PPC Macs, Intel Performance Primitives, etc.) These days, however, libjpeg-turbo is the only underlying technology used by TurboVNC, so TurboJPEG's purpose is largely just code simplicity and flexibility. Thus, since there is no real need for libvncserver to use any technology other than libjpeg-turbo for compressing JPEG, the TurboJPEG wrapper for libjpeg-turbo has been included in-tree so that libvncserver can be directly linked with libjpeg-turbo. This is convenient because many modern Linux distros (Fedora, Ubuntu, etc.) now ship libjpeg-turbo as their default libjpeg library. libvncserver/rfbserver.c -- Added logic to check for the TurboVNC fine-grained quality level and subsampling encodings and to map Tight (0-9) quality levels to appropriate fine-grained quality level and subsampling values if communicating with a TightVNC/TigerVNC viewer. libvncserver/turbo.c -- TurboVNC encoder (compiled instead of libvncserver/tight.c) rfb/rfb.h -- Added support for the TurboVNC subsampling level rfb/rfbproto.h -- Added constants for the TurboVNC fine quality level and subsampling encodings as well as the rfbTightNoZlib constant and notes on its usage.
13 years ago
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
if(*jpegSubsamp<0)
_throw("tjDecompressHeader2(): Could not determine subsampling type for JPEG image");
if(*width<1 || *height<1)
_throw("tjDecompressHeader2(): Invalid data returned in header");
Add TurboVNC encoding support. TurboVNC is a variant of TightVNC that uses the same client/server protocol (RFB version 3.8t), and thus it is fully cross-compatible with TightVNC and TigerVNC (with one exception, which is noted below.) Both the TightVNC and TurboVNC encoders analyze each rectangle, pick out regions of solid color to send separately, and send the remaining subrectangles using mono, indexed color, JPEG, or raw encoding, depending on the number of colors in the subrectangle. However, TurboVNC uses a fundamentally different selection algorithm to determine the appropriate subencoding to use for each subrectangle. Thus, while it sends a protocol stream that can be decoded by any TightVNC-compatible viewer, the mix of subencoding types in this protocol stream will be different from those generated by a TightVNC server. The research that led to TurboVNC is described in the following report: http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf. In summary: 20 RFB captures, representing "common" 2D and 3D application workloads (the 3D workloads were run using VirtualGL), were studied using the TightVNC encoder in isolation. Some of the analysis features in the TightVNC encoder, such as smoothness detection, were found to generate a lot of CPU usage with little or no benefit in compression, so those features were disabled. JPEG encoding was accelerated using libjpeg-turbo (which achieves a 2-4x speedup over plain libjpeg on modern x86 or ARM processors.) Finally, the "palette threshold" (minimum number of colors that the subrectangle must have before it is compressed using JPEG or raw) was adjusted to account for the fact that JPEG encoding is now quite a bit faster (meaning that we can now use it more without a CPU penalty.) TurboVNC has additional optimizations, such as the ability to count colors and encode JPEG images directly from the framebuffer without first translating the pixels into RGB. The TurboVNC encoder compares quite favorably in terms of compression ratio with TightVNC and generally encodes a great deal faster (often an order of magnitude or more.) The version of the TurboVNC encoder included in this patch is roughly equivalent to the one found in version 0.6 of the Unix TurboVNC Server, with a few minor patches integrated from TurboVNC 1.1. TurboVNC 1.0 added multi-threading capabilities, which can be added in later if desired (at the expense of making libvncserver depend on libpthread.) Because TurboVNC uses a fundamentally different mix of subencodings than TightVNC, because it uses the identical protocol (and thus a viewer really has no idea whether it's talking to a TightVNC or TurboVNC server), and because it doesn't support rfbTightPng (and in fact conflicts with it-- see below), the TurboVNC and TightVNC encoders cannot be enabled simultaneously. Compatibility: In *most* cases, a TurboVNC-enabled viewer is fully compatible with a TightVNC server, and vice versa. TurboVNC supports pseudo-encodings for specifying a fine-grained (1-100) quality scale and specifying chrominance subsampling. If a TurboVNC viewer sends those to a TightVNC server, then the TightVNC server ignores them, so the TurboVNC viewer also sends the quality on a 0-9 scale that the TightVNC server can understand. Similarly, the TurboVNC server checks first for fine-grained quality and subsampling pseudo-encodings from the viewer, and failing to receive those, it then checks for the TightVNC 0-9 quality pseudo-encoding. There is one case in which the two systems are not compatible, and that is when a TightVNC or TigerVNC viewer requests compression level 0 without JPEG from a TurboVNC server. For performance reasons, this causes the TurboVNC server to send images directly to the viewer, bypassing Zlib. When the TurboVNC server does this, it also sets bits 7-4 in the compression control byte to rfbTightNoZlib (0x0A), which is unfortunately the same value as rfbTightPng. Older TightVNC viewers that don't handle PNG will assume that the stream is uncompressed but still encapsulated in a Zlib structure, whereas newer PNG-supporting TightVNC viewers will assume that the stream is PNG. In either case, the viewer will probably crash. Since most VNC viewers don't expose compression level 0 in the GUI, this is a relatively rare situation. Description of changes: configure.ac -- Added support for libjpeg-turbo. If passed an argument of --with-turbovnc, configure will now run (or, if cross-compiling, just link) a test program that determines whether the libjpeg library being used is libjpeg-turbo. libjpeg-turbo must be used when building the TurboVNC encoder, because the TurboVNC encoder relies on the libjpeg-turbo colorspace extensions in order to compress images directly out of the framebuffer (which may be, for instance, BGRA rather than RGB.) libjpeg-turbo can optionally be used with the TightVNC encoder as well, but the speedup will only be marginal (the report linked above explains why in more detail, but basically it's because of Amdahl's Law. The TightVNC encoder was designed with the assumption that JPEG had a very high CPU cost, and thus JPEG is used only sparingly.) -- Added a new configure variable, JPEG_LDFLAGS. This is necessitated by the fact that libjpeg-turbo often distributes libjpeg.a and libjpeg.so in /opt/libjpeg-turbo/lib32 or /opt/libjpeg-turbo/lib64, and many people prefer to statically link with it. Thus, more flexibility is needed than is provided by --with-jpeg. If JPEG_LDFLAGS is specified, then it overrides the changes to LDFLAGS enacted by --with-jpeg (but --with-jpeg is still used to set the include path.) The addition of JPEG_LDFLAGS necessitated replacing AC_CHECK_LIB with AC_LINK_IFELSE (because AC_CHECK_LIB automatically sets LIBS to -ljpeg, which is not what we want if we're, for instance, linking statically with libjpeg-turbo.) -- configure does not check for PNG support if TurboVNC encoding is enabled. This prevents the rfbSendRectEncodingTightPng() function from being compiled in, since the TurboVNC encoder doesn't (and can't) support it. common/turbojpeg.c, common/turbojpeg.h -- TurboJPEG is a simple API used to compress and decompress JPEG images in memory. It was originally implemented because it was desirable to use different types of underlying technologies to compress JPEG on different platforms (mediaLib on SPARC, Quicktime on PPC Macs, Intel Performance Primitives, etc.) These days, however, libjpeg-turbo is the only underlying technology used by TurboVNC, so TurboJPEG's purpose is largely just code simplicity and flexibility. Thus, since there is no real need for libvncserver to use any technology other than libjpeg-turbo for compressing JPEG, the TurboJPEG wrapper for libjpeg-turbo has been included in-tree so that libvncserver can be directly linked with libjpeg-turbo. This is convenient because many modern Linux distros (Fedora, Ubuntu, etc.) now ship libjpeg-turbo as their default libjpeg library. libvncserver/rfbserver.c -- Added logic to check for the TurboVNC fine-grained quality level and subsampling encodings and to map Tight (0-9) quality levels to appropriate fine-grained quality level and subsampling values if communicating with a TightVNC/TigerVNC viewer. libvncserver/turbo.c -- TurboVNC encoder (compiled instead of libvncserver/tight.c) rfb/rfb.h -- Added support for the TurboVNC subsampling level rfb/rfbproto.h -- Added constants for the TurboVNC fine quality level and subsampling encodings as well as the rfbTightNoZlib constant and notes on its usage.
13 years ago
bailout:
return retval;
}
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
DLLEXPORT int DLLCALL tjDecompressHeader(tjhandle handle,
unsigned char *jpegBuf, unsigned long jpegSize, int *width, int *height)
{
int jpegSubsamp;
return tjDecompressHeader2(handle, jpegBuf, jpegSize, width, height,
&jpegSubsamp);
}
Add TurboVNC encoding support. TurboVNC is a variant of TightVNC that uses the same client/server protocol (RFB version 3.8t), and thus it is fully cross-compatible with TightVNC and TigerVNC (with one exception, which is noted below.) Both the TightVNC and TurboVNC encoders analyze each rectangle, pick out regions of solid color to send separately, and send the remaining subrectangles using mono, indexed color, JPEG, or raw encoding, depending on the number of colors in the subrectangle. However, TurboVNC uses a fundamentally different selection algorithm to determine the appropriate subencoding to use for each subrectangle. Thus, while it sends a protocol stream that can be decoded by any TightVNC-compatible viewer, the mix of subencoding types in this protocol stream will be different from those generated by a TightVNC server. The research that led to TurboVNC is described in the following report: http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf. In summary: 20 RFB captures, representing "common" 2D and 3D application workloads (the 3D workloads were run using VirtualGL), were studied using the TightVNC encoder in isolation. Some of the analysis features in the TightVNC encoder, such as smoothness detection, were found to generate a lot of CPU usage with little or no benefit in compression, so those features were disabled. JPEG encoding was accelerated using libjpeg-turbo (which achieves a 2-4x speedup over plain libjpeg on modern x86 or ARM processors.) Finally, the "palette threshold" (minimum number of colors that the subrectangle must have before it is compressed using JPEG or raw) was adjusted to account for the fact that JPEG encoding is now quite a bit faster (meaning that we can now use it more without a CPU penalty.) TurboVNC has additional optimizations, such as the ability to count colors and encode JPEG images directly from the framebuffer without first translating the pixels into RGB. The TurboVNC encoder compares quite favorably in terms of compression ratio with TightVNC and generally encodes a great deal faster (often an order of magnitude or more.) The version of the TurboVNC encoder included in this patch is roughly equivalent to the one found in version 0.6 of the Unix TurboVNC Server, with a few minor patches integrated from TurboVNC 1.1. TurboVNC 1.0 added multi-threading capabilities, which can be added in later if desired (at the expense of making libvncserver depend on libpthread.) Because TurboVNC uses a fundamentally different mix of subencodings than TightVNC, because it uses the identical protocol (and thus a viewer really has no idea whether it's talking to a TightVNC or TurboVNC server), and because it doesn't support rfbTightPng (and in fact conflicts with it-- see below), the TurboVNC and TightVNC encoders cannot be enabled simultaneously. Compatibility: In *most* cases, a TurboVNC-enabled viewer is fully compatible with a TightVNC server, and vice versa. TurboVNC supports pseudo-encodings for specifying a fine-grained (1-100) quality scale and specifying chrominance subsampling. If a TurboVNC viewer sends those to a TightVNC server, then the TightVNC server ignores them, so the TurboVNC viewer also sends the quality on a 0-9 scale that the TightVNC server can understand. Similarly, the TurboVNC server checks first for fine-grained quality and subsampling pseudo-encodings from the viewer, and failing to receive those, it then checks for the TightVNC 0-9 quality pseudo-encoding. There is one case in which the two systems are not compatible, and that is when a TightVNC or TigerVNC viewer requests compression level 0 without JPEG from a TurboVNC server. For performance reasons, this causes the TurboVNC server to send images directly to the viewer, bypassing Zlib. When the TurboVNC server does this, it also sets bits 7-4 in the compression control byte to rfbTightNoZlib (0x0A), which is unfortunately the same value as rfbTightPng. Older TightVNC viewers that don't handle PNG will assume that the stream is uncompressed but still encapsulated in a Zlib structure, whereas newer PNG-supporting TightVNC viewers will assume that the stream is PNG. In either case, the viewer will probably crash. Since most VNC viewers don't expose compression level 0 in the GUI, this is a relatively rare situation. Description of changes: configure.ac -- Added support for libjpeg-turbo. If passed an argument of --with-turbovnc, configure will now run (or, if cross-compiling, just link) a test program that determines whether the libjpeg library being used is libjpeg-turbo. libjpeg-turbo must be used when building the TurboVNC encoder, because the TurboVNC encoder relies on the libjpeg-turbo colorspace extensions in order to compress images directly out of the framebuffer (which may be, for instance, BGRA rather than RGB.) libjpeg-turbo can optionally be used with the TightVNC encoder as well, but the speedup will only be marginal (the report linked above explains why in more detail, but basically it's because of Amdahl's Law. The TightVNC encoder was designed with the assumption that JPEG had a very high CPU cost, and thus JPEG is used only sparingly.) -- Added a new configure variable, JPEG_LDFLAGS. This is necessitated by the fact that libjpeg-turbo often distributes libjpeg.a and libjpeg.so in /opt/libjpeg-turbo/lib32 or /opt/libjpeg-turbo/lib64, and many people prefer to statically link with it. Thus, more flexibility is needed than is provided by --with-jpeg. If JPEG_LDFLAGS is specified, then it overrides the changes to LDFLAGS enacted by --with-jpeg (but --with-jpeg is still used to set the include path.) The addition of JPEG_LDFLAGS necessitated replacing AC_CHECK_LIB with AC_LINK_IFELSE (because AC_CHECK_LIB automatically sets LIBS to -ljpeg, which is not what we want if we're, for instance, linking statically with libjpeg-turbo.) -- configure does not check for PNG support if TurboVNC encoding is enabled. This prevents the rfbSendRectEncodingTightPng() function from being compiled in, since the TurboVNC encoder doesn't (and can't) support it. common/turbojpeg.c, common/turbojpeg.h -- TurboJPEG is a simple API used to compress and decompress JPEG images in memory. It was originally implemented because it was desirable to use different types of underlying technologies to compress JPEG on different platforms (mediaLib on SPARC, Quicktime on PPC Macs, Intel Performance Primitives, etc.) These days, however, libjpeg-turbo is the only underlying technology used by TurboVNC, so TurboJPEG's purpose is largely just code simplicity and flexibility. Thus, since there is no real need for libvncserver to use any technology other than libjpeg-turbo for compressing JPEG, the TurboJPEG wrapper for libjpeg-turbo has been included in-tree so that libvncserver can be directly linked with libjpeg-turbo. This is convenient because many modern Linux distros (Fedora, Ubuntu, etc.) now ship libjpeg-turbo as their default libjpeg library. libvncserver/rfbserver.c -- Added logic to check for the TurboVNC fine-grained quality level and subsampling encodings and to map Tight (0-9) quality levels to appropriate fine-grained quality level and subsampling values if communicating with a TightVNC/TigerVNC viewer. libvncserver/turbo.c -- TurboVNC encoder (compiled instead of libvncserver/tight.c) rfb/rfb.h -- Added support for the TurboVNC subsampling level rfb/rfbproto.h -- Added constants for the TurboVNC fine quality level and subsampling encodings as well as the rfbTightNoZlib constant and notes on its usage.
13 years ago
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
DLLEXPORT tjscalingfactor* DLLCALL tjGetScalingFactors(int *numscalingfactors)
Add TurboVNC encoding support. TurboVNC is a variant of TightVNC that uses the same client/server protocol (RFB version 3.8t), and thus it is fully cross-compatible with TightVNC and TigerVNC (with one exception, which is noted below.) Both the TightVNC and TurboVNC encoders analyze each rectangle, pick out regions of solid color to send separately, and send the remaining subrectangles using mono, indexed color, JPEG, or raw encoding, depending on the number of colors in the subrectangle. However, TurboVNC uses a fundamentally different selection algorithm to determine the appropriate subencoding to use for each subrectangle. Thus, while it sends a protocol stream that can be decoded by any TightVNC-compatible viewer, the mix of subencoding types in this protocol stream will be different from those generated by a TightVNC server. The research that led to TurboVNC is described in the following report: http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf. In summary: 20 RFB captures, representing "common" 2D and 3D application workloads (the 3D workloads were run using VirtualGL), were studied using the TightVNC encoder in isolation. Some of the analysis features in the TightVNC encoder, such as smoothness detection, were found to generate a lot of CPU usage with little or no benefit in compression, so those features were disabled. JPEG encoding was accelerated using libjpeg-turbo (which achieves a 2-4x speedup over plain libjpeg on modern x86 or ARM processors.) Finally, the "palette threshold" (minimum number of colors that the subrectangle must have before it is compressed using JPEG or raw) was adjusted to account for the fact that JPEG encoding is now quite a bit faster (meaning that we can now use it more without a CPU penalty.) TurboVNC has additional optimizations, such as the ability to count colors and encode JPEG images directly from the framebuffer without first translating the pixels into RGB. The TurboVNC encoder compares quite favorably in terms of compression ratio with TightVNC and generally encodes a great deal faster (often an order of magnitude or more.) The version of the TurboVNC encoder included in this patch is roughly equivalent to the one found in version 0.6 of the Unix TurboVNC Server, with a few minor patches integrated from TurboVNC 1.1. TurboVNC 1.0 added multi-threading capabilities, which can be added in later if desired (at the expense of making libvncserver depend on libpthread.) Because TurboVNC uses a fundamentally different mix of subencodings than TightVNC, because it uses the identical protocol (and thus a viewer really has no idea whether it's talking to a TightVNC or TurboVNC server), and because it doesn't support rfbTightPng (and in fact conflicts with it-- see below), the TurboVNC and TightVNC encoders cannot be enabled simultaneously. Compatibility: In *most* cases, a TurboVNC-enabled viewer is fully compatible with a TightVNC server, and vice versa. TurboVNC supports pseudo-encodings for specifying a fine-grained (1-100) quality scale and specifying chrominance subsampling. If a TurboVNC viewer sends those to a TightVNC server, then the TightVNC server ignores them, so the TurboVNC viewer also sends the quality on a 0-9 scale that the TightVNC server can understand. Similarly, the TurboVNC server checks first for fine-grained quality and subsampling pseudo-encodings from the viewer, and failing to receive those, it then checks for the TightVNC 0-9 quality pseudo-encoding. There is one case in which the two systems are not compatible, and that is when a TightVNC or TigerVNC viewer requests compression level 0 without JPEG from a TurboVNC server. For performance reasons, this causes the TurboVNC server to send images directly to the viewer, bypassing Zlib. When the TurboVNC server does this, it also sets bits 7-4 in the compression control byte to rfbTightNoZlib (0x0A), which is unfortunately the same value as rfbTightPng. Older TightVNC viewers that don't handle PNG will assume that the stream is uncompressed but still encapsulated in a Zlib structure, whereas newer PNG-supporting TightVNC viewers will assume that the stream is PNG. In either case, the viewer will probably crash. Since most VNC viewers don't expose compression level 0 in the GUI, this is a relatively rare situation. Description of changes: configure.ac -- Added support for libjpeg-turbo. If passed an argument of --with-turbovnc, configure will now run (or, if cross-compiling, just link) a test program that determines whether the libjpeg library being used is libjpeg-turbo. libjpeg-turbo must be used when building the TurboVNC encoder, because the TurboVNC encoder relies on the libjpeg-turbo colorspace extensions in order to compress images directly out of the framebuffer (which may be, for instance, BGRA rather than RGB.) libjpeg-turbo can optionally be used with the TightVNC encoder as well, but the speedup will only be marginal (the report linked above explains why in more detail, but basically it's because of Amdahl's Law. The TightVNC encoder was designed with the assumption that JPEG had a very high CPU cost, and thus JPEG is used only sparingly.) -- Added a new configure variable, JPEG_LDFLAGS. This is necessitated by the fact that libjpeg-turbo often distributes libjpeg.a and libjpeg.so in /opt/libjpeg-turbo/lib32 or /opt/libjpeg-turbo/lib64, and many people prefer to statically link with it. Thus, more flexibility is needed than is provided by --with-jpeg. If JPEG_LDFLAGS is specified, then it overrides the changes to LDFLAGS enacted by --with-jpeg (but --with-jpeg is still used to set the include path.) The addition of JPEG_LDFLAGS necessitated replacing AC_CHECK_LIB with AC_LINK_IFELSE (because AC_CHECK_LIB automatically sets LIBS to -ljpeg, which is not what we want if we're, for instance, linking statically with libjpeg-turbo.) -- configure does not check for PNG support if TurboVNC encoding is enabled. This prevents the rfbSendRectEncodingTightPng() function from being compiled in, since the TurboVNC encoder doesn't (and can't) support it. common/turbojpeg.c, common/turbojpeg.h -- TurboJPEG is a simple API used to compress and decompress JPEG images in memory. It was originally implemented because it was desirable to use different types of underlying technologies to compress JPEG on different platforms (mediaLib on SPARC, Quicktime on PPC Macs, Intel Performance Primitives, etc.) These days, however, libjpeg-turbo is the only underlying technology used by TurboVNC, so TurboJPEG's purpose is largely just code simplicity and flexibility. Thus, since there is no real need for libvncserver to use any technology other than libjpeg-turbo for compressing JPEG, the TurboJPEG wrapper for libjpeg-turbo has been included in-tree so that libvncserver can be directly linked with libjpeg-turbo. This is convenient because many modern Linux distros (Fedora, Ubuntu, etc.) now ship libjpeg-turbo as their default libjpeg library. libvncserver/rfbserver.c -- Added logic to check for the TurboVNC fine-grained quality level and subsampling encodings and to map Tight (0-9) quality levels to appropriate fine-grained quality level and subsampling values if communicating with a TightVNC/TigerVNC viewer. libvncserver/turbo.c -- TurboVNC encoder (compiled instead of libvncserver/tight.c) rfb/rfb.h -- Added support for the TurboVNC subsampling level rfb/rfbproto.h -- Added constants for the TurboVNC fine quality level and subsampling encodings as well as the rfbTightNoZlib constant and notes on its usage.
13 years ago
{
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
if(numscalingfactors==NULL)
{
snprintf(errStr, JMSG_LENGTH_MAX,
"tjGetScalingFactors(): Invalid argument");
return NULL;
}
*numscalingfactors=NUMSF;
return (tjscalingfactor *)sf;
Add TurboVNC encoding support. TurboVNC is a variant of TightVNC that uses the same client/server protocol (RFB version 3.8t), and thus it is fully cross-compatible with TightVNC and TigerVNC (with one exception, which is noted below.) Both the TightVNC and TurboVNC encoders analyze each rectangle, pick out regions of solid color to send separately, and send the remaining subrectangles using mono, indexed color, JPEG, or raw encoding, depending on the number of colors in the subrectangle. However, TurboVNC uses a fundamentally different selection algorithm to determine the appropriate subencoding to use for each subrectangle. Thus, while it sends a protocol stream that can be decoded by any TightVNC-compatible viewer, the mix of subencoding types in this protocol stream will be different from those generated by a TightVNC server. The research that led to TurboVNC is described in the following report: http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf. In summary: 20 RFB captures, representing "common" 2D and 3D application workloads (the 3D workloads were run using VirtualGL), were studied using the TightVNC encoder in isolation. Some of the analysis features in the TightVNC encoder, such as smoothness detection, were found to generate a lot of CPU usage with little or no benefit in compression, so those features were disabled. JPEG encoding was accelerated using libjpeg-turbo (which achieves a 2-4x speedup over plain libjpeg on modern x86 or ARM processors.) Finally, the "palette threshold" (minimum number of colors that the subrectangle must have before it is compressed using JPEG or raw) was adjusted to account for the fact that JPEG encoding is now quite a bit faster (meaning that we can now use it more without a CPU penalty.) TurboVNC has additional optimizations, such as the ability to count colors and encode JPEG images directly from the framebuffer without first translating the pixels into RGB. The TurboVNC encoder compares quite favorably in terms of compression ratio with TightVNC and generally encodes a great deal faster (often an order of magnitude or more.) The version of the TurboVNC encoder included in this patch is roughly equivalent to the one found in version 0.6 of the Unix TurboVNC Server, with a few minor patches integrated from TurboVNC 1.1. TurboVNC 1.0 added multi-threading capabilities, which can be added in later if desired (at the expense of making libvncserver depend on libpthread.) Because TurboVNC uses a fundamentally different mix of subencodings than TightVNC, because it uses the identical protocol (and thus a viewer really has no idea whether it's talking to a TightVNC or TurboVNC server), and because it doesn't support rfbTightPng (and in fact conflicts with it-- see below), the TurboVNC and TightVNC encoders cannot be enabled simultaneously. Compatibility: In *most* cases, a TurboVNC-enabled viewer is fully compatible with a TightVNC server, and vice versa. TurboVNC supports pseudo-encodings for specifying a fine-grained (1-100) quality scale and specifying chrominance subsampling. If a TurboVNC viewer sends those to a TightVNC server, then the TightVNC server ignores them, so the TurboVNC viewer also sends the quality on a 0-9 scale that the TightVNC server can understand. Similarly, the TurboVNC server checks first for fine-grained quality and subsampling pseudo-encodings from the viewer, and failing to receive those, it then checks for the TightVNC 0-9 quality pseudo-encoding. There is one case in which the two systems are not compatible, and that is when a TightVNC or TigerVNC viewer requests compression level 0 without JPEG from a TurboVNC server. For performance reasons, this causes the TurboVNC server to send images directly to the viewer, bypassing Zlib. When the TurboVNC server does this, it also sets bits 7-4 in the compression control byte to rfbTightNoZlib (0x0A), which is unfortunately the same value as rfbTightPng. Older TightVNC viewers that don't handle PNG will assume that the stream is uncompressed but still encapsulated in a Zlib structure, whereas newer PNG-supporting TightVNC viewers will assume that the stream is PNG. In either case, the viewer will probably crash. Since most VNC viewers don't expose compression level 0 in the GUI, this is a relatively rare situation. Description of changes: configure.ac -- Added support for libjpeg-turbo. If passed an argument of --with-turbovnc, configure will now run (or, if cross-compiling, just link) a test program that determines whether the libjpeg library being used is libjpeg-turbo. libjpeg-turbo must be used when building the TurboVNC encoder, because the TurboVNC encoder relies on the libjpeg-turbo colorspace extensions in order to compress images directly out of the framebuffer (which may be, for instance, BGRA rather than RGB.) libjpeg-turbo can optionally be used with the TightVNC encoder as well, but the speedup will only be marginal (the report linked above explains why in more detail, but basically it's because of Amdahl's Law. The TightVNC encoder was designed with the assumption that JPEG had a very high CPU cost, and thus JPEG is used only sparingly.) -- Added a new configure variable, JPEG_LDFLAGS. This is necessitated by the fact that libjpeg-turbo often distributes libjpeg.a and libjpeg.so in /opt/libjpeg-turbo/lib32 or /opt/libjpeg-turbo/lib64, and many people prefer to statically link with it. Thus, more flexibility is needed than is provided by --with-jpeg. If JPEG_LDFLAGS is specified, then it overrides the changes to LDFLAGS enacted by --with-jpeg (but --with-jpeg is still used to set the include path.) The addition of JPEG_LDFLAGS necessitated replacing AC_CHECK_LIB with AC_LINK_IFELSE (because AC_CHECK_LIB automatically sets LIBS to -ljpeg, which is not what we want if we're, for instance, linking statically with libjpeg-turbo.) -- configure does not check for PNG support if TurboVNC encoding is enabled. This prevents the rfbSendRectEncodingTightPng() function from being compiled in, since the TurboVNC encoder doesn't (and can't) support it. common/turbojpeg.c, common/turbojpeg.h -- TurboJPEG is a simple API used to compress and decompress JPEG images in memory. It was originally implemented because it was desirable to use different types of underlying technologies to compress JPEG on different platforms (mediaLib on SPARC, Quicktime on PPC Macs, Intel Performance Primitives, etc.) These days, however, libjpeg-turbo is the only underlying technology used by TurboVNC, so TurboJPEG's purpose is largely just code simplicity and flexibility. Thus, since there is no real need for libvncserver to use any technology other than libjpeg-turbo for compressing JPEG, the TurboJPEG wrapper for libjpeg-turbo has been included in-tree so that libvncserver can be directly linked with libjpeg-turbo. This is convenient because many modern Linux distros (Fedora, Ubuntu, etc.) now ship libjpeg-turbo as their default libjpeg library. libvncserver/rfbserver.c -- Added logic to check for the TurboVNC fine-grained quality level and subsampling encodings and to map Tight (0-9) quality levels to appropriate fine-grained quality level and subsampling values if communicating with a TightVNC/TigerVNC viewer. libvncserver/turbo.c -- TurboVNC encoder (compiled instead of libvncserver/tight.c) rfb/rfb.h -- Added support for the TurboVNC subsampling level rfb/rfbproto.h -- Added constants for the TurboVNC fine quality level and subsampling encodings as well as the rfbTightNoZlib constant and notes on its usage.
13 years ago
}
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
DLLEXPORT int DLLCALL tjDecompress2(tjhandle handle, unsigned char *jpegBuf,
unsigned long jpegSize, unsigned char *dstBuf, int width, int pitch,
int height, int pixelFormat, int flags)
Add TurboVNC encoding support. TurboVNC is a variant of TightVNC that uses the same client/server protocol (RFB version 3.8t), and thus it is fully cross-compatible with TightVNC and TigerVNC (with one exception, which is noted below.) Both the TightVNC and TurboVNC encoders analyze each rectangle, pick out regions of solid color to send separately, and send the remaining subrectangles using mono, indexed color, JPEG, or raw encoding, depending on the number of colors in the subrectangle. However, TurboVNC uses a fundamentally different selection algorithm to determine the appropriate subencoding to use for each subrectangle. Thus, while it sends a protocol stream that can be decoded by any TightVNC-compatible viewer, the mix of subencoding types in this protocol stream will be different from those generated by a TightVNC server. The research that led to TurboVNC is described in the following report: http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf. In summary: 20 RFB captures, representing "common" 2D and 3D application workloads (the 3D workloads were run using VirtualGL), were studied using the TightVNC encoder in isolation. Some of the analysis features in the TightVNC encoder, such as smoothness detection, were found to generate a lot of CPU usage with little or no benefit in compression, so those features were disabled. JPEG encoding was accelerated using libjpeg-turbo (which achieves a 2-4x speedup over plain libjpeg on modern x86 or ARM processors.) Finally, the "palette threshold" (minimum number of colors that the subrectangle must have before it is compressed using JPEG or raw) was adjusted to account for the fact that JPEG encoding is now quite a bit faster (meaning that we can now use it more without a CPU penalty.) TurboVNC has additional optimizations, such as the ability to count colors and encode JPEG images directly from the framebuffer without first translating the pixels into RGB. The TurboVNC encoder compares quite favorably in terms of compression ratio with TightVNC and generally encodes a great deal faster (often an order of magnitude or more.) The version of the TurboVNC encoder included in this patch is roughly equivalent to the one found in version 0.6 of the Unix TurboVNC Server, with a few minor patches integrated from TurboVNC 1.1. TurboVNC 1.0 added multi-threading capabilities, which can be added in later if desired (at the expense of making libvncserver depend on libpthread.) Because TurboVNC uses a fundamentally different mix of subencodings than TightVNC, because it uses the identical protocol (and thus a viewer really has no idea whether it's talking to a TightVNC or TurboVNC server), and because it doesn't support rfbTightPng (and in fact conflicts with it-- see below), the TurboVNC and TightVNC encoders cannot be enabled simultaneously. Compatibility: In *most* cases, a TurboVNC-enabled viewer is fully compatible with a TightVNC server, and vice versa. TurboVNC supports pseudo-encodings for specifying a fine-grained (1-100) quality scale and specifying chrominance subsampling. If a TurboVNC viewer sends those to a TightVNC server, then the TightVNC server ignores them, so the TurboVNC viewer also sends the quality on a 0-9 scale that the TightVNC server can understand. Similarly, the TurboVNC server checks first for fine-grained quality and subsampling pseudo-encodings from the viewer, and failing to receive those, it then checks for the TightVNC 0-9 quality pseudo-encoding. There is one case in which the two systems are not compatible, and that is when a TightVNC or TigerVNC viewer requests compression level 0 without JPEG from a TurboVNC server. For performance reasons, this causes the TurboVNC server to send images directly to the viewer, bypassing Zlib. When the TurboVNC server does this, it also sets bits 7-4 in the compression control byte to rfbTightNoZlib (0x0A), which is unfortunately the same value as rfbTightPng. Older TightVNC viewers that don't handle PNG will assume that the stream is uncompressed but still encapsulated in a Zlib structure, whereas newer PNG-supporting TightVNC viewers will assume that the stream is PNG. In either case, the viewer will probably crash. Since most VNC viewers don't expose compression level 0 in the GUI, this is a relatively rare situation. Description of changes: configure.ac -- Added support for libjpeg-turbo. If passed an argument of --with-turbovnc, configure will now run (or, if cross-compiling, just link) a test program that determines whether the libjpeg library being used is libjpeg-turbo. libjpeg-turbo must be used when building the TurboVNC encoder, because the TurboVNC encoder relies on the libjpeg-turbo colorspace extensions in order to compress images directly out of the framebuffer (which may be, for instance, BGRA rather than RGB.) libjpeg-turbo can optionally be used with the TightVNC encoder as well, but the speedup will only be marginal (the report linked above explains why in more detail, but basically it's because of Amdahl's Law. The TightVNC encoder was designed with the assumption that JPEG had a very high CPU cost, and thus JPEG is used only sparingly.) -- Added a new configure variable, JPEG_LDFLAGS. This is necessitated by the fact that libjpeg-turbo often distributes libjpeg.a and libjpeg.so in /opt/libjpeg-turbo/lib32 or /opt/libjpeg-turbo/lib64, and many people prefer to statically link with it. Thus, more flexibility is needed than is provided by --with-jpeg. If JPEG_LDFLAGS is specified, then it overrides the changes to LDFLAGS enacted by --with-jpeg (but --with-jpeg is still used to set the include path.) The addition of JPEG_LDFLAGS necessitated replacing AC_CHECK_LIB with AC_LINK_IFELSE (because AC_CHECK_LIB automatically sets LIBS to -ljpeg, which is not what we want if we're, for instance, linking statically with libjpeg-turbo.) -- configure does not check for PNG support if TurboVNC encoding is enabled. This prevents the rfbSendRectEncodingTightPng() function from being compiled in, since the TurboVNC encoder doesn't (and can't) support it. common/turbojpeg.c, common/turbojpeg.h -- TurboJPEG is a simple API used to compress and decompress JPEG images in memory. It was originally implemented because it was desirable to use different types of underlying technologies to compress JPEG on different platforms (mediaLib on SPARC, Quicktime on PPC Macs, Intel Performance Primitives, etc.) These days, however, libjpeg-turbo is the only underlying technology used by TurboVNC, so TurboJPEG's purpose is largely just code simplicity and flexibility. Thus, since there is no real need for libvncserver to use any technology other than libjpeg-turbo for compressing JPEG, the TurboJPEG wrapper for libjpeg-turbo has been included in-tree so that libvncserver can be directly linked with libjpeg-turbo. This is convenient because many modern Linux distros (Fedora, Ubuntu, etc.) now ship libjpeg-turbo as their default libjpeg library. libvncserver/rfbserver.c -- Added logic to check for the TurboVNC fine-grained quality level and subsampling encodings and to map Tight (0-9) quality levels to appropriate fine-grained quality level and subsampling values if communicating with a TightVNC/TigerVNC viewer. libvncserver/turbo.c -- TurboVNC encoder (compiled instead of libvncserver/tight.c) rfb/rfb.h -- Added support for the TurboVNC subsampling level rfb/rfbproto.h -- Added constants for the TurboVNC fine quality level and subsampling encodings as well as the rfbTightNoZlib constant and notes on its usage.
13 years ago
{
int i, retval=0; JSAMPROW *row_pointer=NULL;
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
int jpegwidth, jpegheight, scaledw, scaledh;
#ifndef JCS_EXTENSIONS
unsigned char *rgbBuf=NULL;
unsigned char *_dstBuf=NULL; int _pitch=0;
#endif
Add TurboVNC encoding support. TurboVNC is a variant of TightVNC that uses the same client/server protocol (RFB version 3.8t), and thus it is fully cross-compatible with TightVNC and TigerVNC (with one exception, which is noted below.) Both the TightVNC and TurboVNC encoders analyze each rectangle, pick out regions of solid color to send separately, and send the remaining subrectangles using mono, indexed color, JPEG, or raw encoding, depending on the number of colors in the subrectangle. However, TurboVNC uses a fundamentally different selection algorithm to determine the appropriate subencoding to use for each subrectangle. Thus, while it sends a protocol stream that can be decoded by any TightVNC-compatible viewer, the mix of subencoding types in this protocol stream will be different from those generated by a TightVNC server. The research that led to TurboVNC is described in the following report: http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf. In summary: 20 RFB captures, representing "common" 2D and 3D application workloads (the 3D workloads were run using VirtualGL), were studied using the TightVNC encoder in isolation. Some of the analysis features in the TightVNC encoder, such as smoothness detection, were found to generate a lot of CPU usage with little or no benefit in compression, so those features were disabled. JPEG encoding was accelerated using libjpeg-turbo (which achieves a 2-4x speedup over plain libjpeg on modern x86 or ARM processors.) Finally, the "palette threshold" (minimum number of colors that the subrectangle must have before it is compressed using JPEG or raw) was adjusted to account for the fact that JPEG encoding is now quite a bit faster (meaning that we can now use it more without a CPU penalty.) TurboVNC has additional optimizations, such as the ability to count colors and encode JPEG images directly from the framebuffer without first translating the pixels into RGB. The TurboVNC encoder compares quite favorably in terms of compression ratio with TightVNC and generally encodes a great deal faster (often an order of magnitude or more.) The version of the TurboVNC encoder included in this patch is roughly equivalent to the one found in version 0.6 of the Unix TurboVNC Server, with a few minor patches integrated from TurboVNC 1.1. TurboVNC 1.0 added multi-threading capabilities, which can be added in later if desired (at the expense of making libvncserver depend on libpthread.) Because TurboVNC uses a fundamentally different mix of subencodings than TightVNC, because it uses the identical protocol (and thus a viewer really has no idea whether it's talking to a TightVNC or TurboVNC server), and because it doesn't support rfbTightPng (and in fact conflicts with it-- see below), the TurboVNC and TightVNC encoders cannot be enabled simultaneously. Compatibility: In *most* cases, a TurboVNC-enabled viewer is fully compatible with a TightVNC server, and vice versa. TurboVNC supports pseudo-encodings for specifying a fine-grained (1-100) quality scale and specifying chrominance subsampling. If a TurboVNC viewer sends those to a TightVNC server, then the TightVNC server ignores them, so the TurboVNC viewer also sends the quality on a 0-9 scale that the TightVNC server can understand. Similarly, the TurboVNC server checks first for fine-grained quality and subsampling pseudo-encodings from the viewer, and failing to receive those, it then checks for the TightVNC 0-9 quality pseudo-encoding. There is one case in which the two systems are not compatible, and that is when a TightVNC or TigerVNC viewer requests compression level 0 without JPEG from a TurboVNC server. For performance reasons, this causes the TurboVNC server to send images directly to the viewer, bypassing Zlib. When the TurboVNC server does this, it also sets bits 7-4 in the compression control byte to rfbTightNoZlib (0x0A), which is unfortunately the same value as rfbTightPng. Older TightVNC viewers that don't handle PNG will assume that the stream is uncompressed but still encapsulated in a Zlib structure, whereas newer PNG-supporting TightVNC viewers will assume that the stream is PNG. In either case, the viewer will probably crash. Since most VNC viewers don't expose compression level 0 in the GUI, this is a relatively rare situation. Description of changes: configure.ac -- Added support for libjpeg-turbo. If passed an argument of --with-turbovnc, configure will now run (or, if cross-compiling, just link) a test program that determines whether the libjpeg library being used is libjpeg-turbo. libjpeg-turbo must be used when building the TurboVNC encoder, because the TurboVNC encoder relies on the libjpeg-turbo colorspace extensions in order to compress images directly out of the framebuffer (which may be, for instance, BGRA rather than RGB.) libjpeg-turbo can optionally be used with the TightVNC encoder as well, but the speedup will only be marginal (the report linked above explains why in more detail, but basically it's because of Amdahl's Law. The TightVNC encoder was designed with the assumption that JPEG had a very high CPU cost, and thus JPEG is used only sparingly.) -- Added a new configure variable, JPEG_LDFLAGS. This is necessitated by the fact that libjpeg-turbo often distributes libjpeg.a and libjpeg.so in /opt/libjpeg-turbo/lib32 or /opt/libjpeg-turbo/lib64, and many people prefer to statically link with it. Thus, more flexibility is needed than is provided by --with-jpeg. If JPEG_LDFLAGS is specified, then it overrides the changes to LDFLAGS enacted by --with-jpeg (but --with-jpeg is still used to set the include path.) The addition of JPEG_LDFLAGS necessitated replacing AC_CHECK_LIB with AC_LINK_IFELSE (because AC_CHECK_LIB automatically sets LIBS to -ljpeg, which is not what we want if we're, for instance, linking statically with libjpeg-turbo.) -- configure does not check for PNG support if TurboVNC encoding is enabled. This prevents the rfbSendRectEncodingTightPng() function from being compiled in, since the TurboVNC encoder doesn't (and can't) support it. common/turbojpeg.c, common/turbojpeg.h -- TurboJPEG is a simple API used to compress and decompress JPEG images in memory. It was originally implemented because it was desirable to use different types of underlying technologies to compress JPEG on different platforms (mediaLib on SPARC, Quicktime on PPC Macs, Intel Performance Primitives, etc.) These days, however, libjpeg-turbo is the only underlying technology used by TurboVNC, so TurboJPEG's purpose is largely just code simplicity and flexibility. Thus, since there is no real need for libvncserver to use any technology other than libjpeg-turbo for compressing JPEG, the TurboJPEG wrapper for libjpeg-turbo has been included in-tree so that libvncserver can be directly linked with libjpeg-turbo. This is convenient because many modern Linux distros (Fedora, Ubuntu, etc.) now ship libjpeg-turbo as their default libjpeg library. libvncserver/rfbserver.c -- Added logic to check for the TurboVNC fine-grained quality level and subsampling encodings and to map Tight (0-9) quality levels to appropriate fine-grained quality level and subsampling values if communicating with a TightVNC/TigerVNC viewer. libvncserver/turbo.c -- TurboVNC encoder (compiled instead of libvncserver/tight.c) rfb/rfb.h -- Added support for the TurboVNC subsampling level rfb/rfbproto.h -- Added constants for the TurboVNC fine quality level and subsampling encodings as well as the rfbTightNoZlib constant and notes on its usage.
13 years ago
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
getinstance(handle);
if((this->init&DECOMPRESS)==0)
_throw("tjDecompress2(): Instance has not been initialized for decompression");
Add TurboVNC encoding support. TurboVNC is a variant of TightVNC that uses the same client/server protocol (RFB version 3.8t), and thus it is fully cross-compatible with TightVNC and TigerVNC (with one exception, which is noted below.) Both the TightVNC and TurboVNC encoders analyze each rectangle, pick out regions of solid color to send separately, and send the remaining subrectangles using mono, indexed color, JPEG, or raw encoding, depending on the number of colors in the subrectangle. However, TurboVNC uses a fundamentally different selection algorithm to determine the appropriate subencoding to use for each subrectangle. Thus, while it sends a protocol stream that can be decoded by any TightVNC-compatible viewer, the mix of subencoding types in this protocol stream will be different from those generated by a TightVNC server. The research that led to TurboVNC is described in the following report: http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf. In summary: 20 RFB captures, representing "common" 2D and 3D application workloads (the 3D workloads were run using VirtualGL), were studied using the TightVNC encoder in isolation. Some of the analysis features in the TightVNC encoder, such as smoothness detection, were found to generate a lot of CPU usage with little or no benefit in compression, so those features were disabled. JPEG encoding was accelerated using libjpeg-turbo (which achieves a 2-4x speedup over plain libjpeg on modern x86 or ARM processors.) Finally, the "palette threshold" (minimum number of colors that the subrectangle must have before it is compressed using JPEG or raw) was adjusted to account for the fact that JPEG encoding is now quite a bit faster (meaning that we can now use it more without a CPU penalty.) TurboVNC has additional optimizations, such as the ability to count colors and encode JPEG images directly from the framebuffer without first translating the pixels into RGB. The TurboVNC encoder compares quite favorably in terms of compression ratio with TightVNC and generally encodes a great deal faster (often an order of magnitude or more.) The version of the TurboVNC encoder included in this patch is roughly equivalent to the one found in version 0.6 of the Unix TurboVNC Server, with a few minor patches integrated from TurboVNC 1.1. TurboVNC 1.0 added multi-threading capabilities, which can be added in later if desired (at the expense of making libvncserver depend on libpthread.) Because TurboVNC uses a fundamentally different mix of subencodings than TightVNC, because it uses the identical protocol (and thus a viewer really has no idea whether it's talking to a TightVNC or TurboVNC server), and because it doesn't support rfbTightPng (and in fact conflicts with it-- see below), the TurboVNC and TightVNC encoders cannot be enabled simultaneously. Compatibility: In *most* cases, a TurboVNC-enabled viewer is fully compatible with a TightVNC server, and vice versa. TurboVNC supports pseudo-encodings for specifying a fine-grained (1-100) quality scale and specifying chrominance subsampling. If a TurboVNC viewer sends those to a TightVNC server, then the TightVNC server ignores them, so the TurboVNC viewer also sends the quality on a 0-9 scale that the TightVNC server can understand. Similarly, the TurboVNC server checks first for fine-grained quality and subsampling pseudo-encodings from the viewer, and failing to receive those, it then checks for the TightVNC 0-9 quality pseudo-encoding. There is one case in which the two systems are not compatible, and that is when a TightVNC or TigerVNC viewer requests compression level 0 without JPEG from a TurboVNC server. For performance reasons, this causes the TurboVNC server to send images directly to the viewer, bypassing Zlib. When the TurboVNC server does this, it also sets bits 7-4 in the compression control byte to rfbTightNoZlib (0x0A), which is unfortunately the same value as rfbTightPng. Older TightVNC viewers that don't handle PNG will assume that the stream is uncompressed but still encapsulated in a Zlib structure, whereas newer PNG-supporting TightVNC viewers will assume that the stream is PNG. In either case, the viewer will probably crash. Since most VNC viewers don't expose compression level 0 in the GUI, this is a relatively rare situation. Description of changes: configure.ac -- Added support for libjpeg-turbo. If passed an argument of --with-turbovnc, configure will now run (or, if cross-compiling, just link) a test program that determines whether the libjpeg library being used is libjpeg-turbo. libjpeg-turbo must be used when building the TurboVNC encoder, because the TurboVNC encoder relies on the libjpeg-turbo colorspace extensions in order to compress images directly out of the framebuffer (which may be, for instance, BGRA rather than RGB.) libjpeg-turbo can optionally be used with the TightVNC encoder as well, but the speedup will only be marginal (the report linked above explains why in more detail, but basically it's because of Amdahl's Law. The TightVNC encoder was designed with the assumption that JPEG had a very high CPU cost, and thus JPEG is used only sparingly.) -- Added a new configure variable, JPEG_LDFLAGS. This is necessitated by the fact that libjpeg-turbo often distributes libjpeg.a and libjpeg.so in /opt/libjpeg-turbo/lib32 or /opt/libjpeg-turbo/lib64, and many people prefer to statically link with it. Thus, more flexibility is needed than is provided by --with-jpeg. If JPEG_LDFLAGS is specified, then it overrides the changes to LDFLAGS enacted by --with-jpeg (but --with-jpeg is still used to set the include path.) The addition of JPEG_LDFLAGS necessitated replacing AC_CHECK_LIB with AC_LINK_IFELSE (because AC_CHECK_LIB automatically sets LIBS to -ljpeg, which is not what we want if we're, for instance, linking statically with libjpeg-turbo.) -- configure does not check for PNG support if TurboVNC encoding is enabled. This prevents the rfbSendRectEncodingTightPng() function from being compiled in, since the TurboVNC encoder doesn't (and can't) support it. common/turbojpeg.c, common/turbojpeg.h -- TurboJPEG is a simple API used to compress and decompress JPEG images in memory. It was originally implemented because it was desirable to use different types of underlying technologies to compress JPEG on different platforms (mediaLib on SPARC, Quicktime on PPC Macs, Intel Performance Primitives, etc.) These days, however, libjpeg-turbo is the only underlying technology used by TurboVNC, so TurboJPEG's purpose is largely just code simplicity and flexibility. Thus, since there is no real need for libvncserver to use any technology other than libjpeg-turbo for compressing JPEG, the TurboJPEG wrapper for libjpeg-turbo has been included in-tree so that libvncserver can be directly linked with libjpeg-turbo. This is convenient because many modern Linux distros (Fedora, Ubuntu, etc.) now ship libjpeg-turbo as their default libjpeg library. libvncserver/rfbserver.c -- Added logic to check for the TurboVNC fine-grained quality level and subsampling encodings and to map Tight (0-9) quality levels to appropriate fine-grained quality level and subsampling values if communicating with a TightVNC/TigerVNC viewer. libvncserver/turbo.c -- TurboVNC encoder (compiled instead of libvncserver/tight.c) rfb/rfb.h -- Added support for the TurboVNC subsampling level rfb/rfbproto.h -- Added constants for the TurboVNC fine quality level and subsampling encodings as well as the rfbTightNoZlib constant and notes on its usage.
13 years ago
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
if(jpegBuf==NULL || jpegSize<=0 || dstBuf==NULL || width<0 || pitch<0
|| height<0 || pixelFormat<0 || pixelFormat>=TJ_NUMPF)
_throw("tjDecompress2(): Invalid argument");
Add TurboVNC encoding support. TurboVNC is a variant of TightVNC that uses the same client/server protocol (RFB version 3.8t), and thus it is fully cross-compatible with TightVNC and TigerVNC (with one exception, which is noted below.) Both the TightVNC and TurboVNC encoders analyze each rectangle, pick out regions of solid color to send separately, and send the remaining subrectangles using mono, indexed color, JPEG, or raw encoding, depending on the number of colors in the subrectangle. However, TurboVNC uses a fundamentally different selection algorithm to determine the appropriate subencoding to use for each subrectangle. Thus, while it sends a protocol stream that can be decoded by any TightVNC-compatible viewer, the mix of subencoding types in this protocol stream will be different from those generated by a TightVNC server. The research that led to TurboVNC is described in the following report: http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf. In summary: 20 RFB captures, representing "common" 2D and 3D application workloads (the 3D workloads were run using VirtualGL), were studied using the TightVNC encoder in isolation. Some of the analysis features in the TightVNC encoder, such as smoothness detection, were found to generate a lot of CPU usage with little or no benefit in compression, so those features were disabled. JPEG encoding was accelerated using libjpeg-turbo (which achieves a 2-4x speedup over plain libjpeg on modern x86 or ARM processors.) Finally, the "palette threshold" (minimum number of colors that the subrectangle must have before it is compressed using JPEG or raw) was adjusted to account for the fact that JPEG encoding is now quite a bit faster (meaning that we can now use it more without a CPU penalty.) TurboVNC has additional optimizations, such as the ability to count colors and encode JPEG images directly from the framebuffer without first translating the pixels into RGB. The TurboVNC encoder compares quite favorably in terms of compression ratio with TightVNC and generally encodes a great deal faster (often an order of magnitude or more.) The version of the TurboVNC encoder included in this patch is roughly equivalent to the one found in version 0.6 of the Unix TurboVNC Server, with a few minor patches integrated from TurboVNC 1.1. TurboVNC 1.0 added multi-threading capabilities, which can be added in later if desired (at the expense of making libvncserver depend on libpthread.) Because TurboVNC uses a fundamentally different mix of subencodings than TightVNC, because it uses the identical protocol (and thus a viewer really has no idea whether it's talking to a TightVNC or TurboVNC server), and because it doesn't support rfbTightPng (and in fact conflicts with it-- see below), the TurboVNC and TightVNC encoders cannot be enabled simultaneously. Compatibility: In *most* cases, a TurboVNC-enabled viewer is fully compatible with a TightVNC server, and vice versa. TurboVNC supports pseudo-encodings for specifying a fine-grained (1-100) quality scale and specifying chrominance subsampling. If a TurboVNC viewer sends those to a TightVNC server, then the TightVNC server ignores them, so the TurboVNC viewer also sends the quality on a 0-9 scale that the TightVNC server can understand. Similarly, the TurboVNC server checks first for fine-grained quality and subsampling pseudo-encodings from the viewer, and failing to receive those, it then checks for the TightVNC 0-9 quality pseudo-encoding. There is one case in which the two systems are not compatible, and that is when a TightVNC or TigerVNC viewer requests compression level 0 without JPEG from a TurboVNC server. For performance reasons, this causes the TurboVNC server to send images directly to the viewer, bypassing Zlib. When the TurboVNC server does this, it also sets bits 7-4 in the compression control byte to rfbTightNoZlib (0x0A), which is unfortunately the same value as rfbTightPng. Older TightVNC viewers that don't handle PNG will assume that the stream is uncompressed but still encapsulated in a Zlib structure, whereas newer PNG-supporting TightVNC viewers will assume that the stream is PNG. In either case, the viewer will probably crash. Since most VNC viewers don't expose compression level 0 in the GUI, this is a relatively rare situation. Description of changes: configure.ac -- Added support for libjpeg-turbo. If passed an argument of --with-turbovnc, configure will now run (or, if cross-compiling, just link) a test program that determines whether the libjpeg library being used is libjpeg-turbo. libjpeg-turbo must be used when building the TurboVNC encoder, because the TurboVNC encoder relies on the libjpeg-turbo colorspace extensions in order to compress images directly out of the framebuffer (which may be, for instance, BGRA rather than RGB.) libjpeg-turbo can optionally be used with the TightVNC encoder as well, but the speedup will only be marginal (the report linked above explains why in more detail, but basically it's because of Amdahl's Law. The TightVNC encoder was designed with the assumption that JPEG had a very high CPU cost, and thus JPEG is used only sparingly.) -- Added a new configure variable, JPEG_LDFLAGS. This is necessitated by the fact that libjpeg-turbo often distributes libjpeg.a and libjpeg.so in /opt/libjpeg-turbo/lib32 or /opt/libjpeg-turbo/lib64, and many people prefer to statically link with it. Thus, more flexibility is needed than is provided by --with-jpeg. If JPEG_LDFLAGS is specified, then it overrides the changes to LDFLAGS enacted by --with-jpeg (but --with-jpeg is still used to set the include path.) The addition of JPEG_LDFLAGS necessitated replacing AC_CHECK_LIB with AC_LINK_IFELSE (because AC_CHECK_LIB automatically sets LIBS to -ljpeg, which is not what we want if we're, for instance, linking statically with libjpeg-turbo.) -- configure does not check for PNG support if TurboVNC encoding is enabled. This prevents the rfbSendRectEncodingTightPng() function from being compiled in, since the TurboVNC encoder doesn't (and can't) support it. common/turbojpeg.c, common/turbojpeg.h -- TurboJPEG is a simple API used to compress and decompress JPEG images in memory. It was originally implemented because it was desirable to use different types of underlying technologies to compress JPEG on different platforms (mediaLib on SPARC, Quicktime on PPC Macs, Intel Performance Primitives, etc.) These days, however, libjpeg-turbo is the only underlying technology used by TurboVNC, so TurboJPEG's purpose is largely just code simplicity and flexibility. Thus, since there is no real need for libvncserver to use any technology other than libjpeg-turbo for compressing JPEG, the TurboJPEG wrapper for libjpeg-turbo has been included in-tree so that libvncserver can be directly linked with libjpeg-turbo. This is convenient because many modern Linux distros (Fedora, Ubuntu, etc.) now ship libjpeg-turbo as their default libjpeg library. libvncserver/rfbserver.c -- Added logic to check for the TurboVNC fine-grained quality level and subsampling encodings and to map Tight (0-9) quality levels to appropriate fine-grained quality level and subsampling values if communicating with a TightVNC/TigerVNC viewer. libvncserver/turbo.c -- TurboVNC encoder (compiled instead of libvncserver/tight.c) rfb/rfb.h -- Added support for the TurboVNC subsampling level rfb/rfbproto.h -- Added constants for the TurboVNC fine quality level and subsampling encodings as well as the rfbTightNoZlib constant and notes on its usage.
13 years ago
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
if(flags&TJFLAG_FORCEMMX) putenv("JSIMD_FORCEMMX=1");
else if(flags&TJFLAG_FORCESSE) putenv("JSIMD_FORCESSE=1");
else if(flags&TJFLAG_FORCESSE2) putenv("JSIMD_FORCESSE2=1");
Add TurboVNC encoding support. TurboVNC is a variant of TightVNC that uses the same client/server protocol (RFB version 3.8t), and thus it is fully cross-compatible with TightVNC and TigerVNC (with one exception, which is noted below.) Both the TightVNC and TurboVNC encoders analyze each rectangle, pick out regions of solid color to send separately, and send the remaining subrectangles using mono, indexed color, JPEG, or raw encoding, depending on the number of colors in the subrectangle. However, TurboVNC uses a fundamentally different selection algorithm to determine the appropriate subencoding to use for each subrectangle. Thus, while it sends a protocol stream that can be decoded by any TightVNC-compatible viewer, the mix of subencoding types in this protocol stream will be different from those generated by a TightVNC server. The research that led to TurboVNC is described in the following report: http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf. In summary: 20 RFB captures, representing "common" 2D and 3D application workloads (the 3D workloads were run using VirtualGL), were studied using the TightVNC encoder in isolation. Some of the analysis features in the TightVNC encoder, such as smoothness detection, were found to generate a lot of CPU usage with little or no benefit in compression, so those features were disabled. JPEG encoding was accelerated using libjpeg-turbo (which achieves a 2-4x speedup over plain libjpeg on modern x86 or ARM processors.) Finally, the "palette threshold" (minimum number of colors that the subrectangle must have before it is compressed using JPEG or raw) was adjusted to account for the fact that JPEG encoding is now quite a bit faster (meaning that we can now use it more without a CPU penalty.) TurboVNC has additional optimizations, such as the ability to count colors and encode JPEG images directly from the framebuffer without first translating the pixels into RGB. The TurboVNC encoder compares quite favorably in terms of compression ratio with TightVNC and generally encodes a great deal faster (often an order of magnitude or more.) The version of the TurboVNC encoder included in this patch is roughly equivalent to the one found in version 0.6 of the Unix TurboVNC Server, with a few minor patches integrated from TurboVNC 1.1. TurboVNC 1.0 added multi-threading capabilities, which can be added in later if desired (at the expense of making libvncserver depend on libpthread.) Because TurboVNC uses a fundamentally different mix of subencodings than TightVNC, because it uses the identical protocol (and thus a viewer really has no idea whether it's talking to a TightVNC or TurboVNC server), and because it doesn't support rfbTightPng (and in fact conflicts with it-- see below), the TurboVNC and TightVNC encoders cannot be enabled simultaneously. Compatibility: In *most* cases, a TurboVNC-enabled viewer is fully compatible with a TightVNC server, and vice versa. TurboVNC supports pseudo-encodings for specifying a fine-grained (1-100) quality scale and specifying chrominance subsampling. If a TurboVNC viewer sends those to a TightVNC server, then the TightVNC server ignores them, so the TurboVNC viewer also sends the quality on a 0-9 scale that the TightVNC server can understand. Similarly, the TurboVNC server checks first for fine-grained quality and subsampling pseudo-encodings from the viewer, and failing to receive those, it then checks for the TightVNC 0-9 quality pseudo-encoding. There is one case in which the two systems are not compatible, and that is when a TightVNC or TigerVNC viewer requests compression level 0 without JPEG from a TurboVNC server. For performance reasons, this causes the TurboVNC server to send images directly to the viewer, bypassing Zlib. When the TurboVNC server does this, it also sets bits 7-4 in the compression control byte to rfbTightNoZlib (0x0A), which is unfortunately the same value as rfbTightPng. Older TightVNC viewers that don't handle PNG will assume that the stream is uncompressed but still encapsulated in a Zlib structure, whereas newer PNG-supporting TightVNC viewers will assume that the stream is PNG. In either case, the viewer will probably crash. Since most VNC viewers don't expose compression level 0 in the GUI, this is a relatively rare situation. Description of changes: configure.ac -- Added support for libjpeg-turbo. If passed an argument of --with-turbovnc, configure will now run (or, if cross-compiling, just link) a test program that determines whether the libjpeg library being used is libjpeg-turbo. libjpeg-turbo must be used when building the TurboVNC encoder, because the TurboVNC encoder relies on the libjpeg-turbo colorspace extensions in order to compress images directly out of the framebuffer (which may be, for instance, BGRA rather than RGB.) libjpeg-turbo can optionally be used with the TightVNC encoder as well, but the speedup will only be marginal (the report linked above explains why in more detail, but basically it's because of Amdahl's Law. The TightVNC encoder was designed with the assumption that JPEG had a very high CPU cost, and thus JPEG is used only sparingly.) -- Added a new configure variable, JPEG_LDFLAGS. This is necessitated by the fact that libjpeg-turbo often distributes libjpeg.a and libjpeg.so in /opt/libjpeg-turbo/lib32 or /opt/libjpeg-turbo/lib64, and many people prefer to statically link with it. Thus, more flexibility is needed than is provided by --with-jpeg. If JPEG_LDFLAGS is specified, then it overrides the changes to LDFLAGS enacted by --with-jpeg (but --with-jpeg is still used to set the include path.) The addition of JPEG_LDFLAGS necessitated replacing AC_CHECK_LIB with AC_LINK_IFELSE (because AC_CHECK_LIB automatically sets LIBS to -ljpeg, which is not what we want if we're, for instance, linking statically with libjpeg-turbo.) -- configure does not check for PNG support if TurboVNC encoding is enabled. This prevents the rfbSendRectEncodingTightPng() function from being compiled in, since the TurboVNC encoder doesn't (and can't) support it. common/turbojpeg.c, common/turbojpeg.h -- TurboJPEG is a simple API used to compress and decompress JPEG images in memory. It was originally implemented because it was desirable to use different types of underlying technologies to compress JPEG on different platforms (mediaLib on SPARC, Quicktime on PPC Macs, Intel Performance Primitives, etc.) These days, however, libjpeg-turbo is the only underlying technology used by TurboVNC, so TurboJPEG's purpose is largely just code simplicity and flexibility. Thus, since there is no real need for libvncserver to use any technology other than libjpeg-turbo for compressing JPEG, the TurboJPEG wrapper for libjpeg-turbo has been included in-tree so that libvncserver can be directly linked with libjpeg-turbo. This is convenient because many modern Linux distros (Fedora, Ubuntu, etc.) now ship libjpeg-turbo as their default libjpeg library. libvncserver/rfbserver.c -- Added logic to check for the TurboVNC fine-grained quality level and subsampling encodings and to map Tight (0-9) quality levels to appropriate fine-grained quality level and subsampling values if communicating with a TightVNC/TigerVNC viewer. libvncserver/turbo.c -- TurboVNC encoder (compiled instead of libvncserver/tight.c) rfb/rfb.h -- Added support for the TurboVNC subsampling level rfb/rfbproto.h -- Added constants for the TurboVNC fine quality level and subsampling encodings as well as the rfbTightNoZlib constant and notes on its usage.
13 years ago
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
if(setjmp(this->jerr.setjmp_buffer))
{
/* If we get here, the JPEG code has signaled an error. */
Add TurboVNC encoding support. TurboVNC is a variant of TightVNC that uses the same client/server protocol (RFB version 3.8t), and thus it is fully cross-compatible with TightVNC and TigerVNC (with one exception, which is noted below.) Both the TightVNC and TurboVNC encoders analyze each rectangle, pick out regions of solid color to send separately, and send the remaining subrectangles using mono, indexed color, JPEG, or raw encoding, depending on the number of colors in the subrectangle. However, TurboVNC uses a fundamentally different selection algorithm to determine the appropriate subencoding to use for each subrectangle. Thus, while it sends a protocol stream that can be decoded by any TightVNC-compatible viewer, the mix of subencoding types in this protocol stream will be different from those generated by a TightVNC server. The research that led to TurboVNC is described in the following report: http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf. In summary: 20 RFB captures, representing "common" 2D and 3D application workloads (the 3D workloads were run using VirtualGL), were studied using the TightVNC encoder in isolation. Some of the analysis features in the TightVNC encoder, such as smoothness detection, were found to generate a lot of CPU usage with little or no benefit in compression, so those features were disabled. JPEG encoding was accelerated using libjpeg-turbo (which achieves a 2-4x speedup over plain libjpeg on modern x86 or ARM processors.) Finally, the "palette threshold" (minimum number of colors that the subrectangle must have before it is compressed using JPEG or raw) was adjusted to account for the fact that JPEG encoding is now quite a bit faster (meaning that we can now use it more without a CPU penalty.) TurboVNC has additional optimizations, such as the ability to count colors and encode JPEG images directly from the framebuffer without first translating the pixels into RGB. The TurboVNC encoder compares quite favorably in terms of compression ratio with TightVNC and generally encodes a great deal faster (often an order of magnitude or more.) The version of the TurboVNC encoder included in this patch is roughly equivalent to the one found in version 0.6 of the Unix TurboVNC Server, with a few minor patches integrated from TurboVNC 1.1. TurboVNC 1.0 added multi-threading capabilities, which can be added in later if desired (at the expense of making libvncserver depend on libpthread.) Because TurboVNC uses a fundamentally different mix of subencodings than TightVNC, because it uses the identical protocol (and thus a viewer really has no idea whether it's talking to a TightVNC or TurboVNC server), and because it doesn't support rfbTightPng (and in fact conflicts with it-- see below), the TurboVNC and TightVNC encoders cannot be enabled simultaneously. Compatibility: In *most* cases, a TurboVNC-enabled viewer is fully compatible with a TightVNC server, and vice versa. TurboVNC supports pseudo-encodings for specifying a fine-grained (1-100) quality scale and specifying chrominance subsampling. If a TurboVNC viewer sends those to a TightVNC server, then the TightVNC server ignores them, so the TurboVNC viewer also sends the quality on a 0-9 scale that the TightVNC server can understand. Similarly, the TurboVNC server checks first for fine-grained quality and subsampling pseudo-encodings from the viewer, and failing to receive those, it then checks for the TightVNC 0-9 quality pseudo-encoding. There is one case in which the two systems are not compatible, and that is when a TightVNC or TigerVNC viewer requests compression level 0 without JPEG from a TurboVNC server. For performance reasons, this causes the TurboVNC server to send images directly to the viewer, bypassing Zlib. When the TurboVNC server does this, it also sets bits 7-4 in the compression control byte to rfbTightNoZlib (0x0A), which is unfortunately the same value as rfbTightPng. Older TightVNC viewers that don't handle PNG will assume that the stream is uncompressed but still encapsulated in a Zlib structure, whereas newer PNG-supporting TightVNC viewers will assume that the stream is PNG. In either case, the viewer will probably crash. Since most VNC viewers don't expose compression level 0 in the GUI, this is a relatively rare situation. Description of changes: configure.ac -- Added support for libjpeg-turbo. If passed an argument of --with-turbovnc, configure will now run (or, if cross-compiling, just link) a test program that determines whether the libjpeg library being used is libjpeg-turbo. libjpeg-turbo must be used when building the TurboVNC encoder, because the TurboVNC encoder relies on the libjpeg-turbo colorspace extensions in order to compress images directly out of the framebuffer (which may be, for instance, BGRA rather than RGB.) libjpeg-turbo can optionally be used with the TightVNC encoder as well, but the speedup will only be marginal (the report linked above explains why in more detail, but basically it's because of Amdahl's Law. The TightVNC encoder was designed with the assumption that JPEG had a very high CPU cost, and thus JPEG is used only sparingly.) -- Added a new configure variable, JPEG_LDFLAGS. This is necessitated by the fact that libjpeg-turbo often distributes libjpeg.a and libjpeg.so in /opt/libjpeg-turbo/lib32 or /opt/libjpeg-turbo/lib64, and many people prefer to statically link with it. Thus, more flexibility is needed than is provided by --with-jpeg. If JPEG_LDFLAGS is specified, then it overrides the changes to LDFLAGS enacted by --with-jpeg (but --with-jpeg is still used to set the include path.) The addition of JPEG_LDFLAGS necessitated replacing AC_CHECK_LIB with AC_LINK_IFELSE (because AC_CHECK_LIB automatically sets LIBS to -ljpeg, which is not what we want if we're, for instance, linking statically with libjpeg-turbo.) -- configure does not check for PNG support if TurboVNC encoding is enabled. This prevents the rfbSendRectEncodingTightPng() function from being compiled in, since the TurboVNC encoder doesn't (and can't) support it. common/turbojpeg.c, common/turbojpeg.h -- TurboJPEG is a simple API used to compress and decompress JPEG images in memory. It was originally implemented because it was desirable to use different types of underlying technologies to compress JPEG on different platforms (mediaLib on SPARC, Quicktime on PPC Macs, Intel Performance Primitives, etc.) These days, however, libjpeg-turbo is the only underlying technology used by TurboVNC, so TurboJPEG's purpose is largely just code simplicity and flexibility. Thus, since there is no real need for libvncserver to use any technology other than libjpeg-turbo for compressing JPEG, the TurboJPEG wrapper for libjpeg-turbo has been included in-tree so that libvncserver can be directly linked with libjpeg-turbo. This is convenient because many modern Linux distros (Fedora, Ubuntu, etc.) now ship libjpeg-turbo as their default libjpeg library. libvncserver/rfbserver.c -- Added logic to check for the TurboVNC fine-grained quality level and subsampling encodings and to map Tight (0-9) quality levels to appropriate fine-grained quality level and subsampling values if communicating with a TightVNC/TigerVNC viewer. libvncserver/turbo.c -- TurboVNC encoder (compiled instead of libvncserver/tight.c) rfb/rfb.h -- Added support for the TurboVNC subsampling level rfb/rfbproto.h -- Added constants for the TurboVNC fine quality level and subsampling encodings as well as the rfbTightNoZlib constant and notes on its usage.
13 years ago
retval=-1;
goto bailout;
}
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
this->jsrc.bytes_in_buffer=jpegSize;
this->jsrc.next_input_byte=jpegBuf;
jpeg_read_header(dinfo, TRUE);
if(setDecompDefaults(dinfo, pixelFormat)==-1)
{
retval=-1; goto bailout;
}
Add TurboVNC encoding support. TurboVNC is a variant of TightVNC that uses the same client/server protocol (RFB version 3.8t), and thus it is fully cross-compatible with TightVNC and TigerVNC (with one exception, which is noted below.) Both the TightVNC and TurboVNC encoders analyze each rectangle, pick out regions of solid color to send separately, and send the remaining subrectangles using mono, indexed color, JPEG, or raw encoding, depending on the number of colors in the subrectangle. However, TurboVNC uses a fundamentally different selection algorithm to determine the appropriate subencoding to use for each subrectangle. Thus, while it sends a protocol stream that can be decoded by any TightVNC-compatible viewer, the mix of subencoding types in this protocol stream will be different from those generated by a TightVNC server. The research that led to TurboVNC is described in the following report: http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf. In summary: 20 RFB captures, representing "common" 2D and 3D application workloads (the 3D workloads were run using VirtualGL), were studied using the TightVNC encoder in isolation. Some of the analysis features in the TightVNC encoder, such as smoothness detection, were found to generate a lot of CPU usage with little or no benefit in compression, so those features were disabled. JPEG encoding was accelerated using libjpeg-turbo (which achieves a 2-4x speedup over plain libjpeg on modern x86 or ARM processors.) Finally, the "palette threshold" (minimum number of colors that the subrectangle must have before it is compressed using JPEG or raw) was adjusted to account for the fact that JPEG encoding is now quite a bit faster (meaning that we can now use it more without a CPU penalty.) TurboVNC has additional optimizations, such as the ability to count colors and encode JPEG images directly from the framebuffer without first translating the pixels into RGB. The TurboVNC encoder compares quite favorably in terms of compression ratio with TightVNC and generally encodes a great deal faster (often an order of magnitude or more.) The version of the TurboVNC encoder included in this patch is roughly equivalent to the one found in version 0.6 of the Unix TurboVNC Server, with a few minor patches integrated from TurboVNC 1.1. TurboVNC 1.0 added multi-threading capabilities, which can be added in later if desired (at the expense of making libvncserver depend on libpthread.) Because TurboVNC uses a fundamentally different mix of subencodings than TightVNC, because it uses the identical protocol (and thus a viewer really has no idea whether it's talking to a TightVNC or TurboVNC server), and because it doesn't support rfbTightPng (and in fact conflicts with it-- see below), the TurboVNC and TightVNC encoders cannot be enabled simultaneously. Compatibility: In *most* cases, a TurboVNC-enabled viewer is fully compatible with a TightVNC server, and vice versa. TurboVNC supports pseudo-encodings for specifying a fine-grained (1-100) quality scale and specifying chrominance subsampling. If a TurboVNC viewer sends those to a TightVNC server, then the TightVNC server ignores them, so the TurboVNC viewer also sends the quality on a 0-9 scale that the TightVNC server can understand. Similarly, the TurboVNC server checks first for fine-grained quality and subsampling pseudo-encodings from the viewer, and failing to receive those, it then checks for the TightVNC 0-9 quality pseudo-encoding. There is one case in which the two systems are not compatible, and that is when a TightVNC or TigerVNC viewer requests compression level 0 without JPEG from a TurboVNC server. For performance reasons, this causes the TurboVNC server to send images directly to the viewer, bypassing Zlib. When the TurboVNC server does this, it also sets bits 7-4 in the compression control byte to rfbTightNoZlib (0x0A), which is unfortunately the same value as rfbTightPng. Older TightVNC viewers that don't handle PNG will assume that the stream is uncompressed but still encapsulated in a Zlib structure, whereas newer PNG-supporting TightVNC viewers will assume that the stream is PNG. In either case, the viewer will probably crash. Since most VNC viewers don't expose compression level 0 in the GUI, this is a relatively rare situation. Description of changes: configure.ac -- Added support for libjpeg-turbo. If passed an argument of --with-turbovnc, configure will now run (or, if cross-compiling, just link) a test program that determines whether the libjpeg library being used is libjpeg-turbo. libjpeg-turbo must be used when building the TurboVNC encoder, because the TurboVNC encoder relies on the libjpeg-turbo colorspace extensions in order to compress images directly out of the framebuffer (which may be, for instance, BGRA rather than RGB.) libjpeg-turbo can optionally be used with the TightVNC encoder as well, but the speedup will only be marginal (the report linked above explains why in more detail, but basically it's because of Amdahl's Law. The TightVNC encoder was designed with the assumption that JPEG had a very high CPU cost, and thus JPEG is used only sparingly.) -- Added a new configure variable, JPEG_LDFLAGS. This is necessitated by the fact that libjpeg-turbo often distributes libjpeg.a and libjpeg.so in /opt/libjpeg-turbo/lib32 or /opt/libjpeg-turbo/lib64, and many people prefer to statically link with it. Thus, more flexibility is needed than is provided by --with-jpeg. If JPEG_LDFLAGS is specified, then it overrides the changes to LDFLAGS enacted by --with-jpeg (but --with-jpeg is still used to set the include path.) The addition of JPEG_LDFLAGS necessitated replacing AC_CHECK_LIB with AC_LINK_IFELSE (because AC_CHECK_LIB automatically sets LIBS to -ljpeg, which is not what we want if we're, for instance, linking statically with libjpeg-turbo.) -- configure does not check for PNG support if TurboVNC encoding is enabled. This prevents the rfbSendRectEncodingTightPng() function from being compiled in, since the TurboVNC encoder doesn't (and can't) support it. common/turbojpeg.c, common/turbojpeg.h -- TurboJPEG is a simple API used to compress and decompress JPEG images in memory. It was originally implemented because it was desirable to use different types of underlying technologies to compress JPEG on different platforms (mediaLib on SPARC, Quicktime on PPC Macs, Intel Performance Primitives, etc.) These days, however, libjpeg-turbo is the only underlying technology used by TurboVNC, so TurboJPEG's purpose is largely just code simplicity and flexibility. Thus, since there is no real need for libvncserver to use any technology other than libjpeg-turbo for compressing JPEG, the TurboJPEG wrapper for libjpeg-turbo has been included in-tree so that libvncserver can be directly linked with libjpeg-turbo. This is convenient because many modern Linux distros (Fedora, Ubuntu, etc.) now ship libjpeg-turbo as their default libjpeg library. libvncserver/rfbserver.c -- Added logic to check for the TurboVNC fine-grained quality level and subsampling encodings and to map Tight (0-9) quality levels to appropriate fine-grained quality level and subsampling values if communicating with a TightVNC/TigerVNC viewer. libvncserver/turbo.c -- TurboVNC encoder (compiled instead of libvncserver/tight.c) rfb/rfb.h -- Added support for the TurboVNC subsampling level rfb/rfbproto.h -- Added constants for the TurboVNC fine quality level and subsampling encodings as well as the rfbTightNoZlib constant and notes on its usage.
13 years ago
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
if(flags&TJFLAG_FASTUPSAMPLE) dinfo->do_fancy_upsampling=FALSE;
Add TurboVNC encoding support. TurboVNC is a variant of TightVNC that uses the same client/server protocol (RFB version 3.8t), and thus it is fully cross-compatible with TightVNC and TigerVNC (with one exception, which is noted below.) Both the TightVNC and TurboVNC encoders analyze each rectangle, pick out regions of solid color to send separately, and send the remaining subrectangles using mono, indexed color, JPEG, or raw encoding, depending on the number of colors in the subrectangle. However, TurboVNC uses a fundamentally different selection algorithm to determine the appropriate subencoding to use for each subrectangle. Thus, while it sends a protocol stream that can be decoded by any TightVNC-compatible viewer, the mix of subencoding types in this protocol stream will be different from those generated by a TightVNC server. The research that led to TurboVNC is described in the following report: http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf. In summary: 20 RFB captures, representing "common" 2D and 3D application workloads (the 3D workloads were run using VirtualGL), were studied using the TightVNC encoder in isolation. Some of the analysis features in the TightVNC encoder, such as smoothness detection, were found to generate a lot of CPU usage with little or no benefit in compression, so those features were disabled. JPEG encoding was accelerated using libjpeg-turbo (which achieves a 2-4x speedup over plain libjpeg on modern x86 or ARM processors.) Finally, the "palette threshold" (minimum number of colors that the subrectangle must have before it is compressed using JPEG or raw) was adjusted to account for the fact that JPEG encoding is now quite a bit faster (meaning that we can now use it more without a CPU penalty.) TurboVNC has additional optimizations, such as the ability to count colors and encode JPEG images directly from the framebuffer without first translating the pixels into RGB. The TurboVNC encoder compares quite favorably in terms of compression ratio with TightVNC and generally encodes a great deal faster (often an order of magnitude or more.) The version of the TurboVNC encoder included in this patch is roughly equivalent to the one found in version 0.6 of the Unix TurboVNC Server, with a few minor patches integrated from TurboVNC 1.1. TurboVNC 1.0 added multi-threading capabilities, which can be added in later if desired (at the expense of making libvncserver depend on libpthread.) Because TurboVNC uses a fundamentally different mix of subencodings than TightVNC, because it uses the identical protocol (and thus a viewer really has no idea whether it's talking to a TightVNC or TurboVNC server), and because it doesn't support rfbTightPng (and in fact conflicts with it-- see below), the TurboVNC and TightVNC encoders cannot be enabled simultaneously. Compatibility: In *most* cases, a TurboVNC-enabled viewer is fully compatible with a TightVNC server, and vice versa. TurboVNC supports pseudo-encodings for specifying a fine-grained (1-100) quality scale and specifying chrominance subsampling. If a TurboVNC viewer sends those to a TightVNC server, then the TightVNC server ignores them, so the TurboVNC viewer also sends the quality on a 0-9 scale that the TightVNC server can understand. Similarly, the TurboVNC server checks first for fine-grained quality and subsampling pseudo-encodings from the viewer, and failing to receive those, it then checks for the TightVNC 0-9 quality pseudo-encoding. There is one case in which the two systems are not compatible, and that is when a TightVNC or TigerVNC viewer requests compression level 0 without JPEG from a TurboVNC server. For performance reasons, this causes the TurboVNC server to send images directly to the viewer, bypassing Zlib. When the TurboVNC server does this, it also sets bits 7-4 in the compression control byte to rfbTightNoZlib (0x0A), which is unfortunately the same value as rfbTightPng. Older TightVNC viewers that don't handle PNG will assume that the stream is uncompressed but still encapsulated in a Zlib structure, whereas newer PNG-supporting TightVNC viewers will assume that the stream is PNG. In either case, the viewer will probably crash. Since most VNC viewers don't expose compression level 0 in the GUI, this is a relatively rare situation. Description of changes: configure.ac -- Added support for libjpeg-turbo. If passed an argument of --with-turbovnc, configure will now run (or, if cross-compiling, just link) a test program that determines whether the libjpeg library being used is libjpeg-turbo. libjpeg-turbo must be used when building the TurboVNC encoder, because the TurboVNC encoder relies on the libjpeg-turbo colorspace extensions in order to compress images directly out of the framebuffer (which may be, for instance, BGRA rather than RGB.) libjpeg-turbo can optionally be used with the TightVNC encoder as well, but the speedup will only be marginal (the report linked above explains why in more detail, but basically it's because of Amdahl's Law. The TightVNC encoder was designed with the assumption that JPEG had a very high CPU cost, and thus JPEG is used only sparingly.) -- Added a new configure variable, JPEG_LDFLAGS. This is necessitated by the fact that libjpeg-turbo often distributes libjpeg.a and libjpeg.so in /opt/libjpeg-turbo/lib32 or /opt/libjpeg-turbo/lib64, and many people prefer to statically link with it. Thus, more flexibility is needed than is provided by --with-jpeg. If JPEG_LDFLAGS is specified, then it overrides the changes to LDFLAGS enacted by --with-jpeg (but --with-jpeg is still used to set the include path.) The addition of JPEG_LDFLAGS necessitated replacing AC_CHECK_LIB with AC_LINK_IFELSE (because AC_CHECK_LIB automatically sets LIBS to -ljpeg, which is not what we want if we're, for instance, linking statically with libjpeg-turbo.) -- configure does not check for PNG support if TurboVNC encoding is enabled. This prevents the rfbSendRectEncodingTightPng() function from being compiled in, since the TurboVNC encoder doesn't (and can't) support it. common/turbojpeg.c, common/turbojpeg.h -- TurboJPEG is a simple API used to compress and decompress JPEG images in memory. It was originally implemented because it was desirable to use different types of underlying technologies to compress JPEG on different platforms (mediaLib on SPARC, Quicktime on PPC Macs, Intel Performance Primitives, etc.) These days, however, libjpeg-turbo is the only underlying technology used by TurboVNC, so TurboJPEG's purpose is largely just code simplicity and flexibility. Thus, since there is no real need for libvncserver to use any technology other than libjpeg-turbo for compressing JPEG, the TurboJPEG wrapper for libjpeg-turbo has been included in-tree so that libvncserver can be directly linked with libjpeg-turbo. This is convenient because many modern Linux distros (Fedora, Ubuntu, etc.) now ship libjpeg-turbo as their default libjpeg library. libvncserver/rfbserver.c -- Added logic to check for the TurboVNC fine-grained quality level and subsampling encodings and to map Tight (0-9) quality levels to appropriate fine-grained quality level and subsampling values if communicating with a TightVNC/TigerVNC viewer. libvncserver/turbo.c -- TurboVNC encoder (compiled instead of libvncserver/tight.c) rfb/rfb.h -- Added support for the TurboVNC subsampling level rfb/rfbproto.h -- Added constants for the TurboVNC fine quality level and subsampling encodings as well as the rfbTightNoZlib constant and notes on its usage.
13 years ago
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
jpegwidth=dinfo->image_width; jpegheight=dinfo->image_height;
if(width==0) width=jpegwidth;
if(height==0) height=jpegheight;
for(i=0; i<NUMSF; i++)
Add TurboVNC encoding support. TurboVNC is a variant of TightVNC that uses the same client/server protocol (RFB version 3.8t), and thus it is fully cross-compatible with TightVNC and TigerVNC (with one exception, which is noted below.) Both the TightVNC and TurboVNC encoders analyze each rectangle, pick out regions of solid color to send separately, and send the remaining subrectangles using mono, indexed color, JPEG, or raw encoding, depending on the number of colors in the subrectangle. However, TurboVNC uses a fundamentally different selection algorithm to determine the appropriate subencoding to use for each subrectangle. Thus, while it sends a protocol stream that can be decoded by any TightVNC-compatible viewer, the mix of subencoding types in this protocol stream will be different from those generated by a TightVNC server. The research that led to TurboVNC is described in the following report: http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf. In summary: 20 RFB captures, representing "common" 2D and 3D application workloads (the 3D workloads were run using VirtualGL), were studied using the TightVNC encoder in isolation. Some of the analysis features in the TightVNC encoder, such as smoothness detection, were found to generate a lot of CPU usage with little or no benefit in compression, so those features were disabled. JPEG encoding was accelerated using libjpeg-turbo (which achieves a 2-4x speedup over plain libjpeg on modern x86 or ARM processors.) Finally, the "palette threshold" (minimum number of colors that the subrectangle must have before it is compressed using JPEG or raw) was adjusted to account for the fact that JPEG encoding is now quite a bit faster (meaning that we can now use it more without a CPU penalty.) TurboVNC has additional optimizations, such as the ability to count colors and encode JPEG images directly from the framebuffer without first translating the pixels into RGB. The TurboVNC encoder compares quite favorably in terms of compression ratio with TightVNC and generally encodes a great deal faster (often an order of magnitude or more.) The version of the TurboVNC encoder included in this patch is roughly equivalent to the one found in version 0.6 of the Unix TurboVNC Server, with a few minor patches integrated from TurboVNC 1.1. TurboVNC 1.0 added multi-threading capabilities, which can be added in later if desired (at the expense of making libvncserver depend on libpthread.) Because TurboVNC uses a fundamentally different mix of subencodings than TightVNC, because it uses the identical protocol (and thus a viewer really has no idea whether it's talking to a TightVNC or TurboVNC server), and because it doesn't support rfbTightPng (and in fact conflicts with it-- see below), the TurboVNC and TightVNC encoders cannot be enabled simultaneously. Compatibility: In *most* cases, a TurboVNC-enabled viewer is fully compatible with a TightVNC server, and vice versa. TurboVNC supports pseudo-encodings for specifying a fine-grained (1-100) quality scale and specifying chrominance subsampling. If a TurboVNC viewer sends those to a TightVNC server, then the TightVNC server ignores them, so the TurboVNC viewer also sends the quality on a 0-9 scale that the TightVNC server can understand. Similarly, the TurboVNC server checks first for fine-grained quality and subsampling pseudo-encodings from the viewer, and failing to receive those, it then checks for the TightVNC 0-9 quality pseudo-encoding. There is one case in which the two systems are not compatible, and that is when a TightVNC or TigerVNC viewer requests compression level 0 without JPEG from a TurboVNC server. For performance reasons, this causes the TurboVNC server to send images directly to the viewer, bypassing Zlib. When the TurboVNC server does this, it also sets bits 7-4 in the compression control byte to rfbTightNoZlib (0x0A), which is unfortunately the same value as rfbTightPng. Older TightVNC viewers that don't handle PNG will assume that the stream is uncompressed but still encapsulated in a Zlib structure, whereas newer PNG-supporting TightVNC viewers will assume that the stream is PNG. In either case, the viewer will probably crash. Since most VNC viewers don't expose compression level 0 in the GUI, this is a relatively rare situation. Description of changes: configure.ac -- Added support for libjpeg-turbo. If passed an argument of --with-turbovnc, configure will now run (or, if cross-compiling, just link) a test program that determines whether the libjpeg library being used is libjpeg-turbo. libjpeg-turbo must be used when building the TurboVNC encoder, because the TurboVNC encoder relies on the libjpeg-turbo colorspace extensions in order to compress images directly out of the framebuffer (which may be, for instance, BGRA rather than RGB.) libjpeg-turbo can optionally be used with the TightVNC encoder as well, but the speedup will only be marginal (the report linked above explains why in more detail, but basically it's because of Amdahl's Law. The TightVNC encoder was designed with the assumption that JPEG had a very high CPU cost, and thus JPEG is used only sparingly.) -- Added a new configure variable, JPEG_LDFLAGS. This is necessitated by the fact that libjpeg-turbo often distributes libjpeg.a and libjpeg.so in /opt/libjpeg-turbo/lib32 or /opt/libjpeg-turbo/lib64, and many people prefer to statically link with it. Thus, more flexibility is needed than is provided by --with-jpeg. If JPEG_LDFLAGS is specified, then it overrides the changes to LDFLAGS enacted by --with-jpeg (but --with-jpeg is still used to set the include path.) The addition of JPEG_LDFLAGS necessitated replacing AC_CHECK_LIB with AC_LINK_IFELSE (because AC_CHECK_LIB automatically sets LIBS to -ljpeg, which is not what we want if we're, for instance, linking statically with libjpeg-turbo.) -- configure does not check for PNG support if TurboVNC encoding is enabled. This prevents the rfbSendRectEncodingTightPng() function from being compiled in, since the TurboVNC encoder doesn't (and can't) support it. common/turbojpeg.c, common/turbojpeg.h -- TurboJPEG is a simple API used to compress and decompress JPEG images in memory. It was originally implemented because it was desirable to use different types of underlying technologies to compress JPEG on different platforms (mediaLib on SPARC, Quicktime on PPC Macs, Intel Performance Primitives, etc.) These days, however, libjpeg-turbo is the only underlying technology used by TurboVNC, so TurboJPEG's purpose is largely just code simplicity and flexibility. Thus, since there is no real need for libvncserver to use any technology other than libjpeg-turbo for compressing JPEG, the TurboJPEG wrapper for libjpeg-turbo has been included in-tree so that libvncserver can be directly linked with libjpeg-turbo. This is convenient because many modern Linux distros (Fedora, Ubuntu, etc.) now ship libjpeg-turbo as their default libjpeg library. libvncserver/rfbserver.c -- Added logic to check for the TurboVNC fine-grained quality level and subsampling encodings and to map Tight (0-9) quality levels to appropriate fine-grained quality level and subsampling values if communicating with a TightVNC/TigerVNC viewer. libvncserver/turbo.c -- TurboVNC encoder (compiled instead of libvncserver/tight.c) rfb/rfb.h -- Added support for the TurboVNC subsampling level rfb/rfbproto.h -- Added constants for the TurboVNC fine quality level and subsampling encodings as well as the rfbTightNoZlib constant and notes on its usage.
13 years ago
{
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
scaledw=TJSCALED(jpegwidth, sf[i]);
scaledh=TJSCALED(jpegheight, sf[i]);
if(scaledw<=width && scaledh<=height)
break;
}
if(scaledw>width || scaledh>height)
_throw("tjDecompress2(): Could not scale down to desired image dimensions");
width=scaledw; height=scaledh;
dinfo->scale_num=sf[i].num;
dinfo->scale_denom=sf[i].denom;
jpeg_start_decompress(dinfo);
if(pitch==0) pitch=dinfo->output_width*tjPixelSize[pixelFormat];
#ifndef JCS_EXTENSIONS
if(pixelFormat!=TJPF_GRAY &&
(RGB_RED!=tjRedOffset[pixelFormat] ||
RGB_GREEN!=tjGreenOffset[pixelFormat] ||
RGB_BLUE!=tjBlueOffset[pixelFormat] ||
RGB_PIXELSIZE!=tjPixelSize[pixelFormat]))
{
rgbBuf=(unsigned char *)malloc(width*height*3);
if(!rgbBuf) _throw("tjDecompress2(): Memory allocation failure");
_pitch=pitch; pitch=width*3;
_dstBuf=dstBuf; dstBuf=rgbBuf;
Add TurboVNC encoding support. TurboVNC is a variant of TightVNC that uses the same client/server protocol (RFB version 3.8t), and thus it is fully cross-compatible with TightVNC and TigerVNC (with one exception, which is noted below.) Both the TightVNC and TurboVNC encoders analyze each rectangle, pick out regions of solid color to send separately, and send the remaining subrectangles using mono, indexed color, JPEG, or raw encoding, depending on the number of colors in the subrectangle. However, TurboVNC uses a fundamentally different selection algorithm to determine the appropriate subencoding to use for each subrectangle. Thus, while it sends a protocol stream that can be decoded by any TightVNC-compatible viewer, the mix of subencoding types in this protocol stream will be different from those generated by a TightVNC server. The research that led to TurboVNC is described in the following report: http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf. In summary: 20 RFB captures, representing "common" 2D and 3D application workloads (the 3D workloads were run using VirtualGL), were studied using the TightVNC encoder in isolation. Some of the analysis features in the TightVNC encoder, such as smoothness detection, were found to generate a lot of CPU usage with little or no benefit in compression, so those features were disabled. JPEG encoding was accelerated using libjpeg-turbo (which achieves a 2-4x speedup over plain libjpeg on modern x86 or ARM processors.) Finally, the "palette threshold" (minimum number of colors that the subrectangle must have before it is compressed using JPEG or raw) was adjusted to account for the fact that JPEG encoding is now quite a bit faster (meaning that we can now use it more without a CPU penalty.) TurboVNC has additional optimizations, such as the ability to count colors and encode JPEG images directly from the framebuffer without first translating the pixels into RGB. The TurboVNC encoder compares quite favorably in terms of compression ratio with TightVNC and generally encodes a great deal faster (often an order of magnitude or more.) The version of the TurboVNC encoder included in this patch is roughly equivalent to the one found in version 0.6 of the Unix TurboVNC Server, with a few minor patches integrated from TurboVNC 1.1. TurboVNC 1.0 added multi-threading capabilities, which can be added in later if desired (at the expense of making libvncserver depend on libpthread.) Because TurboVNC uses a fundamentally different mix of subencodings than TightVNC, because it uses the identical protocol (and thus a viewer really has no idea whether it's talking to a TightVNC or TurboVNC server), and because it doesn't support rfbTightPng (and in fact conflicts with it-- see below), the TurboVNC and TightVNC encoders cannot be enabled simultaneously. Compatibility: In *most* cases, a TurboVNC-enabled viewer is fully compatible with a TightVNC server, and vice versa. TurboVNC supports pseudo-encodings for specifying a fine-grained (1-100) quality scale and specifying chrominance subsampling. If a TurboVNC viewer sends those to a TightVNC server, then the TightVNC server ignores them, so the TurboVNC viewer also sends the quality on a 0-9 scale that the TightVNC server can understand. Similarly, the TurboVNC server checks first for fine-grained quality and subsampling pseudo-encodings from the viewer, and failing to receive those, it then checks for the TightVNC 0-9 quality pseudo-encoding. There is one case in which the two systems are not compatible, and that is when a TightVNC or TigerVNC viewer requests compression level 0 without JPEG from a TurboVNC server. For performance reasons, this causes the TurboVNC server to send images directly to the viewer, bypassing Zlib. When the TurboVNC server does this, it also sets bits 7-4 in the compression control byte to rfbTightNoZlib (0x0A), which is unfortunately the same value as rfbTightPng. Older TightVNC viewers that don't handle PNG will assume that the stream is uncompressed but still encapsulated in a Zlib structure, whereas newer PNG-supporting TightVNC viewers will assume that the stream is PNG. In either case, the viewer will probably crash. Since most VNC viewers don't expose compression level 0 in the GUI, this is a relatively rare situation. Description of changes: configure.ac -- Added support for libjpeg-turbo. If passed an argument of --with-turbovnc, configure will now run (or, if cross-compiling, just link) a test program that determines whether the libjpeg library being used is libjpeg-turbo. libjpeg-turbo must be used when building the TurboVNC encoder, because the TurboVNC encoder relies on the libjpeg-turbo colorspace extensions in order to compress images directly out of the framebuffer (which may be, for instance, BGRA rather than RGB.) libjpeg-turbo can optionally be used with the TightVNC encoder as well, but the speedup will only be marginal (the report linked above explains why in more detail, but basically it's because of Amdahl's Law. The TightVNC encoder was designed with the assumption that JPEG had a very high CPU cost, and thus JPEG is used only sparingly.) -- Added a new configure variable, JPEG_LDFLAGS. This is necessitated by the fact that libjpeg-turbo often distributes libjpeg.a and libjpeg.so in /opt/libjpeg-turbo/lib32 or /opt/libjpeg-turbo/lib64, and many people prefer to statically link with it. Thus, more flexibility is needed than is provided by --with-jpeg. If JPEG_LDFLAGS is specified, then it overrides the changes to LDFLAGS enacted by --with-jpeg (but --with-jpeg is still used to set the include path.) The addition of JPEG_LDFLAGS necessitated replacing AC_CHECK_LIB with AC_LINK_IFELSE (because AC_CHECK_LIB automatically sets LIBS to -ljpeg, which is not what we want if we're, for instance, linking statically with libjpeg-turbo.) -- configure does not check for PNG support if TurboVNC encoding is enabled. This prevents the rfbSendRectEncodingTightPng() function from being compiled in, since the TurboVNC encoder doesn't (and can't) support it. common/turbojpeg.c, common/turbojpeg.h -- TurboJPEG is a simple API used to compress and decompress JPEG images in memory. It was originally implemented because it was desirable to use different types of underlying technologies to compress JPEG on different platforms (mediaLib on SPARC, Quicktime on PPC Macs, Intel Performance Primitives, etc.) These days, however, libjpeg-turbo is the only underlying technology used by TurboVNC, so TurboJPEG's purpose is largely just code simplicity and flexibility. Thus, since there is no real need for libvncserver to use any technology other than libjpeg-turbo for compressing JPEG, the TurboJPEG wrapper for libjpeg-turbo has been included in-tree so that libvncserver can be directly linked with libjpeg-turbo. This is convenient because many modern Linux distros (Fedora, Ubuntu, etc.) now ship libjpeg-turbo as their default libjpeg library. libvncserver/rfbserver.c -- Added logic to check for the TurboVNC fine-grained quality level and subsampling encodings and to map Tight (0-9) quality levels to appropriate fine-grained quality level and subsampling values if communicating with a TightVNC/TigerVNC viewer. libvncserver/turbo.c -- TurboVNC encoder (compiled instead of libvncserver/tight.c) rfb/rfb.h -- Added support for the TurboVNC subsampling level rfb/rfbproto.h -- Added constants for the TurboVNC fine quality level and subsampling encodings as well as the rfbTightNoZlib constant and notes on its usage.
13 years ago
}
#endif
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
if((row_pointer=(JSAMPROW *)malloc(sizeof(JSAMPROW)
*dinfo->output_height))==NULL)
_throw("tjDecompress2(): Memory allocation failure");
for(i=0; i<(int)dinfo->output_height; i++)
{
if(flags&TJFLAG_BOTTOMUP)
row_pointer[i]=&dstBuf[(dinfo->output_height-i-1)*pitch];
else row_pointer[i]=&dstBuf[i*pitch];
}
while(dinfo->output_scanline<dinfo->output_height)
Add TurboVNC encoding support. TurboVNC is a variant of TightVNC that uses the same client/server protocol (RFB version 3.8t), and thus it is fully cross-compatible with TightVNC and TigerVNC (with one exception, which is noted below.) Both the TightVNC and TurboVNC encoders analyze each rectangle, pick out regions of solid color to send separately, and send the remaining subrectangles using mono, indexed color, JPEG, or raw encoding, depending on the number of colors in the subrectangle. However, TurboVNC uses a fundamentally different selection algorithm to determine the appropriate subencoding to use for each subrectangle. Thus, while it sends a protocol stream that can be decoded by any TightVNC-compatible viewer, the mix of subencoding types in this protocol stream will be different from those generated by a TightVNC server. The research that led to TurboVNC is described in the following report: http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf. In summary: 20 RFB captures, representing "common" 2D and 3D application workloads (the 3D workloads were run using VirtualGL), were studied using the TightVNC encoder in isolation. Some of the analysis features in the TightVNC encoder, such as smoothness detection, were found to generate a lot of CPU usage with little or no benefit in compression, so those features were disabled. JPEG encoding was accelerated using libjpeg-turbo (which achieves a 2-4x speedup over plain libjpeg on modern x86 or ARM processors.) Finally, the "palette threshold" (minimum number of colors that the subrectangle must have before it is compressed using JPEG or raw) was adjusted to account for the fact that JPEG encoding is now quite a bit faster (meaning that we can now use it more without a CPU penalty.) TurboVNC has additional optimizations, such as the ability to count colors and encode JPEG images directly from the framebuffer without first translating the pixels into RGB. The TurboVNC encoder compares quite favorably in terms of compression ratio with TightVNC and generally encodes a great deal faster (often an order of magnitude or more.) The version of the TurboVNC encoder included in this patch is roughly equivalent to the one found in version 0.6 of the Unix TurboVNC Server, with a few minor patches integrated from TurboVNC 1.1. TurboVNC 1.0 added multi-threading capabilities, which can be added in later if desired (at the expense of making libvncserver depend on libpthread.) Because TurboVNC uses a fundamentally different mix of subencodings than TightVNC, because it uses the identical protocol (and thus a viewer really has no idea whether it's talking to a TightVNC or TurboVNC server), and because it doesn't support rfbTightPng (and in fact conflicts with it-- see below), the TurboVNC and TightVNC encoders cannot be enabled simultaneously. Compatibility: In *most* cases, a TurboVNC-enabled viewer is fully compatible with a TightVNC server, and vice versa. TurboVNC supports pseudo-encodings for specifying a fine-grained (1-100) quality scale and specifying chrominance subsampling. If a TurboVNC viewer sends those to a TightVNC server, then the TightVNC server ignores them, so the TurboVNC viewer also sends the quality on a 0-9 scale that the TightVNC server can understand. Similarly, the TurboVNC server checks first for fine-grained quality and subsampling pseudo-encodings from the viewer, and failing to receive those, it then checks for the TightVNC 0-9 quality pseudo-encoding. There is one case in which the two systems are not compatible, and that is when a TightVNC or TigerVNC viewer requests compression level 0 without JPEG from a TurboVNC server. For performance reasons, this causes the TurboVNC server to send images directly to the viewer, bypassing Zlib. When the TurboVNC server does this, it also sets bits 7-4 in the compression control byte to rfbTightNoZlib (0x0A), which is unfortunately the same value as rfbTightPng. Older TightVNC viewers that don't handle PNG will assume that the stream is uncompressed but still encapsulated in a Zlib structure, whereas newer PNG-supporting TightVNC viewers will assume that the stream is PNG. In either case, the viewer will probably crash. Since most VNC viewers don't expose compression level 0 in the GUI, this is a relatively rare situation. Description of changes: configure.ac -- Added support for libjpeg-turbo. If passed an argument of --with-turbovnc, configure will now run (or, if cross-compiling, just link) a test program that determines whether the libjpeg library being used is libjpeg-turbo. libjpeg-turbo must be used when building the TurboVNC encoder, because the TurboVNC encoder relies on the libjpeg-turbo colorspace extensions in order to compress images directly out of the framebuffer (which may be, for instance, BGRA rather than RGB.) libjpeg-turbo can optionally be used with the TightVNC encoder as well, but the speedup will only be marginal (the report linked above explains why in more detail, but basically it's because of Amdahl's Law. The TightVNC encoder was designed with the assumption that JPEG had a very high CPU cost, and thus JPEG is used only sparingly.) -- Added a new configure variable, JPEG_LDFLAGS. This is necessitated by the fact that libjpeg-turbo often distributes libjpeg.a and libjpeg.so in /opt/libjpeg-turbo/lib32 or /opt/libjpeg-turbo/lib64, and many people prefer to statically link with it. Thus, more flexibility is needed than is provided by --with-jpeg. If JPEG_LDFLAGS is specified, then it overrides the changes to LDFLAGS enacted by --with-jpeg (but --with-jpeg is still used to set the include path.) The addition of JPEG_LDFLAGS necessitated replacing AC_CHECK_LIB with AC_LINK_IFELSE (because AC_CHECK_LIB automatically sets LIBS to -ljpeg, which is not what we want if we're, for instance, linking statically with libjpeg-turbo.) -- configure does not check for PNG support if TurboVNC encoding is enabled. This prevents the rfbSendRectEncodingTightPng() function from being compiled in, since the TurboVNC encoder doesn't (and can't) support it. common/turbojpeg.c, common/turbojpeg.h -- TurboJPEG is a simple API used to compress and decompress JPEG images in memory. It was originally implemented because it was desirable to use different types of underlying technologies to compress JPEG on different platforms (mediaLib on SPARC, Quicktime on PPC Macs, Intel Performance Primitives, etc.) These days, however, libjpeg-turbo is the only underlying technology used by TurboVNC, so TurboJPEG's purpose is largely just code simplicity and flexibility. Thus, since there is no real need for libvncserver to use any technology other than libjpeg-turbo for compressing JPEG, the TurboJPEG wrapper for libjpeg-turbo has been included in-tree so that libvncserver can be directly linked with libjpeg-turbo. This is convenient because many modern Linux distros (Fedora, Ubuntu, etc.) now ship libjpeg-turbo as their default libjpeg library. libvncserver/rfbserver.c -- Added logic to check for the TurboVNC fine-grained quality level and subsampling encodings and to map Tight (0-9) quality levels to appropriate fine-grained quality level and subsampling values if communicating with a TightVNC/TigerVNC viewer. libvncserver/turbo.c -- TurboVNC encoder (compiled instead of libvncserver/tight.c) rfb/rfb.h -- Added support for the TurboVNC subsampling level rfb/rfbproto.h -- Added constants for the TurboVNC fine quality level and subsampling encodings as well as the rfbTightNoZlib constant and notes on its usage.
13 years ago
{
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
jpeg_read_scanlines(dinfo, &row_pointer[dinfo->output_scanline],
dinfo->output_height-dinfo->output_scanline);
Add TurboVNC encoding support. TurboVNC is a variant of TightVNC that uses the same client/server protocol (RFB version 3.8t), and thus it is fully cross-compatible with TightVNC and TigerVNC (with one exception, which is noted below.) Both the TightVNC and TurboVNC encoders analyze each rectangle, pick out regions of solid color to send separately, and send the remaining subrectangles using mono, indexed color, JPEG, or raw encoding, depending on the number of colors in the subrectangle. However, TurboVNC uses a fundamentally different selection algorithm to determine the appropriate subencoding to use for each subrectangle. Thus, while it sends a protocol stream that can be decoded by any TightVNC-compatible viewer, the mix of subencoding types in this protocol stream will be different from those generated by a TightVNC server. The research that led to TurboVNC is described in the following report: http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf. In summary: 20 RFB captures, representing "common" 2D and 3D application workloads (the 3D workloads were run using VirtualGL), were studied using the TightVNC encoder in isolation. Some of the analysis features in the TightVNC encoder, such as smoothness detection, were found to generate a lot of CPU usage with little or no benefit in compression, so those features were disabled. JPEG encoding was accelerated using libjpeg-turbo (which achieves a 2-4x speedup over plain libjpeg on modern x86 or ARM processors.) Finally, the "palette threshold" (minimum number of colors that the subrectangle must have before it is compressed using JPEG or raw) was adjusted to account for the fact that JPEG encoding is now quite a bit faster (meaning that we can now use it more without a CPU penalty.) TurboVNC has additional optimizations, such as the ability to count colors and encode JPEG images directly from the framebuffer without first translating the pixels into RGB. The TurboVNC encoder compares quite favorably in terms of compression ratio with TightVNC and generally encodes a great deal faster (often an order of magnitude or more.) The version of the TurboVNC encoder included in this patch is roughly equivalent to the one found in version 0.6 of the Unix TurboVNC Server, with a few minor patches integrated from TurboVNC 1.1. TurboVNC 1.0 added multi-threading capabilities, which can be added in later if desired (at the expense of making libvncserver depend on libpthread.) Because TurboVNC uses a fundamentally different mix of subencodings than TightVNC, because it uses the identical protocol (and thus a viewer really has no idea whether it's talking to a TightVNC or TurboVNC server), and because it doesn't support rfbTightPng (and in fact conflicts with it-- see below), the TurboVNC and TightVNC encoders cannot be enabled simultaneously. Compatibility: In *most* cases, a TurboVNC-enabled viewer is fully compatible with a TightVNC server, and vice versa. TurboVNC supports pseudo-encodings for specifying a fine-grained (1-100) quality scale and specifying chrominance subsampling. If a TurboVNC viewer sends those to a TightVNC server, then the TightVNC server ignores them, so the TurboVNC viewer also sends the quality on a 0-9 scale that the TightVNC server can understand. Similarly, the TurboVNC server checks first for fine-grained quality and subsampling pseudo-encodings from the viewer, and failing to receive those, it then checks for the TightVNC 0-9 quality pseudo-encoding. There is one case in which the two systems are not compatible, and that is when a TightVNC or TigerVNC viewer requests compression level 0 without JPEG from a TurboVNC server. For performance reasons, this causes the TurboVNC server to send images directly to the viewer, bypassing Zlib. When the TurboVNC server does this, it also sets bits 7-4 in the compression control byte to rfbTightNoZlib (0x0A), which is unfortunately the same value as rfbTightPng. Older TightVNC viewers that don't handle PNG will assume that the stream is uncompressed but still encapsulated in a Zlib structure, whereas newer PNG-supporting TightVNC viewers will assume that the stream is PNG. In either case, the viewer will probably crash. Since most VNC viewers don't expose compression level 0 in the GUI, this is a relatively rare situation. Description of changes: configure.ac -- Added support for libjpeg-turbo. If passed an argument of --with-turbovnc, configure will now run (or, if cross-compiling, just link) a test program that determines whether the libjpeg library being used is libjpeg-turbo. libjpeg-turbo must be used when building the TurboVNC encoder, because the TurboVNC encoder relies on the libjpeg-turbo colorspace extensions in order to compress images directly out of the framebuffer (which may be, for instance, BGRA rather than RGB.) libjpeg-turbo can optionally be used with the TightVNC encoder as well, but the speedup will only be marginal (the report linked above explains why in more detail, but basically it's because of Amdahl's Law. The TightVNC encoder was designed with the assumption that JPEG had a very high CPU cost, and thus JPEG is used only sparingly.) -- Added a new configure variable, JPEG_LDFLAGS. This is necessitated by the fact that libjpeg-turbo often distributes libjpeg.a and libjpeg.so in /opt/libjpeg-turbo/lib32 or /opt/libjpeg-turbo/lib64, and many people prefer to statically link with it. Thus, more flexibility is needed than is provided by --with-jpeg. If JPEG_LDFLAGS is specified, then it overrides the changes to LDFLAGS enacted by --with-jpeg (but --with-jpeg is still used to set the include path.) The addition of JPEG_LDFLAGS necessitated replacing AC_CHECK_LIB with AC_LINK_IFELSE (because AC_CHECK_LIB automatically sets LIBS to -ljpeg, which is not what we want if we're, for instance, linking statically with libjpeg-turbo.) -- configure does not check for PNG support if TurboVNC encoding is enabled. This prevents the rfbSendRectEncodingTightPng() function from being compiled in, since the TurboVNC encoder doesn't (and can't) support it. common/turbojpeg.c, common/turbojpeg.h -- TurboJPEG is a simple API used to compress and decompress JPEG images in memory. It was originally implemented because it was desirable to use different types of underlying technologies to compress JPEG on different platforms (mediaLib on SPARC, Quicktime on PPC Macs, Intel Performance Primitives, etc.) These days, however, libjpeg-turbo is the only underlying technology used by TurboVNC, so TurboJPEG's purpose is largely just code simplicity and flexibility. Thus, since there is no real need for libvncserver to use any technology other than libjpeg-turbo for compressing JPEG, the TurboJPEG wrapper for libjpeg-turbo has been included in-tree so that libvncserver can be directly linked with libjpeg-turbo. This is convenient because many modern Linux distros (Fedora, Ubuntu, etc.) now ship libjpeg-turbo as their default libjpeg library. libvncserver/rfbserver.c -- Added logic to check for the TurboVNC fine-grained quality level and subsampling encodings and to map Tight (0-9) quality levels to appropriate fine-grained quality level and subsampling values if communicating with a TightVNC/TigerVNC viewer. libvncserver/turbo.c -- TurboVNC encoder (compiled instead of libvncserver/tight.c) rfb/rfb.h -- Added support for the TurboVNC subsampling level rfb/rfbproto.h -- Added constants for the TurboVNC fine quality level and subsampling encodings as well as the rfbTightNoZlib constant and notes on its usage.
13 years ago
}
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
jpeg_finish_decompress(dinfo);
#ifndef JCS_EXTENSIONS
fromRGB(rgbBuf, _dstBuf, width, _pitch, height, pixelFormat);
#endif
Add TurboVNC encoding support. TurboVNC is a variant of TightVNC that uses the same client/server protocol (RFB version 3.8t), and thus it is fully cross-compatible with TightVNC and TigerVNC (with one exception, which is noted below.) Both the TightVNC and TurboVNC encoders analyze each rectangle, pick out regions of solid color to send separately, and send the remaining subrectangles using mono, indexed color, JPEG, or raw encoding, depending on the number of colors in the subrectangle. However, TurboVNC uses a fundamentally different selection algorithm to determine the appropriate subencoding to use for each subrectangle. Thus, while it sends a protocol stream that can be decoded by any TightVNC-compatible viewer, the mix of subencoding types in this protocol stream will be different from those generated by a TightVNC server. The research that led to TurboVNC is described in the following report: http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf. In summary: 20 RFB captures, representing "common" 2D and 3D application workloads (the 3D workloads were run using VirtualGL), were studied using the TightVNC encoder in isolation. Some of the analysis features in the TightVNC encoder, such as smoothness detection, were found to generate a lot of CPU usage with little or no benefit in compression, so those features were disabled. JPEG encoding was accelerated using libjpeg-turbo (which achieves a 2-4x speedup over plain libjpeg on modern x86 or ARM processors.) Finally, the "palette threshold" (minimum number of colors that the subrectangle must have before it is compressed using JPEG or raw) was adjusted to account for the fact that JPEG encoding is now quite a bit faster (meaning that we can now use it more without a CPU penalty.) TurboVNC has additional optimizations, such as the ability to count colors and encode JPEG images directly from the framebuffer without first translating the pixels into RGB. The TurboVNC encoder compares quite favorably in terms of compression ratio with TightVNC and generally encodes a great deal faster (often an order of magnitude or more.) The version of the TurboVNC encoder included in this patch is roughly equivalent to the one found in version 0.6 of the Unix TurboVNC Server, with a few minor patches integrated from TurboVNC 1.1. TurboVNC 1.0 added multi-threading capabilities, which can be added in later if desired (at the expense of making libvncserver depend on libpthread.) Because TurboVNC uses a fundamentally different mix of subencodings than TightVNC, because it uses the identical protocol (and thus a viewer really has no idea whether it's talking to a TightVNC or TurboVNC server), and because it doesn't support rfbTightPng (and in fact conflicts with it-- see below), the TurboVNC and TightVNC encoders cannot be enabled simultaneously. Compatibility: In *most* cases, a TurboVNC-enabled viewer is fully compatible with a TightVNC server, and vice versa. TurboVNC supports pseudo-encodings for specifying a fine-grained (1-100) quality scale and specifying chrominance subsampling. If a TurboVNC viewer sends those to a TightVNC server, then the TightVNC server ignores them, so the TurboVNC viewer also sends the quality on a 0-9 scale that the TightVNC server can understand. Similarly, the TurboVNC server checks first for fine-grained quality and subsampling pseudo-encodings from the viewer, and failing to receive those, it then checks for the TightVNC 0-9 quality pseudo-encoding. There is one case in which the two systems are not compatible, and that is when a TightVNC or TigerVNC viewer requests compression level 0 without JPEG from a TurboVNC server. For performance reasons, this causes the TurboVNC server to send images directly to the viewer, bypassing Zlib. When the TurboVNC server does this, it also sets bits 7-4 in the compression control byte to rfbTightNoZlib (0x0A), which is unfortunately the same value as rfbTightPng. Older TightVNC viewers that don't handle PNG will assume that the stream is uncompressed but still encapsulated in a Zlib structure, whereas newer PNG-supporting TightVNC viewers will assume that the stream is PNG. In either case, the viewer will probably crash. Since most VNC viewers don't expose compression level 0 in the GUI, this is a relatively rare situation. Description of changes: configure.ac -- Added support for libjpeg-turbo. If passed an argument of --with-turbovnc, configure will now run (or, if cross-compiling, just link) a test program that determines whether the libjpeg library being used is libjpeg-turbo. libjpeg-turbo must be used when building the TurboVNC encoder, because the TurboVNC encoder relies on the libjpeg-turbo colorspace extensions in order to compress images directly out of the framebuffer (which may be, for instance, BGRA rather than RGB.) libjpeg-turbo can optionally be used with the TightVNC encoder as well, but the speedup will only be marginal (the report linked above explains why in more detail, but basically it's because of Amdahl's Law. The TightVNC encoder was designed with the assumption that JPEG had a very high CPU cost, and thus JPEG is used only sparingly.) -- Added a new configure variable, JPEG_LDFLAGS. This is necessitated by the fact that libjpeg-turbo often distributes libjpeg.a and libjpeg.so in /opt/libjpeg-turbo/lib32 or /opt/libjpeg-turbo/lib64, and many people prefer to statically link with it. Thus, more flexibility is needed than is provided by --with-jpeg. If JPEG_LDFLAGS is specified, then it overrides the changes to LDFLAGS enacted by --with-jpeg (but --with-jpeg is still used to set the include path.) The addition of JPEG_LDFLAGS necessitated replacing AC_CHECK_LIB with AC_LINK_IFELSE (because AC_CHECK_LIB automatically sets LIBS to -ljpeg, which is not what we want if we're, for instance, linking statically with libjpeg-turbo.) -- configure does not check for PNG support if TurboVNC encoding is enabled. This prevents the rfbSendRectEncodingTightPng() function from being compiled in, since the TurboVNC encoder doesn't (and can't) support it. common/turbojpeg.c, common/turbojpeg.h -- TurboJPEG is a simple API used to compress and decompress JPEG images in memory. It was originally implemented because it was desirable to use different types of underlying technologies to compress JPEG on different platforms (mediaLib on SPARC, Quicktime on PPC Macs, Intel Performance Primitives, etc.) These days, however, libjpeg-turbo is the only underlying technology used by TurboVNC, so TurboJPEG's purpose is largely just code simplicity and flexibility. Thus, since there is no real need for libvncserver to use any technology other than libjpeg-turbo for compressing JPEG, the TurboJPEG wrapper for libjpeg-turbo has been included in-tree so that libvncserver can be directly linked with libjpeg-turbo. This is convenient because many modern Linux distros (Fedora, Ubuntu, etc.) now ship libjpeg-turbo as their default libjpeg library. libvncserver/rfbserver.c -- Added logic to check for the TurboVNC fine-grained quality level and subsampling encodings and to map Tight (0-9) quality levels to appropriate fine-grained quality level and subsampling values if communicating with a TightVNC/TigerVNC viewer. libvncserver/turbo.c -- TurboVNC encoder (compiled instead of libvncserver/tight.c) rfb/rfb.h -- Added support for the TurboVNC subsampling level rfb/rfbproto.h -- Added constants for the TurboVNC fine quality level and subsampling encodings as well as the rfbTightNoZlib constant and notes on its usage.
13 years ago
bailout:
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
if(dinfo->global_state>DSTATE_START) jpeg_abort_decompress(dinfo);
#ifndef JCS_EXTENSIONS
if(rgbBuf) free(rgbBuf);
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
#endif
Add TurboVNC encoding support. TurboVNC is a variant of TightVNC that uses the same client/server protocol (RFB version 3.8t), and thus it is fully cross-compatible with TightVNC and TigerVNC (with one exception, which is noted below.) Both the TightVNC and TurboVNC encoders analyze each rectangle, pick out regions of solid color to send separately, and send the remaining subrectangles using mono, indexed color, JPEG, or raw encoding, depending on the number of colors in the subrectangle. However, TurboVNC uses a fundamentally different selection algorithm to determine the appropriate subencoding to use for each subrectangle. Thus, while it sends a protocol stream that can be decoded by any TightVNC-compatible viewer, the mix of subencoding types in this protocol stream will be different from those generated by a TightVNC server. The research that led to TurboVNC is described in the following report: http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf. In summary: 20 RFB captures, representing "common" 2D and 3D application workloads (the 3D workloads were run using VirtualGL), were studied using the TightVNC encoder in isolation. Some of the analysis features in the TightVNC encoder, such as smoothness detection, were found to generate a lot of CPU usage with little or no benefit in compression, so those features were disabled. JPEG encoding was accelerated using libjpeg-turbo (which achieves a 2-4x speedup over plain libjpeg on modern x86 or ARM processors.) Finally, the "palette threshold" (minimum number of colors that the subrectangle must have before it is compressed using JPEG or raw) was adjusted to account for the fact that JPEG encoding is now quite a bit faster (meaning that we can now use it more without a CPU penalty.) TurboVNC has additional optimizations, such as the ability to count colors and encode JPEG images directly from the framebuffer without first translating the pixels into RGB. The TurboVNC encoder compares quite favorably in terms of compression ratio with TightVNC and generally encodes a great deal faster (often an order of magnitude or more.) The version of the TurboVNC encoder included in this patch is roughly equivalent to the one found in version 0.6 of the Unix TurboVNC Server, with a few minor patches integrated from TurboVNC 1.1. TurboVNC 1.0 added multi-threading capabilities, which can be added in later if desired (at the expense of making libvncserver depend on libpthread.) Because TurboVNC uses a fundamentally different mix of subencodings than TightVNC, because it uses the identical protocol (and thus a viewer really has no idea whether it's talking to a TightVNC or TurboVNC server), and because it doesn't support rfbTightPng (and in fact conflicts with it-- see below), the TurboVNC and TightVNC encoders cannot be enabled simultaneously. Compatibility: In *most* cases, a TurboVNC-enabled viewer is fully compatible with a TightVNC server, and vice versa. TurboVNC supports pseudo-encodings for specifying a fine-grained (1-100) quality scale and specifying chrominance subsampling. If a TurboVNC viewer sends those to a TightVNC server, then the TightVNC server ignores them, so the TurboVNC viewer also sends the quality on a 0-9 scale that the TightVNC server can understand. Similarly, the TurboVNC server checks first for fine-grained quality and subsampling pseudo-encodings from the viewer, and failing to receive those, it then checks for the TightVNC 0-9 quality pseudo-encoding. There is one case in which the two systems are not compatible, and that is when a TightVNC or TigerVNC viewer requests compression level 0 without JPEG from a TurboVNC server. For performance reasons, this causes the TurboVNC server to send images directly to the viewer, bypassing Zlib. When the TurboVNC server does this, it also sets bits 7-4 in the compression control byte to rfbTightNoZlib (0x0A), which is unfortunately the same value as rfbTightPng. Older TightVNC viewers that don't handle PNG will assume that the stream is uncompressed but still encapsulated in a Zlib structure, whereas newer PNG-supporting TightVNC viewers will assume that the stream is PNG. In either case, the viewer will probably crash. Since most VNC viewers don't expose compression level 0 in the GUI, this is a relatively rare situation. Description of changes: configure.ac -- Added support for libjpeg-turbo. If passed an argument of --with-turbovnc, configure will now run (or, if cross-compiling, just link) a test program that determines whether the libjpeg library being used is libjpeg-turbo. libjpeg-turbo must be used when building the TurboVNC encoder, because the TurboVNC encoder relies on the libjpeg-turbo colorspace extensions in order to compress images directly out of the framebuffer (which may be, for instance, BGRA rather than RGB.) libjpeg-turbo can optionally be used with the TightVNC encoder as well, but the speedup will only be marginal (the report linked above explains why in more detail, but basically it's because of Amdahl's Law. The TightVNC encoder was designed with the assumption that JPEG had a very high CPU cost, and thus JPEG is used only sparingly.) -- Added a new configure variable, JPEG_LDFLAGS. This is necessitated by the fact that libjpeg-turbo often distributes libjpeg.a and libjpeg.so in /opt/libjpeg-turbo/lib32 or /opt/libjpeg-turbo/lib64, and many people prefer to statically link with it. Thus, more flexibility is needed than is provided by --with-jpeg. If JPEG_LDFLAGS is specified, then it overrides the changes to LDFLAGS enacted by --with-jpeg (but --with-jpeg is still used to set the include path.) The addition of JPEG_LDFLAGS necessitated replacing AC_CHECK_LIB with AC_LINK_IFELSE (because AC_CHECK_LIB automatically sets LIBS to -ljpeg, which is not what we want if we're, for instance, linking statically with libjpeg-turbo.) -- configure does not check for PNG support if TurboVNC encoding is enabled. This prevents the rfbSendRectEncodingTightPng() function from being compiled in, since the TurboVNC encoder doesn't (and can't) support it. common/turbojpeg.c, common/turbojpeg.h -- TurboJPEG is a simple API used to compress and decompress JPEG images in memory. It was originally implemented because it was desirable to use different types of underlying technologies to compress JPEG on different platforms (mediaLib on SPARC, Quicktime on PPC Macs, Intel Performance Primitives, etc.) These days, however, libjpeg-turbo is the only underlying technology used by TurboVNC, so TurboJPEG's purpose is largely just code simplicity and flexibility. Thus, since there is no real need for libvncserver to use any technology other than libjpeg-turbo for compressing JPEG, the TurboJPEG wrapper for libjpeg-turbo has been included in-tree so that libvncserver can be directly linked with libjpeg-turbo. This is convenient because many modern Linux distros (Fedora, Ubuntu, etc.) now ship libjpeg-turbo as their default libjpeg library. libvncserver/rfbserver.c -- Added logic to check for the TurboVNC fine-grained quality level and subsampling encodings and to map Tight (0-9) quality levels to appropriate fine-grained quality level and subsampling values if communicating with a TightVNC/TigerVNC viewer. libvncserver/turbo.c -- TurboVNC encoder (compiled instead of libvncserver/tight.c) rfb/rfb.h -- Added support for the TurboVNC subsampling level rfb/rfbproto.h -- Added constants for the TurboVNC fine quality level and subsampling encodings as well as the rfbTightNoZlib constant and notes on its usage.
13 years ago
if(row_pointer) free(row_pointer);
return retval;
}
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
DLLEXPORT int DLLCALL tjDecompress(tjhandle handle, unsigned char *jpegBuf,
unsigned long jpegSize, unsigned char *dstBuf, int width, int pitch,
int height, int pixelSize, int flags)
Add TurboVNC encoding support. TurboVNC is a variant of TightVNC that uses the same client/server protocol (RFB version 3.8t), and thus it is fully cross-compatible with TightVNC and TigerVNC (with one exception, which is noted below.) Both the TightVNC and TurboVNC encoders analyze each rectangle, pick out regions of solid color to send separately, and send the remaining subrectangles using mono, indexed color, JPEG, or raw encoding, depending on the number of colors in the subrectangle. However, TurboVNC uses a fundamentally different selection algorithm to determine the appropriate subencoding to use for each subrectangle. Thus, while it sends a protocol stream that can be decoded by any TightVNC-compatible viewer, the mix of subencoding types in this protocol stream will be different from those generated by a TightVNC server. The research that led to TurboVNC is described in the following report: http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf. In summary: 20 RFB captures, representing "common" 2D and 3D application workloads (the 3D workloads were run using VirtualGL), were studied using the TightVNC encoder in isolation. Some of the analysis features in the TightVNC encoder, such as smoothness detection, were found to generate a lot of CPU usage with little or no benefit in compression, so those features were disabled. JPEG encoding was accelerated using libjpeg-turbo (which achieves a 2-4x speedup over plain libjpeg on modern x86 or ARM processors.) Finally, the "palette threshold" (minimum number of colors that the subrectangle must have before it is compressed using JPEG or raw) was adjusted to account for the fact that JPEG encoding is now quite a bit faster (meaning that we can now use it more without a CPU penalty.) TurboVNC has additional optimizations, such as the ability to count colors and encode JPEG images directly from the framebuffer without first translating the pixels into RGB. The TurboVNC encoder compares quite favorably in terms of compression ratio with TightVNC and generally encodes a great deal faster (often an order of magnitude or more.) The version of the TurboVNC encoder included in this patch is roughly equivalent to the one found in version 0.6 of the Unix TurboVNC Server, with a few minor patches integrated from TurboVNC 1.1. TurboVNC 1.0 added multi-threading capabilities, which can be added in later if desired (at the expense of making libvncserver depend on libpthread.) Because TurboVNC uses a fundamentally different mix of subencodings than TightVNC, because it uses the identical protocol (and thus a viewer really has no idea whether it's talking to a TightVNC or TurboVNC server), and because it doesn't support rfbTightPng (and in fact conflicts with it-- see below), the TurboVNC and TightVNC encoders cannot be enabled simultaneously. Compatibility: In *most* cases, a TurboVNC-enabled viewer is fully compatible with a TightVNC server, and vice versa. TurboVNC supports pseudo-encodings for specifying a fine-grained (1-100) quality scale and specifying chrominance subsampling. If a TurboVNC viewer sends those to a TightVNC server, then the TightVNC server ignores them, so the TurboVNC viewer also sends the quality on a 0-9 scale that the TightVNC server can understand. Similarly, the TurboVNC server checks first for fine-grained quality and subsampling pseudo-encodings from the viewer, and failing to receive those, it then checks for the TightVNC 0-9 quality pseudo-encoding. There is one case in which the two systems are not compatible, and that is when a TightVNC or TigerVNC viewer requests compression level 0 without JPEG from a TurboVNC server. For performance reasons, this causes the TurboVNC server to send images directly to the viewer, bypassing Zlib. When the TurboVNC server does this, it also sets bits 7-4 in the compression control byte to rfbTightNoZlib (0x0A), which is unfortunately the same value as rfbTightPng. Older TightVNC viewers that don't handle PNG will assume that the stream is uncompressed but still encapsulated in a Zlib structure, whereas newer PNG-supporting TightVNC viewers will assume that the stream is PNG. In either case, the viewer will probably crash. Since most VNC viewers don't expose compression level 0 in the GUI, this is a relatively rare situation. Description of changes: configure.ac -- Added support for libjpeg-turbo. If passed an argument of --with-turbovnc, configure will now run (or, if cross-compiling, just link) a test program that determines whether the libjpeg library being used is libjpeg-turbo. libjpeg-turbo must be used when building the TurboVNC encoder, because the TurboVNC encoder relies on the libjpeg-turbo colorspace extensions in order to compress images directly out of the framebuffer (which may be, for instance, BGRA rather than RGB.) libjpeg-turbo can optionally be used with the TightVNC encoder as well, but the speedup will only be marginal (the report linked above explains why in more detail, but basically it's because of Amdahl's Law. The TightVNC encoder was designed with the assumption that JPEG had a very high CPU cost, and thus JPEG is used only sparingly.) -- Added a new configure variable, JPEG_LDFLAGS. This is necessitated by the fact that libjpeg-turbo often distributes libjpeg.a and libjpeg.so in /opt/libjpeg-turbo/lib32 or /opt/libjpeg-turbo/lib64, and many people prefer to statically link with it. Thus, more flexibility is needed than is provided by --with-jpeg. If JPEG_LDFLAGS is specified, then it overrides the changes to LDFLAGS enacted by --with-jpeg (but --with-jpeg is still used to set the include path.) The addition of JPEG_LDFLAGS necessitated replacing AC_CHECK_LIB with AC_LINK_IFELSE (because AC_CHECK_LIB automatically sets LIBS to -ljpeg, which is not what we want if we're, for instance, linking statically with libjpeg-turbo.) -- configure does not check for PNG support if TurboVNC encoding is enabled. This prevents the rfbSendRectEncodingTightPng() function from being compiled in, since the TurboVNC encoder doesn't (and can't) support it. common/turbojpeg.c, common/turbojpeg.h -- TurboJPEG is a simple API used to compress and decompress JPEG images in memory. It was originally implemented because it was desirable to use different types of underlying technologies to compress JPEG on different platforms (mediaLib on SPARC, Quicktime on PPC Macs, Intel Performance Primitives, etc.) These days, however, libjpeg-turbo is the only underlying technology used by TurboVNC, so TurboJPEG's purpose is largely just code simplicity and flexibility. Thus, since there is no real need for libvncserver to use any technology other than libjpeg-turbo for compressing JPEG, the TurboJPEG wrapper for libjpeg-turbo has been included in-tree so that libvncserver can be directly linked with libjpeg-turbo. This is convenient because many modern Linux distros (Fedora, Ubuntu, etc.) now ship libjpeg-turbo as their default libjpeg library. libvncserver/rfbserver.c -- Added logic to check for the TurboVNC fine-grained quality level and subsampling encodings and to map Tight (0-9) quality levels to appropriate fine-grained quality level and subsampling values if communicating with a TightVNC/TigerVNC viewer. libvncserver/turbo.c -- TurboVNC encoder (compiled instead of libvncserver/tight.c) rfb/rfb.h -- Added support for the TurboVNC subsampling level rfb/rfbproto.h -- Added constants for the TurboVNC fine quality level and subsampling encodings as well as the rfbTightNoZlib constant and notes on its usage.
13 years ago
{
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
13 years ago
return tjDecompress2(handle, jpegBuf, jpegSize, dstBuf, width, pitch,
height, getPixelFormat(pixelSize, flags), flags);
Add TurboVNC encoding support. TurboVNC is a variant of TightVNC that uses the same client/server protocol (RFB version 3.8t), and thus it is fully cross-compatible with TightVNC and TigerVNC (with one exception, which is noted below.) Both the TightVNC and TurboVNC encoders analyze each rectangle, pick out regions of solid color to send separately, and send the remaining subrectangles using mono, indexed color, JPEG, or raw encoding, depending on the number of colors in the subrectangle. However, TurboVNC uses a fundamentally different selection algorithm to determine the appropriate subencoding to use for each subrectangle. Thus, while it sends a protocol stream that can be decoded by any TightVNC-compatible viewer, the mix of subencoding types in this protocol stream will be different from those generated by a TightVNC server. The research that led to TurboVNC is described in the following report: http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf. In summary: 20 RFB captures, representing "common" 2D and 3D application workloads (the 3D workloads were run using VirtualGL), were studied using the TightVNC encoder in isolation. Some of the analysis features in the TightVNC encoder, such as smoothness detection, were found to generate a lot of CPU usage with little or no benefit in compression, so those features were disabled. JPEG encoding was accelerated using libjpeg-turbo (which achieves a 2-4x speedup over plain libjpeg on modern x86 or ARM processors.) Finally, the "palette threshold" (minimum number of colors that the subrectangle must have before it is compressed using JPEG or raw) was adjusted to account for the fact that JPEG encoding is now quite a bit faster (meaning that we can now use it more without a CPU penalty.) TurboVNC has additional optimizations, such as the ability to count colors and encode JPEG images directly from the framebuffer without first translating the pixels into RGB. The TurboVNC encoder compares quite favorably in terms of compression ratio with TightVNC and generally encodes a great deal faster (often an order of magnitude or more.) The version of the TurboVNC encoder included in this patch is roughly equivalent to the one found in version 0.6 of the Unix TurboVNC Server, with a few minor patches integrated from TurboVNC 1.1. TurboVNC 1.0 added multi-threading capabilities, which can be added in later if desired (at the expense of making libvncserver depend on libpthread.) Because TurboVNC uses a fundamentally different mix of subencodings than TightVNC, because it uses the identical protocol (and thus a viewer really has no idea whether it's talking to a TightVNC or TurboVNC server), and because it doesn't support rfbTightPng (and in fact conflicts with it-- see below), the TurboVNC and TightVNC encoders cannot be enabled simultaneously. Compatibility: In *most* cases, a TurboVNC-enabled viewer is fully compatible with a TightVNC server, and vice versa. TurboVNC supports pseudo-encodings for specifying a fine-grained (1-100) quality scale and specifying chrominance subsampling. If a TurboVNC viewer sends those to a TightVNC server, then the TightVNC server ignores them, so the TurboVNC viewer also sends the quality on a 0-9 scale that the TightVNC server can understand. Similarly, the TurboVNC server checks first for fine-grained quality and subsampling pseudo-encodings from the viewer, and failing to receive those, it then checks for the TightVNC 0-9 quality pseudo-encoding. There is one case in which the two systems are not compatible, and that is when a TightVNC or TigerVNC viewer requests compression level 0 without JPEG from a TurboVNC server. For performance reasons, this causes the TurboVNC server to send images directly to the viewer, bypassing Zlib. When the TurboVNC server does this, it also sets bits 7-4 in the compression control byte to rfbTightNoZlib (0x0A), which is unfortunately the same value as rfbTightPng. Older TightVNC viewers that don't handle PNG will assume that the stream is uncompressed but still encapsulated in a Zlib structure, whereas newer PNG-supporting TightVNC viewers will assume that the stream is PNG. In either case, the viewer will probably crash. Since most VNC viewers don't expose compression level 0 in the GUI, this is a relatively rare situation. Description of changes: configure.ac -- Added support for libjpeg-turbo. If passed an argument of --with-turbovnc, configure will now run (or, if cross-compiling, just link) a test program that determines whether the libjpeg library being used is libjpeg-turbo. libjpeg-turbo must be used when building the TurboVNC encoder, because the TurboVNC encoder relies on the libjpeg-turbo colorspace extensions in order to compress images directly out of the framebuffer (which may be, for instance, BGRA rather than RGB.) libjpeg-turbo can optionally be used with the TightVNC encoder as well, but the speedup will only be marginal (the report linked above explains why in more detail, but basically it's because of Amdahl's Law. The TightVNC encoder was designed with the assumption that JPEG had a very high CPU cost, and thus JPEG is used only sparingly.) -- Added a new configure variable, JPEG_LDFLAGS. This is necessitated by the fact that libjpeg-turbo often distributes libjpeg.a and libjpeg.so in /opt/libjpeg-turbo/lib32 or /opt/libjpeg-turbo/lib64, and many people prefer to statically link with it. Thus, more flexibility is needed than is provided by --with-jpeg. If JPEG_LDFLAGS is specified, then it overrides the changes to LDFLAGS enacted by --with-jpeg (but --with-jpeg is still used to set the include path.) The addition of JPEG_LDFLAGS necessitated replacing AC_CHECK_LIB with AC_LINK_IFELSE (because AC_CHECK_LIB automatically sets LIBS to -ljpeg, which is not what we want if we're, for instance, linking statically with libjpeg-turbo.) -- configure does not check for PNG support if TurboVNC encoding is enabled. This prevents the rfbSendRectEncodingTightPng() function from being compiled in, since the TurboVNC encoder doesn't (and can't) support it. common/turbojpeg.c, common/turbojpeg.h -- TurboJPEG is a simple API used to compress and decompress JPEG images in memory. It was originally implemented because it was desirable to use different types of underlying technologies to compress JPEG on different platforms (mediaLib on SPARC, Quicktime on PPC Macs, Intel Performance Primitives, etc.) These days, however, libjpeg-turbo is the only underlying technology used by TurboVNC, so TurboJPEG's purpose is largely just code simplicity and flexibility. Thus, since there is no real need for libvncserver to use any technology other than libjpeg-turbo for compressing JPEG, the TurboJPEG wrapper for libjpeg-turbo has been included in-tree so that libvncserver can be directly linked with libjpeg-turbo. This is convenient because many modern Linux distros (Fedora, Ubuntu, etc.) now ship libjpeg-turbo as their default libjpeg library. libvncserver/rfbserver.c -- Added logic to check for the TurboVNC fine-grained quality level and subsampling encodings and to map Tight (0-9) quality levels to appropriate fine-grained quality level and subsampling values if communicating with a TightVNC/TigerVNC viewer. libvncserver/turbo.c -- TurboVNC encoder (compiled instead of libvncserver/tight.c) rfb/rfb.h -- Added support for the TurboVNC subsampling level rfb/rfbproto.h -- Added constants for the TurboVNC fine quality level and subsampling encodings as well as the rfbTightNoZlib constant and notes on its usage.
13 years ago
}