You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
libtdevnc/libvncserver/rfbserver.c

3594 lines
109 KiB

22 years ago
/*
* rfbserver.c - deal with server-side of the RFB protocol.
*/
/*
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
11 years ago
* Copyright (C) 2011-2012 D. R. Commander
* Copyright (C) 2005 Rohit Kumar, Johannes E. Schindelin
* Copyright (C) 2002 RealVNC Ltd.
22 years ago
* OSXvnc Copyright (C) 2001 Dan McGuirk <mcguirk@incompleteness.net>.
* Original Xvnc code Copyright (C) 1999 AT&T Laboratories Cambridge.
* All Rights Reserved.
*
* This is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation; either version 2 of the License, or
* (at your option) any later version.
*
* This software is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this software; if not, write to the Free Software
* Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307,
* USA.
*/
#ifdef __STRICT_ANSI__
#define _BSD_SOURCE
#define _POSIX_SOURCE
#define _XOPEN_SOURCE 600
#endif
#include <stdio.h>
#include <string.h>
#include <rfb/rfb.h>
#include <rfb/rfbregion.h>
#include "private.h"
#ifdef LIBVNCSERVER_HAVE_FCNTL_H
#include <fcntl.h>
#endif
#ifdef WIN32
#include <winsock2.h>
#include <ws2tcpip.h>
#include <io.h>
#define write(sock,buf,len) send(sock,buf,len,0)
#else
#ifdef LIBVNCSERVER_HAVE_UNISTD_H
22 years ago
#include <unistd.h>
#endif
22 years ago
#include <pwd.h>
#ifdef LIBVNCSERVER_HAVE_SYS_SOCKET_H
22 years ago
#include <sys/socket.h>
#endif
#ifdef LIBVNCSERVER_HAVE_NETINET_IN_H
22 years ago
#include <netinet/in.h>
#include <netinet/tcp.h>
#include <netdb.h>
22 years ago
#include <arpa/inet.h>
#endif
#endif
22 years ago
#ifdef DEBUGPROTO
#undef DEBUGPROTO
#define DEBUGPROTO(x) x
#else
#define DEBUGPROTO(x)
#endif
#include <stdarg.h>
#include <scale.h>
/* stst() */
#include <sys/types.h>
#include <sys/stat.h>
#include <unistd.h>
#ifndef WIN32
/* readdir() */
#include <dirent.h>
#endif
/* errno */
#include <errno.h>
/* strftime() */
#include <time.h>
22 years ago
#ifdef LIBVNCSERVER_WITH_WEBSOCKETS
#include "rfbssl.h"
#endif
#ifdef _MSC_VER
#define snprintf _snprintf /* Missing in MSVC */
/* Prevent POSIX deprecation warnings */
#define close _close
#define strdup _strdup
#endif
#ifdef WIN32
#include <direct.h>
#ifdef __MINGW32__
#define mkdir(path, perms) mkdir(path) /* Omit the perms argument to match POSIX signature */
#else /* MSVC and other windows compilers */
#define mkdir(path, perms) _mkdir(path) /* Omit the perms argument to match POSIX signature */
#endif /* __MINGW32__ else... */
#ifndef S_ISDIR
#define S_ISDIR(m) (((m) & S_IFDIR) == S_IFDIR)
#endif
#endif
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
11 years ago
#ifdef LIBVNCSERVER_HAVE_LIBJPEG
Add TurboVNC encoding support. TurboVNC is a variant of TightVNC that uses the same client/server protocol (RFB version 3.8t), and thus it is fully cross-compatible with TightVNC and TigerVNC (with one exception, which is noted below.) Both the TightVNC and TurboVNC encoders analyze each rectangle, pick out regions of solid color to send separately, and send the remaining subrectangles using mono, indexed color, JPEG, or raw encoding, depending on the number of colors in the subrectangle. However, TurboVNC uses a fundamentally different selection algorithm to determine the appropriate subencoding to use for each subrectangle. Thus, while it sends a protocol stream that can be decoded by any TightVNC-compatible viewer, the mix of subencoding types in this protocol stream will be different from those generated by a TightVNC server. The research that led to TurboVNC is described in the following report: http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf. In summary: 20 RFB captures, representing "common" 2D and 3D application workloads (the 3D workloads were run using VirtualGL), were studied using the TightVNC encoder in isolation. Some of the analysis features in the TightVNC encoder, such as smoothness detection, were found to generate a lot of CPU usage with little or no benefit in compression, so those features were disabled. JPEG encoding was accelerated using libjpeg-turbo (which achieves a 2-4x speedup over plain libjpeg on modern x86 or ARM processors.) Finally, the "palette threshold" (minimum number of colors that the subrectangle must have before it is compressed using JPEG or raw) was adjusted to account for the fact that JPEG encoding is now quite a bit faster (meaning that we can now use it more without a CPU penalty.) TurboVNC has additional optimizations, such as the ability to count colors and encode JPEG images directly from the framebuffer without first translating the pixels into RGB. The TurboVNC encoder compares quite favorably in terms of compression ratio with TightVNC and generally encodes a great deal faster (often an order of magnitude or more.) The version of the TurboVNC encoder included in this patch is roughly equivalent to the one found in version 0.6 of the Unix TurboVNC Server, with a few minor patches integrated from TurboVNC 1.1. TurboVNC 1.0 added multi-threading capabilities, which can be added in later if desired (at the expense of making libvncserver depend on libpthread.) Because TurboVNC uses a fundamentally different mix of subencodings than TightVNC, because it uses the identical protocol (and thus a viewer really has no idea whether it's talking to a TightVNC or TurboVNC server), and because it doesn't support rfbTightPng (and in fact conflicts with it-- see below), the TurboVNC and TightVNC encoders cannot be enabled simultaneously. Compatibility: In *most* cases, a TurboVNC-enabled viewer is fully compatible with a TightVNC server, and vice versa. TurboVNC supports pseudo-encodings for specifying a fine-grained (1-100) quality scale and specifying chrominance subsampling. If a TurboVNC viewer sends those to a TightVNC server, then the TightVNC server ignores them, so the TurboVNC viewer also sends the quality on a 0-9 scale that the TightVNC server can understand. Similarly, the TurboVNC server checks first for fine-grained quality and subsampling pseudo-encodings from the viewer, and failing to receive those, it then checks for the TightVNC 0-9 quality pseudo-encoding. There is one case in which the two systems are not compatible, and that is when a TightVNC or TigerVNC viewer requests compression level 0 without JPEG from a TurboVNC server. For performance reasons, this causes the TurboVNC server to send images directly to the viewer, bypassing Zlib. When the TurboVNC server does this, it also sets bits 7-4 in the compression control byte to rfbTightNoZlib (0x0A), which is unfortunately the same value as rfbTightPng. Older TightVNC viewers that don't handle PNG will assume that the stream is uncompressed but still encapsulated in a Zlib structure, whereas newer PNG-supporting TightVNC viewers will assume that the stream is PNG. In either case, the viewer will probably crash. Since most VNC viewers don't expose compression level 0 in the GUI, this is a relatively rare situation. Description of changes: configure.ac -- Added support for libjpeg-turbo. If passed an argument of --with-turbovnc, configure will now run (or, if cross-compiling, just link) a test program that determines whether the libjpeg library being used is libjpeg-turbo. libjpeg-turbo must be used when building the TurboVNC encoder, because the TurboVNC encoder relies on the libjpeg-turbo colorspace extensions in order to compress images directly out of the framebuffer (which may be, for instance, BGRA rather than RGB.) libjpeg-turbo can optionally be used with the TightVNC encoder as well, but the speedup will only be marginal (the report linked above explains why in more detail, but basically it's because of Amdahl's Law. The TightVNC encoder was designed with the assumption that JPEG had a very high CPU cost, and thus JPEG is used only sparingly.) -- Added a new configure variable, JPEG_LDFLAGS. This is necessitated by the fact that libjpeg-turbo often distributes libjpeg.a and libjpeg.so in /opt/libjpeg-turbo/lib32 or /opt/libjpeg-turbo/lib64, and many people prefer to statically link with it. Thus, more flexibility is needed than is provided by --with-jpeg. If JPEG_LDFLAGS is specified, then it overrides the changes to LDFLAGS enacted by --with-jpeg (but --with-jpeg is still used to set the include path.) The addition of JPEG_LDFLAGS necessitated replacing AC_CHECK_LIB with AC_LINK_IFELSE (because AC_CHECK_LIB automatically sets LIBS to -ljpeg, which is not what we want if we're, for instance, linking statically with libjpeg-turbo.) -- configure does not check for PNG support if TurboVNC encoding is enabled. This prevents the rfbSendRectEncodingTightPng() function from being compiled in, since the TurboVNC encoder doesn't (and can't) support it. common/turbojpeg.c, common/turbojpeg.h -- TurboJPEG is a simple API used to compress and decompress JPEG images in memory. It was originally implemented because it was desirable to use different types of underlying technologies to compress JPEG on different platforms (mediaLib on SPARC, Quicktime on PPC Macs, Intel Performance Primitives, etc.) These days, however, libjpeg-turbo is the only underlying technology used by TurboVNC, so TurboJPEG's purpose is largely just code simplicity and flexibility. Thus, since there is no real need for libvncserver to use any technology other than libjpeg-turbo for compressing JPEG, the TurboJPEG wrapper for libjpeg-turbo has been included in-tree so that libvncserver can be directly linked with libjpeg-turbo. This is convenient because many modern Linux distros (Fedora, Ubuntu, etc.) now ship libjpeg-turbo as their default libjpeg library. libvncserver/rfbserver.c -- Added logic to check for the TurboVNC fine-grained quality level and subsampling encodings and to map Tight (0-9) quality levels to appropriate fine-grained quality level and subsampling values if communicating with a TightVNC/TigerVNC viewer. libvncserver/turbo.c -- TurboVNC encoder (compiled instead of libvncserver/tight.c) rfb/rfb.h -- Added support for the TurboVNC subsampling level rfb/rfbproto.h -- Added constants for the TurboVNC fine quality level and subsampling encodings as well as the rfbTightNoZlib constant and notes on its usage.
11 years ago
/*
* Map of quality levels to provide compatibility with TightVNC/TigerVNC
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
11 years ago
* clients. This emulates the behavior of the TigerVNC Server.
Add TurboVNC encoding support. TurboVNC is a variant of TightVNC that uses the same client/server protocol (RFB version 3.8t), and thus it is fully cross-compatible with TightVNC and TigerVNC (with one exception, which is noted below.) Both the TightVNC and TurboVNC encoders analyze each rectangle, pick out regions of solid color to send separately, and send the remaining subrectangles using mono, indexed color, JPEG, or raw encoding, depending on the number of colors in the subrectangle. However, TurboVNC uses a fundamentally different selection algorithm to determine the appropriate subencoding to use for each subrectangle. Thus, while it sends a protocol stream that can be decoded by any TightVNC-compatible viewer, the mix of subencoding types in this protocol stream will be different from those generated by a TightVNC server. The research that led to TurboVNC is described in the following report: http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf. In summary: 20 RFB captures, representing "common" 2D and 3D application workloads (the 3D workloads were run using VirtualGL), were studied using the TightVNC encoder in isolation. Some of the analysis features in the TightVNC encoder, such as smoothness detection, were found to generate a lot of CPU usage with little or no benefit in compression, so those features were disabled. JPEG encoding was accelerated using libjpeg-turbo (which achieves a 2-4x speedup over plain libjpeg on modern x86 or ARM processors.) Finally, the "palette threshold" (minimum number of colors that the subrectangle must have before it is compressed using JPEG or raw) was adjusted to account for the fact that JPEG encoding is now quite a bit faster (meaning that we can now use it more without a CPU penalty.) TurboVNC has additional optimizations, such as the ability to count colors and encode JPEG images directly from the framebuffer without first translating the pixels into RGB. The TurboVNC encoder compares quite favorably in terms of compression ratio with TightVNC and generally encodes a great deal faster (often an order of magnitude or more.) The version of the TurboVNC encoder included in this patch is roughly equivalent to the one found in version 0.6 of the Unix TurboVNC Server, with a few minor patches integrated from TurboVNC 1.1. TurboVNC 1.0 added multi-threading capabilities, which can be added in later if desired (at the expense of making libvncserver depend on libpthread.) Because TurboVNC uses a fundamentally different mix of subencodings than TightVNC, because it uses the identical protocol (and thus a viewer really has no idea whether it's talking to a TightVNC or TurboVNC server), and because it doesn't support rfbTightPng (and in fact conflicts with it-- see below), the TurboVNC and TightVNC encoders cannot be enabled simultaneously. Compatibility: In *most* cases, a TurboVNC-enabled viewer is fully compatible with a TightVNC server, and vice versa. TurboVNC supports pseudo-encodings for specifying a fine-grained (1-100) quality scale and specifying chrominance subsampling. If a TurboVNC viewer sends those to a TightVNC server, then the TightVNC server ignores them, so the TurboVNC viewer also sends the quality on a 0-9 scale that the TightVNC server can understand. Similarly, the TurboVNC server checks first for fine-grained quality and subsampling pseudo-encodings from the viewer, and failing to receive those, it then checks for the TightVNC 0-9 quality pseudo-encoding. There is one case in which the two systems are not compatible, and that is when a TightVNC or TigerVNC viewer requests compression level 0 without JPEG from a TurboVNC server. For performance reasons, this causes the TurboVNC server to send images directly to the viewer, bypassing Zlib. When the TurboVNC server does this, it also sets bits 7-4 in the compression control byte to rfbTightNoZlib (0x0A), which is unfortunately the same value as rfbTightPng. Older TightVNC viewers that don't handle PNG will assume that the stream is uncompressed but still encapsulated in a Zlib structure, whereas newer PNG-supporting TightVNC viewers will assume that the stream is PNG. In either case, the viewer will probably crash. Since most VNC viewers don't expose compression level 0 in the GUI, this is a relatively rare situation. Description of changes: configure.ac -- Added support for libjpeg-turbo. If passed an argument of --with-turbovnc, configure will now run (or, if cross-compiling, just link) a test program that determines whether the libjpeg library being used is libjpeg-turbo. libjpeg-turbo must be used when building the TurboVNC encoder, because the TurboVNC encoder relies on the libjpeg-turbo colorspace extensions in order to compress images directly out of the framebuffer (which may be, for instance, BGRA rather than RGB.) libjpeg-turbo can optionally be used with the TightVNC encoder as well, but the speedup will only be marginal (the report linked above explains why in more detail, but basically it's because of Amdahl's Law. The TightVNC encoder was designed with the assumption that JPEG had a very high CPU cost, and thus JPEG is used only sparingly.) -- Added a new configure variable, JPEG_LDFLAGS. This is necessitated by the fact that libjpeg-turbo often distributes libjpeg.a and libjpeg.so in /opt/libjpeg-turbo/lib32 or /opt/libjpeg-turbo/lib64, and many people prefer to statically link with it. Thus, more flexibility is needed than is provided by --with-jpeg. If JPEG_LDFLAGS is specified, then it overrides the changes to LDFLAGS enacted by --with-jpeg (but --with-jpeg is still used to set the include path.) The addition of JPEG_LDFLAGS necessitated replacing AC_CHECK_LIB with AC_LINK_IFELSE (because AC_CHECK_LIB automatically sets LIBS to -ljpeg, which is not what we want if we're, for instance, linking statically with libjpeg-turbo.) -- configure does not check for PNG support if TurboVNC encoding is enabled. This prevents the rfbSendRectEncodingTightPng() function from being compiled in, since the TurboVNC encoder doesn't (and can't) support it. common/turbojpeg.c, common/turbojpeg.h -- TurboJPEG is a simple API used to compress and decompress JPEG images in memory. It was originally implemented because it was desirable to use different types of underlying technologies to compress JPEG on different platforms (mediaLib on SPARC, Quicktime on PPC Macs, Intel Performance Primitives, etc.) These days, however, libjpeg-turbo is the only underlying technology used by TurboVNC, so TurboJPEG's purpose is largely just code simplicity and flexibility. Thus, since there is no real need for libvncserver to use any technology other than libjpeg-turbo for compressing JPEG, the TurboJPEG wrapper for libjpeg-turbo has been included in-tree so that libvncserver can be directly linked with libjpeg-turbo. This is convenient because many modern Linux distros (Fedora, Ubuntu, etc.) now ship libjpeg-turbo as their default libjpeg library. libvncserver/rfbserver.c -- Added logic to check for the TurboVNC fine-grained quality level and subsampling encodings and to map Tight (0-9) quality levels to appropriate fine-grained quality level and subsampling values if communicating with a TightVNC/TigerVNC viewer. libvncserver/turbo.c -- TurboVNC encoder (compiled instead of libvncserver/tight.c) rfb/rfb.h -- Added support for the TurboVNC subsampling level rfb/rfbproto.h -- Added constants for the TurboVNC fine quality level and subsampling encodings as well as the rfbTightNoZlib constant and notes on its usage.
11 years ago
*/
static const int tight2turbo_qual[10] = {
15, 29, 41, 42, 62, 77, 79, 86, 92, 100
};
static const int tight2turbo_subsamp[10] = {
1, 1, 1, 2, 2, 2, 0, 0, 0, 0
};
#endif
22 years ago
static void rfbProcessClientProtocolVersion(rfbClientPtr cl);
static void rfbProcessClientNormalMessage(rfbClientPtr cl);
static void rfbProcessClientInitMessage(rfbClientPtr cl);
#ifdef LIBVNCSERVER_HAVE_LIBPTHREAD
void rfbIncrClientRef(rfbClientPtr cl)
{
LOCK(cl->refCountMutex);
cl->refCount++;
UNLOCK(cl->refCountMutex);
}
void rfbDecrClientRef(rfbClientPtr cl)
{
LOCK(cl->refCountMutex);
cl->refCount--;
if(cl->refCount<=0) /* just to be sure also < 0 */
TSIGNAL(cl->deleteCond);
UNLOCK(cl->refCountMutex);
}
21 years ago
#else
void rfbIncrClientRef(rfbClientPtr cl) {}
void rfbDecrClientRef(rfbClientPtr cl) {}
#endif
#ifdef LIBVNCSERVER_HAVE_LIBPTHREAD
static MUTEX(rfbClientListMutex);
#endif
22 years ago
struct rfbClientIterator {
rfbClientPtr next;
rfbScreenInfoPtr screen;
rfbBool closedToo;
22 years ago
};
void
rfbClientListInit(rfbScreenInfoPtr rfbScreen)
{
if(sizeof(rfbBool)!=1) {
/* a sanity check */
fprintf(stderr,"rfbBool's size is not 1 (%d)!\n",(int)sizeof(rfbBool));
/* we cannot continue, because rfbBool is supposed to be char everywhere */
exit(1);
}
rfbScreen->clientHead = NULL;
INIT_MUTEX(rfbClientListMutex);
22 years ago
}
rfbClientIteratorPtr
rfbGetClientIterator(rfbScreenInfoPtr rfbScreen)
{
rfbClientIteratorPtr i =
(rfbClientIteratorPtr)malloc(sizeof(struct rfbClientIterator));
i->next = NULL;
i->screen = rfbScreen;
i->closedToo = FALSE;
return i;
}
rfbClientIteratorPtr
rfbGetClientIteratorWithClosed(rfbScreenInfoPtr rfbScreen)
{
rfbClientIteratorPtr i =
(rfbClientIteratorPtr)malloc(sizeof(struct rfbClientIterator));
i->next = NULL;
i->screen = rfbScreen;
i->closedToo = TRUE;
return i;
22 years ago
}
rfbClientPtr
rfbClientIteratorHead(rfbClientIteratorPtr i)
{
#ifdef LIBVNCSERVER_HAVE_LIBPTHREAD
if(i->next != 0) {
rfbDecrClientRef(i->next);
rfbIncrClientRef(i->screen->clientHead);
}
#endif
LOCK(rfbClientListMutex);
i->next = i->screen->clientHead;
UNLOCK(rfbClientListMutex);
return i->next;
}
22 years ago
rfbClientPtr
rfbClientIteratorNext(rfbClientIteratorPtr i)
22 years ago
{
if(i->next == 0) {
LOCK(rfbClientListMutex);
i->next = i->screen->clientHead;
UNLOCK(rfbClientListMutex);
} else {
IF_PTHREADS(rfbClientPtr cl = i->next);
i->next = i->next->next;
IF_PTHREADS(rfbDecrClientRef(cl));
}
#ifdef LIBVNCSERVER_HAVE_LIBPTHREAD
if(!i->closedToo)
while(i->next && i->next->sock<0)
i->next = i->next->next;
if(i->next)
rfbIncrClientRef(i->next);
#endif
return i->next;
22 years ago
}
void
rfbReleaseClientIterator(rfbClientIteratorPtr iterator)
{
IF_PTHREADS(if(iterator->next) rfbDecrClientRef(iterator->next));
21 years ago
free(iterator);
22 years ago
}
/*
* rfbNewClientConnection is called from sockets.c when a new connection
* comes in.
*/
void
rfbNewClientConnection(rfbScreenInfoPtr rfbScreen,
int sock)
22 years ago
{
rfbNewClient(rfbScreen,sock);
22 years ago
}
/*
* rfbReverseConnection is called to make an outward
22 years ago
* connection to a "listening" RFB client.
*/
rfbClientPtr
rfbReverseConnection(rfbScreenInfoPtr rfbScreen,
char *host,
int port)
22 years ago
{
int sock;
rfbClientPtr cl;
if ((sock = rfbConnect(rfbScreen, host, port)) < 0)
22 years ago
return (rfbClientPtr)NULL;
cl = rfbNewClient(rfbScreen, sock);
22 years ago
if (cl) {
cl->reverseConnection = TRUE;
}
return cl;
}
void
rfbSetProtocolVersion(rfbScreenInfoPtr rfbScreen, int major_, int minor_)
{
/* Permit the server to set the version to report */
/* TODO: sanity checking */
if ((major_==3) && (minor_ > 2 && minor_ < 9))
{
rfbScreen->protocolMajorVersion = major_;
rfbScreen->protocolMinorVersion = minor_;
}
else
rfbLog("rfbSetProtocolVersion(%d,%d) set to invalid values\n", major_, minor_);
}
22 years ago
/*
* rfbNewClient is called when a new connection has been made by whatever
* means.
*/
static rfbClientPtr
rfbNewTCPOrUDPClient(rfbScreenInfoPtr rfbScreen,
int sock,
rfbBool isUDP)
22 years ago
{
rfbProtocolVersionMsg pv;
rfbClientIteratorPtr iterator;
rfbClientPtr cl,cl_;
#ifdef LIBVNCSERVER_IPv6
struct sockaddr_storage addr;
#else
struct sockaddr_in addr;
#endif
socklen_t addrlen = sizeof(addr);
rfbProtocolExtension* extension;
22 years ago
22 years ago
cl = (rfbClientPtr)calloc(sizeof(rfbClientRec),1);
22 years ago
cl->screen = rfbScreen;
cl->sock = sock;
cl->viewOnly = FALSE;
/* setup pseudo scaling */
cl->scaledScreen = rfbScreen;
cl->scaledScreen->scaledScreenRefCount++;
rfbResetStats(cl);
22 years ago
cl->clientData = NULL;
cl->clientGoneHook = rfbDoNothingWithClient;
if(isUDP) {
rfbLog(" accepted UDP client\n");
} else {
#ifdef LIBVNCSERVER_IPv6
char host[1024];
#endif
int one=1;
getpeername(sock, (struct sockaddr *)&addr, &addrlen);
#ifdef LIBVNCSERVER_IPv6
if(getnameinfo((struct sockaddr*)&addr, addrlen, host, sizeof(host), NULL, 0, NI_NUMERICHOST) != 0) {
rfbLogPerror("rfbNewClient: error in getnameinfo");
cl->host = strdup("");
}
else
cl->host = strdup(host);
#else
cl->host = strdup(inet_ntoa(addr.sin_addr));
#endif
22 years ago
rfbLog(" other clients:\n");
iterator = rfbGetClientIterator(rfbScreen);
while ((cl_ = rfbClientIteratorNext(iterator)) != NULL) {
rfbLog(" %s\n",cl_->host);
}
rfbReleaseClientIterator(iterator);
22 years ago
if(!rfbSetNonBlocking(sock)) {
close(sock);
return NULL;
}
if (setsockopt(sock, IPPROTO_TCP, TCP_NODELAY,
(char *)&one, sizeof(one)) < 0) {
rfbLogPerror("setsockopt failed: can't set TCP_NODELAY flag, non TCP socket?");
}
FD_SET(sock,&(rfbScreen->allFds));
rfbScreen->maxFd = rfbMax(sock,rfbScreen->maxFd);
INIT_MUTEX(cl->outputMutex);
INIT_MUTEX(cl->refCountMutex);
INIT_MUTEX(cl->sendMutex);
INIT_COND(cl->deleteCond);
22 years ago
cl->state = RFB_PROTOCOL_VERSION;
cl->reverseConnection = FALSE;
cl->readyForSetColourMapEntries = FALSE;
cl->useCopyRect = FALSE;
cl->preferredEncoding = -1;
cl->correMaxWidth = 48;
cl->correMaxHeight = 48;
#ifdef LIBVNCSERVER_HAVE_LIBZ
cl->zrleData = NULL;
#endif
cl->copyRegion = sraRgnCreate();
cl->copyDX = 0;
cl->copyDY = 0;
22 years ago
cl->modifiedRegion =
sraRgnCreateRect(0,0,rfbScreen->width,rfbScreen->height);
22 years ago
INIT_MUTEX(cl->updateMutex);
INIT_COND(cl->updateCond);
22 years ago
cl->requestedRegion = sraRgnCreate();
22 years ago
cl->format = cl->screen->serverFormat;
cl->translateFn = rfbTranslateNone;
cl->translateLookupTable = NULL;
22 years ago
LOCK(rfbClientListMutex);
IF_PTHREADS(cl->refCount = 0);
cl->next = rfbScreen->clientHead;
cl->prev = NULL;
if (rfbScreen->clientHead)
rfbScreen->clientHead->prev = cl;
22 years ago
rfbScreen->clientHead = cl;
UNLOCK(rfbClientListMutex);
22 years ago
#if defined(LIBVNCSERVER_HAVE_LIBZ) || defined(LIBVNCSERVER_HAVE_LIBPNG)
cl->tightQualityLevel = -1;
#ifdef LIBVNCSERVER_HAVE_LIBJPEG
cl->tightCompressLevel = TIGHT_DEFAULT_COMPRESSION;
Replace TightVNC encoder with TurboVNC encoder. This patch is the result of further research and discussion that revealed the following: -- TightPng encoding and the rfbTightNoZlib extension need not conflict. Since TightPng is a separate encoding type, not supported by TurboVNC-compatible viewers, then the rfbTightNoZlib extension can be used solely whenever the encoding type is Tight and disabled with the encoding type is TightPng. -- In the TightVNC encoder, compression levels above 5 are basically useless. On the set of 20 low-level datasets that were used to design the TurboVNC encoder (these include the eight 2D application captures that were also used when designing the TightVNC encoder, as well as 12 3D application captures provided by the VirtualGL Project-- see http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf), moving from Compression Level (CL) 5 to CL 9 in the TightVNC encoder did not increase the compression ratio of any datasets more than 10%, and the compression ratio only increased by more than 5% on four of them. The compression ratio actually decreased a few percent on five of them. In exchange for this paltry increase in compression ratio, the CPU usage, on average, went up by a factor of 5. Thus, for all intents and purposes, TightVNC CL 5 provides the "best useful compression" for that encoder. -- TurboVNC's best compression level (CL 2) compresses 3D and video workloads significantly more "tightly" than TightVNC CL 5 (~70% better, in the aggregate) but does not quite achieve the same level of compression with 2D workloads (~20% worse, in the aggregate.) This decrease in compression ratio may or may not be noticeable, since many of the datasets it affects are not performance-critical (such as the console output of a compilation, etc.) However, for peace of mind, it was still desirable to have a mode that compressed with equal "tightness" to TightVNC CL 5, since we proposed to replace that encoder entirely. -- A new mode was discovered in the TurboVNC encoder that produces, in the aggregate, similar compression ratios on 2D datasets as TightVNC CL 5. That new mode involves using Zlib level 7 (the same level used by TightVNC CL 5) but setting the "palette threshold" to 256, so that indexed color encoding is used whenever possible. This mode reduces bandwidth only marginally (typically 10-20%) relative to TurboVNC CL 2 on low-color workloads, in exchange for nearly doubling CPU usage, and it does not benefit high-color workloads at all (since those are usually encoded with JPEG.) However, it provides a means of reproducing the same "tightness" as the TightVNC encoder on 2D workloads without sacrificing any compression for 3D/video workloads, and without using any more CPU time than necessary. -- The TurboVNC encoder still performs as well or better than the TightVNC encoder when plain libjpeg is used instead of libjpeg-turbo. Specific notes follow: common/turbojpeg.c common/turbojpeg.h: Added code to emulate the libjpeg-turbo colorspace extensions, so that the TurboJPEG wrapper can be used with plain libjpeg as well. This required updating the TurboJPEG wrapper to the latest code from libjpeg-turbo 1.2.0, mainly because the TurboJPEG 1.2 API handles pixel formats in a much cleaner way, which made the conversion code easier to write. It also eases the maintenance to have the wrapper synced as much as possible with the upstream code base (so I can merge any relevant bug fixes that are discovered upstream.) The libvncserver version of the TurboJPEG wrapper is a "lite" version, containing only the JPEG compression/decompression code and not the lossless transform, YUV encoding/decoding, and dynamic buffer allocation features from TurboJPEG 1.2. configure.ac: Removed the --with-turbovnc option. configure still checks for the presence of libjpeg-turbo, but only for the purposes of printing a performance warning if it isn't available. rfb/rfb.h: Fix a bug introduced with the initial TurboVNC encoder patch. We cannot use tightQualityLevel for the TurboVNC 1-100 quality level, because tightQualityLevel is also used by ZRLE. Thus, a new parameter (turboQualityLevel) was created. rfb/rfbproto.h: Remove TurboVNC-specific #ifdefs and language libvncserver/rfbserver.c: Remove TurboVNC-specific #ifdefs. Fix afore-mentioned tightQualityLevel bug. libvncserver/tight.c: Replaced the TightVNC encoder with the TurboVNC encoder. Relative to the initial TurboVNC encoder patch, this patch also: -- Adds TightPng support to the TurboVNC encoder -- Adds the afore-mentioned low-bandwidth mode, which is mapped externally to Compression Level 9 test/*: Included TJUnitTest (a regression test for the TurboJPEG wrapper) as well as TJBench (a benchmark for same.) These are useful for ensuring that the wrapper still functions correctly and performantly if it needs to be modified for whatever reason. Both of these programs are derived from libjpeg-turbo 1.2.0. As with the TurboJPEG wrapper, they do not contain the more advanced features of TurboJPEG 1.2, such as YUV encoding/decoding and lossless transforms.
11 years ago
cl->turboSubsampLevel = TURBO_DEFAULT_SUBSAMP;
{
int i;
for (i = 0; i < 4; i++)
cl->zsActive[i] = FALSE;
}
#endif
#endif
22 years ago
cl->fileTransfer.fd = -1;
cl->enableCursorShapeUpdates = FALSE;
cl->enableCursorPosUpdates = FALSE;
cl->useRichCursorEncoding = FALSE;
cl->enableLastRectEncoding = FALSE;
cl->enableKeyboardLedState = FALSE;
cl->enableSupportedMessages = FALSE;
cl->enableSupportedEncodings = FALSE;
cl->enableServerIdentity = FALSE;
cl->lastKeyboardLedState = -1;
cl->cursorX = rfbScreen->cursorX;
cl->cursorY = rfbScreen->cursorY;
cl->useNewFBSize = FALSE;
22 years ago
#ifdef LIBVNCSERVER_HAVE_LIBZ
cl->compStreamInited = FALSE;
cl->compStream.total_in = 0;
cl->compStream.total_out = 0;
cl->compStream.zalloc = Z_NULL;
cl->compStream.zfree = Z_NULL;
cl->compStream.opaque = Z_NULL;
22 years ago
cl->zlibCompressLevel = 5;
#endif
22 years ago
cl->progressiveSliceY = 0;
cl->extensions = NULL;
cl->lastPtrX = -1;
#ifdef LIBVNCSERVER_WITH_WEBSOCKETS
/*
* Wait a few ms for the client to send one of:
* - Flash policy request
* - WebSockets connection (TLS/SSL or plain)
*/
if (!webSocketsCheck(cl)) {
/* Error reporting handled in webSocketsHandshake */
rfbCloseClient(cl);
rfbClientConnectionGone(cl);
return NULL;
}
#endif
sprintf(pv,rfbProtocolVersionFormat,rfbScreen->protocolMajorVersion,
rfbScreen->protocolMinorVersion);
22 years ago
if (rfbWriteExact(cl, pv, sz_rfbProtocolVersionMsg) < 0) {
22 years ago
rfbLogPerror("rfbNewClient: write");
rfbCloseClient(cl);
rfbClientConnectionGone(cl);
22 years ago
return NULL;
}
22 years ago
}
for(extension = rfbGetExtensionIterator(); extension;
extension=extension->next) {
void* data = NULL;
/* if the extension does not have a newClient method, it wants
* to be initialized later. */
if(extension->newClient && extension->newClient(cl, &data))
rfbEnableExtension(cl, extension, data);
}
rfbReleaseExtensionIterator();
switch (cl->screen->newClientHook(cl)) {
case RFB_CLIENT_ON_HOLD:
cl->onHold = TRUE;
break;
case RFB_CLIENT_ACCEPT:
cl->onHold = FALSE;
break;
case RFB_CLIENT_REFUSE:
rfbCloseClient(cl);
rfbClientConnectionGone(cl);
cl = NULL;
break;
}
22 years ago
return cl;
}
rfbClientPtr
rfbNewClient(rfbScreenInfoPtr rfbScreen,
int sock)
{
return(rfbNewTCPOrUDPClient(rfbScreen,sock,FALSE));
}
rfbClientPtr
rfbNewUDPClient(rfbScreenInfoPtr rfbScreen)
{
return((rfbScreen->udpClient=
rfbNewTCPOrUDPClient(rfbScreen,rfbScreen->udpSock,TRUE)));
}
22 years ago
/*
* rfbClientConnectionGone is called from sockets.c just after a connection
* has gone away.
*/
void
rfbClientConnectionGone(rfbClientPtr cl)
22 years ago
{
#if defined(LIBVNCSERVER_HAVE_LIBZ) && defined(LIBVNCSERVER_HAVE_LIBJPEG)
22 years ago
int i;
#endif
LOCK(rfbClientListMutex);
if (cl->prev)
cl->prev->next = cl->next;
else
cl->screen->clientHead = cl->next;
if (cl->next)
cl->next->prev = cl->prev;
Fix deadlock in threaded mode when using nested rfbClientIteratorNext() calls. Lengthy explanation follows... First, the scenario before this patch: We have three clients 1,2,3 connected. The main thread loops through them using rfbClientIteratorNext() (loop L1) and is currently at client 2 i.e. client 2's cl_2->refCount is 1. At this point we need to loop again through the clients, with cl_2->refCount == 1, i.e. do a loop L2 nested within loop L1. BUT: Now client 2 disconnects, it's clientInput thread terminates its clientOutput thread and calls rfbClientConnectionGone(). This LOCKs clientListMutex and WAITs for cl_2->refCount to become 0. This means this thread waits for the main thread to release cl_2. Waiting, with clientListMutex LOCKed! Meanwhile, the main thread is about to begin the inner rfbClientIteratorNext() loop L2. The first call to rfbClientIteratorNext() LOCKs clientListMutex. BAAM. This mutex is locked by cl2's clientInput thread and is only released when cl_2->refCount becomes 0. The main thread would decrement cl_2->refCount when it would continue with loop L1. But it's waiting for cl2's clientInput thread to release clientListMutex. Which never happens since this one's waiting for the main thread to decrement cl_2->refCount. DEADLOCK. Now, situation with this patch: Same as above, but when client 2 disconnects it's clientInput thread rfbClientConnectionGone(). This again LOCKs clientListMutex, removes cl_2 from the linked list and UNLOCKS clientListMutex. The WAIT for cl_2->refCount to become 0 is _after_ that. Waiting, with clientListMutex UNLOCKed! Therefore, the main thread can continue, do the inner loop L2 (now only looping through 1,3 - 2 was removed from the linked list) and continue with loop L1, finally decrementing cl_2->refCount, allowing cl2's clientInput thread to continue and terminate. The resources held by cl2 are not free()'d by rfbClientConnectionGone until cl2->refCount becomes 0, i.e. loop L1 has released cl2.
11 years ago
UNLOCK(rfbClientListMutex);
#ifdef LIBVNCSERVER_HAVE_LIBPTHREAD
if(cl->screen->backgroundLoop != FALSE) {
int i;
do {
LOCK(cl->refCountMutex);
i=cl->refCount;
if(i>0)
WAIT(cl->deleteCond,cl->refCountMutex);
UNLOCK(cl->refCountMutex);
} while(i>0);
}
#endif
if(cl->sock>=0)
close(cl->sock);
if (cl->scaledScreen!=NULL)
cl->scaledScreen->scaledScreenRefCount--;
#ifdef LIBVNCSERVER_HAVE_LIBZ
rfbFreeZrleData(cl);
#endif
rfbFreeUltraData(cl);
/* free buffers holding pixel data before and after encoding */
free(cl->beforeEncBuf);
free(cl->afterEncBuf);
if(cl->sock>=0)
FD_CLR(cl->sock,&(cl->screen->allFds));
22 years ago
cl->clientGoneHook(cl);
22 years ago
rfbLog("Client %s gone\n",cl->host);
free(cl->host);
#ifdef LIBVNCSERVER_HAVE_LIBZ
22 years ago
/* Release the compression state structures if any. */
if ( cl->compStreamInited ) {
22 years ago
deflateEnd( &(cl->compStream) );
}
#ifdef LIBVNCSERVER_HAVE_LIBJPEG
22 years ago
for (i = 0; i < 4; i++) {
if (cl->zsActive[i])
deflateEnd(&cl->zsStruct[i]);
}
#endif
#endif
22 years ago
if (cl->screen->pointerClient == cl)
cl->screen->pointerClient = NULL;
22 years ago
sraRgnDestroy(cl->modifiedRegion);
21 years ago
sraRgnDestroy(cl->requestedRegion);
sraRgnDestroy(cl->copyRegion);
22 years ago
if (cl->translateLookupTable) free(cl->translateLookupTable);
TINI_COND(cl->updateCond);
TINI_MUTEX(cl->updateMutex);
/* make sure outputMutex is unlocked before destroying */
LOCK(cl->outputMutex);
UNLOCK(cl->outputMutex);
TINI_MUTEX(cl->outputMutex);
LOCK(cl->sendMutex);
UNLOCK(cl->sendMutex);
TINI_MUTEX(cl->sendMutex);
rfbPrintStats(cl);
rfbResetStats(cl);
22 years ago
free(cl);
22 years ago
}
/*
* rfbProcessClientMessage is called when there is data to read from a client.
*/
void
rfbProcessClientMessage(rfbClientPtr cl)
22 years ago
{
switch (cl->state) {
case RFB_PROTOCOL_VERSION:
rfbProcessClientProtocolVersion(cl);
return;
case RFB_SECURITY_TYPE:
rfbProcessClientSecurityType(cl);
return;
22 years ago
case RFB_AUTHENTICATION:
rfbAuthProcessClientMessage(cl);
return;
case RFB_INITIALISATION:
case RFB_INITIALISATION_SHARED:
22 years ago
rfbProcessClientInitMessage(cl);
return;
default:
rfbProcessClientNormalMessage(cl);
return;
}
}
/*
* rfbProcessClientProtocolVersion is called when the client sends its
* protocol version.
*/
static void
rfbProcessClientProtocolVersion(rfbClientPtr cl)
22 years ago
{
rfbProtocolVersionMsg pv;
22 years ago
int n, major_, minor_;
22 years ago
if ((n = rfbReadExact(cl, pv, sz_rfbProtocolVersionMsg)) <= 0) {
22 years ago
if (n == 0)
rfbLog("rfbProcessClientProtocolVersion: client gone\n");
else
rfbLogPerror("rfbProcessClientProtocolVersion: read");
rfbCloseClient(cl);
return;
}
pv[sz_rfbProtocolVersionMsg] = 0;
22 years ago
if (sscanf(pv,rfbProtocolVersionFormat,&major_,&minor_) != 2) {
rfbErr("rfbProcessClientProtocolVersion: not a valid RFB client: %s\n", pv);
rfbCloseClient(cl);
return;
22 years ago
}
rfbLog("Client Protocol Version %d.%d\n", major_, minor_);
22 years ago
22 years ago
if (major_ != rfbProtocolMajorVersion) {
rfbErr("RFB protocol version mismatch - server %d.%d, client %d.%d",
cl->screen->protocolMajorVersion, cl->screen->protocolMinorVersion,
major_,minor_);
rfbCloseClient(cl);
22 years ago
return;
}
/* Check for the minor version use either of the two standard version of RFB */
/*
* UltraVNC Viewer detects FileTransfer compatible servers via rfb versions
* 3.4, 3.6, 3.14, 3.16
* It's a bad method, but it is what they use to enable features...
* maintaining RFB version compatibility across multiple servers is a pain
* Should use something like ServerIdentity encoding
*/
cl->protocolMajorVersion = major_;
cl->protocolMinorVersion = minor_;
rfbLog("Protocol version sent %d.%d, using %d.%d\n",
major_, minor_, rfbProtocolMajorVersion, cl->protocolMinorVersion);
22 years ago
rfbAuthNewClient(cl);
}
void
rfbClientSendString(rfbClientPtr cl, const char *reason)
{
char *buf;
int len = strlen(reason);
rfbLog("rfbClientSendString(\"%s\")\n", reason);
buf = (char *)malloc(4 + len);
((uint32_t *)buf)[0] = Swap32IfLE(len);
memcpy(buf + 4, reason, len);
if (rfbWriteExact(cl, buf, 4 + len) < 0)
rfbLogPerror("rfbClientSendString: write");
free(buf);
rfbCloseClient(cl);
}
22 years ago
/*
* rfbClientConnFailed is called when a client connection has failed either
* because it talks the wrong protocol or it has failed authentication.
*/
void
rfbClientConnFailed(rfbClientPtr cl,
const char *reason)
22 years ago
{
char *buf;
int len = strlen(reason);
rfbLog("rfbClientConnFailed(\"%s\")\n", reason);
buf = (char *)malloc(8 + len);
((uint32_t *)buf)[0] = Swap32IfLE(rfbConnFailed);
((uint32_t *)buf)[1] = Swap32IfLE(len);
22 years ago
memcpy(buf + 8, reason, len);
if (rfbWriteExact(cl, buf, 8 + len) < 0)
22 years ago
rfbLogPerror("rfbClientConnFailed: write");
free(buf);
22 years ago
rfbCloseClient(cl);
}
/*
* rfbProcessClientInitMessage is called when the client sends its
* initialisation message.
*/
static void
rfbProcessClientInitMessage(rfbClientPtr cl)
22 years ago
{
rfbClientInitMsg ci;
union {
char buf[256];
rfbServerInitMsg si;
} u;
22 years ago
int len, n;