There seems to be a locking problem in libvncserver, with respect to how
condition variables are used.
On certain machines in our lab, when using a vncviewer to view a display
that has a very high rate of updates, we will occasionally see the VNC
server process crash. In one stack trace that was obtained, an assertion
had tripped in glibc's pthread_cond_wait, which was called from
clientOutput.
Inspection of clientOutput suggests that WAIT is being called incorrectly.
The mutex that protects a condition variable should always be locked when
calling wait, and on return from the wait will still be locked. The
attached patch fixes the locking around this condition variable, and one
other that I found by grepping the source for similar occurrences.
Signed-off-by: Charles Coffing <ccoffing@novell.com>
rfbEncodingSupportedEncodings - What encodings are supported?
rfbEncodingSupportedMessages - What message types are supported?
rfbEncodingServerIdentity - What is the servers version string?
ie: "x11vnc: 0.8.1 lastmod: 2006-04-25 (LibVNCServer 0.9pre)"
If the client asked for an encoding, and no enabled extension handled it,
LibVNCServer would walk through all extensions, and if they promised to handle
the encoding, execute the extension's newClient() if it was not NULL.
However, if newClient is not NULL, it will be called when a client connects,
and if it returns TRUE, the extension will be enabled. Since all the state of
the extension should be in the client data, there is no good reason why
newClient should return FALSE the first time (thus not enabling the extension),
but TRUE when called just before calling enablePseudoEncoding().
So in effect, the extension got enabled all the time, even if that was not
necessary.
The resolution is to pass a void** to enablePseudoEncoding. This has the
further advantage that enablePseudoEncoding can remalloc() or free() the
data without problems. Though keep in mind that if enablePseudoEncoding()
is called on a not-yet-enabled extension, the passed data points to NULL.
dscho
runge
steven_carr
rohit_99129