Limit lenght to INT_MAX bytes in rfbProcessFileTransferReadBuffer()

This ammends 15bb719c03 fix for a heap
out-of-bound write access in rfbProcessFileTransferReadBuffer() when
reading a transfered file content in a server. The former fix did not
work on platforms with a 32-bit int type (expected by rfbReadExact()).

CVE-2018-15127
<https://github.com/LibVNC/libvncserver/issues/243>
<https://github.com/LibVNC/libvncserver/issues/273>

(cherry picked from commit 09e8fc02f59f16e2583b34fe1a270c238bd9ffec)

libvncserver/rfbserver.c

@@ -83,6 +83,8 @@
 #include <errno.h>
 /* strftime() */
 #include <time.h>
+/* INT_MAX */
+#include <limits.h>
 
 #ifdef LIBVNCSERVER_WITH_WEBSOCKETS
 #include "rfbssl.h"
@@ -1469,8 +1471,11 @@ char *rfbProcessFileTransferReadBuffer(rfbClientPtr cl, uint32_t length)
        0XFFFFFFFF, i.e. SIZE_MAX for 32-bit systems. On 64-bit systems, a length of 0XFFFFFFFF
        will safely be allocated since this check will never trigger and malloc() can digest length+1
        without problems as length is a uint32_t.
+       We also later pass length to rfbReadExact() that expects a signed int type and
+       that might wrap on platforms with a 32-bit int type if length is bigger
+       than 0X7FFFFFFF.
     */
-    if(length == SIZE_MAX) {
+    if(length == SIZE_MAX || length > INT_MAX) {
 	rfbErr("rfbProcessFileTransferReadBuffer: too big file transfer length requested: %u", (unsigned int)length);
 	rfbCloseClient(cl);
 	return NULL;