[Tfug] VMWare Workstation 6 and Fedora Core 9 (beta)

[Brian Murphy]

I am running Fedora Core 9 beta on my UofA desktop.  In order to get  
VMware Workstation 6 to install I had to hack the linux kernel headers  
and a vmnet source file.  While I did it on FC9, this may be true for  
anyone running a 2.6.25 kernel.  I'm posting both for the TFUG list  
users and to document this in google (which I believe scans the list  
archives).  I also don't build a lot of 3rd party software so I'm not  
too worried about the kernel header change I'm making.

Note 1: Make sure you have the Development group (gcc) and  
kernel-devel package installed or else you'll have problems early in  
the vmware-config.pl process.

Note 2: I skipped the vmblock module because it too is buggy and it's  
not necessary for basic vmware operation.

The big problem is that the vmnet kernel module doesn't build.  Online  
suggestions of replacing it with the vmware-any-any update version  
doesn't work.

The first problem is that VMWare depends on the sock_valbool_flag()  
function instead of using sock_set_flag and sock_reset_flag.  A 2.6.25  
kernel patch moved the function from include/net/sock.h to  
net/core/sock.c

(See http://openvz.org/pipermail/devel/2007-November/009116.html)

Take this function and add it back to sock.h.

static inline void sock_valbool_flag(struct sock *sk, int bit, int valbool)
{
        if (valbool)
                sock_set_flag(sk, bit);
        else
                sock_reset_flag(sk, bit);
}


Next, for reasons I didn't dig into, vmware's vmnet-only/filter.c  
isn't picking up values from linux/netfilter_ipv4.h even though the  
file is properly #included.  The error looks something like this:

   CC [M]  /tmp/vmware-config8/vmnet-only/filter.o
/tmp/vmware-config8/vmnet-only/filter.c:48: error: ?NF_IP_LOCAL_IN?  
undeclared here (not in a function)
/tmp/vmware-config8/vmnet-only/filter.c:53: error:  
?NF_IP_POST_ROUTING? undeclared here (not in a function)
/tmp/vmware-config8/vmnet-only/filter.c: In function ?VNetFilterHookFn?:
/tmp/vmware-config8/vmnet-only/filter.c:233: warning: comparison  
between pointer and integer

I just changed the macros to their integer values as seen in  
/usr/src/linux/include/linux/netfilter_ipv4.h.  (NF_IP_LOCAL_IN=1,  
NF_IP_POST_ROUTING=4) Don't miss both instances of NF_IP_POST_ROUTING  
on lines 53 and 233 of filter.c.

Now since vmnet.tar isn't unpacked until the compilation starts, there  
are 2 ways of fixing it.  The first is unpacking  
/usr/lib/vmware/modules/source/vmnet.tar, fixing filter.c, and  
repacking it so the installer will grab your fixed vmnet.tar.  I went  
the quick and dirty route (since I was still working on this hack) and  
hit Control-Z when vmware-config.pl started the vmnet-only  
compilation, looked at the /tmp/vmware-config??/ directory, changed  
the file, and ran fg to resume the build process.  I might eventually  
change vmnet.tar since rawhide frequently puts out new kernels.

vmware-config.pl will finish and load the kernel modules for you.

VMware now starts and took the license key but it can't find some  
shared libraries when it powers on the virtual machine.   
vmware-config.pl warned about this.  Who actually thought it'd be  
right? :-)

Here is the problem:

/usr/lib/vmware/bin/vmware-vmx: error while loading shared libraries:  
libX11.so.6: cannot open shared object file: No such file or directory

Well, it's right there in /usr/lib64!!

# file /usr/lib/vmware/bin/vmware-vmx
/usr/lib/vmware/bin/vmware-vmx: setuid ELF 32-bit LSB executable,  
Intel 80386, version 1 (SYSV), dynamically linked (uses shared libs),  
for GNU/Linux 2.2.5, stripped

grr...32-bit

# ldd /usr/lib/vmware/bin/vmware-vmx
         linux-gate.so.1 =>  (0x00110000)
         libm.so.6 => /lib/libm.so.6 (0x00111000)
         libdl.so.2 => /lib/libdl.so.2 (0x00701000)
         libpthread.so.0 => /lib/libpthread.so.0 (0x006e0000)
         libc.so.6 => /lib/libc.so.6 (0x00549000)
         /lib/ld-linux.so.2 (0x00526000)
         libX11.so.6 => not found
         libXtst.so.6 => not found
         libXext.so.6 => not found
         libXrender.so.1 => not found
         libz.so.1 => not found

# yum install zlib.i386 libXrender.i386 libXext.i386 libXtst.i386 libX11.i386


Lo and behold!!  I have windows xp again!  Woohoo! (yes, sarcasm)

Happy hacking,

Brian Murphy