OK, so that IS the problem. I deleted all the drivers AND uninstalled my ESET anti virus.
I then re-installed the Windows 8.1 x64 xerox scan drivers. Didn't work!! In the device manager, it's listed as an "unknown device". I says something is wrong with the installation of the driver and it cannot continue. No amount of trying or pointing to the drivers worked!!
Then I got the idea of using the Xerox Windows 7 x64 scan driver. THAT WORKED!!! I can scan via TWAIN and WIA. Me happy!!!!
But then I re-installed ESET Smart security, and from that moment on, it quit working again. In the device manager, there is that ESET driver again blocking the access to the scanner driver.
So now I know what the problem is, and how I can solve it. I guess I'll contact ESET now and ask them what to do. If somebody here knows a way of contacting Xerox about this, this might be interesting info for them to know as well...