Ok so it is working now. I think what needed to happen was a restart after the installation of oracle client because crystal server was already running.
Upon install of oracle client, it didn't work right away, so adding the oracle home and THEN restarted after that didn't help either. BUT just adding ORACLE_HOME32 by itself restart server, works but not together with ORACLE_HOME environment variable.
So ended up just deleting those two variable, restart the server and it all works fine.
Now question is does it really need the oracle 64 bit drivers? We won't be using Crystal Reports Enterprise. I'm not sure if something else on the server needs the 64bit drivers...