Centos – skipping incompatible library when trying to build 32 bit application on 64 bit machine

I have tried to make my applications (using a Makefile) as 32 bit applications on a 64 bit machine. The link phase fails because of skipping incompatible message and cannot find -l<library> message.

• Programming language: C
• LSB Version: core-4.1-amd64:core-4.1-ia32:core-4.1-noarch
• OS: CentOS Linux release 7.6.1810 (Core)
• gcc version:

gcc --version
gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-44)
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

• Lot’s of libraries installed, I believe I got all the 32 bit versions.
• yum info on GCC:

sudo yum info gcc
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: download.cf.centos.org
 * epel: d2lzkl7pfhq30w.cloudfront.net
 * extras: download.cf.centos.org
 * updates: download.cf.centos.org
Installed Packages
Name        : gcc
Arch        : x86_64
Version     : 4.8.5
Release     : 44.el7
Size        : 37 M
Repo        : installed
From repo   : base
Summary     : Various compilers (C, C++, Objective-C, Java, ...)
URL         : http://gcc.gnu.org
License     : GPLv3+ and GPLv3+ with exceptions and GPLv2+ with exceptions and LGPLv2+ and BSD
Description : The gcc package contains the GNU Compiler Collection version 4.8.
            : You'll need this package in order to compile C code.

Here is a subset of the results:

/usr/bin/ld: skipping incompatible /home/<User>lib/odbc/3.0/libodbcifV3_10.so when searching for -lodbcifV3_10
/usr/bin/ld: cannot find -lodbcifV3_10
/usr/bin/ld: cannot find -lodbcbaseV3_28
/usr/bin/ld: cannot find -lodbcdV3_28
/usr/bin/ld: cannot find -lodbccV3_28
/usr/bin/ld: skipping incompatible /usr/local/easysoft/unixODBC/lib//libodbc.so when searching for -lodbc
/usr/lib/libaspell.so: error adding symbols: File in wrong format
collect2: error: ld returned 1 exit status
make: *** [wmscentosdev2/webClient] Error 1
[<User>:v10_odbc:V3_28:mysql]> objdump -a /home/<User>lib/odbc/3.0/libodbcifV3_10.so 

/home/<User>lib/odbc/3.0/libodbcifV3_10.so:     file format elf32-i386
/home/<User>lib/odbc/3.0/libodbcifV3_10.so

[<User>:v10_odbc:V3_28:mysql]> objdump -a wmscentosdev2/clientMain.o                        

wmscentosdev2/clientMain.o:     file format elf32-i386
wmscentosdev2/clientMain.o

[<User>:v10_odbc:V3_28:mysql]> 

BTW, ignore the aspell issue, I guess I need to recompile that one, as that is a totally different problem.

I added -m32, and the object files look fine, file says they are 386 files (file format elf32-386). But when it tries to link the files, it cannot find various shared libraries, gives the "skipping incompatible" message. When I look at the shared libraries, they are also the same file format. In fact, they work fine on my production machine. I just cannot compile against them. I even tried adding -march=i686 and -march=i386, on the off chance somehow the linker was still looking at the 64 bit stuff. Note that the things it cannot find are there, right where it says it is searching, it just does not like them for some reason.

I can compile static libraries just fine, so the compile phase seems to work. The resulting ".a" files have the same file format as above, as far as I can see.

Yes, I installed the devel libraries, the static libraries, etc. On CentOS, there is no multilib version of these libraries that I could find. I installed the i686 variants. Maybe the problem is that the libraries were compiled with i386 variant on the older machine? Could that cause a problem?

Lastly, I made a simple hello world app and it compiled fine.

> objdump -a hello64

hello64:     file format elf64-x86-64
hello64

> objdump -a hello64

hello64:     file format elf64-x86-64
hello64

> hello64
Hello World!
> objdump -a hello32

hello32:     file format elf32-i386
hello32

> hello32
Hello World!

Since the above works, I assume it is something burious in my makefiles. They are extremely complex. or else I would just give them here. I suppose something buring in that complexity is causing a problem, but I cannot easily simplify it to isolate the problem. How can I figure out where the mismatch is? Is there a way to tell the linker to be more explicit on where the error is?

BTW, these applications compiled just fine on an older 32 bit system, so it is not the applications themselves.

I might be able to recompile some of these so files, but I cannot recompile all of them – I don’t have the source of one or two.

Sorry if I missed an obvious post. There are a lot of posts for this type of question, and most mention some variation on the above things I already tried. Maybe I just need to hear it explained differently.

Adding reference to GCC:

> ls -lrt /usr/bin/*gcc*
-rwxr-xr-x. 1 root root   2046 Jun 10  2014 /usr/bin/gccmakedep
-rwxr-xr-x. 1 root root  27088 Sep 29  2020 /usr/bin/gcc-ranlib
-rwxr-xr-x. 2 root root 768608 Sep 29  2020 /usr/bin/x86_64-redhat-linux-gcc
-rwxr-xr-x. 1 root root  27088 Sep 29  2020 /usr/bin/gcc-ar
-rwxr-xr-x. 2 root root 768608 Sep 29  2020 /usr/bin/gcc
-rwxr-xr-x. 1 root root  27088 Sep 29  2020 /usr/bin/gcc-nm

A portion of the commands run by make, run with make -Bn. There are 77 objects, so just pulling out one of the compilations, and the final linkage. The log file for this is 911 lines long, 78 distinct "cc" command runs:

... // Other CC commands for other C files
cc -c  -DMALLOC -DRDSJUL -DREPGEN -DSSLOPTB  -DPCKDAE -DWLM -DKAISER -DEAKEYS -DTRKFORMS -DTRKFORMS -g -DORACLE_DB_ACCESS -DODBC_DB_ACCESS -DORACLE_DB_LOG  -std=gnu99  -m32 -march=i386 -D_LINUX_SOURCE -D_POSIX_C_SOURCE=199506L -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -Wall -Wcast-qual -Wstrict-prototypes -Wmissing-prototypes -Wmissing-declarations -nostdinc -I/usr/lib/gcc/i686-redhat-linux/4.4.4/include -I/usr/lib/gcc/i386-redhat-linux/4.1.1/include -I/usr/lib/gcc/x86_64-redhat-linux/4.8.2/include -I/usr/include -pg -g -I. -I/home/wms/v10_odbc/std/lcn/issiobj/h -I/home/wms/v10_odbc/std/lcn/src/lib/base/h -I/home/wms/v10_odbc/std/lcn/src/lib/appl/h -I/home/wms/v10_odbc/std/lcn/src/lib/db/h -I/home/wms/v10_odbc/std/lcn/src/lib/feature/h -I/home/wms/v10_odbc/std/lcn/src/lib/interface -I/home/wms/v10_odbc/std/lcn/src/lib/sheet/h -I/home/wms/v10_odbc/std/lcn/src/h `pkg-config --cflags gtk+-2.0` -I/usr/local/include -I/usr/include/X11R6 -I/opt/glib/lib/glib/include -I/opt/glib/include -I/home/wms/v10_odbc/std/lcn/src/lib/interface/8.1.7/wmscentosdev2 -I/home/wms/v10_odbc/std/lcn/src/issi/interface/lib -I/home/wms/v10_odbc/std/lcn/src/issi/web/libwhmsxml -I/home/wms/v10_odbc/std/lcn/src/issi/inquiry/lib -I/home/wms/v10_odbc/std/lcn/src/issi/order_planning/lib -I/home/wms/v10_odbc/std/lcn/src/issi/frame/lib -I/home/wms/v10_odbc/std/lcn/src/issi/truck/h -I/home/wms/v10_odbc/std/lcn/src/issi/printer/lib `xml2-config --cflags` -DICSYN_NOAUTO clientMain.c -o wmscentosdev2/clientMain.o
... // Other CC commands for other C files
/bin/nice -3 cc wmscentosdev2/itemMessage.o wmscentosdev2/setup.o wmscentosdev2/forecastReview.o wmscentosdev2/user.o wmscentosdev2/webText.o wmscentosdev2/sysparam.o wmscentosdev2/reportxml.o wmscentosdev2/login.o wmscentosdev2/logActivity.o wmscentosdev2/spelling.o wmscentosdev2/reportReorderBySku.o wmscentosdev2/remotecancel.o wmscentosdev2/warehouseActivity.o wmscentosdev2/inquireDescription.o wmscentosdev2/heartBeat.o wmscentosdev2/timePhasedBuckets.o wmscentosdev2/util.o wmscentosdev2/salesOrderDetls.o wmscentosdev2/dispatch.o wmscentosdev2/approval.o wmscentosdev2/UploadCsv.o wmscentosdev2/ntfManagement.o wmscentosdev2/reportProcess.o wmscentosdev2/jasperReport.o wmscentosdev2/genericxml.o wmscentosdev2/supplierPortal.o wmscentosdev2/hostIntegration.o wmscentosdev2/eventTracking.o wmscentosdev2/rorManagement.o wmscentosdev2/mrpDashboard.o wmscentosdev2/reportUsage.o wmscentosdev2/warehouseManagement.o wmscentosdev2/reportUtils.o wmscentosdev2/columnOrderAdmin.o wmscentosdev2/reportOrderSku.o wmscentosdev2/inquire.o wmscentosdev2/manageInventory.o wmscentosdev2/consumption.o wmscentosdev2/reportOrderUser.o wmscentosdev2/cnsManagement.o wmscentosdev2/shopifyAdmin.o wmscentosdev2/question.o wmscentosdev2/reportShipments.o wmscentosdev2/message.o wmscentosdev2/wallet.o wmscentosdev2/remoteorder.o wmscentosdev2/clientMain.o wmscentosdev2/productionSchedTracker.o wmscentosdev2/passwordFetch.o wmscentosdev2/events.o wmscentosdev2/dropdown.o wmscentosdev2/permission.o wmscentosdev2/purchaseOrderDetls.o wmscentosdev2/reportReturns.o wmscentosdev2/reportInventory.o wmscentosdev2/roleAdmin.o wmscentosdev2/remotemsg.o wmscentosdev2/qryitl.o wmscentosdev2/shipmethod.o wmscentosdev2/reportReceiving.o wmscentosdev2/WCJobInq.o wmscentosdev2/order.o wmscentosdev2/limits.o wmscentosdev2/planManagement.o wmscentosdev2/orderQuery.o wmscentosdev2/kanbanAnalysis.o wmscentosdev2/receiptToHost.o wmscentosdev2/details.o wmscentosdev2/dashboard.o wmscentosdev2/reportShipmentsByLoc.o wmscentosdev2/kanbanDashboard.o wmscentosdev2/shipmentActivity.o wmscentosdev2/reportBackorder.o wmscentosdev2/orderInq.o wmscentosdev2/pupManagement.o wmscentosdev2/reportHistoryUsage.o wmscentosdev2/kanbanActivity.o    -L/home/wms/v10_odbc/lib/odbc/3.0 -lodbcifV3_10 -L/home/wms/v10_odbc/lib/odbc/3.0 -L/usr/local/easysoft/unixODBC/lib/ -lodbcbaseV3_28 -lodbcdV3_28 -lodbccV3_28 -lodbc  -pg -g /home/wms/v10_odbc/lib/libwhmsxml.a /usr/lib/libaspell.so /home/wms/v10_odbc/std/lcn/sys/svr/wmscentosdev2/libsvr.o /home/wms/v10_odbc/lib/libpnp.a /home/wms/v10_odbc/lib/libtcif.a /home/wms/v10_odbc/lib/libcms.a /home/wms/v10_odbc/lib/libtcs.a /home/wms/v10_odbc/lib/libframe.a /home/wms/v10_odbc/lib/libqry.a /home/wms/v10_odbc/lib/libfeature.a /home/wms/v10_odbc/lib/libdb.a /home/wms/v10_odbc/lib/libwhmsxml.a /home/wms/v10_odbc/lib/librecv.a /home/wms/v10_odbc/lib/libqry.a /home/wms/v10_odbc/lib/liborder.a  /home/wms/v10_odbc/lib/srclib.a /home/wms/v10_odbc/lib/libe3.a /home/wms/v10_odbc/lib/libisam.a /home/wms/v10_odbc/lib/librds.a  /home/wms/v10_odbc/lib/libwingui.a /home/wms/v10_odbc/lib/libFrmSSgui.a /home/wms/v10_odbc/lib/libsheetgui.a /home/wms/v10_odbc/lib/libFrmSSgui.a /home/wms/v10_odbc/lib/libfrmgui.a /home/wms/v10_odbc/lib/libint.a /home/wms/v10_odbc/lib/libreport.a  /home/wms/v10_odbc/lib/libdb.a /home/wms/v10_odbc/lib/libif.a /home/wms/v10_odbc/lib/libfeature.a  /home/wms/v10_odbc/lib/libdb.a /home/wms/v10_odbc/lib/liborder.a  /home/wms/v10_odbc/lib/srclib.a /home/wms/v10_odbc/lib/liblex.a /home/wms/v10_odbc/lib/liblpdriver.a /home/wms/v10_odbc/lib/libint.a /home/wms/v10_odbc/lib/libwhmsxml.a /home/wms/v10_odbc/lib/libdb.a /home/wms/v10_odbc/lib/liborder.a /home/wms/v10_odbc/lib/libreport.a /home/wms/v10_odbc/lib/libdevicegui.a /home/wms/v10_odbc/lib/libapl.a /home/wms/v10_odbc/lib/libinv.a /home/wms/v10_odbc/lib/libutl.a  /home/wms/v10_odbc/lib/libwingui.a /home/wms/v10_odbc/lib/libsheetgui.a /home/wms/v10_odbc/lib/libfrmgui.a /home/wms/v10_odbc/lib/libwingui.a /home/wms/v10_odbc/lib/libfeature.a  /home/wms/v10_odbc/lib/libgInter.a /home/wms/v10_odbc/lib/libapl.a /home/wms/v10_odbc/lib/liblex.a /home/wms/v10_odbc/lib/librds.a /home/wms/v10_odbc/lib/libutl.a /home/wms/v10_odbc/lib/srclib.a /home/wms/v10_odbc/lib/libscheduling.a  -lcurses `pkg-config --libs gtk+-2.0` /usr/lib/libssl.a /usr/lib/libcrypto.a /usr/lib/libz.a /usr/lib/libcrypt.a `xml2-config --libs` -lgsoap `curl-config --libs` `export PKG_CONFIG_PATH=/lib/pkgconfig;pkg-config --libs json-c` -lfreebl3 `xml2-config --libs` -o wmscentosdev2/webClient

What is interesting is that I don’t see -m32 for the linker phase. So maybe that is the problem.