skip to Main Content

I am trying to compile and run the example from https://docs.python.org/3/extending/embedding.html#very-high-level-embedding , but failed.

My environment is Ubuntu 20.04.2 LTS, with system shipped python3.8(statically built), libpython3-dev and libpython3.8-dev packages installed.


What I’ve tried:

main.c :

#define PY_SSIZE_T_CLEAN
#include <Python.h>

int
main(int argc, char *argv[])
{
    wchar_t *program = Py_DecodeLocale(argv[0], NULL);
    if (program == NULL) {
        fprintf(stderr, "Fatal error: cannot decode argv[0]n");
        exit(1);
    }
    Py_SetProgramName(program);  /* optional but recommended */
    Py_Initialize();
    PyRun_SimpleString("from time import time,ctimen"
                       "print('Today is', ctime(time()))n");
    if (Py_FinalizeEx() < 0) {
        exit(120);
    }
    PyMem_RawFree(program);
    return 0;
}

From https://docs.python.org/3/extending/embedding.html#compiling-and-linking-under-unix-like-systems, get gcc flags.

tian@tian-B250M-Wind:~$ python3-config --cflags
-I/usr/include/python3.8 -I/usr/include/python3.8  -Wno-unused-result -Wsign-compare -g -fdebug-prefix-map=/build/python3.8-4wuY7n/python3.8-3.8.10=. -specs=/usr/share/dpkg/no-pie-compile.specs -fstack-protector -Wformat -Werror=format-security  -DNDEBUG -g -fwrapv -O3 -Wall
tian@tian-B250M-Wind:~$ python3-config --ldflags
-L/usr/lib/python3.8/config-3.8-x86_64-linux-gnu -L/usr/lib  -lcrypt -lpthread -ldl  -lutil -lm -lm

(I don’t know why python3-config output has some duplicated values, that’s not a typing mistake)

gcc {copy cflags output} -o main main.c /path_to_libpython/libpython3.8.a {copy ld flags output}:

gcc -I/usr/include/python3.8 -I/usr/include/python3.8  -Wno-unused-result -Wsign-compare -g -fdebug-prefix-map=/build/python3.8-4wuY7n/python3.8-3.8.10=. -specs=/usr/share/dpkg/no-pie-compile.specs -fstack-protector -Wformat -Werror=format-security  -DNDEBUG -g -fwrapv -O3 -Wall -o main main.c /usr/lib/python3.8/config-3.8-x86_64-linux-gnu/libpython3.8.a -L/usr/lib/python3.8/config-3.8-x86_64-linux-gnu -L/usr/lib  -lcrypt -lpthread -ldl  -lutil -lm -lm

make test1 gives error, nearly all related to -fPIE error:

/usr/bin/ld: /tmp/ccdMZ2Yk.o: relocation R_X86_64_32 against `.rodata.str1.8' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: /usr/lib/python3.8/config-3.8-x86_64-linux-gnu/libpython3.8.a(obmalloc.o): relocation R_X86_64_32 against `.text.hot' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: /usr/lib/python3.8/config-3.8-x86_64-linux-gnu/libpython3.8.a(initconfig.o): relocation R_X86_64_32 against symbol `_PyRuntime' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: /usr/lib/python3.8/config-3.8-x86_64-linux-gnu/libpython3.8.a(pathconfig.o): relocation R_X86_64_32 against symbol `_Py_path_config' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: /usr/lib/python3.8/config-3.8-x86_64-linux-gnu/libpython3.8.a(preconfig.o): relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: /usr/lib/python3.8/config-3.8-x86_64-linux-gnu/libpython3.8.a(pylifecycle.o): relocation R_X86_64_32 against symbol `_PyRuntime' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: /usr/lib/python3.8/config-3.8-x86_64-linux-gnu/libpython3.8.a(pystate.o): relocation R_X86_64_32 against `.rodata.str1.8' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: /usr/lib/python3.8/config-3.8-x86_64-linux-gnu/libpython3.8.a(pythonrun.o): relocation R_X86_64_32 against symbol `_PyParser_Grammar' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: /usr/lib/python3.8/config-3.8-x86_64-linux-gnu/libpython3.8.a(pytime.o): relocation R_X86_64_32S against symbol `PyFloat_Type' can not be used when making a PIE object; recompile with -fPIE
.....

I saw there is a /usr/lib/python3.8/config-3.8-x86_64-linux-gnu/libpython3.8-pic.a and tried link it instead but also failed (log).


Later I tried a Docker image(python:3.10.5-bullseye) with share build python and succeeded.

root@tian-B250M-Wind:/# python3-config --cflags
-I/usr/local/include/python3.10 -I/usr/local/include/python3.10  -Wno-unused-result -Wsign-compare  -DNDEBUG -g -fwrapv -O3 -Wall
root@tian-B250M-Wind:/# python3-config --ldflags
 -L/usr/local/lib  -lcrypt -lpthread -ldl  -lutil -lm -lm
gcc -I/usr/local/include/python3.10 -I/usr/local/include/python3.10  -Wno-unused-result -Wsign-compare  -DNDEBUG -g -fwrapv -O3 -Wall -o main main.c -L/usr/local/lib  -lcrypt -lpthread -ldl  -lutil -lm -lm -lpython3.10

(I add -lpython3.10 in the end)

It compiles and ./main :

root@tian-B250M-Wind:/# ./main
Today is Thu Jul 14 10:39:20 2022

What’s wrong with my compilation for the Ubuntu system shipped, python3.8 one?

Anyway, I just want to validate that I can link the static python library libpythonx.y.a . So if anyone can make that work on a fresh installed machine or with non system shipped python (e.g. self-built static python), I’d like to try.

2

Answers


  1. Chosen as BEST ANSWER

    I made more experiments on both system shipped python and self built python libraries. For self bulilt pythons, it's easy to compile successfully, either for shared built python or statically built python.

    For system shipped python, shared link is easy and what I've missed for static link is that I need to link 2 extra libs -lexpat and -lz. Also there are 2 ways for system shipped python to link successfully:

    1. Add -no-pie, and -lexpat -lz.
    2. No -no-pie, use libpython3.8-pic.a, and -lexpat -lz.

    Check the py38_system_static target below.

    makefile:

    py38_system_failed:
        gcc -I/usr/include/python3.8 -I/usr/include/python3.8  -Wno-unused-result -Wsign-compare -g -fdebug-prefix-map=/build/python3.8-4wuY7n/python3.8-3.8.10=. -specs=/usr/share/dpkg/no-pie-compile.specs -fstack-protector -Wformat -Werror=format-security  -DNDEBUG -g -fwrapv -O3 -Wall -o main main.c /usr/lib/python3.8/config-3.8-x86_64-linux-gnu/libpython3.8.a -L/usr/lib/python3.8/config-3.8-x86_64-linux-gnu -L/usr/lib  -lcrypt -lpthread -ldl  -lutil -lm -lm
    
    py38_system_shared:
        # 36K main
        gcc -I/usr/include/python3.8 -I/usr/include/python3.8  -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -o main main.c -L/usr/lib/python3.8/config-3.8-x86_64-linux-gnu -L/usr/lib -lpython3.8 -lcrypt -lpthread -ldl  -lutil -lm -lm 
    
    py38_system_static:
        # 5.5M main 
        gcc -I/usr/include/python3.8 -I/usr/include/python3.8 -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -no-pie -O3 -Wall -o main main.c /usr/lib/python3.8/config-3.8-x86_64-linux-gnu/libpython3.8.a -L/usr/lib/python3.8/config-3.8-x86_64-linux-gnu -L/usr/lib -lcrypt -lpthread -ldl  -lutil -lm -lm -lexpat -lz
        gcc -I/usr/include/python3.8 -I/usr/include/python3.8 -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -o main main.c /usr/lib/python3.8/config-3.8-x86_64-linux-gnu/libpython3.8-pic.a -L/usr/lib/python3.8/config-3.8-x86_64-linux-gnu -L/usr/lib -lcrypt -lpthread -ldl  -lutil -lm -lm -lexpat -lz
    
    docker_py310_share:
        # 21K main
        gcc -I/usr/local/include/python3.10 -I/usr/local/include/python3.10  -Wno-unused-result -Wsign-compare  -DNDEBUG -g -fwrapv -O3 -Wall -o main main.c -L/usr/local/lib  -lcrypt -lpthread -ldl  -lutil -lm -lm -lpython3.10
    
    py36_static:
        # 15M main
        gcc -I/home/tian/py3.6.12_static/include/python3.6m -I/home/tian/py3.6.12_static/include/python3.6m  -Wno-unused-result -Wsign-compare  -DNDEBUG -g -fwrapv -O3 -Wall -o main main.c -L/home/tian/py3.6.12_static/lib/python3.6/config-3.6m-x86_64-linux-gnu -L/home/tian/py3.6.12_static/lib -lpython3.6m -lpthread -ldl  -lutil -lm  -Xlinker -export-dynamic
    
    py36_shared:
        # 36K main
        gcc -I/home/tian/py3.6.12_shared/include/python3.6m -I/home/tian/py3.6.12_shared/include/python3.6m  -Wno-unused-result -Wsign-compare  -DNDEBUG -g -fwrapv -O3 -Wall -o main main.c -L/home/tian/py3.6.12_shared/lib -lpython3.6m -lpthread -ldl  -lutil -lm  -Xlinker -export-dynamic
    
    py39_static:
        # 22M main
        gcc -I/home/tian/py3.9.13_static/include/python3.9 -I/home/tian/py3.9.13_static/include/python3.9  -Wno-unused-result -Wsign-compare  -DNDEBUG -g -fwrapv -O3 -Wall -o main main.c  -L/home/tian/py3.9.13_static/lib/python3.9/config-3.9-x86_64-linux-gnu -L/home/tian/py3.9.13_static/lib  -lpython3.9 -lcrypt -lpthread -ldl  -lutil -lm -lm 
        gcc -I/home/tian/py3.9.13_static/include/python3.9 -I/home/tian/py3.9.13_static/include/python3.9  -Wno-unused-result -Wsign-compare  -DNDEBUG -g -fwrapv -O3 -Wall -o main2 main.c /home/tian/py3.9.13_static/lib/libpython3.9.a -lcrypt -lpthread -ldl  -lutil -lm -lm 
        
    py39_shared:
        # 36K main
        gcc -I/home/tian/py3.9.13_shared/include/python3.9 -I/home/tian/py3.9.13_shared/include/python3.9  -Wno-unused-result -Wsign-compare  -DNDEBUG -g -fwrapv -O3 -Wall -o main main.c -L/home/tian/py3.9.13_shared/lib  -lpython3.9 -lcrypt -lpthread -ldl  -lutil -lm -lm 
    
    clean:
        rm main && rm main2
    

  2. Problem

    Print the output for gcc -v. What you will find within "Configured With" is the flag --enable-default-pie. PIE (Position Independent Executable) is enabled by default.

    You are including a pre-compiled object that was not built with PIE enabled (python 3.8), therefore PIE must be disabled to compile your code if you choose to use the installed Python on your system.

    The reason why you didn’t encounter this in Docker is because you were building a different version of pre-compiled python 3.10, which does have PIE enabled.

    Solution 1

    Either, add flag -no-pie to LDFlags in the Makefile;

    Or, pass -no-pie to gcc during compile time.

    Solution 2

    Compile your desired Python on your system with PIE flag enabled, then compile your code using the new Python you compiled with PIE.

    References

    https://docs.python.org/3/extending/embedding.html#very-high-level-embedding
    https://www.redhat.com/en/blog/position-independent-executables-pie
    https://linuxtut.com/en/4fc41123ed41cf443a6b/

    Login or Signup to reply.
Please signup or login to give your own answer.
Back To Top
Search