I am trying to compile and run the example from https://docs.python.org/3/extending/embedding.html#very-high-level-embedding , but failed.
My environment is Ubuntu 20.04.2 LTS, with system shipped python3.8(statically built), libpython3-dev
and libpython3.8-dev
packages installed.
What I’ve tried:
main.c :
#define PY_SSIZE_T_CLEAN
#include <Python.h>
int
main(int argc, char *argv[])
{
wchar_t *program = Py_DecodeLocale(argv[0], NULL);
if (program == NULL) {
fprintf(stderr, "Fatal error: cannot decode argv[0]n");
exit(1);
}
Py_SetProgramName(program); /* optional but recommended */
Py_Initialize();
PyRun_SimpleString("from time import time,ctimen"
"print('Today is', ctime(time()))n");
if (Py_FinalizeEx() < 0) {
exit(120);
}
PyMem_RawFree(program);
return 0;
}
From https://docs.python.org/3/extending/embedding.html#compiling-and-linking-under-unix-like-systems, get gcc
flags.
tian@tian-B250M-Wind:~$ python3-config --cflags
-I/usr/include/python3.8 -I/usr/include/python3.8 -Wno-unused-result -Wsign-compare -g -fdebug-prefix-map=/build/python3.8-4wuY7n/python3.8-3.8.10=. -specs=/usr/share/dpkg/no-pie-compile.specs -fstack-protector -Wformat -Werror=format-security -DNDEBUG -g -fwrapv -O3 -Wall
tian@tian-B250M-Wind:~$ python3-config --ldflags
-L/usr/lib/python3.8/config-3.8-x86_64-linux-gnu -L/usr/lib -lcrypt -lpthread -ldl -lutil -lm -lm
(I don’t know why python3-config
output has some duplicated values, that’s not a typing mistake)
gcc {copy cflags output} -o main main.c /path_to_libpython/libpython3.8.a {copy ld flags output}
:
gcc -I/usr/include/python3.8 -I/usr/include/python3.8 -Wno-unused-result -Wsign-compare -g -fdebug-prefix-map=/build/python3.8-4wuY7n/python3.8-3.8.10=. -specs=/usr/share/dpkg/no-pie-compile.specs -fstack-protector -Wformat -Werror=format-security -DNDEBUG -g -fwrapv -O3 -Wall -o main main.c /usr/lib/python3.8/config-3.8-x86_64-linux-gnu/libpython3.8.a -L/usr/lib/python3.8/config-3.8-x86_64-linux-gnu -L/usr/lib -lcrypt -lpthread -ldl -lutil -lm -lm
make test1
gives error, nearly all related to -fPIE
error:
/usr/bin/ld: /tmp/ccdMZ2Yk.o: relocation R_X86_64_32 against `.rodata.str1.8' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: /usr/lib/python3.8/config-3.8-x86_64-linux-gnu/libpython3.8.a(obmalloc.o): relocation R_X86_64_32 against `.text.hot' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: /usr/lib/python3.8/config-3.8-x86_64-linux-gnu/libpython3.8.a(initconfig.o): relocation R_X86_64_32 against symbol `_PyRuntime' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: /usr/lib/python3.8/config-3.8-x86_64-linux-gnu/libpython3.8.a(pathconfig.o): relocation R_X86_64_32 against symbol `_Py_path_config' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: /usr/lib/python3.8/config-3.8-x86_64-linux-gnu/libpython3.8.a(preconfig.o): relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: /usr/lib/python3.8/config-3.8-x86_64-linux-gnu/libpython3.8.a(pylifecycle.o): relocation R_X86_64_32 against symbol `_PyRuntime' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: /usr/lib/python3.8/config-3.8-x86_64-linux-gnu/libpython3.8.a(pystate.o): relocation R_X86_64_32 against `.rodata.str1.8' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: /usr/lib/python3.8/config-3.8-x86_64-linux-gnu/libpython3.8.a(pythonrun.o): relocation R_X86_64_32 against symbol `_PyParser_Grammar' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: /usr/lib/python3.8/config-3.8-x86_64-linux-gnu/libpython3.8.a(pytime.o): relocation R_X86_64_32S against symbol `PyFloat_Type' can not be used when making a PIE object; recompile with -fPIE
.....
I saw there is a /usr/lib/python3.8/config-3.8-x86_64-linux-gnu/libpython3.8-pic.a
and tried link it instead but also failed (log).
Later I tried a Docker image(python:3.10.5-bullseye
) with share build python and succeeded.
root@tian-B250M-Wind:/# python3-config --cflags
-I/usr/local/include/python3.10 -I/usr/local/include/python3.10 -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall
root@tian-B250M-Wind:/# python3-config --ldflags
-L/usr/local/lib -lcrypt -lpthread -ldl -lutil -lm -lm
gcc -I/usr/local/include/python3.10 -I/usr/local/include/python3.10 -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -o main main.c -L/usr/local/lib -lcrypt -lpthread -ldl -lutil -lm -lm -lpython3.10
(I add -lpython3.10
in the end)
It compiles and ./main
:
root@tian-B250M-Wind:/# ./main
Today is Thu Jul 14 10:39:20 2022
What’s wrong with my compilation for the Ubuntu system shipped, python3.8 one?
Anyway, I just want to validate that I can link the static python library libpythonx.y.a
. So if anyone can make that work on a fresh installed machine or with non system shipped python (e.g. self-built static python), I’d like to try.
2
Answers
I made more experiments on both system shipped python and self built python libraries. For self bulilt pythons, it's easy to compile successfully, either for shared built python or statically built python.
For system shipped python, shared link is easy and what I've missed for static link is that I need to link 2 extra libs
-lexpat
and-lz
. Also there are 2 ways for system shipped python to link successfully:-no-pie
, and-lexpat -lz
.-no-pie
, uselibpython3.8-pic.a
, and-lexpat -lz
.Check the
py38_system_static
target below.makefile:
Problem
Print the output for
gcc -v
. What you will find within "Configured With" is the flag--enable-default-pie
. PIE (Position Independent Executable) is enabled by default.You are including a pre-compiled object that was not built with PIE enabled (python 3.8), therefore PIE must be disabled to compile your code if you choose to use the installed Python on your system.
The reason why you didn’t encounter this in Docker is because you were building a different version of pre-compiled python 3.10, which does have PIE enabled.
Solution 1
Either, add flag
-no-pie
toLDFlags
in the Makefile;Or, pass
-no-pie
togcc
during compile time.Solution 2
Compile your desired Python on your system with PIE flag enabled, then compile your code using the new Python you compiled with PIE.
References
https://docs.python.org/3/extending/embedding.html#very-high-level-embedding
https://www.redhat.com/en/blog/position-independent-executables-pie
https://linuxtut.com/en/4fc41123ed41cf443a6b/