On Ubuntu 20.04, sudo apt install linux-headers-generic will install relevent kernel headers to /usr/src. To use with gcc, do a -I /usr/src/{headers} and inside of a C file, do #include "linux/sched/signal.h"
I run into issues with linux/sched/signal.h (and other header files) include other header files with #include <asm/bug.h> ,i.e. <> over "" notation. Therefore, gcc will look for them in /usr/include not /usr/src and obviously not find them. I could cp -R /usr/src/{headers}/include /usr/include however this would override existing kernel header files of different versions.
So, how can I compile with specific kernel headers using gcc?
I run into issues with linux/sched/signal.h (and other header files) include other header files with #include <asm/bug.h> ,i.e. <> over "" notation. Therefore, gcc will look for them in /usr/include not /usr/src and obviously not find them. I could cp -R /usr/src/{headers}/include /usr/include however this would override existing kernel header files of different versions.
So, how can I compile with specific kernel headers using gcc?
Edited by Ryan
on
Reason: Initial post
I'm pretty sure doing -I/usr/src/{headers} will make <asm/bug.h> include too look in /usr/src/{headers}/asm/bug.h location first, and only then in other standard /usr/include location.
Are you sure your /usr/src/{headers}/asm/bug.h file exists?
You can ask gcc to show order of folders it will use for includes. To do so pass -v argument. You will see that if you pass extra path with -I argument that goes on top of the list. For example, on my ArchLinux using gcc -v -I folder_passed_with_I_argument -c test.c produces following list:
As you can see folder_passed_with_I_argument is on the top before all other folders.
Btw - are you building kernel module? Because for that you should use their make build system - as that sets up all paths correctly for you. It basically requires extra folders, nost just root one, for example arch/x86/include for x86 architecture.
Here is information how to build kernel modules: https://www.kernel.org/doc/html/latest/kbuild/modules.html As you can see - you pass path to headers to make with -C and current folder with M= and it will build everything for you automatically without worrying about include paths.
Are you sure your /usr/src/{headers}/asm/bug.h file exists?
You can ask gcc to show order of folders it will use for includes. To do so pass -v argument. You will see that if you pass extra path with -I argument that goes on top of the list. For example, on my ArchLinux using gcc -v -I folder_passed_with_I_argument -c test.c produces following list:
1 2 3 4 5 6 7 | #include "..." search starts here: #include <...> search starts here: folder_passed_with_I_argument /usr/lib/gcc/x86_64-pc-linux-gnu/10.2.0/include /usr/local/include /usr/lib/gcc/x86_64-pc-linux-gnu/10.2.0/include-fixed /usr/include |
As you can see folder_passed_with_I_argument is on the top before all other folders.
Btw - are you building kernel module? Because for that you should use their make build system - as that sets up all paths correctly for you. It basically requires extra folders, nost just root one, for example arch/x86/include for x86 architecture.
Here is information how to build kernel modules: https://www.kernel.org/doc/html/latest/kbuild/modules.html As you can see - you pass path to headers to make with -C and current folder with M= and it will build everything for you automatically without worrying about include paths.
Edited by Mārtiņš Možeiko
on
You're right about the include paths. My mistake.
I'm not building a kernel module. I'm making various system calls in a userspace program. With:
The C code (omitted libc _start replacement for brevity) for myinit:
I found that info.procs == 54, which I thought should be 1. So tried to inspect running processes:
I found that the for_each_process macro was defined in linux/sched/signal.h, which took me down this line of investigation.
Running various find . -type f -iname "..." there seems to be overlap between files in include/asm-generic and arch/x86/include/asm. So I added -I/usr/src/{headers}/arch/x86/include. However, it now fails look for generated/timeconst.h which I can't find. I also get various compilation errors, like empty declaration, expected declaration specifiers before ‘__noreturn’ and expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘{’ token etc.
I'm not building a kernel module. I'm making various system calls in a userspace program. With:
1 | qemu-system-x86_64 -s -S -enable-kvm -m 512M -drive format=raw,file=myos.img -kernel {5.11.6-kernel} -append "root=/dev/sda init=/sbin/myinit nokaslr"
|
The C code (omitted libc _start replacement for brevity) for myinit:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 | #include <linux/sysinfo.h> int x64_syscall_sysinfo(struct sysinfo *sys_info) { int result = 0; __asm__ __volatile__("syscall" : "=a" (result) : "a" (99), "D" (sys_info) : "r11", "rcx", "memory"); return result; } int main(int argc, char *argv[]) { struct sysinfo info = {0}; // Have set up a breakpoint here to inspect contents in debugger x64_syscall_sysinfo(&info); return 0; } |
I found that info.procs == 54, which I thought should be 1. So tried to inspect running processes:
1 2 3 4 | struct task_list *task = NULL;
for_each_process(task) {
u32 pid = task->pid;
}
|
I found that the for_each_process macro was defined in linux/sched/signal.h, which took me down this line of investigation.
Running various find . -type f -iname "..." there seems to be overlap between files in include/asm-generic and arch/x86/include/asm. So I added -I/usr/src/{headers}/arch/x86/include. However, it now fails look for generated/timeconst.h which I can't find. I also get various compilation errors, like empty declaration, expected declaration specifiers before ‘__noreturn’ and expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘{’ token etc.
Edited by Ryan
on
Haven't looked in detail, but that probably also includes there count of currently running kernel threads. Process and thread in most places are represented the same way on Linux. You should check kernel source to see what is actually counted/returned there.
I don't think you can use for_each_process in user space, that is meant for code compiled in kernel (or modules loaded there). In userspace you are supposed to look in /proc filesystem for information about processes. Most of the stuff from linux headers package is meant for kernel-space code. It is pretty useless for userspace (aside maybe some defines & macros).
I don't think you can use for_each_process in user space, that is meant for code compiled in kernel (or modules loaded there). In userspace you are supposed to look in /proc filesystem for information about processes. Most of the stuff from linux headers package is meant for kernel-space code. It is pretty useless for userspace (aside maybe some defines & macros).
Edited by Mārtiņš Možeiko
on
Thanks for the information regarding the use-cases for linux headers.
Follow up questions regarding info struct:
I ask this in relation to Casey mentioning that he will allocate different amounts of memory based on the profile of the system. I thought this would be a way to do this.
Follow up questions regarding info struct:
- info.totalram == 475MB. As I specified 512MB for qemu, why is this not 512MB?
- info.totalram - info.freeram == 449MB. Does this mean the kernel is running with a memory footprint of 475MB - 449MB == 26MB?
I ask this in relation to Casey mentioning that he will allocate different amounts of memory based on the profile of the system. I thought this would be a way to do this.
That only lists usable memory that is not taken by any other device. Not total memory of system.
Maybe memory goes to gpu or some IO mapped range. IO peripherals need to use some memory range for their own purposes.
You can look at dmesg output (kernel messages) to see how it assigns memory - it will list there exactly which address are available and which are not. Here's example how it looks on my ArchLinux, it shows various regions which are reserved and not usable:
There are usually more messages listing more details. And also it shows exactly numbers it can use:
It says here that it has 2.2GB of some reserved memory for me.
To get actual real hardware memory installed in system you can read DMI tables from BIOS. On linux commandline you can do that with sudo dmidecode -t 17. dmidecode utility source is available, you can check it for details if you want to implement same functionality.
Maybe memory goes to gpu or some IO mapped range. IO peripherals need to use some memory range for their own purposes.
You can look at dmesg output (kernel messages) to see how it assigns memory - it will list there exactly which address are available and which are not. Here's example how it looks on my ArchLinux, it shows various regions which are reserved and not usable:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 | [ +0.000000] BIOS-provided physical RAM map: [ +0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009dfff] usable [ +0.000000] BIOS-e820: [mem 0x000000000009e000-0x000000000009efff] reserved [ +0.000000] BIOS-e820: [mem 0x000000000009f000-0x000000000009ffff] usable [ +0.000000] BIOS-e820: [mem 0x00000000000a0000-0x00000000000fffff] reserved [ +0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000003fffffff] usable [ +0.000000] BIOS-e820: [mem 0x0000000040000000-0x00000000403fffff] reserved [ +0.000000] BIOS-e820: [mem 0x0000000040400000-0x000000008b918fff] usable [ +0.000000] BIOS-e820: [mem 0x000000008b919000-0x000000008b919fff] ACPI NVS [ +0.000000] BIOS-e820: [mem 0x000000008b91a000-0x000000008b93ffff] usable [ +0.000000] BIOS-e820: [mem 0x000000008b940000-0x000000008b940fff] reserved [ +0.000000] BIOS-e820: [mem 0x000000008b941000-0x000000008ba7ffff] usable [ +0.000000] BIOS-e820: [mem 0x000000008ba80000-0x000000008d2ebfff] reserved [ +0.000000] BIOS-e820: [mem 0x000000008d2ec000-0x000000008d368fff] ACPI data [ +0.000000] BIOS-e820: [mem 0x000000008d369000-0x000000008d841fff] ACPI NVS [ +0.000000] BIOS-e820: [mem 0x000000008d842000-0x000000008dffefff] reserved [ +0.000000] BIOS-e820: [mem 0x000000008dfff000-0x000000008dffffff] usable [ +0.000000] BIOS-e820: [mem 0x000000008e000000-0x000000009f7fffff] reserved [ +0.000000] BIOS-e820: [mem 0x00000000e0000000-0x00000000efffffff] reserved [ +0.000000] BIOS-e820: [mem 0x00000000fe000000-0x00000000fe010fff] reserved [ +0.000000] BIOS-e820: [mem 0x00000000fec00000-0x00000000fec00fff] reserved [ +0.000000] BIOS-e820: [mem 0x00000000fee00000-0x00000000fee00fff] reserved [ +0.000000] BIOS-e820: [mem 0x00000000ff000000-0x00000000ffffffff] reserved [ +0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000205f7fffff] usable |
There are usually more messages listing more details. And also it shows exactly numbers it can use:
1 | [ +0.251604] Memory: 131464188K/133871732K available (14344K kernel code, 2032K rwdata, 8900K rodata, 1720K init, 4316K bss, 2407284K reserved, 0K cma-reserved) |
It says here that it has 2.2GB of some reserved memory for me.
To get actual real hardware memory installed in system you can read DMI tables from BIOS. On linux commandline you can do that with sudo dmidecode -t 17. dmidecode utility source is available, you can check it for details if you want to implement same functionality.
Edited by Mārtiņš Možeiko
on
Ah I see. Informative as always Mārtiņš.
Sorry to bother you with something unrelated: How would you go about implementing LoadLibraryA() and FreeLibrary() without libc dlopen? I found the syscall uselib (whose man pages said it was obsolete, yet cat /boot/config-$(uname -r) | grep CONFIG_USELIB showed it to be configured in my modern Ubuntu 20.04...) but it only returns a status code. Also, there is no equivalent unload call. So I don't think this is the way to go.
From what I gather, you need to open(), mmap() and munmap() based on ELF header and subsequent program headers. The source for gcc is too complex for me. I just what to load and unload standard ELF shared objects, nothing fancy. What would you recommend?
Sorry to bother you with something unrelated: How would you go about implementing LoadLibraryA() and FreeLibrary() without libc dlopen? I found the syscall uselib (whose man pages said it was obsolete, yet cat /boot/config-$(uname -r) | grep CONFIG_USELIB showed it to be configured in my modern Ubuntu 20.04...) but it only returns a status code. Also, there is no equivalent unload call. So I don't think this is the way to go.
From what I gather, you need to open(), mmap() and munmap() based on ELF header and subsequent program headers. The source for gcc is too complex for me. I just what to load and unload standard ELF shared objects, nothing fancy. What would you recommend?
If you want to dynamically load your own .so files without depending on system .so files then you can do that by loading .so file contents to memory - typically with mmap to map it to memory, but you can do with just read(). Then parsing ELF headers, fixing up all relocations, changing memory protection to execute when necessary, then executing entry code. After that you can lookup symbols from ELF symbol table and get pointer to call. Don't know good source on doing this to elf files, but for windows .dll files this library shows how to do that: https://github.com/fancycode/MemoryModule You just need to read up on ELF format - how is it done there.
Note that doing this you must be sure .so files you load won't be loading other system .so files with regular dynamic loader. Otherwise expect a lot of problems. Basically if you load everything - fine, but once you need something from system (like X11 library, or OpenGL api) - not good.
Here's the relevant source code in musl C library that loads dynamic libraries: https://git.musl-libc.org/cgit/musl/tree/ldso/dynlink.c#n991 It sill does more things than you need, but it may be easier to read it as it is smaller than glibc.
Note that doing this you must be sure .so files you load won't be loading other system .so files with regular dynamic loader. Otherwise expect a lot of problems. Basically if you load everything - fine, but once you need something from system (like X11 library, or OpenGL api) - not good.
Here's the relevant source code in musl C library that loads dynamic libraries: https://git.musl-libc.org/cgit/musl/tree/ldso/dynlink.c#n991 It sill does more things than you need, but it may be easier to read it as it is smaller than glibc.
Edited by Mārtiņš Možeiko
on