Latest posts for tag fedora
Systemd Nspawn and Fedora 44
Fedora 44 is out and I need to add support for it Moncic-CI.
What could possibly go wrong?
rpmbuild cannot build packages
Well, when using systemd-nspawn as a backend, rpmbuild cannot build packages:
+ rpmbuild -ba --define 'srcarchivename hello' /srv/moncic-ci/source/hello/fedora/SPECS/hello.spec
Building target platforms: noarch
Building for target noarch
setting SOURCE_DATE_EPOCH=1748563200
Executing(%mkbuilddir): /bin/sh -e /var/tmp/rpm-tmp.NEQnsa
Executing(%prep): /bin/sh -e /var/tmp/rpm-tmp.aE2raB
+ umask 022
+ cd /root/rpmbuild/BUILD/hello-1.0-build
+ cd /root/rpmbuild/BUILD/hello-1.0-build
+ rm -rf hello
+ /usr/lib/rpm/rpmuncompress -x /root/rpmbuild/SOURCES/hello.tar.gz
/usr/bin/tar: hello/debian: Cannot mkdir: Function not implemented
/usr/bin/tar: hello/debian/changelog: Cannot open: Function not implemented
/usr/bin/tar: hello/debian/compat: Cannot open: Function not implemented
/usr/bin/tar: hello/debian/control: Cannot open: Function not implemented
/usr/bin/tar: hello/debian/copyright: Cannot open: Function not implemented
/usr/bin/tar: hello/debian/hello.install: Cannot open: Function not implemented
/usr/bin/tar: hello/debian/rules: Cannot open: Function not implemented
/usr/bin/tar: hello/debian/source: Cannot mkdir: Function not implemented
/usr/bin/tar: hello/debian/source/format: Cannot open: Function not implemented
/usr/bin/tar: hello/fedora: Cannot mkdir: Function not implemented
/usr/bin/tar: hello/fedora/SPECS: Cannot mkdir: Function not implemented
/usr/bin/tar: hello/fedora/SPECS/hello.spec: Cannot open: Function not implemented
/usr/bin/tar: hello/hello: Cannot open: Function not implemented
/usr/bin/tar: Exiting with failure status due to previous errors
error: Bad exit status from /var/tmp/rpm-tmp.aE2raB (%prep)
Bad exit status from /var/tmp/rpm-tmp.aE2raB (%prep)
Ok, what's wrong with rpmuncompress?
$ /usr/lib/rpm/rpmuncompress -x hello.tar.gz -v
/usr/bin/gzip -dc 'hello.tar.gz' | /usr/bin/tar -xvvof -
Tar cannot untar...
Is that all that it's really doing? Is tar not working now?
$ tar -axf hello.tar.gz
tar: hello/debian: Cannot mkdir: Function not implemented
tar: hello/debian/changelog: Cannot open: Function not implemented
tar: hello/debian/copyright: Cannot open: Function not implemented
tar: hello/debian/hello.install: Cannot open: Function not implemented
tar: hello/debian/rules: Cannot open: Function not implemented
tar: hello/debian/source: Cannot mkdir: Function not implemented
tar: hello/debian/source/format: Cannot open: Function not implemented
tar: hello/debian/compat: Cannot open: Function not implemented
tar: hello/debian/control: Cannot open: Function not implemented
tar: hello/fedora: Cannot mkdir: Function not implemented
tar: hello/fedora/SPECS: Cannot mkdir: Function not implemented
tar: hello/fedora/SPECS/hello.spec: Cannot open: Function not implemented
tar: hello/hello: Cannot open: Function not implemented
tar: Exiting with failure status due to previous errors
Yes. Tar indeed appears not to be working.
$ strace tar -axf hello.tar.gz
…
openat2(AT_FDCWD, "hello/", …, 24) = -1 ENOSYS (Function not implemented)
Oooh, new tar is using openat2, that's really cool! I'm a big fan of that set of syscalls, and, I agree with what Lennart wrote just the other day, that focussing solely on memory stuff ignoring the fs stuff is just bad security engineering. Cheers to tar!
However, there seems to be a seccomp configuration in place that disables openat2.
I looked around the internet for a bit and this systemd pull request seems relevant, although I haven't yet traced all the steps of what's going on to be able to confirm or deny it.
...except when tar can untar
Compare this:
# tar -C / -acf - root | tar -C /tmp -avxf -
root/
root/.ssh/
tar: root/.ssh: Cannot mkdir: Function not implemented
root/.bash_logout
tar: root/.bash_logout: Cannot open: Function not implemented
root/.bash_profile
tar: root/.bash_profile: Cannot open: Function not implemented
root/.bashrc
tar: root/.bashrc: Cannot open: Function not implemented
root/.cshrc
tar: root/.cshrc: Cannot open: Function not implemented
root/.tcshrc
tar: root/.tcshrc: Cannot open: Function not implemented
root/.bash_history
tar: root/.bash_history: Cannot open: Function not implemented
tar: Exiting with failure status due to previous errors
With this:
# tar -C /root -acf - . | tar -C /tmp -avxf -
./
./.ssh/
./.bash_logout
./.bash_profile
./.bashrc
./.cshrc
./.tcshrc
./.bash_history
Now what? Let's strace again.
In the first case, it uses openat2:
# mkdir /tmp/bad && tar -C / -acf - root | strace tar -C /tmp/bad -axf -
…
openat(AT_FDCWD, "/tmp/bad", O_RDONLY|O_CLOEXEC|O_PATH|O_DIRECTORY) = 3
mkdirat(3, "root", 0500) = 0
openat2(3, "root/", {flags=O_RDONLY|O_NOFOLLOW|O_CLOEXEC|O_PATH|O_DIRECTORY, resolve=RESOLVE_BENEATH}, 24) = -1 ENOSYS (Function not implemented)
write(2, "tar: ", 5tar: ) = 5
write(2, "root/.ssh: Cannot mkdir", 23root/.ssh: Cannot mkdir) = 23
write(2, ": Function not implemented", 26: Function not implemented) = 26
write(2, "\n", 1
) = 1
openat2(3, "root/", {flags=O_RDONLY|O_NOFOLLOW|O_CLOEXEC|O_PATH|O_DIRECTORY, resolve=RESOLVE_BENEATH}, 24) = -1 ENOSYS (Function not implemented)
write(2, "tar: ", 5tar: ) = 5
write(2, "root/.bash_logout: Cannot open", 30root/.bash_logout: Cannot open) = 30
write(2, ": Function not implemented", 26: Function not implemented) = 26
write(2, "\n", 1
) = 1
openat2(3, "root/", {flags=O_RDONLY|O_NOFOLLOW|O_CLOEXEC|O_PATH|O_DIRECTORY, resolve=RESOLVE_BENEATH}, 24) = -1 ENOSYS (Function not implemented)
write(2, "tar: ", 5tar: ) = 5
write(2, "root/.bash_profile: Cannot open", 31root/.bash_profile: Cannot open) = 31
write(2, ": Function not implemented", 26: Function not implemented) = 26
write(2, "\n", 1
) = 1
openat2(3, "root/", {flags=O_RDONLY|O_NOFOLLOW|O_CLOEXEC|O_PATH|O_DIRECTORY, resolve=RESOLVE_BENEATH}, 24) = -1 ENOSYS (Function not implemented)
write(2, "tar: ", 5tar: ) = 5
write(2, "root/.bashrc: Cannot open", 25root/.bashrc: Cannot open) = 25
write(2, ": Function not implemented", 26: Function not implemented) = 26
write(2, "\n", 1
) = 1
openat2(3, "root/", {flags=O_RDONLY|O_NOFOLLOW|O_CLOEXEC|O_PATH|O_DIRECTORY, resolve=RESOLVE_BENEATH}, 24) = -1 ENOSYS (Function not implemented)
write(2, "tar: ", 5tar: ) = 5
write(2, "root/.cshrc: Cannot open", 24root/.cshrc: Cannot open) = 24
write(2, ": Function not implemented", 26: Function not implemented) = 26
write(2, "\n", 1
) = 1
openat2(3, "root/", {flags=O_RDONLY|O_NOFOLLOW|O_CLOEXEC|O_PATH|O_DIRECTORY, resolve=RESOLVE_BENEATH}, 24) = -1 ENOSYS (Function not implemented)
write(2, "tar: ", 5tar: ) = 5
write(2, "root/.tcshrc: Cannot open", 25root/.tcshrc: Cannot open) = 25
write(2, ": Function not implemented", 26: Function not implemented) = 26
write(2, "\n", 1
) = 1
openat2(3, "root/", {flags=O_RDONLY|O_NOFOLLOW|O_CLOEXEC|O_PATH|O_DIRECTORY, resolve=RESOLVE_BENEATH}, 24) = -1 ENOSYS (Function not implemented)
write(2, "tar: ", 5tar: ) = 5
write(2, "root/.bash_history: Cannot open", 31root/.bash_history: Cannot open) = 31
write(2, ": Function not implemented", 26: Function not implemented) = 26
write(2, "\n", 1
) = 1
close(0) = 0
utimensat(3, "root", [UTIME_OMIT, {tv_sec=1777462709, tv_nsec=0} /* 2026-04-29T13:38:29+0200 */], AT_SYMLINK_NOFOLLOW) = 0
fchownat(3, "root", 0, 0, AT_SYMLINK_NOFOLLOW) = 0
fchmodat2(3, "root", 0550, AT_SYMLINK_NOFOLLOW) = 0
write(2, "tar: ", 5tar: ) = 5
write(2, "Exiting with failure status due "..., 50Exiting with failure status due to previous errors) = 50
write(2, "\n", 1
In the second case, it does not:
# mkdir /tmp/good && tar -C /root -acf - . | strace tar -C /tmp/good -axf -
…
openat(AT_FDCWD, "/tmp/good", O_RDONLY|O_CLOEXEC|O_PATH|O_DIRECTORY) = 3
mkdirat(3, ".", 0500) = -1 EEXIST (File exists)
newfstatat(3, ".", {st_mode=S_IFDIR|0755, st_size=40, ...}, AT_SYMLINK_NOFOLLOW) = 0
newfstatat(3, ".", {st_mode=S_IFDIR|0755, st_size=40, ...}, AT_SYMLINK_NOFOLLOW) = 0
mkdirat(3, ".ssh", 0700) = 0
utimensat(3, ".ssh", [UTIME_OMIT, {tv_sec=1777462478, tv_nsec=0} /* 2026-04-29T13:34:38+0200 */], AT_SYMLINK_NOFOLLOW) = 0
fchownat(3, ".ssh", 0, 0, AT_SYMLINK_NOFOLLOW) = 0
openat(3, ".bash_logout", O_WRONLY|O_CREAT|O_EXCL|O_NOCTTY|O_NONBLOCK|O_CLOEXEC, 0600) = 4
write(4, "# ~/.bash_logout\n\n", 18) = 18
utimensat(4, NULL, [UTIME_OMIT, {tv_sec=1768608000, tv_nsec=0} /* 2026-01-17T01:00:00+0100 */], 0) = 0
fchown(4, 0, 0) = 0
fchmod(4, 0644) = 0
close(4) = 0
openat(3, ".bash_profile", O_WRONLY|O_CREAT|O_EXCL|O_NOCTTY|O_NONBLOCK|O_CLOEXEC, 0600) = 4
write(4, "# .bash_profile\n\n# Get the alias"..., 144) = 144
utimensat(4, NULL, [UTIME_OMIT, {tv_sec=1768608000, tv_nsec=0} /* 2026-01-17T01:00:00+0100 */], 0) = 0
fchown(4, 0, 0) = 0
fchmod(4, 0644) = 0
close(4) = 0
openat(3, ".bashrc", O_WRONLY|O_CREAT|O_EXCL|O_NOCTTY|O_NONBLOCK|O_CLOEXEC, 0600) = 4
write(4, "# .bashrc\n\n# Source global defin"..., 575) = 575
utimensat(4, NULL, [UTIME_OMIT, {tv_sec=1768608000, tv_nsec=0} /* 2026-01-17T01:00:00+0100 */], 0) = 0
fchown(4, 0, 0) = 0
fchmod(4, 0644) = 0
close(4) = 0
openat(3, ".cshrc", O_WRONLY|O_CREAT|O_EXCL|O_NOCTTY|O_NONBLOCK|O_CLOEXEC, 0600) = 4
write(4, "# .cshrc\n\n# User specific aliase"..., 100) = 100
utimensat(4, NULL, [UTIME_OMIT, {tv_sec=1768608000, tv_nsec=0} /* 2026-01-17T01:00:00+0100 */], 0) = 0
fchown(4, 0, 0) = 0
fchmod(4, 0644) = 0
close(4) = 0
openat(3, ".tcshrc", O_WRONLY|O_CREAT|O_EXCL|O_NOCTTY|O_NONBLOCK|O_CLOEXEC, 0600) = 4
write(4, "# .tcshrc\n\n# User specific alias"..., 129) = 129
utimensat(4, NULL, [UTIME_OMIT, {tv_sec=1768608000, tv_nsec=0} /* 2026-01-17T01:00:00+0100 */], 0) = 0
fchown(4, 0, 0) = 0
fchmod(4, 0644) = 0
close(4) = 0
openat(3, ".bash_history", O_WRONLY|O_CREAT|O_EXCL|O_NOCTTY|O_NONBLOCK|O_CLOEXEC, 0600) = 4
write(4, "dnf install rpmutils\ndnf install"..., 105) = 105
utimensat(4, NULL, [UTIME_OMIT, {tv_sec=1777462709, tv_nsec=0} /* 2026-04-29T13:38:29+0200 */], 0) = 0
fchown(4, 0, 0) = 0
close(4) = 0
close(0) = 0
newfstatat(3, ".", {st_mode=S_IFDIR|0755, st_size=180, ...}, 0) = 0
utimensat(3, ".", [UTIME_OMIT, {tv_sec=1777462709, tv_nsec=0} /* 2026-04-29T13:38:29+0200 */], 0) = 0
fchownat(3, ".", 0, 0, 0) = 0
fchmodat(3, ".", 0550) = 0
Narrowing down the problem
systemd-nspawn -D works
$ sudo systemd-nspawn --ephemeral -D /var/lib/machines/fedora44
[root@fedora44-da319fe9130f3407 ~]# tar -C / -cf - root | tar -C /tmp -xf -
[root@fedora44-da319fe9130f3407 ~]# tar -C /root -cf - . | tar -C /tmp -xf -
machinectl works
$ sudo machinectl start fedora44
$ sudo machinectl shell fedora44
Connected to machine fedora44. Press ^] three times within 1s to exit session.
[root@fedora44 ~]# tar -C / -cf - root | tar -C /tmp -xf -
[root@fedora44 ~]# tar -C /root -cf - . | tar -C /tmp -xf -
[root@fedora44 ~]# cat /etc/os-release
NAME="Fedora Linux"
VERSION="44 (Forty Four)"
…
machinectl + systemd-run works
$ sudo machinectl start fedora44
$ sudo systemd-run --machine fedora44 --wait --tty sh -c 'tar -C / -cf - root | tar -C /tmp -xf -'
Running as unit: run-p431602-i431902.service
Press ^] three times within 1s to disconnect TTY.
Finished with result: success
Main processes terminated with: code=exited, status=0/SUCCESS
Service runtime: 33ms
CPU time consumed: 10ms
Memory peak: 1.3M (swap: 0B)
$ sudo systemd-run --machine fedora44 --wait --tty sh -c 'tar -C /root -cf - . | tar -C /tmp -xf -'
Running as unit: run-p431638-i431938.service
Press ^] three times within 1s to disconnect TTY.
Finished with result: success
Main processes terminated with: code=exited, status=0/SUCCESS
Service runtime: 29ms
CPU time consumed: 10ms
Memory peak: 1.3M (swap: 0B)
systemd-nspawn + systemd-run works
$ sudo systemd-nspawn --ephemeral -D /var/lib/machines/fedora44 --machine test --boot
$ sudo systemd-run --machine test --wait --tty sh -c 'tar -C / -cf - root | tar -C /tmp -xf -'
[sudo] password for enrico:
Running as unit: run-p431915-i432215.service
Press ^] three times within 1s to disconnect TTY.
Finished with result: success
Main processes terminated with: code=exited, status=0/SUCCESS
Service runtime: 62ms
CPU time consumed: 10ms
Memory peak: 2.1M (swap: 0B)
$ sudo systemd-run --machine test --wait --tty sh -c 'tar -C /root -cf - . | tar -C /tmp -xf -'
Running as unit: run-p431995-i432295.service; invocation ID: 99c164ebae684bb99df71f9230ecef7c
Press ^] three times within 1s to disconnect TTY.
Finished with result: success
Main processes terminated with: code=exited, status=0/SUCCESS
Service runtime: 49ms
CPU time consumed: 11ms
Memory peak: 1.8M (swap: 0B)
systemd-nspawn --suppress-sync=true + systemd-run does not work
So, what is Moncic-CI doing that I'm not?
It's a CI running tasks in throwaway ephemeral machines, so it uses
--suppress-sync=true for performance. Let's see...
$ sudo systemd-nspawn --ephemeral --suppress-sync=true -D /var/lib/machines/fedora44 --machine test --boot
$ sudo systemd-run --machine test --wait --tty sh -c 'tar -C / -cf - root | tar -C /tmp -xf -'
Running as unit: run-p432437-i432737.service
Press ^] three times within 1s to disconnect TTY.
tar: root/.ssh: Cannot mkdir: Function not implemented
tar: root/.bash_logout: Cannot open: Function not implemented
tar: root/.bash_profile: Cannot open: Function not implemented
tar: root/.bashrc: Cannot open: Function not implemented
tar: root/.cshrc: Cannot open: Function not implemented
tar: root/.tcshrc: Cannot open: Function not implemented
tar: root/.bash_history: Cannot open: Function not implemented
tar: Exiting with failure status due to previous errors
Finished with result: exit-code
Main processes terminated with: code=exited, status=2/INVALIDARGUMENT
Service runtime: 62ms
CPU time consumed: 13ms
Memory peak: 1.8M (swap: 0B)
Getting warmer there...
systemd-nspawn --suppress-sync=true does not work
$ sudo systemd-nspawn --ephemeral --suppress-sync=true -D /var/lib/machines/fedora44
Spawning container fedora44-233d2e0e6853964c on /var/lib/machines/.#machine.fedora44655d618f26cb4d20.
Press Ctrl-] three times within 1s to kill container.
[root@fedora44-233d2e0e6853964c ~]# tar -C / -cf - root | tar -C /tmp -xf -
tar: root/.ssh: Cannot mkdir: Function not implemented
tar: root/.bash_logout: Cannot open: Function not implemented
tar: root/.bash_profile: Cannot open: Function not implemented
tar: root/.bashrc: Cannot open: Function not implemented
tar: root/.cshrc: Cannot open: Function not implemented
tar: root/.tcshrc: Cannot open: Function not implemented
tar: root/.bash_history: Cannot open: Function not implemented
tar: Exiting with failure status due to previous errors
[root@fedora44-233d2e0e6853964c ~]#
Looks like a systemd-nspawn bug...
...filed as https://github.com/systemd/systemd/issues/41868
Edited to add: Marcin Juszkiewicz pointed out this is already known as RedHat bug #2437037.
...with a simple workaround
...and pushed a commit
that disables (for now) --suppress-sync when building on Fedora 44 containers.
With the workaround integration tests are now finally passing!