Intelligente Lösungen
in neuer Dimension

LXC: Probleme beim Container-Start

Ich verwende auf meinem Arbeitsplatzrechner gelegentlich LXC-Container für Spezial-Aufgaben, beispielsweise für Tests oder das Bauen von Drittpaketen. Dafür habe ich einen separaten Speicherbereich angelegt (“Storage Pool” im Sprachgebrauch von LXC), damit die Container nicht den Rest meines Rechners zumüllen.

Heute lassen sich die Container plötzlich nicht mehr starten. In diesem Artikel beschreibe ich die Maßnahmen, um wieder zu einem funktionierendem LXC zu gelangen. Die Ursachenermittlung für die Störung streife ich nur am Rande.

Fehlermeldung beim Start

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
uli@ulicsl:~$ lxc start build-gitea
Error: Failed to run: /snap/lxd/current/bin/lxd forkstart build-gitea /var/snap/lxd/common/lxd/containers
/var/snap/lxd/common/lxd/logs/build-gitea/lxc.conf: Try `lxc info --show-log build-gitea` for more info

uli@ulicsl:~$ lxc info --show-log build-gitea
Name: build-gitea
Status: STOPPED
Type: container
Architecture: x86_64
Created: 2021/01/19 08:04 CET
Last Used: 2021/12/03 08:22 CET

Log:

lxc build-gitea 20211203072258.714 WARN     conf - conf.c:lxc_map_ids:3579 - newuidmap binary is missing
lxc build-gitea 20211203072258.714 WARN     conf - conf.c:lxc_map_ids:3585 - newgidmap binary is missing
lxc build-gitea 20211203072258.715 WARN     conf - conf.c:lxc_map_ids:3579 - newuidmap binary is missing
lxc build-gitea 20211203072258.715 WARN     conf - conf.c:lxc_map_ids:3585 - newgidmap binary is missing
lxc build-gitea 20211203072258.716 WARN     cgfsng - cgroups/cgfsng.c:fchowmodat:1251 - No such file or directory - Failed to fchownat(40, memory.oom.group, 1000000000, 0, AT_EMPTY_PATH | AT_SYMLINK_NOFOLLOW )
lxc build-gitea 20211203072258.783 ERROR    dir - storage/dir.c:dir_mount:194 - Operation not permitted - Failed to mount "/var/snap/lxd/common/lxd/containers/build-gitea/rootfs" onto "/var/snap/lxd/common/lxc/"
lxc build-gitea 20211203072258.783 ERROR    conf - conf.c:lxc_mount_rootfs:1419 - Failed to mount rootfs "/var/snap/lxd/common/lxd/containers/build-gitea/rootfs" onto "/var/snap/lxd/common/lxc/" with options "(null)"
lxc build-gitea 20211203072258.783 ERROR    conf - conf.c:lxc_setup_rootfs_prepare_root:3951 - Failed to setup rootfs for
lxc build-gitea 20211203072258.783 ERROR    conf - conf.c:lxc_setup:4317 - Failed to setup rootfs
lxc build-gitea 20211203072258.783 ERROR    start - start.c:do_start:1275 - Failed to setup container "build-gitea"
lxc build-gitea 20211203072258.784 ERROR    sync - sync.c:sync_wait:34 - An error occurred in another process (expected sequence number 3)
lxc build-gitea 20211203072258.792 WARN     network - network.c:lxc_delete_network_priv:3617 - Failed to rename interface with index 0 from "eth0" to its initial name "vethfffcb4b3"
lxc build-gitea 20211203072258.792 ERROR    lxccontainer - lxccontainer.c:wait_on_daemonized_start:867 - Received container state "ABORTING" instead of "RUNNING"
lxc build-gitea 20211203072258.792 ERROR    start - start.c:__lxc_start:2074 - Failed to spawn container "build-gitea"
lxc build-gitea 20211203072258.792 WARN     start - start.c:lxc_abort:1039 - No such process - Failed to send SIGKILL via pidfd 41 for process 19013
lxc build-gitea 20211203072303.919 WARN     conf - conf.c:lxc_map_ids:3579 - newuidmap binary is missing
lxc build-gitea 20211203072303.919 WARN     conf - conf.c:lxc_map_ids:3585 - newgidmap binary is missing
lxc 20211203072303.956 ERROR    af_unix - af_unix.c:lxc_abstract_unix_recv_fds_iov:218 - Connection reset by peer - Failed to receive response
lxc 20211203072303.956 ERROR    commands - commands.c:lxc_cmd_rsp_recv_fds:127 - Failed to receive file descriptors

Die Fehlermeldungen sind relativ nichtssagend. Möglicherweise gibt es Probleme mit dem StoragePool?

LXC-Speicherbereiche – StoragePool

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
uli@ulicsl:~$ lxc storage list
+---------+--------+------------------------------+-------------+---------+
|  NAME   | DRIVER |            SOURCE            | DESCRIPTION | USED BY |
+---------+--------+------------------------------+-------------+---------+
| default | btrfs  | /dev/mapper/ubuntu--vg-lxdlv |             | 7       |
+---------+--------+------------------------------+-------------+---------+

uli@ulicsl:~$ lxc storage show default
config:
  size: 15GB
  source: /dev/mapper/ubuntu--vg-lxdlv
  volatile.initial_source: /dev/mapper/ubuntu--vg-lxdlv
description: ""
name: default
driver: btrfs
used_by:
- /1.0/instances/backgroundremover-2110
- /1.0/instances/build-gitea
- /1.0/instances/test-2110
- /1.0/instances/ubuntu-2004
- /1.0/instances/ubuntu-2104
- /1.0/instances/ubuntu-2110
- /1.0/profiles/default
status: Created
locations:
- none

Die Ausgaben sehen unauffällig aus!

Sichtung von /dev/mapper/ubuntu—vg-lxdlv

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
root@ulicsl:~# mkdir mnt
root@ulicsl:~# mount /dev/mapper/ubuntu--vg-lxdlv mnt
root@ulicsl:~# ls mnt
containers  containers-snapshots  custom  custom-snapshots  images  virtual-machines  virtual-machines-snapshots
root@ulicsl:~# ls mnt/containers
build-gitea  test-2110
root@ulicsl:~# cd mnt
root@ulicsl:~/mnt# find . -ls
      256     16 drwxr-xr-x   1 root     root          200 Nov  9 04:35 .
      257      0 drwx--x--x   1 root     root            0 Nov  9 04:35 ./custom
      258      0 drwx--x--x   1 root     root            0 Nov  9 04:35 ./custom-snapshots
      259      0 drwx--x--x   1 root     root            0 Nov  9 04:35 ./images
      260      0 drwx--x--x   1 root     root           40 Nov  9 04:36 ./containers
      264      0 d--x------   1 root     root           22 Nov  9 04:35 ./containers/build-gitea
      265      4 -r--------   1 root     root         3079 Dez  3 08:42 ./containers/build-gitea/backup.yaml
      266      0 d--x------   1 root     root           22 Nov  9 04:36 ./containers/test-2110
      267      4 -r--------   1 root     root         3025 Dez  3 08:53 ./containers/test-2110/backup.yaml
      261      0 drwx--x--x   1 root     root            0 Nov  9 04:35 ./containers-snapshots
      262      0 drwx--x--x   1 root     root            0 Nov  9 04:35 ./virtual-machines
      263      0 drwx--x--x   1 root     root            0 Nov  9 04:35 ./virtual-machines-snapshots
root@ulicsl:~/mnt# cd ..
root@ulicsl:~# umount mnt
root@ulicsl:~# rmdir mnt

Der Inhalt des Speicherbereichs sieht relativ “dünn” aus! Üblicherweise liegen hier pro Container sehr viele Daten ab! Aktuell sehe ich nur

  • build-gitea
  • test-2110

und für diese auch nur ein Backup der Konfigurationsdateien!

Quasi alle Container-Daten scheinen zu fehlen!

Sichtung frühere Aktionen

Am 2021-10-29 zwischen 06:10:55 und 06:14:09 habe ich letztmalig erfolgreich mit “build-gitea” gearbeitet. Also: Dieser Container ging kaputt zwischen 2021-10-29 und 2021-12-03!

Am 2021-11-07 scheine ich zwischen 11:00 und 11:30 Aktionen mit BTRFS und LXDLV unternommen zu haben. Eventuell habe ich da Murks fabriziert? Muß ich bei Gelegenheit im Detail untersuchen…

Container löschen

Löschen der Container klappt nicht so ohne weiteres:

1
2
uli@ulicsl:~$ lxc delete ubuntu-2004
Error: Error deleting storage volume: Failed setting subvolume writable "/var/snap/lxd/common/lxd/storage-pools/default/containers/ubuntu-2004": Failed to run: btrfs property set -ts /var/snap/lxd/common/lxd/storage-pools/default/containers/ubuntu-2004 ro false: ERROR: Could not get subvolume flags: Invalid argument

Wenn ich vorab eine Tabelle von LXD aufräume, dann klappt es:

1
2
3
4
5
6
7
8
9
10
11
uli@ulicsl:~$ lxd sql global "delete from storage_volumes where storage_pool_id=1;"
Rows affected: 6

uli@ulicsl:~$ lxc delete ubuntu-2004
uli@ulicsl:~$ lxc delete ubuntu-2104
...

uli@ulicsl:~$ lxc ls
+------+-------+------+------+------+-----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
+------+-------+------+------+------+-----------+

Container neu anlegen

Leider klappt die Neuanlage nicht:

1
2
3
4
5
uli@ulicsl:~$ lxc launch ubuntu:20.04 ubuntu-2004
Creating ubuntu-2004
Starting ubuntu-2004                        
Error: Failed preparing container for start: Failed to handle idmapped storage: No such file or directory: "/var/snap/lxd/common/lxd/storage-pools/default/containers/ubuntu-2004/rootfs"
Try `lxc info --show-log local:ubuntu-2004` for more info

“Alles” löschen

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
uli@ulicsl:~$ lxd sql global "delete from storage_volumes where storage_pool_id=1;"
uli@ulicsl:~$ lxc delete ubuntu-2004
uli@ulicsl:~$ lxc image list
+-------+--------------+--------+---------------------------------------------+--------------+-----------+----------+-----------------------------+
| ALIAS | FINGERPRINT  | PUBLIC |                 DESCRIPTION                 | ARCHITECTURE |   TYPE    |   SIZE   |         UPLOAD DATE         |
+-------+--------------+--------+---------------------------------------------+--------------+-----------+----------+-----------------------------+
|       | a84023244321 | no     | ubuntu 20.04 LTS amd64 (release) (20211129) | x86_64       | CONTAINER | 380.57MB | Dec 3, 2021 at 9:39am (UTC) |
+-------+--------------+--------+---------------------------------------------+--------------+-----------+----------+-----------------------------+
uli@ulicsl:~$ lxc image delete a840234321
uli@ulicsl:~$ EDITOR=jmacs lxc profile edit default
# root: samt "pool" rauswerfen
uli@ulicsl:~$ lxc storage show default
config:
  size: 50GB
  source: /dev/mapper/ubuntu--vg-lxdlv
  volatile.initial_source: /dev/mapper/ubuntu--vg-lxdlv
description: ""
name: default
driver: btrfs
used_by: []
status: Created
locations:
- none

uli@ulicsl:~$ lxc storage delete default
Storage pool default deleted

Hier die Anpassungen bei “lxc profile edit…”:

1
2
3
4
5
6
7
8
9
10
11
12
--- /tmp/default-ko 2021-12-04 12:46:11.742362472 +0100
+++ /tmp/default-ok  2021-12-04 12:46:00.715994312 +0100
@@ -5,9 +5,5 @@
     name: eth0
     network: lxdnat
     type: nic
-  root:
-    path: /
-    pool: default
-    type: disk
 name: default
 used_by: []

LXD neu initialisieren

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
uli@ulicsl:~ lxd init
Would you like to use LXD clustering? (yes/no) [default=no]:
Do you want to configure a new storage pool? (yes/no) [default=yes]:
Name of the new storage pool [default=default]:
Name of the storage backend to use (ceph, btrfs, dir, lvm, zfs) [default=zfs]: btrfs
Create a new BTRFS pool? (yes/no) [default=yes]: no
Name of the existing BTRFS pool or dataset: /dev/mapper/ubuntu--vg-lxdlv
Would you like to connect to a MAAS server? (yes/no) [default=no]:
Would you like to create a new local network bridge? (yes/no) [default=yes]: no
Would you like to configure LXD to use an existing bridge or host interface? (yes/no) [default=no]:
Would you like the LXD server to be available over the network? (yes/no) [default=no]:
Would you like stale cached images to be updated automatically? (yes/no) [default=yes]
Would you like a YAML "lxd init" preseed to be printed? (yes/no) [default=no]: yes


config: {}
networks: []
storage_pools:
- config:
    source: /dev/mapper/ubuntu--vg-lxdlv
  description: ""
  name: default
  driver: btrfs
profiles:
- config: {}
  description: ""
  devices:
    root:
      path: /
      pool: default
      type: disk
  name: default
projects: []
cluster: null

Container neu anlegen

Nun klappt’s:

1
2
3
uli@ulicsl:~$ lxc launch ubuntu:20.04 ubuntu-2004
Creating ubuntu-2004
Starting ubuntu-2004

Historie und Anmerkung

  • 2021-12-03: Erste Version