opencontainers / opencontainers/runc

Checkpoint fails when using NFS

Open
#2,170 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
13.5k
Forks
2.3k
Avg merge
2d 8h
Merged PRs (30d)
30

Description

Hey :)

Without NFS I succesfully migrated a MySQL container (runc) from host A to host B (I use scp to copy needed data after the dump). Now I want to use NFS to minimalise the downtime. Is it possible?

When container starts, it initializes data. The data goes into a mounted directory and automatically transfers to both hosts. Till here everything works fine. But when I start the checkpoint, it throws an error :/

First I do a pre-dump -> works. After, the normal dump ->error

Error (criu/mount.c:947): mnt: Mount 730 ./run/mysqld (master_id: 428 shared_id: 0) has unreachable sharing. Try --enable-external-masters.

Added the flag "enable-external-masters" in the /etc/criu/runc.conf -> no effect

mount
...
192.168.0.129:/home/vm1/Desktop/nfs on /home/vm1/Desktop/nfs type nfs4 (rw,relatime,vers=4.2,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.0.164,local_lock=none,addr=192.168.0.129)

I hope it make sense what I did here. Also tried with"type": "nfs" -> same result.
Maybe I need to add something to the seccomp section? Some syscall maybe

config.json

"mounts": [
		{
		    "destination": "/var/run/mysqld",
		    "type": "bind",
		    "source": "/home/vm1/Desktop/nfs",
		    "options": ["rw","rbind"]
		},
		{
		    "destination": "/tmp",
		    "type": "bind",
		    "source": "/home/vm1/Desktop/nfs",
		    "options": ["rw","rbind"]
		},
		{
		    "destination": "/var/lib/mysql",
		    "type": "bind",
		    "source": "/home/vm1/Desktop/nfs",
		    "options": ["rw","rbind"]
		}
                ...
]
runc -v
runc version 1.0.0-rc8+dev
commit: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
spec: 1.0.1-dev

dump.log:

(00.004902) 	type ext4 source /dev/sda1 mnt_id 729 s_dev 0x800001 /home/vm1/Desktop/runc/mysql/rootfs @ ./ flags 0x280001 options errors=remount-ro
(00.004917) 	type nfs4 source 192.168.0.129:/home/vm1/Desktop/nfs mnt_id 730 s_dev 0x33 / @ ./run/mysqld flags 0x280000 options vers=4.2,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.0.164,local_lock=none,addr=192.168.0.129
(00.004924) 	type nfs4 source 192.168.0.129:/home/vm1/Desktop/nfs mnt_id 731 s_dev 0x33 / @ ./tmp flags 0x280000 options vers=4.2,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.0.164,local_lock=none,addr=192.168.0.129
(00.004931) 	type nfs4 source 192.168.0.129:/home/vm1/Desktop/nfs mnt_id 732 s_dev 0x33 / @ ./var/lib/mysql flags 0x280000 options vers=4.2,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.0.164,local_lock=none,addr=192.168.0.129
(00.004937) 	type ext4 source /dev/sda1 mnt_id 733 s_dev 0x800001 /home/vm1/Desktop/docker/dump @ ./docker-entrypoint-initdb.d flags 0x280000 options errors=remount-ro
(00.004950) 	type proc source proc mnt_id 734 s_dev 0x34 / @ ./proc flags 0x20000e options 
(00.004954) 	type tmpfs source tmpfs mnt_id 735 s_dev 0x35 / @ ./dev flags 0x1000002 options size=65536k,mode=755
(00.004959) 	type devpts source devpts mnt_id 736 s_dev 0x36 / @ ./dev/pts flags 0x20000a options gid=5,mode=620,ptmxmode=666
(00.004968) 	type tmpfs source shm mnt_id 737 s_dev 0x37 / @ ./dev/shm flags 0x20000e options size=65536k
(00.004972) 	type mqueue source mqueue mnt_id 738 s_dev 0x32 / @ ./dev/mqueue flags 0x20000e options 
(00.004982) 	type sysfs source sysfs mnt_id 739 s_dev 0x38 / @ ./sys flags 0x20000f options 
(00.004986) mnt: Building mountpoints tree
(00.004988) mnt: 	Building plain mount tree
(00.004989) mnt: 		Working on 739->729
(00.004990) mnt: 		Working on 738->735
(00.004992) mnt: 		Working on 737->735
(00.004993) mnt: 		Working on 736->735
(00.004994) mnt: 		Working on 735->729
(00.004995) mnt: 		Working on 734->729
(00.004996) mnt: 		Working on 733->729
(00.004998) mnt: 		Working on 732->729
(00.004999) mnt: 		Working on 731->729
(00.005000) mnt: 		Working on 730->729
(00.005001) mnt: 		Working on 729->611
(00.005002) mnt: 	Resorting siblings on 729
(00.005004) mnt: 	Resorting siblings on 739
(00.005005) mnt: 	Resorting siblings on 735
(00.005006) mnt: 	Resorting siblings on 738
(00.005007) mnt: 	Resorting siblings on 737
(00.005009) mnt: 	Resorting siblings on 736
(00.005011) mnt: 	Resorting siblings on 734
(00.005013) mnt: 	Resorting siblings on 733
(00.005014) mnt: 	Resorting siblings on 732
(00.005015) mnt: 	Resorting siblings on 731
(00.005017) mnt: 	Resorting siblings on 730
(00.005019) mnt: Done:
(00.005020) mnt: [./](729->611)
(00.005022) mnt:  [./var/lib/mysql](732->729)
(00.005024) mnt:  <--
(00.005025) mnt:  [./run/mysqld](730->729)
(00.005026) mnt:  <--
(00.005027) mnt:  [./sys](739->729)
(00.005028) mnt:  <--
(00.005030) mnt:  [./dev](735->729)
(00.005031) mnt:   [./dev/mqueue](738->735)
(00.005032) mnt:   <--
(00.005033) mnt:   [./dev/shm](737->735)
(00.005035) mnt:   <--
(00.005036) mnt:   [./dev/pts](736->735)
(00.005037) mnt:   <--
(00.005038) mnt:  <--
(00.005039) mnt:  [./proc](734->729)
(00.005040) mnt:  <--
(00.005042) mnt:  [./docker-entrypoint-initdb.d](733->729)
(00.005043) mnt:  <--
(00.005044) mnt:  [./tmp](731->729)
(00.005045) mnt:  <--
(00.005046) mnt: <--
(00.005051) mnt: Found /docker-entrypoint-initdb.d mapping for ./docker-entrypoint-initdb.d mountpoint
(00.005053) mnt: Found /var/lib/mysql mapping for ./var/lib/mysql mountpoint
(00.005054) mnt: Found /tmp mapping for ./tmp mountpoint
(00.005056) mnt: Inspecting sharing on 739 shared_id 0 master_id 0 (@./sys)
(00.005058) mnt: Inspecting sharing on 738 shared_id 0 master_id 0 (@./dev/mqueue)
(00.005059) mnt: Inspecting sharing on 737 shared_id 0 master_id 0 (@./dev/shm)
(00.005060) mnt: Inspecting sharing on 736 shared_id 0 master_id 0 (@./dev/pts)
(00.005062) mnt: Inspecting sharing on 735 shared_id 0 master_id 0 (@./dev)
(00.005063) mnt: Inspecting sharing on 734 shared_id 0 master_id 0 (@./proc)
(00.005064) mnt: Inspecting sharing on 733 shared_id 0 master_id -1 (@./docker-entrypoint-initdb.d)
(00.005066) mnt: 	The mount 729 is bind for 733 (@./ -> @./docker-entrypoint-initdb.d)
(00.005067) mnt: Inspecting sharing on 732 shared_id 0 master_id 428 (@./var/lib/mysql)
(00.005069) mnt: 	The mount 731 is bind for 732 (@./tmp -> @./var/lib/mysql)
(00.005070) mnt: 	The mount 730 is bind for 732 (@./run/mysqld -> @./var/lib/mysql)
(00.005071) mnt: Inspecting sharing on 731 shared_id 0 master_id 428 (@./tmp)
(00.005073) mnt: Inspecting sharing on 730 shared_id 0 master_id 428 (@./run/mysqld)
(00.005074) **Error (criu/mount.c:947): mnt: Mount 730 ./run/mysqld (master_id: 428 shared_id: 0) has unreachable sharing. Try --enable-external-masters.**
(00.005082) Unlock network
(00.005083) Running network-unlock scripts
(00.005085) 	RPC
iptables-restore: invalid option -- 'w'
ip6tables-restore: invalid option -- 'w'
(00.006771) Unfreezing tasks into 1
(00.006780) 	Unseizing 6668 into 1
(00.006971) Error (criu/cr-dump.c:1743): Dumping FAILED.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the checkpoint failure at criu/mount.c:947 and the NFS bind mounts shown in config.json and dump.log. Compare the effect of /etc/criu/runc.conf with the reported runc 1.0.0-rc8+dev environment, then determine whether checkpointing completes without the unreachable-sharing error.

Written by the indexing model from the issue text.

Assessment

Tech stack
mysql
Domain
infrastructure
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.