opencontainers / opencontainers/runc
Checkpoint fails when using NFS
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 13.5k
- Forks
- 2.3k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 30
Description
Hey :)
Without NFS I succesfully migrated a MySQL container (runc) from host A to host B (I use scp to copy needed data after the dump). Now I want to use NFS to minimalise the downtime. Is it possible?
When container starts, it initializes data. The data goes into a mounted directory and automatically transfers to both hosts. Till here everything works fine. But when I start the checkpoint, it throws an error :/
First I do a pre-dump -> works. After, the normal dump ->error
Error (criu/mount.c:947): mnt: Mount 730 ./run/mysqld (master_id: 428 shared_id: 0) has unreachable sharing. Try --enable-external-masters.
Added the flag "enable-external-masters" in the /etc/criu/runc.conf -> no effect
mount
...
192.168.0.129:/home/vm1/Desktop/nfs on /home/vm1/Desktop/nfs type nfs4 (rw,relatime,vers=4.2,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.0.164,local_lock=none,addr=192.168.0.129)
I hope it make sense what I did here. Also tried with"type": "nfs" -> same result.
Maybe I need to add something to the seccomp section? Some syscall maybe
config.json
"mounts": [
{
"destination": "/var/run/mysqld",
"type": "bind",
"source": "/home/vm1/Desktop/nfs",
"options": ["rw","rbind"]
},
{
"destination": "/tmp",
"type": "bind",
"source": "/home/vm1/Desktop/nfs",
"options": ["rw","rbind"]
},
{
"destination": "/var/lib/mysql",
"type": "bind",
"source": "/home/vm1/Desktop/nfs",
"options": ["rw","rbind"]
}
...
]
runc -v
runc version 1.0.0-rc8+dev
commit: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
spec: 1.0.1-dev
dump.log:
(00.004902) type ext4 source /dev/sda1 mnt_id 729 s_dev 0x800001 /home/vm1/Desktop/runc/mysql/rootfs @ ./ flags 0x280001 options errors=remount-ro
(00.004917) type nfs4 source 192.168.0.129:/home/vm1/Desktop/nfs mnt_id 730 s_dev 0x33 / @ ./run/mysqld flags 0x280000 options vers=4.2,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.0.164,local_lock=none,addr=192.168.0.129
(00.004924) type nfs4 source 192.168.0.129:/home/vm1/Desktop/nfs mnt_id 731 s_dev 0x33 / @ ./tmp flags 0x280000 options vers=4.2,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.0.164,local_lock=none,addr=192.168.0.129
(00.004931) type nfs4 source 192.168.0.129:/home/vm1/Desktop/nfs mnt_id 732 s_dev 0x33 / @ ./var/lib/mysql flags 0x280000 options vers=4.2,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.0.164,local_lock=none,addr=192.168.0.129
(00.004937) type ext4 source /dev/sda1 mnt_id 733 s_dev 0x800001 /home/vm1/Desktop/docker/dump @ ./docker-entrypoint-initdb.d flags 0x280000 options errors=remount-ro
(00.004950) type proc source proc mnt_id 734 s_dev 0x34 / @ ./proc flags 0x20000e options
(00.004954) type tmpfs source tmpfs mnt_id 735 s_dev 0x35 / @ ./dev flags 0x1000002 options size=65536k,mode=755
(00.004959) type devpts source devpts mnt_id 736 s_dev 0x36 / @ ./dev/pts flags 0x20000a options gid=5,mode=620,ptmxmode=666
(00.004968) type tmpfs source shm mnt_id 737 s_dev 0x37 / @ ./dev/shm flags 0x20000e options size=65536k
(00.004972) type mqueue source mqueue mnt_id 738 s_dev 0x32 / @ ./dev/mqueue flags 0x20000e options
(00.004982) type sysfs source sysfs mnt_id 739 s_dev 0x38 / @ ./sys flags 0x20000f options
(00.004986) mnt: Building mountpoints tree
(00.004988) mnt: Building plain mount tree
(00.004989) mnt: Working on 739->729
(00.004990) mnt: Working on 738->735
(00.004992) mnt: Working on 737->735
(00.004993) mnt: Working on 736->735
(00.004994) mnt: Working on 735->729
(00.004995) mnt: Working on 734->729
(00.004996) mnt: Working on 733->729
(00.004998) mnt: Working on 732->729
(00.004999) mnt: Working on 731->729
(00.005000) mnt: Working on 730->729
(00.005001) mnt: Working on 729->611
(00.005002) mnt: Resorting siblings on 729
(00.005004) mnt: Resorting siblings on 739
(00.005005) mnt: Resorting siblings on 735
(00.005006) mnt: Resorting siblings on 738
(00.005007) mnt: Resorting siblings on 737
(00.005009) mnt: Resorting siblings on 736
(00.005011) mnt: Resorting siblings on 734
(00.005013) mnt: Resorting siblings on 733
(00.005014) mnt: Resorting siblings on 732
(00.005015) mnt: Resorting siblings on 731
(00.005017) mnt: Resorting siblings on 730
(00.005019) mnt: Done:
(00.005020) mnt: [./](729->611)
(00.005022) mnt: [./var/lib/mysql](732->729)
(00.005024) mnt: <--
(00.005025) mnt: [./run/mysqld](730->729)
(00.005026) mnt: <--
(00.005027) mnt: [./sys](739->729)
(00.005028) mnt: <--
(00.005030) mnt: [./dev](735->729)
(00.005031) mnt: [./dev/mqueue](738->735)
(00.005032) mnt: <--
(00.005033) mnt: [./dev/shm](737->735)
(00.005035) mnt: <--
(00.005036) mnt: [./dev/pts](736->735)
(00.005037) mnt: <--
(00.005038) mnt: <--
(00.005039) mnt: [./proc](734->729)
(00.005040) mnt: <--
(00.005042) mnt: [./docker-entrypoint-initdb.d](733->729)
(00.005043) mnt: <--
(00.005044) mnt: [./tmp](731->729)
(00.005045) mnt: <--
(00.005046) mnt: <--
(00.005051) mnt: Found /docker-entrypoint-initdb.d mapping for ./docker-entrypoint-initdb.d mountpoint
(00.005053) mnt: Found /var/lib/mysql mapping for ./var/lib/mysql mountpoint
(00.005054) mnt: Found /tmp mapping for ./tmp mountpoint
(00.005056) mnt: Inspecting sharing on 739 shared_id 0 master_id 0 (@./sys)
(00.005058) mnt: Inspecting sharing on 738 shared_id 0 master_id 0 (@./dev/mqueue)
(00.005059) mnt: Inspecting sharing on 737 shared_id 0 master_id 0 (@./dev/shm)
(00.005060) mnt: Inspecting sharing on 736 shared_id 0 master_id 0 (@./dev/pts)
(00.005062) mnt: Inspecting sharing on 735 shared_id 0 master_id 0 (@./dev)
(00.005063) mnt: Inspecting sharing on 734 shared_id 0 master_id 0 (@./proc)
(00.005064) mnt: Inspecting sharing on 733 shared_id 0 master_id -1 (@./docker-entrypoint-initdb.d)
(00.005066) mnt: The mount 729 is bind for 733 (@./ -> @./docker-entrypoint-initdb.d)
(00.005067) mnt: Inspecting sharing on 732 shared_id 0 master_id 428 (@./var/lib/mysql)
(00.005069) mnt: The mount 731 is bind for 732 (@./tmp -> @./var/lib/mysql)
(00.005070) mnt: The mount 730 is bind for 732 (@./run/mysqld -> @./var/lib/mysql)
(00.005071) mnt: Inspecting sharing on 731 shared_id 0 master_id 428 (@./tmp)
(00.005073) mnt: Inspecting sharing on 730 shared_id 0 master_id 428 (@./run/mysqld)
(00.005074) **Error (criu/mount.c:947): mnt: Mount 730 ./run/mysqld (master_id: 428 shared_id: 0) has unreachable sharing. Try --enable-external-masters.**
(00.005082) Unlock network
(00.005083) Running network-unlock scripts
(00.005085) RPC
iptables-restore: invalid option -- 'w'
ip6tables-restore: invalid option -- 'w'
(00.006771) Unfreezing tasks into 1
(00.006780) Unseizing 6668 into 1
(00.006971) Error (criu/cr-dump.c:1743): Dumping FAILED.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the checkpoint failure at criu/mount.c:947 and the NFS bind mounts shown in config.json and dump.log. Compare the effect of /etc/criu/runc.conf with the reported runc 1.0.0-rc8+dev environment, then determine whether checkpointing completes without the unreachable-sharing error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- mysql
- Domain
- infrastructure
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100