공유 스토리지(RBD) 가 있는 두 개의 컴퓨팅 노드 1과 2가 있고 Ceph실시간 마이그레이션을 설정하려고 하는데 다음 오류로 인해 실패합니다. 무엇이 잘못되었는지는 알 수 없습니다.
Openstack 파이크 16.0.16을 사용하고 있습니다.
[root@compute-01 instances]# cat /etc/libvirt/libvirtd.conf
# Ansible managed
listen_tls = 0
listen_tcp = 1
unix_sock_group = "libvirt"
unix_sock_ro_perms = "0777"
unix_sock_rw_perms = "0770"
auth_unix_ro = "none"
auth_unix_rw = "none"
auth_tcp = "none"
nova.log에 다음 오류가 나타납니다.
VM 실시간 마이그레이션을 처음 수행하면 작동하지만 다음 오류가 발생하고 VM이 Error상태 로 전환됩니다.
C1 ----> C2 (처음에는 작동하지만 오류가 있음)
lt] [instance: 4f4009ee-902d-4ee9-ae99-e9bc55267b32] Received unexpected event network-vif-unplugged-251b70a9-2118-4f95-8b35-e9e52f4392e7 for instance
2018-07-25 17:32:39.573 2833 ERROR nova.compute.manager [req-e0cd3865-151e-4d07-8b94-3a8943dafb57 78ca592a663a487d982a4c412ce4d52e d0d9c227e2d34d3a941e3cd16dea06ed - default default] [instance: 4f4009ee-902d-4ee9-ae99-e9bc55267b32] Live migration failed.: AttributeError: 'Guest' object has no attribute 'migrate_configure_max_speed'
2018-07-25 17:32:39.573 2833 ERROR nova.compute.manager [instance: 4f4009ee-902d-4ee9-ae99-e9bc55267b32] Traceback (most recent call last):
2018-07-25 17:32:39.573 2833 ERROR nova.compute.manager [instance: 4f4009ee-902d-4ee9-ae99-e9bc55267b32] File "/openstack/venvs/nova-16.0.16/lib/python2.7/site-packages/nova/compute/manager.py", line 5580, in _do_live_migration
2018-07-25 17:32:39.573 2833 ERROR nova.compute.manager [instance: 4f4009ee-902d-4ee9-ae99-e9bc55267b32] block_migration, migrate_data)
2018-07-25 17:32:39.573 2833 ERROR nova.compute.manager [instance: 4f4009ee-902d-4ee9-ae99-e9bc55267b32] File "/openstack/venvs/nova-16.0.16/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6436, in live_migration
2018-07-25 17:32:39.573 2833 ERROR nova.compute.manager [instance: 4f4009ee-902d-4ee9-ae99-e9bc55267b32] migrate_data)
2018-07-25 17:32:39.573 2833 ERROR nova.compute.manager [instance: 4f4009ee-902d-4ee9-ae99-e9bc55267b32] File "/openstack/venvs/nova-16.0.16/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6944, in _live_migration
2018-07-25 17:32:39.573 2833 ERROR nova.compute.manager [instance: 4f4009ee-902d-4ee9-ae99-e9bc55267b32] guest.migrate_configure_max_speed(
2018-07-25 17:32:39.573 2833 ERROR nova.compute.manager [instance: 4f4009ee-902d-4ee9-ae99-e9bc55267b32] AttributeError: 'Guest' object has no attribute 'migrate_configure_max_speed'
2018-07-25 17:32:39.573 2833 ERROR nova.compute.manager [instance: 4f4009ee-902d-4ee9-ae99-e9bc55267b32]
2018-07-25 17:32:41.646 2833 WARNING nova.compute.manager [req-eb9e883f-08c3-427d-89c3-cdcf012e7c8b 78ca592a663a487d982a4c412ce4d52e d0d9c227e2d34d3a941e3cd16dea06ed - default default] [instance: 4f4009ee-902d-4ee9-ae99-e9bc55267b32] Received unexpected event network-vif-plugged-251b70a9-2118-4f95-8b35-e9e52f4392e7 for instance
2018-07-25 17:32:49.516 2833 WARNING nova.compute.manager [req-d70c12f9-42fc-43be-ae8e-6dd6b21b1b1f 78ca592a663a487d982a4c412ce4d52e d0d9c227e2d34d3a941e3cd16dea06ed - default default] [instance: 4f4009ee-902d-4ee9-ae99-e9bc55267b32] Received unexpected event network-vif-plugged-251b70a9-2118-4f95-8b35-e9e52f4392e7 for instance
가상 머신의 오류 상태를 수정하려면 이렇게 해야 했습니다.
[root@ostack-infra-01-utility-container-a8dbff46 ~]# nova list
+--------------------------------------+------+--------+------------+-------------+-----------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+------+--------+------------+-------------+-----------------------+
| 4f4009ee-902d-4ee9-ae99-e9bc55267b32 | d1 | ERROR | - | NOSTATE | net-vlan31=10.31.1.10 |
+--------------------------------------+------+--------+------------+-------------+-----------------------+
nova reset-state --active 4f4009ee-902d-4ee9-ae99-e9bc55267b32
마이그레이션이 성공한 후에도 Horizon 대시보드에는 여전히 열려 있는 것으로 표시됩니다.C1
이제 VM이 C2에서 실행 중이고 C1으로 다시 이동해야 합니다. 다음 오류가 발생합니다. VM을 마이그레이션한 후 nova가 파일을 정리하지 않은 것 같습니다. 또는 상태가 잘못되었기 때문일 수도 있습니다. 이전 파일을 정리하지 않았습니다.
2018-07-25 19:42:04.143 8785 ERROR nova.compute.manager [req-285a7f42-99a1-47a2-86c1-79140fcfca31 040960f6067d42c2b52c3fcac9ebde6d 2349c3efbf8a4c6ba6dc3b961160c81b - default default] [instance: aa58095d-7027-488e-901e-f3259353de0d] Pre live migration failed at ostack-compute-02.v1v0x.net: DestinationDiskExists_Remote: The supplied disk path (/var/lib/nova/instances/aa58095d-7027-488e-901e-f3259353de0d) already exists, it is expected not to exist.
Traceback (most recent call last):
File "/openstack/venvs/nova-16.0.16/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 160, in _process_incoming
res = self.dispatcher.dispatch(message)
File "/openstack/venvs/nova-16.0.16/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 213, in dispatch
return self._do_dispatch(endpoint, method, ctxt, args)
File "/openstack/venvs/nova-16.0.16/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _do_dispatch
result = func(ctxt, **new_args)
File "/openstack/venvs/nova-16.0.16/lib/python2.7/site-packages/nova/exception_wrapper.py", line 76, in wrapped
function_name, call_dict, binary)
File "/openstack/venvs/nova-16.0.16/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
self.force_reraise()
File "/openstack/venvs/nova-16.0.16/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
six.reraise(self.type_, self.value, self.tb)
File "/openstack/venvs/nova-16.0.16/lib/python2.7/site-packages/nova/exception_wrapper.py", line 67, in wrapped
return f(self, context, *args, **kw)
File "/openstack/venvs/nova-16.0.16/lib/python2.7/site-packages/nova/compute/utils.py", line 880, in decorated_function
return function(self, context, *args, **kwargs)
File "/openstack/venvs/nova-16.0.16/lib/python2.7/site-packages/nova/compute/manager.py", line 218, in decorated_function
kwargs['instance'], e, sys.exc_info())
File "/openstack/venvs/nova-16.0.16/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
self.force_reraise()
File "/openstack/venvs/nova-16.0.16/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
six.reraise(self.type_, self.value, self.tb)
File "/openstack/venvs/nova-16.0.16/lib/python2.7/site-packages/nova/compute/manager.py", line 206, in decorated_function
return function(self, context, *args, **kwargs)
File "/openstack/venvs/nova-16.0.16/lib/python2.7/site-packages/nova/compute/manager.py", line 5507, in pre_live_migration
migrate_data)
File "/openstack/venvs/nova-16.0.16/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7072, in pre_live_migration
raise exception.DestinationDiskExists(path=instance_dir)
DestinationDiskExists: The supplied disk path (/var/lib/nova/instances/aa58095d-7027-488e-901e-f3259353de0d) already exists, it is expected not to exist.
2018-07-25 19:42:04.143 8785 ERROR nova.compute.manager [instance: aa58095d-7027-488e-901e-f3259353de0d] Traceback (most recent call last):
2018-07-25 19:42:04.143 8785 ERROR nova.compute.manager [instance: aa58095d-7027-488e-901e-f3259353de0d] File "/openstack/venvs/nova-16.0.16/lib/python2.7/site-packages/nova/compute/manager.py", line 5562, in _do_live_migration
2018-07-25 19:42:04.143 8785 ERROR nova.compute.manager [instance: aa58095d-7027-488e-901e-f3259353de0d] block_migration, disk, dest, migrate_data)
2018-07-25 19:42:04.143 8785 ERROR nova.compute.manager [instance: aa58095d-7027-488e-901e-f3259353de0d] File "/openstack/venvs/nova-16.0.16/lib/python2.7/site-packages/nova/compute/rpcapi.py", line 745, in pre_live_migration
2018-07-25 19:42:04.143 8785 ERROR nova.compute.manager [instance: aa58095d-7027-488e-901e-f3259353de0d] disk=disk, migrate_data=migrate_data)
2018-07-25 19:42:04.143 8785 ERROR nova.compute.manager [instance: aa58095d-7027-488e-901e-f3259353de0d] File "/openstack/venvs/nova-16.0.16/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 169, in call
2018-07-25 19:42:04.143 8785 ERROR nova.compute.manager [instance: aa58095d-7027-488e-901e-f3259353de0d] retry=self.retry)
2018-07-25 19:42:04.143 8785 ERROR nova.compute.manager [instance: aa58095d-7027-488e-901e-f3259353de0d] File "/openstack/venvs/nova-16.0.16/lib/python2.7/site-packages/oslo_messaging/transport.py", line 123, in _send
2018-07-25 19:42:04.143 8785 ERROR nova.compute.manager [instance: aa58095d-7027-488e-901e-f3259353de0d] timeout=timeout, retry=retry)
2018-07-25 19:42:04.143 8785 ERROR nova.compute.manager [instance: aa58095d-7027-488e-901e-f3259353de0d] File "/openstack/venvs/nova-16.0.16/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 566, in send
2018-07-25 19:42:04.143 8785 ERROR nova.compute.manager [instance: aa58095d-7027-488e-901e-f3259353de0d] retry=retry)
2018-07-25 19:42:04.143 8785 ERROR nova.compute.manager [instance: aa58095d-7027-488e-901e-f3259353de0d] File "/openstack/venvs/nova-16.0.16/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 557, in _send
2018-07-25 19:42:04.143 8785 ERROR nova.compute.manager [instance: aa58095d-7027-488e-901e-f3259353de0d] raise result
2018-07-25 19:42:04.143 8785 ERROR nova.compute.manager [instance: aa58095d-7027-488e-901e-f3259353de0d] DestinationDiskExists_Remote: The supplied disk path (/var/lib/nova/instances/aa58095d-7027-488e-901e-f3259353de0d) already exists, it is expected not to exist.
2018-07-25 19:42:04.143 8785 ERROR nova.compute.manager [instance: aa58095d-7027-488e-901e-f3259353de0d] Traceback (most recent call last):
2018-07-25 19:42:04.143 8785 ERROR nova.compute.manager [instance: aa58095d-7027-488e-901e-f3259353de0d]
2018-07-25 19:42:04.143 8785 ERROR nova.compute.manager [instance: aa58095d-7027-488e-901e-f3259353de0d] File "/openstack/venvs/nova-16.0.16/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 160, in _process_incoming
2018-07-25 19:42:04.143 8785 ERROR nova.compute.manager [instance: aa58095d-7027-488e-901e-f3259353de0d] res = self.dispatcher.dispatch(message)
2018-07-25 19:42:04.143 8785 ERROR nova.compute.manager [instance: aa58095d-7027-488e-901e-f3259353de0d]
2018-07-25 19:42:04.143 8785 ERROR nova.compute.manager [instance: aa58095d-7027-488e-901e-f3259353de0d] File "/openstack/venvs/nova-16.0.16/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 213, in dispatch
2018-07-25 19:42:04.143 8785 ERROR nova.compute.manager [instance: aa58095d-7027-488e-901e-f3259353de0d] return self._do_dispatch(endpoint, method, ctxt, args)
2018-07-25 19:42:04.143 8785 ERROR nova.compute.manager [instance: aa58095d-7027-488e-901e-f3259353de0d]
2018-07-25 19:42:04.143 8785 ERROR nova.compute.manager [instance: aa58095d-7027-488e-901e-f3259353de0d] File "/openstack/venvs/nova-16.0.16/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _do_dispatch
2018-07-25 19:42:04.143 8785 ERROR nova.compute.manager [instance: aa58095d-7027-488e-901e-f3259353de0d] result = func(ctxt, **new_args)
2018-07-25 19:42:04.143 8785 ERROR nova.compute.manager [instance: aa58095d-7027-488e-901e-f3259353de0d]
2018-07-25 19:42:04.143 8785 ERROR nova.compute.manager [instance: aa58095d-7027-488e-901e-f3259353de0d] File "/openstack/venvs/nova-16.0.16/lib/python2.7/site-packages/nova/exception_wrapper.py", line 76, in wrapped
2018-07-25 19:42:04.143 8785 ERROR nova.compute.manager [instance: aa58095d-7027-488e-901e-f3259353de0d] function_name, call_dict, binary)
2018-07-25 19:42:04.143 8785 ERROR nova.compute.manager [instance: aa58095d-7027-488e-901e-f3259353de0d]
2018-07-25 19:42:04.143 8785 ERROR nova.compute.manager [instance: aa58095d-7027-488e-901e-f3259353de0d] File "/openstack/venvs/nova-16.0.16/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-07-25 19:42:04.143 8785 ERROR nova.compute.manager [instance: aa58095d-7027-488e-901e-f3259353de0d] self.force_reraise()
2018-07-25 19:42:04.143 8785 ERROR nova.compute.manager [instance: aa58095d-7027-488e-901e-f3259353de0d]
2018-07-25 19:42:04.143 8785 ERROR nova.compute.manager [instance: aa58095d-7027-488e-901e-f3259353de0d] File "/openstack/venvs/nova-16.0.16/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-07-25 19:42:04.143 8785 ERROR nova.compute.manager [instance: aa58095d-7027-488e-901e-f3259353de0d] six.reraise(self.type_, self.value, self.tb)
2018-07-25 19:42:04.143 8785 ERROR nova.compute.manager [instance: aa58095d-7027-488e-901e-f3259353de0d]
2018-07-25 19:42:04.143 8785 ERROR nova.compute.manager [instance: aa58095d-7027-488e-901e-f3259353de0d] File "/openstack/venvs/nova-16.0.16/lib/python2.7/site-packages/nova/exception_wrapper.py", line 67, in wrapped
2018-07-25 19:42:04.143 8785 ERROR nova.compute.manager [instance: aa58095d-7027-488e-901e-f3259353de0d] return f(self, context, *args, **kw)
2018-07-25 19:42:04.143 8785 ERROR nova.compute.manager [instance: aa58095d-7027-488e-901e-f3259353de0d]
2018-07-25 19:42:04.143 8785 ERROR nova.compute.manager [instance: aa58095d-7027-488e-901e-f3259353de0d] File "/openstack/venvs/nova-16.0.16/lib/python2.7/site-packages/nova/compute/utils.py", line 880, in decorated_function
2018-07-25 19:42:04.143 8785 ERROR nova.compute.manager [instance: aa58095d-7027-488e-901e-f3259353de0d] return function(self, context, *args, **kwargs)
2018-07-25 19:42:04.143 8785 ERROR nova.compute.manager [instance: aa58095d-7027-488e-901e-f3259353de0d]
2018-07-25 19:42:04.143 8785 ERROR nova.compute.manager [instance: aa58095d-7027-488e-901e-f3259353de0d] File "/openstack/venvs/nova-16.0.16/lib/python2.7/site-packages/nova/compute/manager.py", line 218, in decorated_function
2018-07-25 19:42:04.143 8785 ERROR nova.compute.manager [instance: aa58095d-7027-488e-901e-f3259353de0d] kwargs['instance'], e, sys.exc_info())
2018-07-25 19:42:04.143 8785 ERROR nova.compute.manager [instance: aa58095d-7027-488e-901e-f3259353de0d]
2018-07-25 19:42:04.143 8785 ERROR nova.compute.manager [instance: aa58095d-7027-488e-901e-f3259353de0d] File "/openstack/venvs/nova-16.0.16/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-07-25 19:42:04.143 8785 ERROR nova.compute.manager [instance: aa58095d-7027-488e-901e-f3259353de0d] self.force_reraise()
2018-07-25 19:42:04.143 8785 ERROR nova.compute.manager [instance: aa58095d-7027-488e-901e-f3259353de0d]
2018-07-25 19:42:04.143 8785 ERROR nova.compute.manager [instance: aa58095d-7027-488e-901e-f3259353de0d] File "/openstack/venvs/nova-16.0.16/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-07-25 19:42:04.143 8785 ERROR nova.compute.manager [instance: aa58095d-7027-488e-901e-f3259353de0d] six.reraise(self.type_, self.value, self.tb)
2018-07-25 19:42:04.143 8785 ERROR nova.compute.manager [instance: aa58095d-7027-488e-901e-f3259353de0d]
2018-07-25 19:42:04.143 8785 ERROR nova.compute.manager [instance: aa58095d-7027-488e-901e-f3259353de0d] File "/openstack/venvs/nova-16.0.16/lib/python2.7/site-packages/nova/compute/manager.py", line 206, in decorated_function
2018-07-25 19:42:04.143 8785 ERROR nova.compute.manager [instance: aa58095d-7027-488e-901e-f3259353de0d] return function(self, context, *args, **kwargs)
2018-07-25 19:42:04.143 8785 ERROR nova.compute.manager [instance: aa58095d-7027-488e-901e-f3259353de0d]
2018-07-25 19:42:04.143 8785 ERROR nova.compute.manager [instance: aa58095d-7027-488e-901e-f3259353de0d] File "/openstack/venvs/nova-16.0.16/lib/python2.7/site-packages/nova/compute/manager.py", line 5507, in pre_live_migration
2018-07-25 19:42:04.143 8785 ERROR nova.compute.manager [instance: aa58095d-7027-488e-901e-f3259353de0d] migrate_data)
2018-07-25 19:42:04.143 8785 ERROR nova.compute.manager [instance: aa58095d-7027-488e-901e-f3259353de0d]
2018-07-25 19:42:04.143 8785 ERROR nova.compute.manager [instance: aa58095d-7027-488e-901e-f3259353de0d] File "/openstack/venvs/nova-16.0.16/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7072, in pre_live_migration
2018-07-25 19:42:04.143 8785 ERROR nova.compute.manager [instance: aa58095d-7027-488e-901e-f3259353de0d] raise exception.DestinationDiskExists(path=instance_dir)
2018-07-25 19:42:04.143 8785 ERROR nova.compute.manager [instance: aa58095d-7027-488e-901e-f3259353de0d]
2018-07-25 19:42:04.143 8785 ERROR nova.compute.manager [instance: aa58095d-7027-488e-901e-f3259353de0d] DestinationDiskExists: The supplied disk path (/var/lib/nova/instances/aa58095d-7027-488e-901e-f3259353de0d) already exists, it is expected not to exist.
2018-07-25 19:42:04.143 8785 ERROR nova.compute.manager [instance: aa58095d-7027-488e-901e-f3259353de0d]
2018-07-25 19:42:04.143 8785 ERROR nova.compute.manager [instance: aa58095d-7027-488e-901e-f3259353de0d]
2018-07-25 19:42:06.908 8785 WARNING nova.compute.manager [req-cda02056-75c7-463c-ac7e-2925ba2cd29c 78ca592a663a487d982a4c412ce4d52e d0d9c227e2d34d3a941e3cd16dea06ed - default default] [instance: aa58095d-7027-488e-901e-f3259353de0d] Received unexpected event network-vif-unplugged-7a356ab1-e0d3-4a69-9aa9-f71329caa17f for instance
2018-07-25 19:42:07.821 8785 ERROR nova.virt.libvirt.driver [req-cda02056-75c7-463c-ac7e-2925ba2cd29c 78ca592a663a487d982a4c412ce4d52e d0d9c227e2d34d3a941e3cd16dea06ed - default default] [instance: aa58095d-7027-488e-901e-f3259353de0d] Live Migration failure: Domain not found: no domain with matching name 'instance-00000056': libvirtError: Domain not found: no domain with matching name 'instance-00000056'
2018-07-25 19:42:08.294 8785 WARNING nova.compute.manager [req-5d3abd38-c8a9-499d-b97f-1d9e748a19d3 78ca592a663a487d982a4c412ce4d52e d0d9c227e2d34d3a941e3cd16dea06ed - default default] [instance: aa58095d-7027-488e-901e-f3259353de0d] Received unexpected event network-vif-plugged-7a356ab1-e0d3-4a69-9aa9-f71329caa17f for instance
2018-07-25 19:42:14.934 8785 WARNING nova.compute.manager [req-e9442ba9-1700-4221-a495-31d76a2b5bf0 78ca592a663a487d982a4c412ce4d52e d0d9c227e2d34d3a941e3cd16dea06ed - default default] [instance: aa58095d-7027-488e-901e-f3259353de0d] Received unexpected event network-vif-plugged-7a356ab1-e0d3-4a69-9aa9-f71329caa17f for instance
답변1
원래 오류는 libvirt 드라이버 호출에서 발생합니다 guest.migrate_configure_max_speed.
https://github.com/openstack/nova/blob/stable/pike/nova/virt/libvirt/driver.py#L6944
파이크까지 (https://github.com/openstack/nova/commit/23446a9552b5be3b040278646149a0f481d0a005) 그리고 여기에 누락되었기 때문에 Pike로 백포트되지 않은 것 같습니다.
https://github.com/openstack/nova/blob/stable/pike/nova/virt/libvirt/guest.py
guest.migrate_configure_max_speed이 커밋에서는 드라이버에 대한 호출이 다음에 추가되었습니다.https://github.com/openstack/nova/commit/ff747792b8f5aefe1bebb01bdf49dacc01353348
no-op를 사용하여 guest.py를 수동으로 패치하여 오류를 수정할 수 있습니다.
def migrate_configure_max_speed(self, bandwidth):
pass