[Error][Centos7/Openstack]Openstack

2023.10.06

 

저번에 Comptue에 Nova를 설치하고 나서 마지막 단계에서 문제가 생겼다.

# systemctl enable libvirtd.service openstack-nova-compute.service
# systemctl start libvirtd.service openstack-nova-compute.service

다음과 같은 명령어를 실행핬는데 실행이 되지 않았다.

 

상태를 확인해보니 다음과 같은 상태였다

[root@compute ~]# systemctl status libvirtd.service openstack-nova-compute.service
● libvirtd.service - Virtualization daemon
   Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled)   Active: active (running) since Sat 2023-09-30 06:13:51 KST; 10s ago
     Docs: man:libvirtd(8)
           https://libvirt.org
 Main PID: 20150 (libvirtd)
    Tasks: 17 (limit: 32768)
   CGroup: /system.slice/libvirtd.service
           └─20150 /usr/sbin/libvirtd

Sep 30 06:13:41 compute systemd[1]: Starting Virtualization daemon...
Sep 30 06:13:51 compute systemd[1]: Started Virtualization daemon.
Sep 30 06:13:52 compute libvirtd[20150]: 2023-09-29 21:13:52.036+0000: 20167: info : li...rg)Sep 30 06:13:52 compute libvirtd[20150]: 2023-09-29 21:13:52.036+0000: 20167: info : ho...uteSep 30 06:13:52 compute libvirtd[20150]: 2023-09-29 21:13:52.036+0000: 20167: error : v...orySep 30 06:13:52 compute libvirtd[20150]: 2023-09-29 21:13:52.045+0000: 20167: error : v...ory



● openstack-nova-compute.service - OpenStack Nova Compute Server
   Loaded: loaded (/usr/lib/systemd/system/openstack-nova-compute.service; enabled; vendor preset: disabled)
   Active: activating (start) since Sat 2023-09-30 06:13:51 KST; 10s ago
 Main PID: 20168 (nova-compute)
    Tasks: 1
   CGroup: /system.slice/openstack-nova-compute.service
           └─20168 /usr/bin/python2 /usr/bin/nova-compute

Sep 30 06:13:51 compute systemd[1]: Starting OpenStack Nova Compute Server...
Hint: Some lines were ellipsized, use -l to show in full.

openstack-nova-compute.service의 Demon이 실행되지 않는것이었다.  원인을 찾기 위해서 로그를 확인했다.

 

[root@compute ~]# tail /var/log/nova/nova-compute.log
2023-09-30 06:23:56.685 20229 ERROR oslo.messaging._drivers.impl_rabbit [req-115b6b12-7ade-4362-84a1-986cb99e7e34 - - - - -] Connection failed: [Errno 113] EHOSTUNREACH (retrying in 26.0 seconds): error: [Errno 113] EHOSTUNREACH
2023-09-30 06:24:22.736 20229 ERROR oslo.messaging._drivers.impl_rabbit [req-115b6b12-7ade-4362-84a1-986cb99e7e34 - - - - -] Connection failed: [Errno 113] EHOSTUNREACH (retrying in 28.0 seconds): error: [Errno 113] EHOSTUNREACH
2023-09-30 06:24:50.795 20229 ERROR oslo.messaging._drivers.impl_rabbit [req-115b6b12-7ade-4362-84a1-986cb99e7e34 - - - - -] Connection failed: [Errno 113] EHOSTUNREACH (retrying in 30.0 seconds): error: [Errno 113] EHOSTUNREACH
2023-09-30 06:25:20.868 20229 ERROR oslo.messaging._drivers.impl_rabbit [req-115b6b12-7ade-4362-84a1-986cb99e7e34 - - - - -] Connection failed: [Errno 113] EHOSTUNREACH (retrying in 32.0 seconds): error: [Errno 113] EHOSTUNREACH
2023-09-30 06:25:52.946 20229 ERROR oslo.messaging._drivers.impl_rabbit [req-115b6b12-7ade-4362-84a1-986cb99e7e34 - - - - -] Connection failed: [Errno 113] EHOSTUNREACH (retrying in 32.0 seconds): error: [Errno 113] EHOSTUNREACH
2023-09-30 06:26:25.007 20229 ERROR oslo.messaging._drivers.impl_rabbit [req-115b6b12-7ade-4362-84a1-986cb99e7e34 - - - - -] Connection failed: [Errno 113] EHOSTUNREACH (retrying in 32.0 seconds): error: [Errno 113] EHOSTUNREACH
2023-09-30 06:26:58.047 20229 ERROR oslo.messaging._drivers.impl_rabbit [req-115b6b12-7ade-4362-84a1-986cb99e7e34 - - - - -] Connection failed: [Errno 113] EHOSTUNREACH (retrying in 32.0 seconds): error: [Errno 113] EHOSTUNREACH
2023-09-30 06:27:31.128 20229 ERROR oslo.messaging._drivers.impl_rabbit [req-115b6b12-7ade-4362-84a1-986cb99e7e34 - - - - -] Connection failed: [Errno 113] EHOSTUNREACH (retrying in 32.0 seconds): error: [Errno 113] EHOSTUNREACH
2023-09-30 06:28:04.870 20229 ERROR oslo.messaging._drivers.impl_rabbit [req-115b6b12-7ade-4362-84a1-986cb99e7e34 - - - - -] Connection failed: [Errno 113] EHOSTUNREACH (retrying in 32.0 seconds): error: [Errno 113] EHOSTUNREACH
2023-09-30 06:28:37.769 20229 ERROR oslo.messaging._drivers.impl_rabbit [req-115b6b12-7ade-4362-84a1-986cb99e7e34 - - - - -] Connection failed: [Errno 113] EHOSTUNREACH (retrying in 32.0 seconds): error: [Errno 113] EHOSTUNREACH

이와 같은 에러 로그로 계속 뜨는 것이다. oslo.messaging 쪽과 rabbitmq 설정 시 오류가 난 줄 알고 Controller 쪽에 설정을 다시 살펴봤지만, 아무리봐도 오타 혹은 주석처리 문제는 아니였다.

 

그래서 Chat gpt에게 물어봤는데, 당시에는 글이 눈에 읽히지도 않고 머리가 돌아가지 않아서 잘몰랐다. 시간이 지나고 다시 천천히 보니깐 포트문제일 수도 있다는 것이다.

 

 

가정 1) Hostname 문제인가

Virtuallbox에는 Compute1로 설정하고 os 상에 다르게 설정되어서 그런가 싶어 각각 숫자를 넣어서 수정했다.

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
# controller
192.168.56.10   controller

# compute1
192.168.56.11   compute1

# compute2
192.168.56.12   compute2

[root@compute1 ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

# controller
192.168.56.10   controller

# compute1
192.168.56.11   compute1

# compute2
192.168.56.12   compute2

=> 하지만 똑같은 증상이 반복되었다.

 

 

가정 2) 네트워크적인 문제인 것인가?

네트워크적인 문제로 서로 통신이 안되서 그런것인가라고 생각해봤다. 그러기엔 초반에 ping으로 통신이 잘 됐는데 싶다가, 혹여 시간이 지나서 네트워크에 문제가 생긴것인가라는 생각에 서로 통신을 다시 해보았다.

[root@controller ~]# ping 192.168.56.10 -c 3
PING 192.168.56.10 (192.168.56.10) 56(84) bytes of data.
64 bytes from 192.168.56.10: icmp_seq=1 ttl=64 time=0.067 ms
64 bytes from 192.168.56.10: icmp_seq=2 ttl=64 time=0.076 ms
64 bytes from 192.168.56.10: icmp_seq=3 ttl=64 time=0.124 ms

--- 192.168.56.10 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1999ms
rtt min/avg/max/mdev = 0.067/0.089/0.124/0.025 ms

[root@compute1 ~]# ping 192.168.56.10 -c 3
PING 192.168.56.10 (192.168.56.10) 56(84) bytes of data.
64 bytes from 192.168.56.10: icmp_seq=1 ttl=64 time=0.529 ms
64 bytes from 192.168.56.10: icmp_seq=2 ttl=64 time=0.661 ms
64 bytes from 192.168.56.10: icmp_seq=3 ttl=64 time=0.878 ms

--- 192.168.56.10 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2026ms
rtt min/avg/max/mdev = 0.529/0.689/0.878/0.145 ms

=> 똑같은 증상이 반복되었다.

 

 

 

 

가정 3) 방화벽 문제인 것인가

그래서 일단, Controller 의 방화벽을 끄고 다시 진행해보기로 했다.

[root@controller ~]# systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
   Active: active (running) since Tue 2023-10-03 10:44:54 KST; 14h ago
     Docs: man:firewalld(1)
 Main PID: 730 (firewalld)
   CGroup: /system.slice/firewalld.service
           └─730 /usr/bin/python2 -Es /usr/sbin/firewalld --nofork --nopid

Oct 03 10:44:53 controller systemd[1]: Starting firewalld - dynamic firewall daemon...
Oct 03 10:44:54 controller systemd[1]: Started firewalld - dynamic firewall daemon.
Oct 03 10:44:54 controller firewalld[730]: WARNING: AllowZoneDrifting is enabled. This is considered an insecure... now.Hint: Some lines were ellipsized, use -l to show in full.

Controller에 방화벽은 잘 수행되고 있는 상태이다.

 

[root@controller ~]# systemctl stop firewalld

이제 명령어를 통해 방화벽을 중지한다.

 

[root@controller ~]# systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled)
   Active: inactive (dead) since Wed 2023-10-04 00:51:05 KST; 1s ago
     Docs: man:firewalld(1)
  Process: 730 ExecStart=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS (code=exited, status=0/SUCCESS)
 Main PID: 730 (code=exited, status=0/SUCCESS)

Oct 03 10:44:53 controller systemd[1]: Starting firewalld - dynamic firewall daemon...
Oct 03 10:44:54 controller systemd[1]: Started firewalld - dynamic firewall daemon.
Oct 03 10:44:54 controller firewalld[730]: WARNING: AllowZoneDrifting is enabled. This is considered an insecure... now.Oct 04 00:51:04 controller systemd[1]: Stopping firewalld - dynamic firewall daemon...
Oct 04 00:51:05 controller systemd[1]: Stopped firewalld - dynamic firewall daemon.
Hint: Some lines were ellipsized, use -l to show in full.

제대로 중지되었다.

 

 

그래서 Compute Node에 서비스를 재시작한다.

[root@compute1 ~]# systemctl restart libvirtd.service openstack-nova-compute.service

아무런 에러없이 제대로 명령어가 실행되는 것 같다

 

[root@compute1 ~]# systemctl status libvirtd.service openstack-nova-compute.service
● libvirtd.service - Virtualization daemon
   Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled)   Active: active (running) since Sat 2023-09-30 06:55:07 KST; 7s ago
     Docs: man:libvirtd(8)
           https://libvirt.org
 Main PID: 20490 (libvirtd)
    Tasks: 17 (limit: 32768)
   CGroup: /system.slice/libvirtd.service
           └─20490 /usr/sbin/libvirtd

Sep 30 06:55:06 compute1 systemd[1]: Starting Virtualization daemon...
Sep 30 06:55:07 compute1 systemd[1]: Started Virtualization daemon.
Sep 30 06:55:07 compute1 libvirtd[20490]: 2023-09-29 21:55:07.086+0000: 20506: info : l...rg)Sep 30 06:55:07 compute1 libvirtd[20490]: 2023-09-29 21:55:07.086+0000: 20506: info : h...te1Sep 30 06:55:07 compute1 libvirtd[20490]: 2023-09-29 21:55:07.086+0000: 20506: error : ...orySep 30 06:55:07 compute1 libvirtd[20490]: 2023-09-29 21:55:07.091+0000: 20506: error : ...orySep 30 06:55:08 compute1 libvirtd[20490]: 2023-09-29 21:55:08.893+0000: 20494: error : ...ory
● openstack-nova-compute.service - OpenStack Nova Compute Server
   Loaded: loaded (/usr/lib/systemd/system/openstack-nova-compute.service; enabled; vendor preset: disabled)
   Active: active (running) since Sat 2023-09-30 06:55:08 KST; 5s ago
 Main PID: 20507 (nova-compute)
    Tasks: 22
   CGroup: /system.slice/openstack-nova-compute.service
           └─20507 /usr/bin/python2 /usr/bin/nova-compute

Sep 30 06:55:07 compute1 systemd[1]: Starting OpenStack Nova Compute Server...
Sep 30 06:55:08 compute1 systemd[1]: Started OpenStack Nova Compute Server.
Hint: Some lines were ellipsized, use -l to show in full.

데몬 상태를 확인해본 결과 서비스가 잘 수행되고 있다. 그렇다면 Controller 에서 Compute Node를 인식해야한다.

 

그래서 Controller에서 확인해본다.

[root@controller ~]# . admin-openrc

[root@controller ~]# openstack compute service list --service nova-compute
+----+--------------+----------+------+---------+-------+----------------------------+
| ID | Binary       | Host     | Zone | Status  | State | Updated At                 |
+----+--------------+----------+------+---------+-------+----------------------------+
|  6 | nova-compute | compute1 | nova | enabled | up    | 2023-10-03T15:55:35.000000 |
+----+--------------+----------+------+---------+-------+----------------------------+

Controller에서 Compute를 잘 인식하고 있다.

 

 

결론

Controller에서 차단해서 정보를 가지고 오지 못하는 문제이다. 해서 관련된 포트를 열어주기로 했다.

 

 

해결

1. (Controller) 방화벽을 실행한다.

[root@controller ~]# systemctl restart firewalld

[root@controller ~]# systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2023-10-04 01:07:41 KST; 5s ago
     Docs: man:firewalld(1)
 Main PID: 31321 (firewalld)
   CGroup: /system.slice/firewalld.service
           └─31321 /usr/bin/python2 -Es /usr/sbin/firewalld --nofork --nopid

Oct 04 01:07:41 controller systemd[1]: Starting firewalld - dynamic firewall daemon...
Oct 04 01:07:41 controller systemd[1]: Started firewalld - dynamic firewall daemon.
Oct 04 01:07:41 controller firewalld[31321]: WARNING: AllowZoneDrifting is enabled. This...w.
Hint: Some lines were ellipsized, use -l to show in full.

 

2.  (Controller) 5672 포트를 열어준다.

[root@controller ~]# firewall-cmd --permanent --zone=public --add-port=5672/tcp
success

 

** Controller 방확벽을 중지하고 실행한다면 다음과 같은 명령어를 볼 수 있다. ** 

[root@controller ~]# firewall-cmd --permanent --zone=public --add-port=5672
FirewallD is not running

 

3. 

[root@controller ~]# firewall-cmd --reload
success

[root@controller ~]# firewall-cmd --list-ports
5672/tcp

 

4. (Controller) 방화벽을 재시작해준다.

[root@controller ~]# systemctl restart firewalld

 

5. (Compute) 서비스를 재시작하고 상태를 확인한다.

[root@compute1 ~]# systemctl restart libvirtd.service openstack-nova-compute.service


[root@compute1 ~]# systemctl status libvirtd.service openstack-nova-compute.service
● libvirtd.service - Virtualization daemon
   Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled)   Active: active (running) since Wed 2023-10-04 01:44:55 KST; 10s ago
     Docs: man:libvirtd(8)
           https://libvirt.org
 Main PID: 27899 (libvirtd)
    Tasks: 17 (limit: 32768)
   CGroup: /system.slice/libvirtd.service
           └─27899 /usr/sbin/libvirtd

Oct 04 01:44:55 compute1 systemd[1]: Starting Virtualization daemon...
Oct 04 01:44:55 compute1 systemd[1]: Started Virtualization daemon.
Oct 04 01:44:56 compute1 libvirtd[27899]: 2023-10-03 16:44:56.058+0000: 27915: info : l...rg)Oct 04 01:44:56 compute1 libvirtd[27899]: 2023-10-03 16:44:56.058+0000: 27915: info : h...te1Oct 04 01:44:56 compute1 libvirtd[27899]: 2023-10-03 16:44:56.058+0000: 27915: error : ...oryOct 04 01:44:56 compute1 libvirtd[27899]: 2023-10-03 16:44:56.062+0000: 27915: error : ...oryOct 04 01:44:57 compute1 libvirtd[27899]: 2023-10-03 16:44:57.850+0000: 27902: error : ...ory
● openstack-nova-compute.service - OpenStack Nova Compute Server
   Loaded: loaded (/usr/lib/systemd/system/openstack-nova-compute.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2023-10-04 01:44:57 KST; 8s ago
 Main PID: 27916 (nova-compute)
    Tasks: 22
   CGroup: /system.slice/openstack-nova-compute.service
           └─27916 /usr/bin/python2 /usr/bin/nova-compute

Oct 04 01:44:55 compute1 systemd[1]: Starting OpenStack Nova Compute Server...
Oct 04 01:44:57 compute1 systemd[1]: Started OpenStack Nova Compute Server.
Hint: Some lines were ellipsized, use -l to show in full.

잘 수행되고 있음을 확인할 수 있다.

 

 

6.  (Controller) Compute를 잘 인식하고 있는지 확인해본다.

[root@controller ~]# openstack compute service list --service nova-compute
+----+--------------+----------+------+---------+-------+----------------------------+
| ID | Binary       | Host     | Zone | Status  | State | Updated At                 |
+----+--------------+----------+------+---------+-------+----------------------------+
|  6 | nova-compute | compute1 | nova | enabled | up    | 2023-10-03T16:45:23.000000 |
+----+--------------+----------+------+---------+-------+----------------------------+