Eucalyptus EBS

Working enviroment:

 Eucalyptus version. 1.5.1 installed from source on CentOS 5.3 \\x86_64 with xen hypervisor ver 3.0.3 
 2 hosts   front-end with cc   and  one node-controller
 Networking mode MANAGED-NOVLAN
 Eucalyptus run as user eucalyptus

Problem:

EBS volumes don't detach

Simptoms:

Run commands

 $ ec2-attach-volume vol-32D404AD -i i-422F06E6 -d /dev/xvdb
 ATTACHMENT      vol-32D404AD    i-422F06E6      /dev/xvdb       attaching       2009-05-27T08:58:56+0000

 $ ec2-detach-volume vol-32D404AD
 ADATTACHMENT    vol-32D404AD    i-422F06E6      xvdb            2009-05-27T08:59:03+0000

 $ ec2-describe-volumes
 VOLUME  vol-32D404AD    1                       in-use  2009-05-25T13:09:21+0000
 ATTACHMENT      vol-32D404AD    i-422F06E6      xvdb            2009-05-27T09:02:15+0000
 VOLUME  vol-326C049E    1                       available       2009-05-26T09:35:10+0000

Volume still in-use

Errors in log files on node-controller nc.log

 [Wed May 27 12:00:00 2009][025754][EUCAINFO  ] doDetachVolume() invoked (id=i-422F06E6 vol=vol-32D404AD remote=/dev/etherd/e0.1  local=xvdb force=0)
 [Wed May 27 12:00:00 2009][025754][EUCADEBUG ] xml to detach: <disk type='block'><driver name='phy'/><source dev='/dev/etherd/e0.1'/><target dev='xvdb'/></disk>
 [Wed May 27 12:00:00 2009][025754][EUCADEBUG ] Running command: /opt/eucalyptus/usr/share/eucalyptus/euca_rootwrap virsh detach-device i-422F06E6 /tmp/detachxml.5VaNRe
 [Wed May 27 12:00:00 2009][025754][EUCADEBUG ] first workaround command failed (255), trying second workaround...
 [Wed May 27 12:00:00 2009][025754][EUCADEBUG ] Running command: sudo xm block-detach i-422F06E6 xvdb
 [Wed May 27 12:00:00 2009][025754][EUCAERROR ] DetachVolume() failed (err=1) 
 [Wed May 27 12:00:00 2009][025754][EUCAERROR ] ERROR: doDetachVolume() failed error=1

httpd-nc_error_log

 sudo: sorry, you must have a tty to run sudo
 error: Failed to open '/tmp/detachxml.dAz5aB': No such file or directory

Solution 1

Enable second workaround which run command sudo xm block-detach

Step 1

Add user eucalyptus to sudoers file

 Cmnd_Alias VIRTUAL = /opt/libvirt62/bin/virsh, /usr/sbin/xm
 Defaults:eucalyptus     !requiretty
 eucalyptus   ALL = NOPASSWD: VIRTUAL

Step 2

Restart node-controller

Step 3

View in nc.log when we see that ec2-detach-volume start working,

 [Wed May 27 12:11:31 2009][026168][EUCAINFO  ] doDetachVolume() invoked (id=i-422F06E6 vol=vol-32D404AD remote=/dev/etherd/e0.1 local=xvdb force=0)
 [Wed May 27 12:11:31 2009][026168][EUCADEBUG ] Running command: /opt/eucalyptus/usr/share/eucalyptus/euca_rootwrap virsh detach-device i-422F06E6 /tmp/detachxml.Wjt34X
 [Wed May 27 12:11:31 2009][026168][EUCADEBUG ] first workaround command failed (255), trying second workaround...
 [Wed May 27 12:11:31 2009][026168][EUCADEBUG ] Running command: sudo xm block-detach i-422F06E6 xvdb
 [Wed May 27 12:11:31 2009][026168][EUCAINFO  ] detached /dev/etherd/e0.1 as xvdb in domain i-422F06E6

Solution 2

Enable first workaround which run command virsh detach-device

Step 1

On file httpd-nc_error_log we have messages

 error: Failed to open '/tmp/detachxml.dAz5aB': No such file or directory
 error: Failed to open '/tmp/detachxml.dkUSuw': No such file or directory

to fix this applay patch:

 +++ eucalyptus-1.5.1/node/handlers_xen.c        2009-05-27 11:40:59.000000000 +0300
 @@ -1158,10 +1158,10 @@
                if (fd > 0) {
                 write(fd, xml, strlen(xml));
                 close(fd);
 -               unlink(tmpfile);
                snprintf(cmd, 1024, "%s detach-device %s %s",virsh_command_path, instanceId, tmpfile);
                logprintfl(EUCADEBUG, "Running command: %s\n", cmd);
                err = WEXITSTATUS(system(cmd));
 +               unlink(tmpfile);
                if (err) {
                  logprintfl(EUCADEBUG, "first workaround command failed (%d), trying second workaround...\n", err);
                  snprintf(cmd, 1024, "%s block-detach %s %s", xm_command_path, instanceId, localDevReal);

Step 2

Stop node-controller

Compile new library

 cd eucalyptus-1.5.1/node/
 make

Copy new libEucalyptusNC.so to /opt/eucalyptus/packages/axis2c-1.5.0/services/EucalyptusNC/

Step 3

Add to /opt/eucalyptus/etc/init.d/eucalyptus-nc

 export VIRSH_DEFAULT_CONNECT_URI='xen+unix:///'
 PATH=$PATH:/opt/libvirt/bin

Start node-controller

Step 4 View logfile

 nc.log
 [Wed May 27 13:44:22 2009][026877][EUCAINFO  ] doDetachVolume() invoked (id=i-422F06E6 vol=vol-32D404AD remote=/dev/etherd/e0.1 local=xvdb force=0)
 [Wed May 27 13:44:22 2009][026877][EUCADEBUG ] Running command: /opt/eucalyptus/usr/share/eucalyptus/euca_rootwrap virsh detach-device i-422F06E6 /tmp/detachxml.EKUiEE
 [Wed May 27 13:44:22 2009][026877][EUCAINFO  ] detached /dev/etherd/e0.1 as xvdb in domain i-422F06E6

BalticCloud

Eucalyptus EBS