Swap on xeon phi

Swap on xeon phi

I have a Xeon Phi card running MPSS Gold update 3.

I want to use swap but mic_virtblk will not load into the kernel

message is cannot find mic_virtblk.

Do I need to use MPSS 3.2.3 to use swap?

23 posts / 0 new
Last post
For more complete information about compiler optimizations, see our Optimization Notice.

Did you read the article?

https://software.intel.com/en-us/blogs/2013/07/11/avoiding-potential-pro...

I suppose this is a potential hurdle in porting applications which needed a swap partition in the past, but it seems worth while to explore methods to reduce memory and disk usage (although the KNL would seem to change the rules again within 2 years).

Tried to follow the instructions in the article, but as I said in my original post

mic_virtblk.ko will not load into the mic kernel.

is there something else I need to do first that is missing from the article?

 

 

Also,

This path is not present on my host machine.

        /sys/devices/virtual/mic/mic0

up o

        /sys/devices/virtual/mic

is present but only contains /ctrl and scif

do I have the correct version of MPSS to set up a swap space?  (gold update 3)

Your contents for that directory appear to be normal even for current MPSS.

Unfortunately, your version of MPSS is too old for us to check. (We have no test systems with that old of an MPSS.)

Please update your version. If you still have an issue, let us know and we'll do what we can.

Regards
---
Taylor
 

Oh, and please wait for a few (business) days before updating your version of MPSS.  ;-)

Regards
---
Taylor
 

 

Found this in the rel notes for gold update 3

Intel Tracking ID:  4845954
Affected OS:        All Linux
Description:        [Driver] Virtualization is no longer working for
                    coprocessors
Notes:              Investigating

Is this resolved in the latest version?

Phil Sharrock

According to the bug tracking system, this issue is resolved and closed as of this release.

 

Loaded MPSS 3.2.3.

Still will not load mic_virtblk.ko into kernel, same error as before.

Also when compiling on Phi card the linux command 'ar' (via busybox) does not accept

the -r option, this option is needed when adding items to archives.

Phil Sharrock

Sorry, I should have said closed as of the MPSS 3.3 release. And I guess I need to update that blog. The complete directions for using a virtio block device are now in the MPSS User's Guide, rather than the readme and some of the directory locations appear to have changed. Sigh.

As far as ar not having a replace option, I can see how that would be painful. I'll look into this.

 

Even following the instructions in the MPSS users guide, modprobe mic_virtblk fails to load the driver

I get the same error message file not found/not present

Uh, oh. I think I know what is wrong. With 3.2 and continuing into 3.3, a number of files which are considered optional on the coprocessor have to moved to rpm files that must be explicitly added if you want to use them. I found mic_virtblk in  mpss-modules-3.3-1.knightscorner.rpm. Under MPSS 3.3, to load these files you need to follow the instructions in section 11.3 in the MPSS User's Guide.

When I use these extra rpm files on the coprocessor, I prefer to follow section 11.3.3, making a directory containing only the rpm files I want uploaded to the coprocessor and including in these the files needed to run zypper on the coprocessor (see section 11.3). That way I can specify the directory name rather than the individual filenames on the micctrl command and not need to rerun the command each time I add or remove files from the directory. If, after I reboot, I log onto the coprocessor and find a directory called /RPMs-to-install, it means the rpm files did not successfully install. In this case, I install the files needed to run zypper and use zypper to check for any unsatisfied dependencies or errors, then fix those.

 

Tried to install the suggested rpm file.

Got the message already installed

Swap on MIC works for me in MPSS 3.3. I only installed the RPMs inside of "mpss-3.3/" and the kernel module. The difference in my procedure versus Frances' article are:

  • The path to the virtblk_file is different in MPSS 3.3
  • I can format the drive from the host
  • If you are changing virtblk_file, you should unload and reload module mic on the host
  • There is no need to modprobe mic_virtblk, it loads automatically if virtblk_file existed when MPSS booted
  • I used an LVM volume instead of a file on the host

Here is the sequence of commands that I used:

[root@c001-n002 ~]# service mpss unload
Shutting down Intel(R) MPSS:                               [  OK  ]
Removing MIC Module:                                       [  OK  ]
[root@c001-n002 ~]# modprobe mic
[root@c001-n002 ~]# mkswap /dev/mapper/vg_phi-lvol0 # Formatting a logical volume for swap
mkswap: /dev/mapper/vg_phi-lvol0: warning: don't erase bootbits sectors
        on whole disk. Use -f to force.
Setting up swapspace version 1, size = 31457276 KiB
no label, UUID=6b0609cc-4b8b-4bc4-8e35-7a0f52260ffa
[root@c001-n002 ~]# echo /dev/mapper/vg_phi-lvol0 > /sys/class/mic/mic0/virtblk_file 
[root@c001-n002 ~]# service mpss start
Starting Intel(R) MPSS:                                    [  OK  ]
mic0: online (mode: linux image: /usr/share/mpss/boot/bzImage-knightscorner)
mic1: online (mode: linux image: /usr/share/mpss/boot/bzImage-knightscorner)
mic2: online (mode: linux image: /usr/share/mpss/boot/bzImage-knightscorner)
mic3: online (mode: linux image: /usr/share/mpss/boot/bzImage-knightscorner)
[root@c001-n002 ~]# ssh mic0
[root@c001-n002-mic0 ~]# swapon /dev/vda 
[root@c001-n002-mic0 ~]# cat /proc/swaps 
Filename				Type		Size	Used	Priority
/dev/vda                                partition	31457276	0	-1
[root@c001-n002-mic0 ~]# 

 

See also our article on file I/O on Xeon Phi for performance benchmarks: http://research.colfaxinternational.com/post/2014/07/28/io.aspx

Quick update: I have just tested using a file instead of a logical volume on host. That configuration also worked for me.

Phil,

Any update? Did you get it to work?

Regards
--
Taylor
 

 

Yes that worked. swap now available.

now just got to stop the Phi card locking up when most of memory used.

no communication with card after a few minutes.

Phil

Hi, i'm trying to use virtblk with mpss 3.51 but

- no luck using the user guide method

- no luck using Andrey's method above

errors:

/dev/vda - - - No such file or directory

and when doing  [root@mic0 ~]# modprobe mic_virtblk

FATAL: Error inserting mic_virtblk (/lib/modules/2.6.38.8+mpss3.5.1/extra/virtio/mic_virtblk.ko): No such device or address

Is there an updated procedure ? Many thanks in advance !

 

just tried it on my mpss 3.5.1 install and it "just works".

Did you install the mpss-modules package? this RPM includes the module mic_virtblk.ko:

[root@mic0 ~]# rpm -qf /lib/modules/2.6.38.8+mpss3.5.1/extra/virtio/mic_virtblk.ko
mpss-modules-3.5.1-1.knightscorner

 

Also, make sure to initialize the swap device prior to using it:

[root@mic0 ~]# swapon /dev/vda
swapon: /dev/vda: Invalid argument
[root@mic0 ~]# mkswap /dev/vda
Setting up swapspace version 1, size = 10485755904 bytes
[root@mic0 ~]# swapon /dev/vda

[root@mic0 ~]# cat /proc/swaps      
Filename                                Type            Size    Used    Priority
/dev/vda                                partition       10239996        0       -1

 

I just double checked on a couple systems that I use and both had /lib/modules/2.6.38.8+mpss<version#>/extra/virtio/mic_virtio/mic_virtblk.ko installed by default. When the mic_virtblk kernel module loads, it creates /dev/vda. What modprobe is complaining about is not that it cannot find the kernel module but that it cannot find the device on the host that is providing the actual swap space. Without that space the module can't load.

So first check on the host to be sure the device you are using exists - /dev/<disk_name>, /dev/<disk_partition_name>, /dev/mapper/<logical_volume_name> or the file name , if you are using a file instead of an actual device.

Make sure /sys/class/mic/mic0/virtblk_file on the host exists and that the content of that file is the full path name to the device you are using.

If the file /sys/class/mic/mic0/virblk_file did not exist when the coprocessor was booted, stop and restart the mpss. 

If all that checks out, mic_virtblk.ko really should load.

Thank you JJK and Frances,

true the error message got me confused (Frances: <What modprobe is complaining about is not that it cannot find the kernel module but that it cannot find the device on the host that is providing the actual swap space. Without that space the module can't load.>   )

the /var/log/messages seems to confirm this:

Jul 13 10:03:45 mic0 user.warn kernel: [    7.021619] Module mic_virtblk loaded at 0xffffffffa007a000
Jul 13 10:03:45 mic0 user.err kernel: [    7.029411] Have you set virtblk file?

But i still didn't get the module to load. I'm mostly trying to mount an ext2 partition now, not to swap. But i tried swap too and the point of failure is the same.

_________________________________________________

Here is my complete attempt to mount an ext2 partition sdb1 (on freshly formatted disk sdb) : 

("phi" is my host, mic0 is one of the 8 coprocessors. RHEL7.1 , MPSS 3.5.1, Supermicro 4028 TRT server containing a PCIe splitter/expansion)

[root@phi ~]# lsblk
NAME          MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda             8:0    0 119.2G  0 disk
├─sda1          8:1    0   500M  0 part /boot
└─sda2          8:2    0 118.8G  0 part
  ├─rhel-swap 253:0    0  11.9G  0 lvm  [SWAP]
  ├─rhel-root 253:1    0    50G  0 lvm  /
  └─rhel-home 253:2    0  56.8G  0 lvm  /home
sdb             8:16   0  59.6G  0 disk
[root@phi ~]# fdisk /dev/sdb
Welcome to fdisk (util-linux 2.23.2).

Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Command (m for help): N
Partition type:
   p   primary (0 primary, 0 extended, 4 free)
   e   extended
Select (default p): p
Partition number (1-4, default 1):
First sector (2048-125045423, default 2048):
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-125045423, default 125045423):
Using default value 125045423
Partition 1 of type Linux and of size 59.6 GiB is set

Command (m for help): p

Disk /dev/sdb: 64.0 GB, 64023257088 bytes, 125045424 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0xa054008f

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1            2048   125045423    62521688   83  Linux

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.
[root@phi ~]# mkfs.ext2 /dev/sdb1
mke2fs 1.42.9 (28-Dec-2013)
/dev/sdb1 is mounted; will not make a filesystem here!
[root@phi ~]# lsblk
NAME          MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda             8:0    0 119.2G  0 disk
├─sda1          8:1    0   500M  0 part /boot
└─sda2          8:2    0 118.8G  0 part
  ├─rhel-swap 253:0    0  11.9G  0 lvm  [SWAP]
  ├─rhel-root 253:1    0    50G  0 lvm  /
  └─rhel-home 253:2    0  56.8G  0 lvm  /home
sdb             8:16   0  59.6G  0 disk
└─sdb1          8:17   0  59.6G  0 part /mnt/sdb1
[root@phi ~]# umount /dev/sdb1
[root@phi ~]# mkfs.ext2 /dev/sdb1
mke2fs 1.42.9 (28-Dec-2013)
Discarding device blocks: done
warning: 86 blocks unused.

Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
3915216 inodes, 15630336 blocks
781516 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
477 block groups
32768 blocks per group, 32768 fragments per group
8208 inodes per group
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
        4096000, 7962624, 11239424

Allocating group tables: done
Writing inode tables: done
Writing superblocks and filesystem accounting information: done

[root@phi ~]# mount /dev/sdb1 /mnt/sdb1
[root@phi ~]# mount
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime,seclabel)
devtmpfs on /dev type devtmpfs (rw,nosuid,seclabel,size=32839680k,nr_inodes=8209920,mode=755)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,seclabel)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,seclabel,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,seclabel,mode=755)
tmpfs on /sys/fs/cgroup type tmpfs (rw,nosuid,nodev,noexec,seclabel,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpuacct,cpu)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/net_cls type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
configfs on /sys/kernel/config type configfs (rw,relatime)
/dev/mapper/rhel-root on / type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
selinuxfs on /sys/fs/selinux type selinuxfs (rw,relatime)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=35,pgrp=1,timeout=300,minproto=5,maxproto=5,direct)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
mqueue on /dev/mqueue type mqueue (rw,relatime,seclabel)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,seclabel)
/dev/sda1 on /boot type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
/dev/mapper/rhel-home on /home type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
/dev/sdb1 on /mnt/sdb1 type ext2 (rw,relatime,seclabel)
[root@phi ~]# service mpss unload
Shutting down Intel(R) MPSS:                               [  OK  ]
Removing MIC Module:                                       [  OK  ]
[root@phi ~]# modprobe mic
[root@phi ~]# echo /dev/sdb1 > /sys/class/mic/mic0/virtblk_file
[root@phi ~]# cat /sys/class/mic/mic0/virtblk_file
/dev/sdb1
[root@phi ~]# service mpss start
Starting mpss (via systemctl):                             [  OK  ]
[root@phi ~]# ssh mic0
Password:
[root@mic0 ~]# mkdir /mnt/vda
[root@mic0 ~]# mount /dev/vda /mnt/vda
mount: mounting /dev/vda on /mnt/vda failed: No such file or directory
[root@mic0 ~]#

from /var/log/messages:

Jul 13 10:03:45 mic0 user.warn kernel: [    7.021619] Module mic_virtblk loaded at 0xffffffffa007a000
Jul 13 10:03:45 mic0 user.err kernel: [    7.029411] Have you set virtblk file?

______________________________________________________________________________

I have also noticed that /sys/class/mic/mic0 redirects me to a directory "/sys/devices/pci0000:00/0000:00:02.0/0000:02:00.0/0000:03:08.0/0000:04:00.0/mic/mic0"

It does however contain the virtblk_file with the line i echoed in, before mpss start and after start too.

I am not sure that you can use a block device /dev/sdb1 as a value for virtblk_file ; I've created a file on disk, formatted it as an EXT2 filesystem and mounted it. That works fine with mpss 3.5.1:

# service mpss stop
# dd if=/dev/zero of=/var/mpss/mic0.virtblk_file bs=1024k count=10000
# echo /var/mpss/mic0.virtblk_file > /sys/class/mic/mic0/virtblk_file
# service mpss start

Then, on the Phi, I can format /dev/vda, mount it, etc.

And, on my box, /sys/class/mic/mic0 is a symlink to /sys/devices/pci0000:80/0000:80:03.0/0000:87:00.0/0000:88:10.0/0000:8a:00.0/mic/mic0 , so the value you are seeing does not surprise me.

 

Leave a Comment

Please sign in to add a comment. Not a member? Join today