file open failed in offload mode

file open failed in offload mode


I tried the following code in offload mode. However, it returns "file open failed". I know that I should set the environment variable about ./proxyfs/  However, I don't know whether I have successfully set it. Can anyone please tell me how to set it and how to test whether it's set successfully?



#pragma offload_attribute(push,target(mic))
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#pragma offload_attribute(pop)

int main()
	FILE *fp;
	char buffer[7];
	#pragma offload target(mic) nocopy(fp)
		fp = fopen("./proxyfs/myfile.txt","wb");
		if (fp==NULL)
			fprintf(stderr,"Failed to open myfile.txt for write\n");
	#pragma offload target(mic) nocopy(fp) out(buffer)
		fp = fopen("./proxyfs/myfile.txt","rb");
		if (fp==NULL)
			fprintf(stderr,"Failed to open myfile.txt for write\n");
		if (strcmp(buffer,"Hello\n")!=0)
			fprintf(stderr,"File incorrectly read back on coproc\n");

	return 0;


11 posts / 0 new
Last post
For more complete information about compiler optimizations, see our Optimization Notice.

The MIC_PROXY_FS_ROOT environment variable associated with the proxyfs was depreciated over a year ago or so. You will have to consider using NFS. You can refer to earlier inquiries here and here.

The program fails due to the "./proxyfs" sub-directory not existing. If you drop that portion then the code runs and creates a local file on the coprocessor within the current working directory associated with the offload execution process.


when I delete "./proxyfs", it works. However, can I get access to the *.txt file from outside the program?

I tried  find command on mic0 and get nothing.

So if I want to get access to the file from outside the program, I have to use the NFS stuff, right? thanks

The offload process executes on the coprocessor using a temporary working directory located under /tmp/coi_procs/…  .That directory and all contents are removed at program termination. If you want/need to create files on the coprocessor and keep them (temporarily) for use later outside the program then create a local directory (perhaps under /tmp) on the coprocessor and modify the program to create/modify files there, or just create/modify the files directly in /tmp. Note that /tmp is purged when the coprocessor boots so it’s not meant for long term file saving.  Alternatively, you can perform file I/O to a NFS mounted filesystem where your files would be available after program termination from the server.

Thanks for your explanation. I have two more groups of questions:

1. About the temporary directory/files.

Since you mentioned the local directory /tmp. Is this directory on the host or on the co-processor? I tried the command  find $home -name 'tmp' on both the host and the co-processor, but found nothing. (From your explanation, I guess it should be on the co-processor, but I cannot find it.) Or do I need to explicitly create such a directory? If so, where, the co-processor or the host, should I create the directory? Should the created directory be directly under the path /home/user_name/ ?

2. About the NFS setting.

I've read this readme web-page, however, there are some instructions that I don't understand. In the section 7.6, it refers to the port 111 and 2049. I run command rpcinfo -p on the host and find there are lines like

proto port service

tcp    111  portmapper

udp   111  portmapper

tcp    2049 nfs

udp   2049 nfs

Does this mean that the required ports have been enabled? And the manual gives an example about "/mic0fs". Is this directory located on the host or the co-processor? It seems that I should create such a directory, Is it necessary to  create the directory directly under the path "/home/user_name/" ?

Thanks again.

Regarding the questions:

1. /tmp is on the coprocessor. It is a system provided directory (really a sym-link) and thus you do not need to create it. It will be clearly visible/accessible after ssh-ing to the coprocessor (see below).  I do not know what login id you used to connect to the card but /tmp resides at “/” (root) level not within $home and unless you have explicitly set it, that variable is not defined by default. $HOME is but on the coprocessor under the “root” login id it is set to /root.

[root@xxxxxxxx-mic0 etc]# ls -l /
drwxr-xr-x    2 root     root          1660 Dec 31  1969 bin
drwxr-xr-x    2 root     root            80 Dec 31  1969 boot
drwxr-xr-x   13 root     root          2520 Jun 24 12:10 dev
drwxr-xr-x   27 root     root          1440 Dec 18  2013 etc
drwxr-xr-x    4 root     root          4096 Jun  3 18:22 home
-rwxr-xr-x    1 root     root          3512 Dec 18  2013 init
drwxr-xr-x    3 root     root            60 Dec 31  1969 lib
drwxr-xr-x    4 root     root          1520 Jun 24 12:10 lib64
drwxr-xr-x   10 root     root           200 Dec 31  1969 media
drwxr-xr-x    2 root     root           120 Dec 31  1969 mnt
dr-xr-xr-x  980 root     root             0 Dec 31  1969 proc
drwxr-xr-x    3 root     root           100 Jul 11 04:29 root
drwxr-xr-x    2 root     root          1620 Dec 31  1969 sbin
drwxr-xr-x    2 root     root           340 Jun 24 12:10 sep3.15
drwxr-xr-x   12 root     root             0 Dec 31  1969 sys
lrwxrwxrwx    1 root     root             8 Dec 31  1969 tmp -> /var/tmp
drwxrwxrwx   11 root     root           220 Dec 31  1969 usr
drwxr-xr-x    7 root     root           240 Dec 31  1969 var


2. I won’t profess to be a network/NFS expert and there are others who frequent this forum who can help and hopefully will chime in.

The readme you cited is for the MPSS 2.1 release, so if you are using that older release you should consider upgrading to the newer MPSS. I believe a newer update is due to be available in a few business days so you may want to await announcement of that and then upgrade as I think you would be well served by updating your MPSS.

rcpinfo provides info about RCP services running on the system. The -p includes the port # binding. Even with what you show it is still possible iptables could block NFS access, but I believe generally under a default Linux installation ports are not blocked. So, unless you are knowingly doing something specific to alter iptables, perhaps assume you have the ports open and usable w/NFS and proceed with configuring and trying to mount the filesystem of interest on the coprocessor.

In the example you cited from the readme, /mic0fs is a directory that resides on the host at the “/” (root) level. It is expected to have to create that on the host first and then setup both the host and coprocessor configurations necessary to NFS mount it to coprocessor. The example uses coprocessor 0, also known by the name “mic0”.

If your interest is in mounting user home directories from the host, then you would not create any such “mic0fs” sub-directory under the user’s home but instead just setup the host and coprocessor NFS configurations to export the user home directory from the host and mount it on the card. Doing this may also start getting a need for enabling normal user accounts on the card. That’s another can of worms…..

Here are some good references available that can assist with setting up NFS and adding User accounts on the coprocessor:

System Administration for the Intel® Xeon Phi™ Coprocessor
Intel® Manycore Platform Software Stack (Intel® MPSS) Cluster Setup Guide
Intel® Manycore Platform Software Stack (Intel® MPSS) User’s Guide


I tried modifying the /tmp directory directly, it works.

I've almost finished the sect 7.6 of the readme document. However, I have a question about 7.6 3-d, which is "Chown michome directories as appropriate". I've never used the chown command before. This command seems to be related to ownership of a directory. I set the nfs settings according to the readme via root account. Normally, I log on the system via a sudo account called newuser. In this case, how do I use chown  to set the ownership of the michome directory?

And, after I finished steps of the sect 7.6, do I need to modify the code posted above to perform offload I/O?


Before completing step 7.6 3-d, there may be more to step 7.6 3-c that is needed. Step c had you create the /srv/michome directory. Under this directory you need to also create home areas for any users. At least one you can create is the micuser. You can also create a newuser area here (based on your setup) and then modify your program (in your initial post) to perform I/O (during offload) into that newuser home directory. e.g. Use these open statements in your original program:

fp = fopen("/home/newuser/myfile.txt","wb");  // In the first offload section
fp = fopen("/home/newuser/myfile.txt","rb");   // In the second offload section

Because the offload process runs as micuser you need to make certain the newuser home area has sufficient permission to allow the micuser id to read/write that area.

So, in step c, if you created two home directories, micuser and newuser, then in step d you use chown to change the owner of each to their respective owner id, and then you use chmod to enable group access permissions to the newuser area. So you might do something like this on the host while working under a normal user account id:

user-account> sudo mkdir -p /srv/michome/micuser
user-account> sudo chown micuser /srv/michome/micuser
user-account> sudo mkdir -p /srv/michome/newuser
user-account> sudo chown newuser /srv/michome/newuser
user-account> sudo chmod 775 /srv/michome/newuser

The above “775” in the last chmod command enables group access for the group associated with the newuser home area. I do not know what the default group is for your newuser account id, but to enable your offload process to read/write into the newuser home area, you need to add the group associated with the newuser home area to the micuser account id. Let’s assume the group associated with the newuser home area is newuser. You would grant the micuser id group access using this command:

user-account> sudo usermod -a -G newuser micuser  

When using the usermod command above, please make sure you identify the group association for the newuser home area on your system first and then use that group name in the command above.

If you supplement the instructions 7.6 3 with the added steps above then that should enable your offload program to perform I/O into the newuser home area.

I met another problem with the newuser account.

I used to be able to log in to mic0 via newuser account smoothly. However, maybe after I run 7.6.3-efgh, the problem appeared.

It's fine when I log in to the host via newuser. However, when I logged on to mic0, it asked for password. I can log on to mic0 after typing the password, with a line Could not chdir to home directory /home/newuser: No such file or directory. I found no /home/newuser on mic0.
This line disappeared after I created /srv/michome/newuser on the host, but I still need password to log in to mic0 via newuser, while root account needs no password to log in.

I found no /home/newuser/.ssh on mic0. When I tried mkdir .ssh it returned mkdir: can't create directory '.ssh' : Permission denied.
Then I tried to modified /srv/michome/newuser on the host via newuser with sudo privilege. I created /srv/michome/newuser/.ssh/ and authorized_keys. And I have /home/newuser/.ssh/id_rsa on the host. But I still need password to log in to mic0.

I wonder if there're still other required procedures to do or I mis-paired the key. Currently, I haven't find I make any mistakes in pairing the key.

Sorry to have more questions:

1. Now I can use offload to write to the directory /newuser. (THANKS!) However, I cannot use offload mode to get access to /newuser/data, that is, it's sub-directories. Is there any way to get access to its sub-directories?

2. on the host,

scp ./file newuser@mic0:/home/newuser/dir1/dir2/

cp ./file /srv/michome/newuser/dir1/dir2/

Is there any difference between these two commands?


ps, in your second piece of code, I think "src" is a typo for "srv", right?

Yes you are correct that in the series of commands to make the directories and set owner those should have used “srv” not “src”. I corrected those in the earlier post.

Glad to hear you have the offload I/O working.

Regarding question #1, the issue is likely the permissions/group access on the sub-directories. Check that the dir1 and dir2 have “775” permissions the same group ownership as their parent (i.e. /srv/michome/newuser). You can change the group ownership using chgrp. I don’t know what the default group is for your newuser id, but let’s say it is “newuser”, so then to set the permissions/group ownership for the sub-directories so offload will work you might do this:

user-account> sudo chmod -R 775 /srv/michome/newuser
user-account> sudo chgrp -R newuser /srv/michome/newuser

Regarding question #2, the difference between the two commands is that the first has the added levels of “authentication”, ssh and NFS, and the second command copies the file locally on the host.


Leave a Comment

Please sign in to add a comment. Not a member? Join today