Accessing Uploaded Datasets from Tensorflow notebook

Accessing Uploaded Datasets from Tensorflow notebook

Hi

I have been using the SDK for sometime now. Everything is working fine. But now as the tool supports Tensorflow only in the advanced mode (jupyter notbooks) I want to use it, and access a dataset that I uploaded and used several times with Caffe and the GUI. So far I can't find the right way to do that. I found another post in this forum that explains how to access the lmdb files (the format in which the image dataset is stored) from caffe notebooks, but the same approach doesn't work with tensorflow. Also that approach will allow me to access the dataset file, but file. It will not be easy to use this for training.

uuid = "4f316909-3ab5-4bca-bdec-f5976509e0e2" 
lmdb_file = "/workspace/dlsdk/jobs/caffe/datasets/%s/train.txt_LMDB" % uuid
lmdb_env = lmdb.open(lmdb_file)
lmdb_txn = lmdb_env.begin()
lmdb_cursor = lmdb_txn.cursor()
datum = caffe_pb2.Datum()

In summary I want to train a network using a dataset that I uploaded using the GUI from a tensorflow notebook. Is there a method that will allow me to do that.

Ahmed 

3 posts / 0 new
Last post
For more complete information about compiler optimizations, see our Optimization Notice.

Hi Ahmed,
There are a few issues here:
1. The dataset uploaded and created in Caffe is located in the Caffe docker container.
2. In order to use it in the TensorFlow container, you need to manually move it
3. The code you are referring to in your post is used in Caffe notebooks in order to extract images from the LMDB. The caffe framework is not installed on the TF notebook (different docker container...).

I think it is best to search for a solution online (if there is) for using the LMDB as a dataset for TensorFlow. Then you if you move the LMDB file itself, you will be able to use it.

Barak.

Hi Barak,

Thank you for the quick reply.

For the first and second points, apparently, either both running on the same container or sharing the same file-system (at least for me). I can access the files saved by Caffe using the tensor flow terminal. The tensor flow notebooks and the dataset files are on the same file system. So I don't need to move any files. 

For the third point, it is clear, that tensor flow container doesn't have access to this LMDB library. 

I will search again, hopping that I can find a solution.

Thanks again

Ahmed

Leave a Comment

Please sign in to add a comment. Not a member? Join today