shape mismatch when optimize resnet_v1_50.pb

shape mismatch when optimize resnet_v1_50.pb

Hi, 

I downloaded resnet_v1_50_2016_08_28.tar.gz from  http://download.tensorflow.org/models/resnet_v1_50_2016_08_28.tar.gz, and use dldt 2018_R5 to optimize it. Get shape mismatch error. Please help.

Commands:

python3 tf_models/research/slim/export_inference_graph.py \
    --model_name resnet_v1_50 \
    --output_file ov_irs/resnet_v1_50.pb

python3 ~/dldt/model-optimizer/mo/utils/summarize_graph.py --input_model ov_irs/resnet_v1_50.pb

wget http://download.tensorflow.org/models/resnet_v1_50_2016_08_28.tar.gz
tar xvf resnet_v1_50_2016_08_28.tar.gz -C ov_irs

python3 ~/dldt/model-optimizer/mo_tf.py --input_model ./ov_irs/resnet_v1_50.pb --input_checkpoint ./ov_irs/resnet_v1_50.ckpt -b 1 --mean_value [103.94,116.78,123.68] --scale 1

output:

Model Optimizer arguments:
Common parameters:
        - Path to the Input Model:      /home/glorysdj/./ov_irs/resnet_v1_50.pb
        - Path for generated IR:        /home/glorysdj/.
        - IR output name:       resnet_v1_50
        - Log level:    ERROR
        - Batch:        1
        - Input layers:         Not specified, inherited from the model
        - Output layers:        Not specified, inherited from the model
        - Input shapes:         Not specified, inherited from the model
        - Mean values:  [103.94,116.78,123.68]
        - Scale values:         Not specified
        - Scale factor:         1.0
        - Precision of IR:      FP32
        - Enable fusing:        True
        - Enable grouped convolutions fusing:   True
        - Move mean values to preprocess section:       False
        - Reverse input channels:       False
TensorFlow specific parameters:
        - Input model in text protobuf format:  False
        - Offload unsupported operations:       False
        - Path to model dump for TensorBoard:   None
        - List of shared libraries with TensorFlow custom layers implementation:        None
        - Update the configuration file with input/output node names:   None
        - Use configuration file used to generate the model with Object Detection API:  None
        - Operations to offload:        None
        - Patterns to offload:  None
        - Use the config file:  None
Model Optimizer version:        1.5.12.49d067a0
[ FRAMEWORK ERROR ]  Cannot load input model: Restoring from checkpoint failed. This is most likely due to a mismatch between the current graph and the graph from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:

Assign requires shapes of both tensors to match. lhs shape= [1,1,2048,1001] rhs shape= [1,1,2048,1000]
         [[node save/Assign_266 (defined at /home/glorysdj/dldt/model-optimizer/mo/front/tf/loader.py:105)  = Assign[T=DT_FLOAT, _class=["loc:@resnet_v1_50/logits/weights"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:CPU:0"](resnet_v1_50/logits/weights, save/RestoreV2:266)]]

Caused by op 'save/Assign_266', defined at:
  File "/home/glorysdj/dldt/model-optimizer/mo_tf.py", line 31, in <module>
    sys.exit(main(get_tf_cli_parser(), 'tf'))
  File "/home/glorysdj/dldt/model-optimizer/mo/main.py", line 325, in main
    return driver(argv)
  File "/home/glorysdj/dldt/model-optimizer/mo/main.py", line 267, in driver
    mean_scale_values=mean_scale)
  File "/home/glorysdj/dldt/model-optimizer/mo/pipeline/tf.py", line 127, in tf2nx
    saved_model_tags=argv.saved_model_tags)
  File "/home/glorysdj/dldt/model-optimizer/mo/front/tf/loader.py", line 208, in load_tf_graph_def
    graph_def = freeze_checkpoint(graph_def=graph_def, checkpoint=checkpoint, output_node_names=outputs)
  File "/home/glorysdj/dldt/model-optimizer/mo/front/tf/loader.py", line 105, in freeze_checkpoint
    tf.train.Saver(var_list=var_list).restore(sess, checkpoint)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/saver.py", line 1102, in __init__
    self.build()
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/saver.py", line 1114, in build
    self._build(self._filename, build_save=True, build_restore=True)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/saver.py", line 1151, in _build
    build_save=build_save, build_restore=build_restore)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/saver.py", line 795, in _build_internal
    restore_sequentially, reshape)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/saver.py", line 428, in _AddRestoreOps
    assign_ops.append(saveable.restore(saveable_tensors, shapes))
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/saver.py", line 119, in restore
    self.op.get_shape().is_fully_defined())
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/state_ops.py", line 221, in assign
    validate_shape=validate_shape)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gen_state_ops.py", line 61, in assign
    use_locking=use_locking, name=name)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/util/deprecation.py", line 488, in new_func
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 3274, in create_op
    op_def=op_def)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 1770, in __init__
    self._traceback = tf_stack.extract_stack()

InvalidArgumentError (see above for traceback): Restoring from checkpoint failed. This is most likely due to a mismatch between the current graph and the graph from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:

Assign requires shapes of both tensors to match. lhs shape= [1,1,2048,1001] rhs shape= [1,1,2048,1000]
         [[node save/Assign_266 (defined at /home/glorysdj/dldt/model-optimizer/mo/front/tf/loader.py:105)  = Assign[T=DT_FLOAT, _class=["loc:@resnet_v1_50/logits/weights"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:CPU:0"](resnet_v1_50/logits/weights, save/RestoreV2:266)]]

 

11 posts / 0 new
Last post
For more complete information about compiler optimizations, see our Optimization Notice.

Dear Dongjie S,

Please make sure your Tensorflow model is a frozen protobuf first before you use Model Optimizer.  Below are instructions on using Model Optimizer to convert a Tensorflow model:

https://docs.openvinotoolkit.org/latest/_docs_MO_DG_prepare_model_convert_model_Convert_Model_From_TensorFlow.html

Also you are using an older version of OpenVino (though not too old). I strongly encourage you to download the latest 2019 R1 release.

Thanks !

Shubha

Dear Dongjie S,

I encourted exactly the same error as you described "Assign requires shapes of both tensors to match. lhs shape= [1,1,2048,1001] rhs shape= [1,1,2048,1000]" when modifying resnet_v1_50.pb model using OpenVINO 2019 R1. 

I also downloaded resnet_v1_50_2016_08_28.tar.gz from  http://download.tensorflow.org/models/resnet_v1_50_2016_08_28.tar.gz, from where our guide https://docs.openvinotoolkit.org/latest/_docs_MO_DG_prepare_model_conver... suggests.

So did you solve this issue?  Looking forward to your reply.

Thanks,

Una

Dear everyone,

By now we have 2019R2. In fact it was just released last week. Please update to the latest release and try again.

Thanks,

Shubha

Hi Shubha, I'm using 2019R2, I still suffer same issue, any idea how to solve this? Thanks

Dear Bin X,

Can you kindly be more explicit about your error ? This thread is pretty old. Is it a mismatch error using Model Optimizer ? And are you using the resnet_v1_50 model just like the Original Poster ? What is your exact model optimizer command ?

Thanks,

Shubha

Hi Shubha, I tried 2019R2 and internal 2019R3, all fail with same issue as Dongjie mentioned:

1. wget http://download.tensorflow.org/models/ resnet_v1_50_2016_08_28.tar.gz

2. tar xvf resnet_v1_50_2016_08_28.tar.gz

3. python3 research/slim/export_inference_graph.py     --model_name resnet_v1_50     --output_file resnet_v1_50_inference_graph.pb

4. python3 /opt/intel/openvino_2019.1.144/deployment_tools/model_optimizer/mo/utils/summarize_graph.py --input_model ./resnet_v1_50_inference_graph.pb

5. python3 /usr/lib/python3.6/site-packages/tensorflow/python/tools/freeze_graph.py --input_graph ./resnet_v1_50_inference_graph.pb --input_checkpoint resnet_v1_50.ckpt --input_binary=true --output_graph frozen_resnet_v1_50.pb --output_node_names=resnet_v1_50/predictions/Reshape_1

then error showed as attachment

Attachments: 

AttachmentSize
Downloadimage/png Untitled.png327.77 KB

Dear Bin X.,

Why do you think that this is an OpenVino problem ? In looking at your *.jpg it's clearly a Tensorflow (freeze_graph.py) error. In fact I found this Stack Overflow link about Tensorflow Assign error about the very same error you're getting.

This is the Model Optimizer Documentation for Tensorflow Slim .  It looks like you followed the steps in this document, based on what I see in your post.  But in your steps, you decided to use  freeze_graph.py while the MO document does not recommend this step. Instead it jumps immediately to (keep in mind it's a different model than yours):

<MODEL_OPTIMIZER_INSTALL_DIR>/mo_tf.py --input_model ./inception_v1_inference_graph.pb --input_checkpoint ./inception_v1.ckpt -b 1 --mean_value [127.5,127.5,127.5] --scale 127.5

Hope it helps,

Thanks,

Shubha

 

Hi Shubha, I run tf_mo.py directly as you pointed out, but still encounter similar issue, are you able to duplicate this problem?

[root@localhost 00464629]# python3.6 /opt/intel/openvino_2019.2.245/deployment_tools/model_optimizer/mo_tf.py --input_model ./resnet_v1_50_inference_graph.pb --input_checkpoint ./resnet_v1_50.ckpt -b 1 --mean_value [127.5,127.5,127.5] --scale 127.5
Model Optimizer arguments:
Common parameters:
    - Path to the Input Model:     /home/xiebin/00464629/./resnet_v1_50_inference_graph.pb
    - Path for generated IR:     /home/xiebin/00464629/.
    - IR output name:     resnet_v1_50_inference_graph
    - Log level:     ERROR
    - Batch:     1
    - Input layers:     Not specified, inherited from the model
    - Output layers:     Not specified, inherited from the model
    - Input shapes:     Not specified, inherited from the model
    - Mean values:     [127.5,127.5,127.5]
    - Scale values:     Not specified
    - Scale factor:     127.5
    - Precision of IR:     FP32
    - Enable fusing:     True
    - Enable grouped convolutions fusing:     True
    - Move mean values to preprocess section:     False
    - Reverse input channels:     False
TensorFlow specific parameters:
    - Input model in text protobuf format:     False
    - Path to model dump for TensorBoard:     None
    - List of shared libraries with TensorFlow custom layers implementation:     None
    - Update the configuration file with input/output node names:     None
    - Use configuration file used to generate the model with Object Detection API:     None
    - Operations to offload:     None
    - Patterns to offload:     None
    - Use the config file:     None
Model Optimizer version:     2019.2.0-436-gf5827d4
[ FRAMEWORK ERROR ]  Cannot load input model: Assign requires shapes of both tensors to match. lhs shape= [1001] rhs shape= [1000]
     [[Node: save/Assign_265 = Assign[T=DT_FLOAT, _class=["loc:@resnet_v1_50/logits/biases"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:CPU:0"](resnet_v1_50/logits/biases, save/RestoreV2:265)]]

Caused by op 'save/Assign_265', defined at:
  File "/opt/intel/openvino_2019.2.245/deployment_tools/model_optimizer/mo_tf.py", line 31, in <module>
    sys.exit(main(get_tf_cli_parser(), 'tf'))
  File "/opt/intel/openvino_2019.2.245/deployment_tools/model_optimizer/mo/main.py", line 302, in main
    return driver(argv)
  File "/opt/intel/openvino_2019.2.245/deployment_tools/model_optimizer/mo/main.py", line 251, in driver
    is_binary=not argv.input_model_is_text)
  File "/opt/intel/openvino_2019.2.245/deployment_tools/model_optimizer/mo/pipeline/tf.py", line 82, in tf2nx
    saved_model_tags=argv.saved_model_tags)
  File "/opt/intel/openvino_2019.2.245/deployment_tools/model_optimizer/mo/front/tf/loader.py", line 206, in load_tf_graph_def
    graph_def = freeze_checkpoint(graph_def=graph_def, checkpoint=checkpoint, output_node_names=outputs)
  File "/opt/intel/openvino_2019.2.245/deployment_tools/model_optimizer/mo/front/tf/loader.py", line 103, in freeze_checkpoint
    tf.train.Saver(var_list=var_list).restore(sess, checkpoint)
  File "/usr/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1284, in __init__
    self.build()
  File "/usr/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1296, in build
    self._build(self._filename, build_save=True, build_restore=True)
  File "/usr/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1333, in _build
    build_save=build_save, build_restore=build_restore)
  File "/usr/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 781, in _build_internal
    restore_sequentially, reshape)
  File "/usr/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 422, in _AddRestoreOps
    assign_ops.append(saveable.restore(saveable_tensors, shapes))
  File "/usr/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 113, in restore
    self.op.get_shape().is_fully_defined())
  File "/usr/lib/python3.6/site-packages/tensorflow/python/ops/state_ops.py", line 219, in assign
    validate_shape=validate_shape)
  File "/usr/lib/python3.6/site-packages/tensorflow/python/ops/gen_state_ops.py", line 60, in assign
    use_locking=use_locking, name=name)
  File "/usr/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "/usr/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3414, in create_op
    op_def=op_def)
  File "/usr/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1740, in __init__
    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access

InvalidArgumentError (see above for traceback): Assign requires shapes of both tensors to match. lhs shape= [1001] rhs shape= [1000]
     [[Node: save/Assign_265 = Assign[T=DT_FLOAT, _class=["loc:@resnet_v1_50/logits/biases"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:CPU:0"](resnet_v1_50/logits/biases, save/RestoreV2:265)]]

Dear Bin X.

Thanks for posting the detailed exception log. But it's clear that it's still not a Model Optimizer issue.  Please take a look. Model Optimizer is simply passing Tensorflow's error along:

  File "/opt/intel/openvino_2019.2.245/deployment_tools/model_optimizer/mo/front/tf/loader.py", line 206, in load_tf_graph_def
    graph_def = freeze_checkpoint(graph_def=graph_def, checkpoint=checkpoint, output_node_names=outputs)
  File "/opt/intel/openvino_2019.2.245/deployment_tools/model_optimizer/mo/front/tf/loader.py", line 103, in freeze_checkpoint
    tf.train.Saver(var_list=var_list).restore(sess, checkpoint)
  File "/usr/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1284, in __init__
    self.build()
  File "/usr/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1296, in build
    self._build(self._filename, build_save=True, build_restore=True)
  File "/usr/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1333, in _build
    build_save=build_save, build_restore=build_restore)
  File "/usr/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 781, in _build_internal
    restore_sequentially, reshape)
  File "/usr/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 422, in _AddRestoreOps
    assign_ops.append(saveable.restore(saveable_tensors, shapes))

Hope it helps.  It looks like your Tensorflow Model is encountering difficulties getting frozen. Why that is, I don't know.

Thanks,

Shubha

Hi Bin X.

Can you solve this problem? I have the same problem as you. If you have already solved it,can you share how you solved it.

Thanks,

Yujia

Leave a Comment

Please sign in to add a comment. Not a member? Join today