BUG - Subgraph replacement attempt to iterate over nodes that have been removed

BUG - Subgraph replacement attempt to iterate over nodes that have been removed

Hi. I'm seeing exceptions thrown when using the subgraph replacement.

In "repalcement.py" you have a function

   def find_and_replace_pattern(self, graph: nx.MultiDiGraph):
        for match in find_pattern_matches(graph, **self.pattern()):
            match = {v: k for k, v in match.items()} 
            for k in match:
                match[k] = Node(graph, match[k])
            self.replace_sub_graph(graph, match)

 

If this finds  a subgraph to match and replace, then the "replace_sub_graph(...)" method in my extension derived from FrontReplacementSubgraph is called.

If this calls "replace_node(..)" in graph.py, then that removes a node from the graph. This is modifying the graph while the "find_and_replace_pattern()" is iterating over it - and causes an exception when the iterator tries to find the node that has already been removed in the graph.

I have worked around this with changes to replace_node(...) in graph.py - not calling graph.remove_node() but simply removing the outgoing edges from the node that is being replaced. I suspect this is not how you intend the code to work, but it is letting me proceed.

I will attempt to create a customer agnostic example that I can share. Note that the behaviour of the script is non-deterministic since find_pattern_matches() will search the graph in different orders on different runs, so sometimes the pattern of execution is ok for a given graph, sometimes it throws the exception.

If you have a better arena for me to raise issues with model_optimiser as an Intel employee then I'd be happy to follow your advice on how to register and track issues. 

 

 

2 posts / 0 new
Last post
For more complete information about compiler optimizations, see our Optimization Notice.

Attached a zip file with a trivial test. 

Test input in "input"

XML Results with no subgraph replacement in Results/ncr_mult_no_subgraph_replacement.xml

The results I need are Results/ncr_mult_required.xml

My extensions are ncr_graph.py - copy to extensions/front - and neg_concat_relu.py - copy to extensions/ops

I changed caffe.py to call .. (See another forum post)

FrontReplacementSubgraph.find_and_replace_all_patterns(graph) # rik

Without the change to graph.py I get ...

[ 2018-06-14 11:59:59,971 ] [ DEBUG ] [ ncr_graph:39 ]  replace_sub_graph NCR

***replace_node ncr3/relu

[ 2018-06-14 12:00:02,823 ] [ DEBUG ] [ ncr_graph:39 ]  replace_sub_graph NCR

***replace_node ncr2/relu

***replace_node ncr1/relu

[ 2018-06-14 12:00:04,401 ] [ DEBUG ] [ ncr_graph:39 ]  replace_sub_graph NCR

[ 2018-06-14 12:00:05,150 ] [ ERROR ] [ main:230 ]  -------------------------------------------------

[ 2018-06-14 12:00:05,151 ] [ ERROR ] [ main:231 ]  ----------------- INTERNAL ERROR ----------------

[ 2018-06-14 12:00:05,151 ] [ ERROR ] [ main:232 ]  Unexpected exception happened.

[ 2018-06-14 12:00:05,151 ] [ ERROR ] [ main:233 ]  Please contact Model Optimizer developers and forward the following information:

[ 2018-06-14 12:00:05,151 ] [ ERROR ] [ main:234 ]  'ncr2/relu'

[ 2018-06-14 12:00:05,157 ] [ ERROR ] [ main:235 ]  Traceback (most recent call last):

  File "C:\Intel\computer_vision_sdk_2018.1.249\deployment_tools\model_optimizer\mo\main.py", line 222, in main

    return driver(argv)

  File "C:\Intel\computer_vision_sdk_2018.1.249\deployment_tools\model_optimizer\mo\main.py", line 202, in driver

    custom_layers_mapping_path=custom_layers_mapping_path)

  File "C:\Intel\computer_vision_sdk_2018.1.249\deployment_tools\model_optimizer\mo\pipeline\caffe.py", line 112, in driver

    FrontReplacementSubgraph.find_and_replace_all_patterns(graph) # rik

  File "C:\Intel\computer_vision_sdk_2018.1.249\deployment_tools\model_optimizer\mo\utils\replacement_pattern.py", line 42, in find_and_replace_all_patterns

    replacer.find_and_replace_pattern(graph)

  File "C:\Intel\computer_vision_sdk_2018.1.249\deployment_tools\model_optimizer\mo\front\common\replacement.py", line 121, in find_and_replace_pattern

    for match in find_pattern_matches(graph, **self.pattern()):

  File "C:\Users\rjallen\AppData\Roaming\Python\Python36\site-packages\networkx\algorithms\isomorphism\isomorphvf2.py", line 372, in subgraph_isomorphisms_iter

    for mapping in self.match():

  File "C:\Users\rjallen\AppData\Roaming\Python\Python36\site-packages\networkx\algorithms\isomorphism\isomorphvf2.py", line 308, in match

    if self.syntactic_feasibility(G1_node, G2_node):

  File "C:\Users\rjallen\AppData\Roaming\Python\Python36\site-packages\networkx\algorithms\isomorphism\isomorphvf2.py", line 632, in syntactic_feasibility

    for predecessor in self.G1.pred[G1_node]:

KeyError: 'ncr2/relu'

 

Hope this makes sense and I'm not wasting your time.

Attachments: 

AttachmentSize
Downloadapplication/zip Multi subgraph replace.zip11.1 KB

Leave a Comment

Please sign in to add a comment. Not a member? Join today