Understanding Tensorflow Control Dependencies

I am trying to gain a stronger grasp of TensorFlow. I came across the concept of control dependencies. I understand that the order of ops as specified by us is not really relevant

Solution 1:

In the snippet you have posted, tf.control_dependencies is not having any effect. The function creates a context where new operations are created with a control dependency to the given operations, but in your code there are no new operations within the context, just evaluation of previously existing operations.

In most cases, control flow in TensorFlow is "obvious", in the sense that there is only one way to make a computation correctly. However, when stateful objects (i.e. variables) are involved, there are situations that may be ambiguous. Consider the following example:

import tensorflow as tf

v1 = tf.Variable(0)
v2 = tf.Variable(0)
upd1 = tf.assign(v1, v2 + 1)
upd2 = tf.assign(v2, v1 + 1)
init = tf.global_variables_initializer()

v1 and v2 are both variables initialized to 0 and then updated. However, each use the value of the other variable in the update. In a regular Python program things would run sequentially, so upd1 would run first (so v1 would be 1) and upd2 after (so v2 would be 2, because v1 was 1). But TensorFlow does not record the order in which operations are created, only their dependencies. So it may also happen that upd2 runs before upd1 (so v1 would be 2 and v2 would be 1) or that both update values (v2 + 1 and v1 + 1) are computed before the assignments (so both v1 and v2 would be 1 in the end). Indeed, if I run it several times:

for i in range(10):
    with tf.Session() as sess:
        sess.run(init)
        sess.run([upd1, upd2])
        print(*sess.run([v1, v2]))

I do not always get the same result (personally I get 1 1 and 2 1, although technically 1 2 would also be possible). If for example you wanted to compute the new value for v2 after v1 has been updated, you could just do the following:

import tensorflow as tf

v1 = tf.Variable(0)
v2 = tf.Variable(0)
upd1 = tf.assign(v1, v2 + 1)
upd2 = tf.assign(v2, upd1 + 1)
init = tf.global_variables_initializer()

Here the new value v2 is computed using upd1, which is guaranteed to be the value of the variable after the update. So here upd2 would have an implicit dependency to the assignment, and so things would work as expected.

But what if you wanted to always compute the new values for v1 and v2 using the non-updated variable values (that is, consistently end up with both v1 and v2 being 1)? In that case you can use tf.control_dependencies:

import tensorflow as tf

v1 = tf.Variable(0)
v2 = tf.Variable(0)
new_v1 = v2 + 1
new_v2 = v1 + 1
with tf.control_dependencies([new_v1, new_v2]):
    upd1 = tf.assign(v1, new_v1)
    upd2 = tf.assign(v2, new_v2)
init = tf.global_variables_initializer()

Here, the assignment operations cannot happen until the new values for v1 and v2 have been computed, so their final values will always be 1 in both cases.