How to Use 1D Convolution in Tensorflow
TensorFlow now supports 1D convolution since version r0.11, using
Consider a basic example with an input of length
10, and dimension
16. The batch size is
32. We therefore have a placeholder with input shape
[batch_size, 10, 16].
batch_size = 32 x = tf.placeholder(tf.float32, [batch_size, 10, 16])
We then create a filter with width 3, and we take
16 channels as input, and output also
filter = tf.zeros([3, 16, 16]) # these should be real values, not 0
Finally we apply
tf.nn.conv1d with a stride and a padding:
- stride: integer s
- padding: this works like in 2D, you can choose between
SAMEwill output the same input length, while
VALIDwill not add zero padding.
For our example we take a stride of 2, and a valid padding.
output = tf.nn.conv1d(x, filter, stride=2, padding="VALID")
The output shape should be
[batch_size, 4, 16].
padding="SAME", we would have had an output shape of
[batch_size, 5, 16].
For previous versions of TensorFlow, you can just use 2D convolutions while setting the height of the inputs and the filters to
Math behind 1D convolution with advanced examples
To calculate 1D convolution by hand, you slide your kernel over the input, calculate the element-wise multiplications and sum them up.
The easiest way is for padding=0, stride=1
So if your
input = [1, 0, 2, 3, 0, 1, 1] and
kernel = [2, 1, 3] the result of the convolution is
[8, 11, 7, 9, 4], which is calculated in the following way:
- 8 = 1 * 2 + 0 * 1 + 2 * 3
- 11 = 0 * 2 + 2 * 1 + 3 * 3
- 7 = 2 * 2 + 3 * 1 + 0 * 3
- 9 = 3 * 2 + 0 * 1 + 1 * 3
- 4 = 0 * 2 + 1 * 1 + 1 * 3
TF's conv1d function calculates convolutions in batches, so in order to do this in TF, we need to provide the data in the correct format (doc explains that input should be in
[batch, in_width, in_channels], it also explains how kernel should look like). So
import tensorflow as tf i = tf.constant([1, 0, 2, 3, 0, 1, 1], dtype=tf.float32, name='i') k = tf.constant([2, 1, 3], dtype=tf.float32, name='k') print i, '\n', k, '\n' data = tf.reshape(i, [1, int(i.shape), 1], name='data') kernel = tf.reshape(k, [int(k.shape), 1, 1], name='kernel') print data, '\n', kernel, '\n' res = tf.squeeze(tf.nn.conv1d(data, kernel, 1, 'VALID')) with tf.Session() as sess: print sess.run(res)
which will give you the same answer we calculated previously:
[ 8. 11. 7. 9. 4.]
Convolution with padding
Padding is just a fancy way to tell append and prepend your input with some value. In most of the cases, this value is 0, and this is why most of the time people name it zero-padding. TF support 'VALID' and 'SAME' zero-padding, for arbitrary padding you need to use tf.pad(). 'VALID' padding means no padding at all, where the same means that the output will have the same size as the input. Let's calculate the convolution with
padding=1 on the same example (notice that for our kernel this is 'SAME' padding). To do this we just append our array with 1 zero at the beginning/end:
input = [0, 1, 0, 2, 3, 0, 1, 1, 0].
Here you can notice that you do not need to recalculate everything: all the elements stay the same except of the first/last one which are:
- 1 = 0 * 2 + 1 * 1 + 0 * 3
- 3 = 1 * 2 + 1 * 1 + 0 * 3
So the result is
[1, 8, 11, 7, 9, 4, 3] which is the same as calculated with TF:
res = tf.squeeze(tf.nn.conv1d(data, kernel, 1, 'SAME')) with tf.Session() as sess: print sess.run(res)
Convolution with strides
Strides allow you to skip elements while sliding. In all our previous examples we slided 1 element, now you can slide
s elements at a time. Because we will use a previous example, there is a trick: sliding by
n elements is equivalent to sliding by 1 element and selecting every n-th element.
So if we use our previous example with
padding=1 and change
stride to 2, you just take the previous result
[1, 8, 11, 7, 9, 4, 3] and leave each 2-nd element, which will result in
[1, 11, 9, 3]. You can do this in TF in the following way:
res = tf.squeeze(tf.nn.conv1d(data, kernel, 2, 'SAME')) with tf.Session() as sess: print sess.run(res)