Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
Tips and Traps¶
The broadcast concept in numpy is essentially a way to “virtually” duplicate data in a numpy array so that it is “virtually” reshaped to be compatible with another numpy array for a certain operation. Do not confused yourself about it with the broadcast concept in Spark which sends a full copy of a (small) DataFrame to each work node for
BroadCastJoin.numpy.expand_dims expands the shape of an array and returns a veiw (no copy is made). It is very useful to help broadcasting arrays.
numpy.expand_dims¶
numpy.expand_dims returns a view (no copy is made) to the expanded array.
The example below illustrate this.
Create a 2-d array a1.
a1 = np.array([[1, 2, 3], [4, 5, 6]])
a1array([[1, 2, 3],
[4, 5, 6]])Expand the dimension of a1 to be (2, 3, 1).
a2 = np.expand_dims(a1, axis=2)
a2.shape(2, 3, 1)a2[:, :, 0]array([[1, 2, 3],
[4, 5, 6]])Update an element of a2.
a2[0, 0, 0] = 1000Notice that a1 is updated too.
a1array([[1000, 2, 3],
[ 4, 5, 6]])numpy.reshape and numpy
Both functions reshape the dimension of an array without changing the data. A view (instead of a copy) of the original array is returned. The example below illustrates this.
Create a 2-d array a1.
a1 = np.array([[1, 2, 3], [4, 5, 6]])
a1array([[1, 2, 3],
[4, 5, 6]])Reshape the array a1.
a2 = a1.reshape((2, 3, 1))
a2.shape(2, 3, 1)a2[:, :, 0]array([[1, 2, 3],
[4, 5, 6]])Update an element of a2.
You can pass the shape parameters as individual parameters instead of passing it as a tuple.
a1.reshape(2, 3, 1)array([[[1000],
[ 2],
[ 3]],
[[ 4],
[ 5],
[ 6]]])a2[0, 0, 0] = 1000Notice that a1 is updated too.
a1array([[1000, 2, 3],
[ 4, 5, 6]])Use numpy.expand_dims to Help Broadcast Arrays¶
All of numpy.expand_dims, numpy.reshape and numpy.array.reshape
can be used to reshape an array to help broadcasting.
The below illustrates how to use numpy.expand_dims to help broadcast numpy arrays
using an example of manipulating images.
Read in an image.
!wget https://user-images.githubusercontent.com/824507/128439087-0c935d86-bb34-4c2c-8e69-6d78b3022833.png -O 4s.jpg--2021-08-05 17:48:59-- https://user-images.githubusercontent.com/824507/128439087-0c935d86-bb34-4c2c-8e69-6d78b3022833.png
Resolving user-images.githubusercontent.com (user-images.githubusercontent.com)... 185.199.110.133, 185.199.111.133, 185.199.109.133, ...
Connecting to user-images.githubusercontent.com (user-images.githubusercontent.com)|185.199.110.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 4588 (4.5K) [image/png]
Saving to: ‘4s.jpg’
4s.jpg 100%[===================>] 4.48K --.-KB/s in 0s
2021-08-05 17:48:59 (17.2 MB/s) - ‘4s.jpg’ saved [4588/4588]
from PIL import Image
import numpy as np
img = Image.open("4s.jpg")
img
Convert the image to a numpy array.
arr = np.array(img)
arr.shape(54, 37, 3)Get the sum of channels.
channel_sum = arr.sum(axis=2, dtype=np.float32) + 0.01
channel_sum.shape(54, 37)Now suppose we want to calculate the ratio of each channel to this sum.
It won’t work if we use arr / channel_sum as the dimensions of the 2 arrays
are not compatible for broadcasting.
One solution is to expand the dimension of channel_sum to (54, 37, 1)
which is compatible for broadcasting with arr.
Notice that numpy.expand_dims returns a view (no copy is made) of the dim-expanded array.
np.expand_dims(channel_sum, axis=2).shape(54, 37, 1)ratios = arr / np.expand_dims(channel_sum, axis=2)
ratios.shape(54, 37, 3)If the values of the 3 channes are close enough (by comparing the max/min values of the ratios), make the corresponding pixles white.
ratio_max = ratios.max(axis=2)
ratio_min = ratios.min(axis=2)
mask = (ratio_max - ratio_min) < 0.35
arr[mask, :] = 255Image.fromarray(arr)
Notice that the slight shading effect in the original picture is removed.