Most WANTED PyTorch Tensor operations!

In this post, we will introduce ourselves to PyTorch. It is developed by Facebook AI research lab in 2016. The library is mostly used for computer vision, machine learning, and deep learning applications. Currently, I am participating in a bootcamp called “Deep Learning with PyTorch: Zero to GANs”. In the scope of this program, we will learn and implement many things by using this library. As I progress through the bootcamp, I aim to share what I have learned here!

If you have no idea about PyTorch, don’t worry. I am also a complete beginner in deep learning. We will go through definitions and explanations in between subjects and hopefully learn together. Let’s get started!

What is PyTorch?

PyTorch is a Python-based scientific computing package targeted at two sets of audiences:

• A replacement for NumPy to use the power of GPUs
• a deep learning research platform that provides maximum flexibility and speed. [0]

The definition above raises this question: What the hell is GPU’s and what does it mean to use the power of them?

Well, GPU stands for Graphical Processing Unit. They are said to be more powerful and fast when compared to CPU’s (Central Processing Unit) for doing some specific tasks.
*This is a huge topic, so we won’t go into details here. If you are curious about it, kindly check out this link for more info: https://bit.ly/2JjYnh8

PyTorch uses tensors to do numerical computations. It includes many functions that will help us to do these calculations easily.

A tensor is a number, vector, matrix, or any n-dimensional array.

To decide which functions to introduce, I used Google Trends! This platform allows us to see trends of people’s searches on Google. Here’s what I got when I typed “PyTorch tensor”.

Interest over time for a year

Looks like China loves PyTorch!

From the analysis, people mostly search for the following functions for PyTorch tensors:

Here you can see the top-10 searches about PyTorch tensors.

So, we will cover the following functions:

• torch.from_numpy() — .numpy()
• torch.size() — torch.shape()
• torch.rand()
• .clone()
• .reshape()
• torch.Tensor.tolist()

How to turn a numpy array into PyTorch tensor (and vice-versa)?

It is relatively easy to convert numpy arrays to tensors. We will use torch.from_numpy() function. Let’s see it on an example.

Numpy is another library that makes scientific mathematical operations easier.

np is an alias used for numpy here. This way we can use numpy methods with typing just np instead of numpy. We could have used anything, but it’s best to follow common conventions to write more readable and understandable code.

We created our array, now it’s time to import torch and convert it.

Voila! We have a brand new tensor. Could it be any easier!?

There are a few things to consider when we are converting numpy array to tensor. For example:

Oops… The same element in numpy array also changed. That’s because t1 and nd_arr share the same memory. So any modifications on either will affect the other. Watch out!

from_numpy() supports only some specific data types.

We got a Type Error because only the following data types are supported: float64, float32, float16, complex64, complex128, int64, int32, int16, int8, uint8, and bool.

Another important thing to keep in mind: the tensors we converted from numpy arrays are not resizable. In other words, you cannot change dimensions of the tensors.

For more information about this function, you can check out the documentation: https://bit.ly/33E9Cs3

Now we have learned how to convert numpy arrays to pytorch tensors, it’s time to do it the other way. Turn tensor to numpy array! For this, we will create a new tensor.

For this process, we will use .numpy() function.

Since numpy array supported data types are also supported by PyTorch, we won’t get a type error.

Don’t forget : Tensor and converted numpy array shares the same memory location.

This operation is called as “the numpy bridge”. For more information, check out the documentation: https://bit.ly/33Drcw6

What is the size & shape of my tensor?

Size and shape concepts are a bit different when we compare numpy and PyTorch. Size refers to an integer, the number of elements, in a numpy array. Shape refers you to a tuple (a,b) where:
- a=number of rows in the array and
- b= number of columns in the array.

For PyTorch, using size() and shape has no difference. This is how we learn about the size/shape of a tensor.

As you can see, concepts are abit different for numpy and PyTorch. Here’s a good resorce to learn more about it: https://bit.ly/2IbbEZ8

How to form a tensor with random numbers?

PyTorch has many different functions allowing us to create randomized tensors.

• torch.rand() : Returns a tensor filled with random numbers from a uniform distribution on the interval [0, 1)
• torch.rand_like(input) : Returns a tensor with the same size as input that is filled with random numbers from a uniform distribution on the interval [0, 1).
• torch.randint() : Returns a tensor filled with random integers generated uniformly between low (inclusive) and high (exclusive).
• torch.randint_like(input) : Does the exact same operation with randint but specifies its sizes with the input tensor.
• torch.randn() : Returns a tensor filled with random numbers from a normal distribution with mean 0 and variance 1 (also called the standard normal distribution).
• torch.randn_like(input) : Does the exact same operation with randn but specifies its size with the input tensor.
• torch.randperm() : Returns a random permutation of integers from 0 to n - 1.

We will only implement torch.rand() here as the other functions are very similar. Check out the documentation for further info: https://bit.ly/3mxzHjP

If you run the cell again, you will see different elements .

The specified size must be of type integer.

How to take copies/clone of a tensor?

There are more than one ways we can clone a tensor [1].

y = tensor.new_tensor(x) #a

y = x.detach().clone() or x.clone().detach() #b

y = torch.empty_like(x).copy_(x) #c

y = torch.tensor(x) #d

According to stackoverflow, b is explicitly preferred over others. So we will implement that now. You can learn more about the reason behind it in this post: https://bit.ly/2JCwyRp

In this version, we detach the tensor from its computational path and clone or we first clone and then detach. If you first detach the tensor and then clone it, the computation path is not copied, the other way around it is copied and then abandoned. Thus, .detach().clone() is very slightly more efficient.[1]

We easily cloned our tensor!

Also, when we clone a tensor the new one does not share the same memory location with the original one. But some cloning methods share the same memory, make sure you pick the one you need!

Another way of doing the same operation is this, but it’s not the best practice. So it’s better to avoid using this one.

When you use this approach it also works, but PyTorch throws a little warning that looks like this: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).

The difference between two methods is explained very well in here:
https://bit.ly/3qvWv5J

How to reshape a tensor?

PyTorch offers different ways to manipulate sizes of tensors. We will introduce one of them here and leave the remaining options to you. Check them out here: https://bit.ly/3lHnoQN

We will use torch.reshape() method. It basically returns a tensor that has the same number of elements but in a different shape. In other words, number of rows and columns are different.

When you specify the new size in a tuple, it automatically creates new tensor for you.

Be careful, the resized tensor and original one share the same memory location!

How to create a tensor from list (and vice versa)?

Turning lists to tensors are really easy. The process is very similar to turning numpy array to a tensor.

As long as lists contain appropriate data types and sizes, tensors can be created easily. Let’s see it over an example:

Make sure your dimensions are appropriate to form a tensor!

We can also convert tensors into lists by using torch.Tensor.tolist() function.

That’s all for now! Hope you enjoyed this tutorial. Stay tuned for the upcoming ones and don’t forget to practice them by yourself.

The link to the notebook including codes of this post can be found here: https://jovian.ai/semanurkps/most-wanted-tensor-operations

Jovian AI — Deep Learning with PyTorch: Zero to GANs
https://jovian.ai/learn/deep-learning-with-pytorch-zero-to-gans

REFERENCES

[0] https://pytorch.org/tutorials/beginner/blitz/tensor_tutorial.html

[1] https://stackoverflow.com/questions/55266154/pytorch-preferred-way-to-copy-a-tensor