조마조마

Pytorch provides two main ways of doing this:

• The less recommended way is to save the entire model object as follows:

torch.save(model, PATH_TO_MODEL)

And then, the saved model can be later read as follows:

model = torch.load(PATH_TO_MODEL)

Although this approach looks the most straightforward, this can be problematic in some cases. This is because we are not only saving the model parameters, but also the model classes and directory structure used in our source code. If our class signatures or directory structures change later, loading the model will fail in potentially unfixable ways.

• The second and more recommended way is to only save the model parameters as follows:

torch.save(model.state_dict(), PATH_TO_MODEL)

Later, when we need to restore the model, first we instantiate an empty model object and then load the model parameters into that model object as follows:

model = Net()
model.load_state_dict(torch.load(PATH_TO_MODEL)

We will use the morte recommended way to save the model as shown in the following code:

PATH_TO_MODEL = "./convnet.pth"
torch.save(model.state_dict(), PATH_TO_MODEL)

The convnet.pth file is essentially a pickle file containing model parameters.

※ reference: Mastering Pytorch

COLORS = np.random.randint(0, 255, size=(len(classes), 3), dtype="uint8")

# draw loop
	color = [int(c) for c in COLORS[idx]]

※ reference: pytorch computer vision codebook

define an Adam optmizer object with a learning rate of 1e-4:

from torch import optim
opt = optim.Adam(model_resnet18.parameters(), lr=1e-4)

we can read the current value of the learning rate using the following function:

def get_lr(opt):
	for param_group in opt.param_groups:
    	return param_group['lr']
        
current_lr = get_lr(opt)
print('current  lr = {}'.format(current_lr)

define a learning scheduler using the CosineAnnealingLR method:

from torch.optim.lr_scheduler import CosineAnnealingLR
lr_schedular = CosineAnnealingLR(opt, T_max=2, eta_min=1e-5)

※ reference: pytorch computer vision codebook

let's get the weight of the first layer:

for w in model_resnet18.parameters():
	w = w.data.cpu()
    print(w.shape)
    break

then, normalize the weights:

min_w = torch.min(w)
w1 = (-1 / (2 * min_w)) * w + 0.5
print(torch.min(w1).item(), torch.max(w1).item())

next, make a grid and display it:

grid_size = len(w1)
x_grid = [w1[i] for i in range(grid_size)]
x_grid = utils.make_grid(x_grid, nrow=8, padding=1)
print(x_grid.shape)

plt.figure(figsize=(10, 10))
show(x_grid)

※ reference: pytorch computer vision codebook

# a deep copy of weights for the best performing model
best_model_wts = copy.deepcopy(model.state_dict())

# initialize best loss to a large value
best_loss=float('inf')

# main loop
	.
	.
	.
	# store best model
	if val_loss < best_loss:
		best_loss = val loss
    	best_model_wts = copy.deepcopy(model.state_dict())
    	# store weights into a local file
    	torch.save(model.state_dict(), path2weights)
    	print("Copied best model weights!")

Once training is complete, we'll want to store the trained parameters in a file for deployment and future use.

There are two ways of doing so.

Let's look at the first method:

1. First, we will store the model parameters or state_dict in a file:

# define path2weights
path2weights="./models/weights.pt"

# store state_dict to file
torch.save(model.state_dict(), path2weights)

2. To load the model parameters from the file, we will define an object of the Net class:

# define model: weights are randomly inintiated
_model = Net()

3. Then, we will load state_dict from the file:

weights=torch.load(path2weights)

4. Next, we will set state_dict to the model:

_model.load_state_dict(weights)

In the first method, we stored state_dict or model parameters only.

Whenever we need the trained model for deployment, we have to create an object of the model, then load the parameters from the file, and then set the parameters to the model.

This is the recommended method by PyTorch creators.

Let's look at the second method:

1. First, we will store the model in a file:

# define a path2model
path2model="./models/model.pt"

# store model and weights into a file
torch.save(model, path2model)

2. To load the model parameters from the file, we will define an object of the Net class:

#define model: weights are randomly initiated
_model = Net()

3. Then, we will load the model from the local file:

_model=torch.load(path2model)

In second method, we stored the model into a file.

In other words, we stored both the model and state_dict into one file. 

Whenever we need the trained model for depolyment, we need to create an object of the Net class. 

Then, we loaded the model from the file. 

So, there is no actual benefit of doing this compared to the previous method.

※ reference: pytorch computer vision code book

use torchsummary package to get summary of the model to see the output shape and the number of parameters in each layer.

1. install the torchsummary package:

pip install torchsummary

2. let's get the model summary using torchsummary:

from torchsummary import summary
summary(model, input_size=(channels, H, W))

----------------------------------------------------------------
        Layer (type)               Output Shape         Param #
================================================================
            Conv2d-1           [-1, 10, 24, 24]             260
            Conv2d-2             [-1, 20, 8, 8]           5,020
         Dropout2d-3             [-1, 20, 8, 8]               0
            Linear-4                   [-1, 50]          16,050
            Linear-5                   [-1, 10]             510
================================================================
Total params: 21,840
Trainable params: 21,840
Non-trainable params: 0
----------------------------------------------------------------
Input size (MB): 0.00
Forward/backward pass size (MB): 0.06
Params size (MB): 0.08
Estimated Total Size (MB): 0.15
----------------------------------------------------------------

※ reference: github.com/sksq96/pytorch-summary

rtx 30xx번대 ampere 아키텍처 정식 지원 pytorch 1.7.1(stable) 버전이 나왔다.

기존 1.7.0 버전에서는 30xx번대에서 jit 오류를 뿜어댔으나

나를 포함한 다른 유저들의 열화와 같은 성원에

nightly 버전(1.8)부터 해결되더니 드디어 stable(1.7.1) 버전도 나왔다.

아마 기존에는 anaconda의 cuda 버전이 낮아서 안되지 않았나 싶다.

각설하고, 윈도우 환경에서 pytorch 1.7.1 + cuda 11.0 + cudnn 8.0.5로 기존 코드들 깔끔하게 동작한다.

30xx번대 그래픽 카드 사용자들은 pytorch 업데이트 해서 사용해보자.