simple example for EDA(Exploratory Data Analysis) using Tensorflow data validation

 

refer to this page for more detail

: https://www.tensorflow.org/tfx/data_validation/get_started

..

!pip install tensorflow_data_validation

import tensorflow_data_validation as tfdv

stats = tfdv.generate_statistics_from_tfrecord(data_location=path)

tfdv.visualize_statistics(stats)

..

Thank you.

ModuleNotFoundError: No module named 'onnxruntime'

install onnxruntime

> pip install onnxruntime

Measuring processing time python

 Measure processing time

..

#metho #1

import time

start = time.time()

print("hello")

end = time.time()

print(end - start)

#method #2

from timeit import default_timer as timer

start = timer()

# ...

end = timer()

print(end - start) # Time in seconds, e.g. 5.38091952400282

..

Thank you.

www.marearts.com

Object of type float32 is not JSON serializable

 

Converting float to avoid error, refer to code:

..

import json

face_dict = {'x1': 240.54083251953125, 'y1': 470.02429199218744, 'x2': 479.535400390625, 'y2': 655.3250732421875, 'LeyeX': 382.76947021484375, 'LeyeY': 538.7545166015624, 'ReyeX': 383.48541259765625, 'ReyeY': 621.1448364257811, 'NoseX': 332.7269287109375, 'NoseY': 590.6889648437499, 'LlipsX': 300.84881591796875, 'LlipsY': 542.9485473632811, 'RlipsX': 301.3223876953125, 'RlipsY': 615.5052490234374, 'conf': 0.9999992}

data_convert = {k:float(v) for k,v in face_dict.items()}

with open('./data_convert.json', 'w') as fp:

json.dump(data_convert, fp, indent=5)

..

Another solution is:

..

with open('./data_convert.json', 'w') as fp:

json.dump(str(face_dict), fp, indent=5)

..

But in this case, json is saved as string.

Thank you.

www.marearts.com

rectangle, box face -> mosaic, pixelate

 

refer to mosaic function

..

def mosaic(img, rect, size):

(x1, y1, x2, y2) = rect

w = x2 - x1

h = y2 - y1

i_rect = img[y1:y2, x1:x2]

i_small = cv2.resize(i_rect, ( size, size))

i_mos = cv2.resize(i_small, (w, h), interpolation=cv2.INTER_AREA)

img2 = img.copy()

img2[y1:y2, x1:x2] = i_mos

return img2

#.... detect face

for face_dict in faces_dict:

x1,y1 = ( int(face_dict['x1']), int(face_dict['y1']))

x2,y2 = ( int(face_dict['x2']), int(face_dict['y2']))

# image = anonymize_face_pixelate(image[y1:y2, x1:x2, :] , blocks=3)

image = mosaic(image, (x1, y1, x2, y2), 10 )

#.... face mosaic

..

Ex) result

Thank you.

www.marearts.com

Remove duplicated dict element in list, remove same dict element in tow list

 refer to code:

..

a=[{'a':1, 'b':3}, {'a':2, 'b':4}]

b=[{'a':3, 'b':3}, {'a':2, 'b':4}]

a.extend(b)

[dict(t) for t in {tuple(d.items()) for d in a}]

>> [{'a': 1, 'b': 3}, {'a': 2, 'b': 4}, {'a': 3, 'b': 3}]

..

Thank you.

www.marearts.com

python calculate intersect of union area between two box

 refer to code:

..

def IoU(box1, box2):

# box = (x1, y1, x2, y2)

box1_area = (box1[2] - box1[0] + 1) * (box1[3] - box1[1] + 1)

box2_area = (box2[2] - box2[0] + 1) * (box2[3] - box2[1] + 1)

# obtain x1, y1, x2, y2 of the intersection

x1 = max(box1[0], box2[0])

y1 = max(box1[1], box2[1])

x2 = min(box1[2], box2[2])

y2 = min(box1[3], box2[3])

x11, y11, x12, y12 = box1

x21, y21, x22, y22 = box2

if x21<x11 and y21<y11 and x22>x12 and y22>y12:

return 1.0

if x21>x11 and y21>y11 and x22<x12 and y22<y12:

return 1.0

# compute the width and height of the intersection

w = max(0, x2 - x1 + 1)

h = max(0, y2 - y1 + 1)

inter = w * h

iou = inter / (box1_area + box2_area - inter)

return iou

..

Thank you

www.marearts.com

python yaml to dict

 refer to code

..

import yaml

with open('hparams.yaml', 'r') as stream:

try:

parsed_yaml=yaml.safe_load(stream)

print(parsed_yaml, type(parsed_yaml))

except yaml.YAMLError as exc:

print(exc)

..

> example output

{'batch_size': 1000, 'data_path': ['../npy/x_train_coin_eth.npy', '../npy/y_train_coin_eth.npy', '../npy/x_val_coin_eth.npy', '../npy/y_val_coin_eth.npy', '../npy/x_test_coin_eth.npy', '../npy/y_test_coin_eth.npy'], 'encoder_hidden_dims': [4, 2], 'input_dim': 5, 'input_seq': 128, 'learning_rate': 0.0001, 'num_LSTM': 2, 'num_layers': 1} <class 'dict'>

Thank you.
www.marearts.com

'your model' object has no attribute 'seek'. You can only torch.load from a file that is seekable. Please pre-load the data into a buffer like io.BytesIO and try to load from it instead.

 

I tired to save sub model in seq-to-seq model which is encoder part.

What I used for save and load is like follow code and I failed with error like title.

* Failed case

..

torch.save(model.lstm_auto_model.lstm_encoder.lstms, 'idx-78-encoder_lstm_encoder_lstms.mare')

torch.load(lstm_abyss_model.lstm_encoder, 'idx-78-encoder_lstm_lstm_encoder.mare')

..

* error message

AttributeError: 'LSTM' object has no attribute 'seek'. You can only torch.load from a file that is seekable. Please pre-load the data into a buffer like io.BytesIO and try to load from it instead.

..

My solution is to use 'state_dict()' to save model.

Refer to bellow code which was solution for me.

..

torch.save(model.lstm_auto_model.lstm_encoder.lstms.state_dict(), 'idx-78-encoder_lstm_encoder_lstms_dict.mare')

lstm_abyss_model.lstm_encoder.lstms.load_state_dict(torch.load('idx-78-encoder_lstm_encoder_lstms_dict.mare'))

..

Thank you.

www.marearts.com

opencv, cv2, rotate image & point

 

refer to code.

sample point is 100,100

firstly, I rotated image as 90 degree and point.

And compare origin image & pt where rotated point is placed in right position.

..

import numpy as np

import cv2

from matplotlib import pyplot as plt

def rotate_image(angle, width, height):

image_center = (width/2, height/2)

rotation_mat = cv2.getRotationMatrix2D(image_center, angle, 1.)

abs_cos = abs(rotation_mat[0,0])

abs_sin = abs(rotation_mat[0,1])

bound_w = int(height * abs_sin + width * abs_cos)

bound_h = int(height * abs_cos + width * abs_sin)

rotation_mat[0, 2] += bound_w/2 - image_center[0]

rotation_mat[1, 2] += bound_h/2 - image_center[1]

return rotation_mat, (bound_w, bound_h)

#open image

image = cv2.imread('img.png')

sample_pt =(100,100)

#get rotate mat, bound

height, width = image.shape[:2]

rotation_mat, (bound_w, bound_h) = rotate_image(90, width, height)

#rotate image

image_90 = cv2.warpAffine(image, rotation_mat, (bound_w, bound_h))

#rotate pt

sample = np.array([100, 100, 1])

sample_90 = np.matmul(rotation_mat, sample)

#origin image & pt

cv2.circle(image, sample_pt, 1, (0, 0, 255), 2, cv2.LINE_AA)

#rotated image & pt

cv2.circle(image_90, (int(sample_90[0]), int(sample_90[1])), 1, (0, 0, 255), 2, cv2.LINE_AA)

image = image[:,:,::-1]

plt.figure(figsize=(10, 10), dpi=100)

plt.imshow(image)

image_90 = image_90[:,:,::-1]

plt.figure(figsize=(10, 10), dpi=100)

plt.imshow(image_90)

..

origin image with point

90 rotated image and point

Thank you.

www.marearts.com

LSTM Autoencoder pytorch

 

Code 

...

import torch

import torch.nn as nn

from torchinfo import summary

import copy

class LSTM(nn.Module):

def __init__(self, input_dim, hidden_dims, num_layers, num_LSTM):

super(LSTM, self).__init__()

self.input_dim = input_dim

self.hidden_dims = hidden_dims

self.num_layers = num_layers

LSTMs=[]

fDim = self.input_dim

for i in range(num_LSTM):

LSTMs.append( nn.LSTM(input_size=fDim, hidden_size=hidden_dims[i], num_layers=self.num_layers, batch_first=True) )

fDim = hidden_dims[i]

self.lstms = nn.ModuleList(LSTMs)

def forward(self, x):

for i, lstm in enumerate(self.lstms):

lstm_out, (hidden_out, cell_out) = lstm(x)

x = lstm_out

last_sequence_hidden_dim = x[:,-1,:] #lstm_out[:,-1,:]

return x, last_sequence_hidden_dim

class regressor(nn.Module):

def __init__(self, input_dim, output_dim, dropout=0.1):

super(regressor, self).__init__()

self.input_dim = input_dim

self.output_dim = output_dim

self.dropout = dropout

self.regressor = self.make_regressor()

def make_regressor(self):

layers = []

layers.append(nn.Dropout(self.dropout))

layers.append(nn.Linear(self.input_dim, self.input_dim // 2))

layers.append(nn.ReLU())

layers.append(nn.Linear(self.input_dim // 2, self.output_dim))

regressor = nn.Sequential(*layers)

return regressor

def forward(self,x):

x = self.regressor(x)

return x

class LSTM_autoencoder(nn.Module):

def __init__(self, input_dim, encoder_hidden_dims, num_layers, num_LSTM, input_seq):

super(LSTM_autoencoder, self).__init__()

self.input_dim = input_dim #5

self.encoder_hidden_dims = copy.deepcopy(encoder_hidden_dims) #[256, 128, 64]

encoder_hidden_dims.reverse()

self.decoder_hidden_dims = copy.deepcopy(encoder_hidden_dims) #[64, 128, 256]

self.num_layers = num_layers #2

self.num_LSTM = num_LSTM #3

self.input_seq = input_seq

#LSTM model encoder

self.lstm_encoder = LSTM(input_dim, self.encoder_hidden_dims, num_layers, num_LSTM)

#LSTM model decoder

self.lstm_decoder = LSTM(self.decoder_hidden_dims[0], self.decoder_hidden_dims, num_layers, num_LSTM)

#LSTM regressor model

self.lstm_regressor = regressor(self.encoder_hidden_dims[0], input_dim)

def forward(self, x):

input_encoder=x

_, output_encoder = self.lstm_encoder(input_encoder)

print(f'1 - lstm encoder input:{input_encoder.shape} output:{output_encoder.shape}')

x_inter = torch.unsqueeze(output_encoder, 1)

intput_decoder = x_inter.repeat(1, self.input_seq, 1)

print(f'2 - input_decoder: {intput_decoder.shape}')

output_decoder, _ = self.lstm_decoder(intput_decoder)

print(f'3 - input decoder: {intput_decoder.shape} output decoder:{output_decoder.shape}')

output_regressor = self.lstm_regressor(output_decoder)

print(f'4 - output_regressor input: {output_decoder.shape} output decoder:{output_regressor.shape}')

return output_regressor

...

Test class and show summary

..

input_dim = 5

num_LSTM = 2

encoder_hidden_dims = [256, 128]

num_layers = 2

input_seq = 140

batch_size=100

lstm_auto_model = LSTM_autoencoder(input_dim, encoder_hidden_dims, num_layers, num_LSTM, input_seq)

summary(lstm_auto_model, input_size=(batch_size, input_seq, input_dim))

..

output

..

Refer to my ugly drawing

Thank you.

www.marearts.com

time-distributed dense (TDD, TimeDistributed) layer in PyTorch

 

refer to code:

..

import torch

m = torch.nn.Linear(256, 5)

#batch, sequence(time), dim

input = torch.randn(100, 140, 256)

output = m(input) #100, 140, 256 -> 100, 140, 5

print(output.size())

#torch.Size([100, 140, 5])

..

Thank you.

www.marearts.com

torch repeat example

 Refer to code

..

batch_size=100

input_seq=140

input_dim=5

rand_input = torch.rand(batch_size, input_seq, input_dim)

repeat_output = rand_input.repeat(1, 1, 1)

print(f'input :{rand_input.shape}, repeat:{repeat_output.shape}')

repeat_output = rand_input.repeat(1, 10, 1)

print(f'input :{rand_input.shape}, repeat:{repeat_output.shape}')

repeat_output = rand_input.repeat(1, 1, 10)

print(f'input :{rand_input.shape}, repeat:{repeat_output.shape}')

repeat_output = rand_input.repeat(10, 1, 1)

print(f'input :{rand_input.shape}, repeat:{repeat_output.shape}')

..

>>

input :torch.Size([100, 140, 5]), repeat:torch.Size([100, 140, 5])
input :torch.Size([100, 140, 5]), repeat:torch.Size([100, 1400, 5])
input :torch.Size([100, 140, 5]), repeat:torch.Size([100, 140, 50])
input :torch.Size([100, 140, 5]), repeat:torch.Size([1000, 140, 5])

Thank you.

www.marearts.com

pytorch module list example

 

ex1)

linears = nn.ModuleList([nn.Linear(10, 10) for i in range(10)])

..

ex2)

linears=[]

for i in range(10):

linears.append( nn.Linear(10, 10) )

nn.ModuleList(linears)

..

print module list

>

ModuleList(
  (0): Linear(in_features=10, out_features=10, bias=True)
  (1): Linear(in_features=10, out_features=10, bias=True)
  (2): Linear(in_features=10, out_features=10, bias=True)
  (3): Linear(in_features=10, out_features=10, bias=True)
  (4): Linear(in_features=10, out_features=10, bias=True)
  (5): Linear(in_features=10, out_features=10, bias=True)
  (6): Linear(in_features=10, out_features=10, bias=True)
  (7): Linear(in_features=10, out_features=10, bias=True)
  (8): Linear(in_features=10, out_features=10, bias=True)
  (9): Linear(in_features=10, out_features=10, bias=True)
)

Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance

 

Try #1

from pytorch_lightning.strategies.ddp import DDPStrategy

trainer = pl.Trainer(

strategy = DDPStrategy(find_unused_parameters=False),

accelerator = 'gpu',

devices = 3

)

..

Try #2

from pytorch_lightning.plugins import DDPPlugin

trainer = pl.Trainer(

val_check_interval=0.1,

gpus=-1,

accelerator="ddp",

callbacks=[checkpoint_callback, early_stop_callback],

plugins=DDPPlugin(find_unused_parameters=False),

precision=16,

)

..

Thank you.

pytorch lightning change checkpoints output dir and version str

 

Use "save_dir",  "name",  "version" params in pl_loggers function.

more detail : https://pytorch-lightning.readthedocs.io/en/stable/extensions/generated/pytorch_lightning.loggers.TensorBoardLogger.html

..

from pytorch_lightning import loggers as pl_loggers

tb_logger = pl_loggers.TensorBoardLogger(save_dir="logs", name='no_default', version='my_version_2')

trainer = Trainer(max_epochs=num_epochs, callbacks=[checkpoint_callback], logger=tb_logger)

..

And you can retrieve what is my logging path by:

tb_logger.log_dir or self.logger.log_dir (in class)

> logs/no_default/my_version_2

Thank you.

www.marearts.com

🙇🏻‍♂️

Get google stock date using pandas.

 

refer to sample code

..

import datetime

import pandas_datareader.data as pdr

# We will look at stock prices over the past year, starting at January 1, 2016

start = (2000, 12, 1)

start = datetime.datetime(*start)

end = datetime.date.today()

google = pdr.DataReader('028050.KS', 'yahoo', start, end)

google.Low.plot(grid=True)

..

Thank you.

www.marearts.com

ModuleNotFoundError: No module named 'pandas_datareader'

 

Simply install package through pip

..

> pip install pandas_datareader

..

Thank you.

skip sanity val check in pytorch lightning

 

set trainer like this:

> trainer = Trainer(num_sanity_val_steps=0)

Thank you.