10/17/2021

install awscli for linux (python)


Try to do these command


>pip install awscli --upgrade --user

>sudo apt install awscli



Thank you.

www.marearts.com

10/13/2021

Get file list in s3 bucket folder (example python code)

 refer to source code

..

import boto3
import botocore

ACCESS_KEY = '**'
SECRET_KEY = '**'

s3_client = boto3.client(
's3',
aws_access_key_id=ACCESS_KEY,
aws_secret_access_key=SECRET_KEY,
region_name = 'eu-west-1'
)

bucket_name = '***'
folder_name = '***'

#check folder exist
try:
result = s3_client.list_objects_v2(Bucket=bucket_name, Prefix=folder_name )
for o in result['Contents']:
print(o['Key']) #file list in folder
except botocore.exceptions.ClientError as e:
print('no folder exist')

..


Thank you 

www.marearts.com



10/12/2021

aws dynamodb scan all item using paginate

 


Refer to code, Thank you!

..


import boto3

ACCESS_KEY = '***'
SECRET_KEY = '***'
dynamo_client = boto3.client(
'dynamodb',
aws_access_key_id=ACCESS_KEY,
aws_secret_access_key=SECRET_KEY,
region_name = 'eu-west-1'
)


def Search_Item(TABLE_NAME):
paginator = dynamo_client.get_paginator('scan')
operation_parameters = {'TableName': TABLE_NAME,}
page_iterator = paginator.paginate(**operation_parameters)
return page_iterator


TABLE_NAME ='****'

page_iterator = Search_Item(TABLE_NAME)
for page in page_iterator:
for item in page['Items']:
print(item['UID']['S'])

..


www.marearts.com

🙇🏻‍♂️


10/03/2021

Docker container stop by image name

 Stop docker container by image name:

imagename='mydockerimage'
docker stop $(docker ps | awk '{split($2,image,":"); print $1, image[1]}' | awk -v image=$imagename '$2 == image {print $1}')

Stop docker container by image name and tag:

imagename='mydockerimage:latest'
docker stop $(docker ps | awk -v image=$imagename '$2 == image {print $1}')

9/27/2021

docker swarm simple command

*install virtualbox

-+-

#install virtualbox on terminal
#osx
brew install virtualbox
#linux
sudo apt install virtualbox

-+-

 

* install docker-machine

...

#osx
curl -L https://github.com/docker/machine/releases/download/v0.16.2/docker-machine-`uname -s`-`uname -m` >/usr/local/bin/docker-machine && chmod +x /usr/local/bin/docker-machine
#linux
curl -L https://github.com/docker/machine/releases/download/v0.16.2/docker-machine-`uname -s`-`uname -m` >/tmp/docker-machine && chmod +x /tmp/docker-machine && sudo cp /tmp/docker-machine /usr/local/bin/docker-machine

...


*docker-machine

...

#docker-machine basic cmd
docker-machine ls
docker-machine rm name
docker-machine kill name
docker-machine create name

...



9/24/2021

python save & load argparse

 


refer to code:

..

from argparse import ArgumentParser
import json

parser = ArgumentParser()
parser.add_argument('--seed', type=int, default=8)
parser.add_argument('--resume', type=str, default='a/b/c.ckpt')
parser.add_argument('--surgery', type=str, default='190', choices=['190', '417'])
args = parser.parse_args()

with open('commandline_args.txt', 'w') as f:
json.dump(args.__dict__, f, indent=2)

parser = ArgumentParser()
args = parser.parse_args()
with open('commandline_args.txt', 'r') as f:
args.__dict__ = json.load(f)

print(args)

..


Thank you!

🙇🏻‍♂️

1D sequence (signal) data resampling (python)

 import numpy as np

import matplotlib.pyplot as plt
from scipy.signal import resample
from scipy.interpolate import interp1d

def ResampleLinear1D(original, targetLen):
original = np.array(original, dtype=np.float)
index_arr = np.linspace(0, len(original)-1, num=targetLen, dtype=np.float)
index_floor = np.array(index_arr, dtype=np.int) #Round down
index_ceil = index_floor + 1
index_rem = index_arr - index_floor #Remain

val1 = original[index_floor]
val2 = original[index_ceil % len(original)]
interp = val1 * (1.0-index_rem) + val2 * index_rem
assert(len(interp) == targetLen)
return interp

if __name__=="__main__":

original = np.sin(np.arange(256)/10.0)
targetLen = 100

# Method 1: Use scipy interp1d (linear interpolation)
# This is the simplest conceptually as it just uses linear interpolation. Scipy
# also offers a range of other interpolation methods.
f = interp1d(np.arange(256), original, 'linear')
plt.plot(np.apply_along_axis(f, 0, np.linspace(0, 255, num=targetLen)))

# Method 2: Use numpy to do linear interpolation
# If you don't have scipy, you can do it in numpy with the above function
plt.plot(ResampleLinear1D(original, targetLen))

# Method 3: Use scipy's resample
# Converts the signal to frequency space (Fourier method), then back. This
# works efficiently on periodic functions but poorly on non-periodic functions.
plt.plot(resample(original, targetLen))

plt.show()

9/22/2021

jupyter notebook to python py

 

run this command

>jupyter nbconvert your_notebook_name.ipynb --to python


thank you.


pytorch cuda definition

Their syntax varies slightly, but they are equivalent:

.to(name)

.to(device)

.cuda()

CPU

to('cpu')

to(torch.device('cpu'))

cpu()

Current GPU

to('cuda')

to(torch.device('cuda'))

cuda()

Specific GPU

to('cuda:1')

to(torch.device('cuda:1'))

cuda(device=1)

Note: the current cuda device is 0 by default, but this can be set with torch.cuda.set_device().

9/21/2021

write classification_report to txt file

Refer to code ^^ 

..

from sklearn.metrics import classification_report

report = classification_report(y_test_true_list, y_test_pred_list, labels=labels, target_names=target_names)

fp = open('report.txt', 'w')
fp.write(report)
fp.close()

..


Thank you.

www.marearts.com


python pil change dpi without save file

 


look at the code! ^^

..

#load image
tif_path = './input_img.tif'
#open image
pil_image = Image.open(tif_path)
#change dpi in memory
temp_mem_file = BytesIO()
page.save(temp_mem_file, format='png', dpi=(100,100))
#save it to pil
temp_mem_file.seek(0)
pil_image = Image.open(temp_mem_file)
#do it someting
#..
#save file
pil_image.save('./output.png')
#www.marearts.com
#study.marearts.com

..


Thank you.


9/20/2021

convert numpy.ndarray object to float


*Here is object dtype numpy.ndarray
print(oneD_pt.dtype, type(oneD_pt), oneD_pt)
> object <class 'numpy.ndarray'> [3 2 1 2 2] 

* convert float dtype numpy.ndarray
oneD_pt=oneD_pt.astype('float')
print(oneD_pt.dtype, type(oneD_pt), oneD_pt)
> float64 <class 'numpy.ndarray'> [3. 2. 1. 2. 2.]


thank you.

python argparse simple example

 

It's useful to save hyper parameter for machine learning


import argparse

paser = argparse.ArgumentParser()

args = paser.parse_args("")

args.batch_size=10

args.epoch=10


Thank you.


vs code server failed to start


try to kill the server which you want access and try to access again


1 .ctrl-shift-p and choose:
2. Remote-SSH: kill VS Code Server on Host...
3. access remote again


Thank you.


9/07/2021

nginx.conf, set to accept last path and poxy it.

 


ex)

www.xxx.yy/aaa/ping

www.xxx.yy/bbb/ping

Those urls will go to /ping proxy.


location ~ ^/.*/(ping|invocations) {
rewrite ^/.*/(.*)$ /$1 last;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_redirect off;
proxy_pass http://gunicorn/$1;
}

9/01/2021

AWS dynamoDB safe searching way, search item by scan and get it by paginator.

 refer to follow code

--

import boto3

dynamo_client = boto3.client('dynamodb')

def Search_Item(TABLE_NAME, uid):
paginator = dynamo_client.get_paginator('scan')
operation_parameters = {
'TableName': TABLE_NAME,
'FilterExpression': 'UID = :search',
'ExpressionAttributeValues': {
':search': {'S': uid},
}
}
page_iterator = paginator.paginate(**operation_parameters)
return page_iterator

UID_list = []
TABLE_NAME ='example_Table'
UID ='example_uid'

page_iterator = Search_Item(TABLE_NAME, UID)
for page in page_iterator:
for item in page['Items']:
UID_list.append(item['UID']['S'])

--


Thank you.

www.marearts.com




8/27/2021

AWS, find all file list in s3 bucket

 Find all file list in s3 bucket

import boto3
BUCKET_INPUT = 'bucket_name'
PREFIX_INPUT = 'some_prefix_folder_name'
item_list_all = []
paginator = s3_client.get_paginator('list_objects')
operation_parameters = {'Bucket': BUCKET_INPUT} #,'Prefix': PREFIX_INPUT}
page_iterator = paginator.paginate(**operation_parameters)
for page in page_iterator:
item_list_all.append(page['Contents'])
key_all = []
for page in item_list_all:
for i in range(len(page)):
key_all.append(page[i]["Key"])
print("The total number of files in the bucket:", len(key_all) )

Thank you.

www.marearts.com


8/11/2021

ERROR: Could not build wheels for tokenizers which use PEP 517 and cannot be installed directly

I met this error when I install simpletransformers or transformers.

I did many try but no luck.

But this was solution to me.

Install Rust before transformer installation.

so..

curl https://sh.rustup.rs -sSf | bash -s -- -y
PATH="/root/.cargo/bin:${PATH}"

and install transformers

Thank you.

www.marearts.com



8/08/2021

unable to prepare context: unable to evaluate symlinks in Dockerfile path: lstat /var/lib/snapd/void/Dockerfile: no such file or directory

 

if you install docker using snap, it can be happen.

so remove docker from snap installation

> sudo snap remove docker


And install docker again using apt

> sudo apt update
> sudo apt upgrade
> sudo apt install docker.io
> sudo systemctl enable --now docker


Thank you.
www.marearts.com


8/04/2021

labelme2yolo, yolo2labelme format converter

 

original page is here:

https://marearts.notion.site/Labelme2Yolo-Yolo2Labelme-3564dd886ac64b5499d6f2784a8a4be8


yolo2labelme main code

if __name__ == "__main__":
#parameters
paser = argparse.ArgumentParser()
args = paser.parse_args("")
##### set params #####
#input : yolo image, label path
args.yolo_images_path = './yolo_data/images/'
args.yolo_labels = './yolo_data/labels/'
#class count
args.yolo_nc = 80
#class names
args.yolo_names = ['aeroplane', 'apple', 'backpack', 'banana', 'baseball bat', 'baseball glove', 'bear', 'bed', 'bench', 'bicycle', 'bird', 'boat', 'book', 'bottle', 'bowl', 'broccoli', 'bus', 'cake', 'car', 'carrot', 'cat', 'cell phone', 'chair', 'clock', 'cow', 'cup', 'diningtable', 'dog', 'donut', 'elephant', 'fire hydrant', 'fork', 'frisbee', 'giraffe', 'hair drier', 'handbag', 'horse', 'hot dog', 'keyboard', 'kite', 'knife', 'laptop', 'microwave', 'motorbike', 'mouse', 'orange', 'oven', 'parking meter', 'person', 'pizza', 'pottedplant', 'refrigerator', 'remote', 'sandwich', 'scissors', 'sheep', 'sink', 'skateboard', 'skis', 'snowboard', 'sofa', 'spoon', 'sports ball', 'stop sign', 'suitcase', 'surfboard', 'teddy bear', 'tennis racket', 'tie', 'toaster', 'toilet', 'toothbrush', 'traffic light', 'train', 'truck', 'tvmonitor', 'umbrella', 'vase', 'wine glass', 'zebra']

#output : labelme output path
args.labelme_path = './output_yolo2labelme/'
args.labelme_save_image = True #False #create jpg images in output path
#######################


#######################
#start processing
yolo2labelme_main(args)
#######################


labelme2yolo main code

#parameters
paser = argparse.ArgumentParser()
args = paser.parse_args("")

##### set params #####
#input : labelme output path
args.labelme_path = './output_yolo2labelme/'
args.labelme_save_image = True #False #create jpg images in output path

#output : yolo image, label path
args.yolo_images_path = './output_yolo_data/images/'
args.yolo_labels = './output_yolo_data/labels/'
#class count
args.yolo_nc = 80
#class names
args.yolo_names = ['aeroplane', 'apple', 'backpack', 'banana', 'baseball bat', 'baseball glove', 'bear', 'bed', 'bench', 'bicycle', 'bird', 'boat', 'book', 'bottle', 'bowl', 'broccoli', 'bus', 'cake', 'car', 'carrot', 'cat', 'cell phone', 'chair', 'clock', 'cow', 'cup', 'diningtable', 'dog', 'donut', 'elephant', 'fire hydrant', 'fork', 'frisbee', 'giraffe', 'hair drier', 'handbag', 'horse', 'hot dog', 'keyboard', 'kite', 'knife', 'laptop', 'microwave', 'motorbike', 'mouse', 'orange', 'oven', 'parking meter', 'person', 'pizza', 'pottedplant', 'refrigerator', 'remote', 'sandwich', 'scissors', 'sheep', 'sink', 'skateboard', 'skis', 'snowboard', 'sofa', 'spoon', 'sports ball', 'stop sign', 'suitcase', 'surfboard', 'teddy bear', 'tennis racket', 'tie', 'toaster', 'toilet', 'toothbrush', 'traffic light', 'train', 'truck', 'tvmonitor', 'umbrella', 'vase', 'wine glass', 'zebra']
#######################


#######################
#start processing
labelme2yolo_main(args)
#######################



Thank you.

www.marearts.com

🙇🏻‍♂️

8/03/2021

make python package zip for lambda layer using docker

 Here is good reference for this subject:

https://aws.amazon.com/premiumsupport/knowledge-center/lambda-layer-simulated-docker/

https://dev.to/matthewvielkind/creating-python-aws-lambda-layers-with-docker-4376



If we use docker then it's very simple.

Let's go through step by step


1. make Main folder & python library folder

ex) In this tutorial, we suppose to need flask package

mkdir flask-layer
cd flask-layer
mkdir -pv python/lib/python3.6/site-packages
or
mkdir -pv python/lib/python3.8/site-packages


2. make requirements.txt and put packages to install by pip

flask==1.1.1


So folder structure looks like this:

├── requirements.txt └── python/ └── lib/ ├── python3.6/ │ └── site-packages/ └── python3.8/ └── site-packages/


3. Run docker to install packages

note, change python version properly

docker run -v "$PWD":/var/task "public.ecr.aws/sam/build-python3.6" /bin/sh -c "pip install -r requirements.txt -t python/lib/python3.6/site-packages/; exit"

 

4. compress package to zip

zip -r yourpack.zip python > /dev/null


5. use zip file for your lambda

That's it all.


Thank you.

www.marearts.com

🙇🏻‍♂️


7/24/2021

yolo data coordinate format, draw rectangle by cv2

yolo v5 data coordinate format


 




ex)

str_v = "32 0.262 0.7878 0.314 0.385"



#read image

cvmat = cv2.imread(img_path)



#get height, width

h,w,_ = cvmat.shape



#extract x1, y1 <- center, width, height

x1 = int( float(str_v.split(' ')[1]) * w )

y1 = int( float(str_v.split(' ')[2]) * h )

xw = int( float(str_v.split(' ')[3]) * w /2)

yw = int( float(str_v.split(' ')[4]) * h /2)



#make x1,y1, x2,y2

start_point = (x1 - xw, y1 - yw )

end_point = (x1 + xw, y1 + yw )



#draw rectangle

cvmat = cv2.rectangle(cvmat, start_point, end_point, (255, 0, 0), 2)



Thank you.

https://study.marearts.com


find all txt file in dir and read line by line in txt. python example source code

 

python code

.

import glob

files = []
start_dir = '/MareArts/dev/dataset/coco_yolov5pytorch/train/labels/'
# output_path = '/Volumes/output/'
pattern = "*.txt"
search_dir = os.path.join(start_dir,pattern)

total = 0
txt_list = glob.glob(search_dir)
for i,v in enumerate(txt_list):
print(i,v)
file1 = open(v, 'r')
Lines = file1.readlines()
count = 0
# Strips the newline character
for line in Lines:
count += 1
print("Line{}: {}".format(count, line.strip()))

.


result

0 /MareArts/dev/dataset/coco_yolov5pytorch/train/labels/000000253429_jpg.rf.7f12980ae02c096d1b9ac0b574d3ac1b.txt
Line1: 10 0.6768149882903981 0.2203125 0.6463700234192038 0.196875
Line2: 36 0.32786885245901637 0.67265625 0.6557377049180327 0.6359375
Line3: 48 0.34543325526932084 0.57578125 0.36533957845433257 0.6765625
1 /MareArts/dev/dataset/coco_yolov5pytorch/train/labels/000000009465_jpg.rf.3d5a6b94f8c1afdc004f2641cd578912.txt
Line1: 36 0.6703125 0.8002309468822171 0.2109375 0.22170900692840648
Line2: 73 0.06015625 0.684757505773672 0.1171875 0.42494226327944573
Line3: 48 0.16484375 0.7621247113163973 0.04375 0.16166281755196305
Line4: 48 0.24921875 0.7829099307159353 0.0296875 0.16628175519630484
Line5: 48 0.2234375 0.7736720554272517 0.0359375 0.17090069284064666
Line6: 48 0.19765625 0.76905311778291 0.03125 0.16628175519630484
Line7: 48 0.1265625 0.7840646651270208 0.075 0.20554272517321015
2 /MareArts/dev/dataset/coco_yolov5pytorch/train/labels/000000009466_jpg.rf.5b49bdd7c037ed299dd3b6fd2531c34d.txt
Line1: 20 0.2625 0.45433255269320844 0.3078125 0.4496487119437939
3 /MareArts/dev/dataset/coco_yolov5pytorch/train/labels/000000009469_jpg.rf.38d5d254d972459fba2431d6ed2baffe.txt
Line1: 27 0.262 0.7878048780487805 0.314 0.3853658536585366
Line2: 48 0.304 0.5146341463414634 0.282 0.9170731707317074
4 /MareArts/dev/dataset/coco_yolov5pytorch/train/labels/000000009474_jpg.rf.fe5b0057cd113427b333ccd60c2d10d5.txt
Line1: 10 0.3203125 0.5530660377358491 0.2953125 0.8702830188679245
5 /MareArts/dev/dataset/coco_yolov5pytorch/train/labels/000000009478_jpg.rf.1584a7148b62cc3612da3407115b14b7.txt
Line1: 33 0.52421875 0.5612745098039216 0.21875 0.44607843137254904
...
...


Thank you.
https://study.marearts.com

check pytorch, Tensorflow can use GPU

 

test tensorflow which can use GPU

#method 1
import tensorflow as tf
tf.test.is_built_with_cuda()
> Ture

#method 2
from tensorflow.python.client import device_lib
device_lib.list_local_devices()
> ..

test pytorch can use GPU

#method 3
import torch
torch.cuda.is_available()
>>> True

torch.cuda.current_device()
>>> 0

torch.cuda.device(0)
>>> <torch.cuda.device at 0x7efce0b03be0>

torch.cuda.device_count()
>>> 1

torch.cuda.get_device_name(0)
>>> 'GeForce GTX 950M'



Thank you.
www.MareArts.com

7/11/2021

Hide current working directory in terminal

 

do it on your terminal

$ export OLD_PS1=$PS1                    # save Long Prompt to OLD_PS1
$ export PS1="\u > "export PS1="\u $ "   # change to PS1 to Short Prompt
$ export PS1=$OLD_PS1                    # restore Long Prompt to PS1

7/09/2021

curlpp/cURLpp.hpp: No such file or directory

issue: 

curlpp/cURLpp.hpp: No such file or directory


solution:

install package which related with curl

> sudo apt-get install pkg-config libcurlpp-dev libcurl4-openssl-dev


* Check header file location

> dpkg -S cURLpp.hpp

libcurlpp-dev:amd64: /usr/include/curlpp/cURLpp.hpp


Thank you.


study.marearts.com

marearts



7/08/2021

C linux get Mac Address

 In my case, I failed to get Mac from eth0 devices.

So I need to check all devices to get Mac address.

This is my solution example code:

(study.marearts.com)


..

#include <sys/ioctl.h>
#include <net/if.h>
#include <unistd.h>
#include <netinet/in.h>
#include <string.h>
#include <stdio.h>
#include <sstream>
#include <string>
#include <iostream>
std::string getMac(){
struct ifreq ifr;
struct ifconf ifc;
char buf[1024];
int success = 0;
int sock = socket(AF_INET, SOCK_DGRAM, IPPROTO_IP);
if (sock == -1) { /* handle error*/ };
ifc.ifc_len = sizeof(buf);
ifc.ifc_buf = buf;
if (ioctl(sock, SIOCGIFCONF, &ifc) == -1) { /* handle error */ }
struct ifreq* it = ifc.ifc_req;
const struct ifreq* const end = it + (ifc.ifc_len / sizeof(struct ifreq));
int a=0;
for (; it != end; ++it) {
strcpy(ifr.ifr_name, it->ifr_name);
if (ioctl(sock, SIOCGIFFLAGS, &ifr) == 0) {
if (! (ifr.ifr_flags & IFF_LOOPBACK)) { // don't count loopback
if (ioctl(sock, SIOCGIFHWADDR, &ifr) == 0) {
success = 1;
break;
}
}
}
else {
/* handle error */
return "0";
}
}
unsigned char mac_address[6];
if (success)
{
memcpy(mac_address, ifr.ifr_hwaddr.sa_data, 6);
std::stringstream ss;
std::string target;
char buffer [50];
sprintf (buffer, "%x-%x-%x-%x-%x-%x", mac_address[0], mac_address[1], mac_address[2], mac_address[3], mac_address[4], mac_address[5]);
ss << buffer;
ss >> target;
return target;
}
return "0";
}
int main()
{
std::cout<< getMac() << std::endl;
}

..

Output is

44-a5-6e-63-a4-bb


Thank you.

marearts



python date time add or subtract

datetime package is very useful.

see the example code..

..
import datetime
print(datetime.datetime.now())
print(datetime.datetime.now().date())
lastHourDateTime = datetime.datetime.now() + datetime.timedelta(days = +10)
print(lastHourDateTime.date())
print(lastHourDateTime.time())
lastHourDateTime = datetime.datetime.now() + datetime.timedelta(hours = 10)
print(lastHourDateTime.date())
print(lastHourDateTime.time())
lastHourDateTime = datetime.datetime.now() + datetime.timedelta(hours = -10)
print(lastHourDateTime.date())
print(lastHourDateTime.time())
..
result
2021-07-08 10:19:44.969082
2021-07-08
2021-07-18
10:19:44.969146
2021-07-08
20:19:44.969164

Thank you.

study.marearts.com


7/07/2021

opencv Vector CameraParams save(yaml file store) and load(yaml file restore ) using FileNode, FileNodeIterator


Refer to load & save function.

..

std::vector<cv::detail::CameraParams> params;

bool loadCameraParams(std::string fileName){

cv::FileStorage fs(fileName+".yaml", cv::FileStorage::READ);

if(!fs.isOpened())
return false;

/////
int nObjects= 0;
fs["NC"] >> nObjects;
params.clear();
params.resize(nObjects);
/////
cv::FileNode fn = fs["cameras"];
int id=0;
for (cv::FileNodeIterator it = fn.begin(); it != fn.end(); it++,id++)
{
cv::FileNode item = *it;

cv::Mat K, R, t;
double ppx, ppy, focal, aspect;
item["K"] >> K;
item["R"] >> R;
item["t"] >> t;
item["ppx"] >> ppx;
item["ppy"] >> ppy;
item["focal"] >> focal;
item["aspect"] >> aspect;

mg_params[id].K() = (cv::Mat)K;
mg_params[id].R = R;
mg_params[id].t = t;
mg_params[id].ppx = (double)ppx;
mg_params[id].ppy = (double)ppy;
mg_params[id].focal = (double)focal;
mg_params[id].aspect = (double)aspect;
}

return true;
}

bool saveCameraParams(std::string fileName){

cv::FileStorage fs(fileName+".yaml", cv::FileStorage::WRITE);
cv::detail::CameraParams cameras = mg_params[0];
fs << "NC" << int(params.size());

fs << "cameras" << "[";
for (const auto& param : mg_params)
{
fs << "{";
fs << "K" << param.K();
fs << "R" << param.R;
fs << "t" << param.t;
fs << "ppx" << param.ppx;
fs << "ppy" << param.ppy;
fs << "focal" << param.focal;
fs << "aspect" << param.aspect;
fs << "}";
}
fs << "]";
fs.release();
return true;
}

..


Thank you!!

☕️