12/19/2021

semantic segmentation vs instance segmentation

 




* semantic segmentation
Finding out which class each pixel belongs to

* instance segmentation
To find out the pixel position of a detected object class based on object detection(localisation)

Is that confuse? ^^

Thank you.
πŸ™‡πŸ»‍♂️

www.marearts.com

12/13/2021

docker MongoDB pull and run

 

#pull image

docker pull mongodb

#run container

docker run --name mongodb -d -p 27017:27017 mongo


ex) access in python 

import pymongo # package for working with MongoDB
client = pymongo.MongoClient("mongodb://localhost:27017/")


Thank you.

www.marearts.com


12/10/2021

Retrive docker container IP address



replace 'container_name_or_id' -> 'you id or container name'

docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' container_name_or_id

Thank you.

mongodb python, Table recode export, loading, save csv, load csv

 refer to below code:


.

#connet db
client = pymongo.MongoClient("mongodb://localhost:27017/")
#connet table
who_table = db["who_table"]

#delete all
who_table.drop()

#load csv to table
df = pd.read_csv('1.csv')
data = df.to_dict('records')
who_table.insert_many(data)
#show all items
for x in who_table.find():
print(x)

#save table to csv
cursor = who_table.find({})
df = pd.DataFrame(list(cursor))
df.to_csv('2.csv', index=False)

.


Thank you.πŸ™‡πŸ»‍♂️

www.marearts.com


12/09/2021

IP 2 Geometry information python

 Here is official page:

https://pypi.org/project/ip2geotools/

https://github.com/tomas-net/ip2geotools


This is my try

..

# pip install ip2geotools
from ip2geotools.databases.noncommercial import DbIpCity
response = DbIpCity.get('165.132.116.105', api_key='free')
print(response)

..

The results :

165.132.116.105

Seodaemun-gu

Seoul

KR

37.5790747

126.9367861


Thank you. πŸ™‡πŸ»‍♂️

www.marearts.com


12/08/2021

increase number in dynamodb field key value

 

Get Access permission

.

import boto3
##########################################################
ACCESS_KEY = 'xxxx'
SECRET_KEY = 'xxx'
dynamo_client = boto3.client('dynamodb',
aws_access_key_id = ACCESS_KEY,
aws_secret_access_key = SECRET_KEY
)

.


sample code

.

#update when db
def update_when_db(who, when):
dynamo_client.update_item(
TableName='president_when', #table name
Key={'when': {'S': when}}, #primary key filed name, #key value, if key file not exist, it will create
UpdateExpression="ADD #counter :increment",
ExpressionAttributeNames={'#counter': who}, #key filed name that you want to increase
ExpressionAttributeValues={':increment': {'N': '1'}} #increase 1
)

.


Thank you.

πŸ™‡πŸ»‍♂️


www.marearts.com




Rotation(orientation) correction when uploading photos from mobile (python, pillow)

 Try to use ImageOps function.

Solution is simple refer to below code:

.

#pip install Pillow
from PIL import Image, ImageOps
image = Image.open('img.jpg') #open image
#### orient fix
image = ImageOps.exif_transpose(image)
#### go futher!

.


Thank you.

πŸ™‡πŸ»‍♂️

www.marearts.com


12/06/2021

Signature & Handwritten Detector on document image

Handwritten & Signature Detector on Document Image


* Live Demo

✍🏼 Signature & Handwritten Detector on Document

http://signhand.marearts.com/


* API explanation

https://www.marearts.com/Signature-Handwritten-Detector-c81f875ddb66490a8db990b50ce6f665


----------------------------------------

🌷 MareArts Free API List 🌷

MareArts 🎬 Live 

http://live.marearts.com


✍🏼 Signature & Handwritten Detector on Document

http://signhand.marearts.com/


πŸš™ Auto Number Plate Recognition - Korean

http://redkongkr.marearts.com/


🚦 Car License Plate detector

http://lpdetector.marearts.com/


🏳️‍🌈 Representative Color Extractor

http://recolor.marearts.com/


πŸ““ OCR Tess-v5

http://tessv5.marearts.com/


πŸ“ Regular Expression Generator

http://regular.marearts.com/


🐠 QRcode Generator

http://qr.marearts.com/


🦧 Barcode Generator

http://barcode.marearts.com/


🌊 Elastic Image Generator

http://elastic.marearts.com/


🏁 Mosaic Image Generator

http://mosaic.marearts.com/


⚽️ PDF to IMG convertor

http://pdf2img.marearts.com/


πŸ€ PDF to TXT convertor

http://pdf2txt.marearts.com/


🏈 PDF to HTML convertor

http://pdf2html.marearts.com/


⚾️ PDF to JSON convertor

http://pdf2json.marearts.com/


πŸ™‡πŸ»‍♂️ Thank you!

https://www.marearts.com

-----------------------------------

12/02/2021

How to combine multiple criterions to a loss function? Multiple loss function for single model.

You can simply reference below code:


ex1)

b = nn.MSELoss()(output_x, x_labels) a = nn.CrossEntropyLoss()(output_y, y_labels) loss = a + b loss.backward()


ex2)

b = nn.MSELoss() a = nn.CrossEntropyLoss() loss_a = a(output_x, x_labels) loss_b = b(output_y, y_labels) loss = loss_a + loss_b loss.backward()


And there are many opinions in here:

https://discuss.pytorch.org/t/how-to-combine-multiple-criterions-to-a-loss-function/348/27


Thank you.

www.marearts.com

πŸ™‡πŸ»‍♂️

12/01/2021

pdf2img, pdf to image, python library

 way #1

--

# pip install pdf2image
from pdf2image import convert_from_path
pdffile = '2081033884.pdf'
pages = convert_from_path(pdffile, 500)
#Saving pages in jpeg format
for i, page in enumerate(pages):
page.save(f'pdf2image_{i}.jpg', 'JPEG')

--



way #2

--

#pip install pymupdf
import fitz
pdffile = '2081033884.pdf'
doc = fitz.open(pdffile)
#split pages
for i, page in enumerate(doc.pages()):
pix = page.get_pixmap()
img_filename = f"fitz_{i}.jpg"
pix.pil_save(img_filename, format="jpeg", dpi=(300,300)) #, ... more PIL parameters)

--


Thank you.

www.marearts.com

πŸ™‡πŸ»‍♂️

11/28/2021

python, simple mkdir and remove folder

import os
#remove folder
os.system('rm -rf ./folder1/folder2/')
#make folder
os.system('mkdir -pv ./folder1/folder2/')

 

Thank you.

www.marearts.com

πŸ™‡πŸ»‍♂️

print pdf from html, python code

#pip install weasyprint
from weasyprint import HTML
#read html
HTML_path = './sample.html'
#convert html2pdf with coordinate
pdf = HTML(HTML_path).write_pdf()
#save pdf
open('sample_weasyprint.pdf', 'wb').write(pdf)

 

Thank you.

www.marearts.com


python polygon list to rectangle

 refer to below code:

--

def returnRect(ploygon):
x_list = []
y_list = []
for v in polygon:
x_list.append(v[0])
y_list.append(v[1])

left = min(x_list)
right = max(x_list)
top = min(y_list)
bottom = max(y_list)
return left, top, right, bottom

polygon = [[776.1818181818181, 221.2121212121212], [895.8787878787879, 219.6969696969697], [892.8484848484848, 198.48484848484847], [879.2121212121211, 146.96969696969697], [795.8787878787879, 150.0], [779.2121212121211, 160.6060606060606]]
x1,y1,x2,y2 = returnRect(polygon)
print(x1,y1,x2,y2)

--


Thank you.

www.marearts.com


11/12/2021

compute_class_weight() takes 1 positional argument but 3 were given

put a explicit parameter name like:

from sklearn.utils import class_weight
class_weights=class_weight.compute_class_weight(class_weight='balanced',classes=np.unique(y),y=y)


Thank you.

www.marearts.com


11/01/2021

.local/lib/python3.8/site-packages/kaggle/api/kaggle_api_extended.py", line 164, in authenticate raise IOError('Could not find {}. Make sure it\'s located in'

If you are using Linux:

1. pip install kaggle
2. cd ~/.kaggle
3. homepage www.kaggle.com -> Your Account -> Create New API token
4. mv ~/Downloads/kaggle.json ./
5. chmod 600 ./kaggle.json

Thank you πŸ™‡πŸ»‍♂️

10/17/2021

install awscli for linux (python)


Try to do these command


>pip install awscli --upgrade --user

>sudo apt install awscli



Thank you.

www.marearts.com

10/13/2021

Get file list in s3 bucket folder (example python code)

 refer to source code

..

import boto3
import botocore

ACCESS_KEY = '**'
SECRET_KEY = '**'

s3_client = boto3.client(
's3',
aws_access_key_id=ACCESS_KEY,
aws_secret_access_key=SECRET_KEY,
region_name = 'eu-west-1'
)

bucket_name = '***'
folder_name = '***'

#check folder exist
try:
result = s3_client.list_objects_v2(Bucket=bucket_name, Prefix=folder_name )
for o in result['Contents']:
print(o['Key']) #file list in folder
except botocore.exceptions.ClientError as e:
print('no folder exist')

..


Thank you 

www.marearts.com



10/12/2021

aws dynamodb scan all item using paginate

 


Refer to code, Thank you!

..


import boto3

ACCESS_KEY = '***'
SECRET_KEY = '***'
dynamo_client = boto3.client(
'dynamodb',
aws_access_key_id=ACCESS_KEY,
aws_secret_access_key=SECRET_KEY,
region_name = 'eu-west-1'
)


def Search_Item(TABLE_NAME):
paginator = dynamo_client.get_paginator('scan')
operation_parameters = {'TableName': TABLE_NAME,}
page_iterator = paginator.paginate(**operation_parameters)
return page_iterator


TABLE_NAME ='****'

page_iterator = Search_Item(TABLE_NAME)
for page in page_iterator:
for item in page['Items']:
print(item['UID']['S'])

..


www.marearts.com

πŸ™‡πŸ»‍♂️


10/03/2021

Docker container stop by image name

 Stop docker container by image name:

imagename='mydockerimage'
docker stop $(docker ps | awk '{split($2,image,":"); print $1, image[1]}' | awk -v image=$imagename '$2 == image {print $1}')

Stop docker container by image name and tag:

imagename='mydockerimage:latest'
docker stop $(docker ps | awk -v image=$imagename '$2 == image {print $1}')

9/27/2021

docker swarm simple command

*install virtualbox

-+-

#install virtualbox on terminal
#osx
brew install virtualbox
#linux
sudo apt install virtualbox

-+-

 

* install docker-machine

...

#osx
curl -L https://github.com/docker/machine/releases/download/v0.16.2/docker-machine-`uname -s`-`uname -m` >/usr/local/bin/docker-machine && chmod +x /usr/local/bin/docker-machine
#linux
curl -L https://github.com/docker/machine/releases/download/v0.16.2/docker-machine-`uname -s`-`uname -m` >/tmp/docker-machine && chmod +x /tmp/docker-machine && sudo cp /tmp/docker-machine /usr/local/bin/docker-machine

...


*docker-machine

...

#docker-machine basic cmd
docker-machine ls
docker-machine rm name
docker-machine kill name
docker-machine create name

...



9/24/2021

python save & load argparse

 


refer to code:

..

from argparse import ArgumentParser
import json

parser = ArgumentParser()
parser.add_argument('--seed', type=int, default=8)
parser.add_argument('--resume', type=str, default='a/b/c.ckpt')
parser.add_argument('--surgery', type=str, default='190', choices=['190', '417'])
args = parser.parse_args()

with open('commandline_args.txt', 'w') as f:
json.dump(args.__dict__, f, indent=2)

parser = ArgumentParser()
args = parser.parse_args()
with open('commandline_args.txt', 'r') as f:
args.__dict__ = json.load(f)

print(args)

..


Thank you!

πŸ™‡πŸ»‍♂️

1D sequence (signal) data resampling (python)

 import numpy as np

import matplotlib.pyplot as plt
from scipy.signal import resample
from scipy.interpolate import interp1d

def ResampleLinear1D(original, targetLen):
original = np.array(original, dtype=np.float)
index_arr = np.linspace(0, len(original)-1, num=targetLen, dtype=np.float)
index_floor = np.array(index_arr, dtype=np.int) #Round down
index_ceil = index_floor + 1
index_rem = index_arr - index_floor #Remain

val1 = original[index_floor]
val2 = original[index_ceil % len(original)]
interp = val1 * (1.0-index_rem) + val2 * index_rem
assert(len(interp) == targetLen)
return interp

if __name__=="__main__":

original = np.sin(np.arange(256)/10.0)
targetLen = 100

# Method 1: Use scipy interp1d (linear interpolation)
# This is the simplest conceptually as it just uses linear interpolation. Scipy
# also offers a range of other interpolation methods.
f = interp1d(np.arange(256), original, 'linear')
plt.plot(np.apply_along_axis(f, 0, np.linspace(0, 255, num=targetLen)))

# Method 2: Use numpy to do linear interpolation
# If you don't have scipy, you can do it in numpy with the above function
plt.plot(ResampleLinear1D(original, targetLen))

# Method 3: Use scipy's resample
# Converts the signal to frequency space (Fourier method), then back. This
# works efficiently on periodic functions but poorly on non-periodic functions.
plt.plot(resample(original, targetLen))

plt.show()

9/22/2021

jupyter notebook to python py

 

run this command

>jupyter nbconvert your_notebook_name.ipynb --to python


thank you.


pytorch cuda definition

Their syntax varies slightly, but they are equivalent:

.to(name)

.to(device)

.cuda()

CPU

to('cpu')

to(torch.device('cpu'))

cpu()

Current GPU

to('cuda')

to(torch.device('cuda'))

cuda()

Specific GPU

to('cuda:1')

to(torch.device('cuda:1'))

cuda(device=1)

Note: the current cuda device is 0 by default, but this can be set with torch.cuda.set_device().

9/21/2021

write classification_report to txt file

Refer to code ^^ 

..

from sklearn.metrics import classification_report

report = classification_report(y_test_true_list, y_test_pred_list, labels=labels, target_names=target_names)

fp = open('report.txt', 'w')
fp.write(report)
fp.close()

..


Thank you.

www.marearts.com


python pil change dpi without save file

 


look at the code! ^^

..

#load image
tif_path = './input_img.tif'
#open image
pil_image = Image.open(tif_path)
#change dpi in memory
temp_mem_file = BytesIO()
page.save(temp_mem_file, format='png', dpi=(100,100))
#save it to pil
temp_mem_file.seek(0)
pil_image = Image.open(temp_mem_file)
#do it someting
#..
#save file
pil_image.save('./output.png')
#www.marearts.com
#study.marearts.com

..


Thank you.


9/20/2021

convert numpy.ndarray object to float


*Here is object dtype numpy.ndarray
print(oneD_pt.dtype, type(oneD_pt), oneD_pt)
> object <class 'numpy.ndarray'> [3 2 1 2 2] 

* convert float dtype numpy.ndarray
oneD_pt=oneD_pt.astype('float')
print(oneD_pt.dtype, type(oneD_pt), oneD_pt)
> float64 <class 'numpy.ndarray'> [3. 2. 1. 2. 2.]


thank you.

python argparse simple example

 

It's useful to save hyper parameter for machine learning


import argparse

paser = argparse.ArgumentParser()

args = paser.parse_args("")

args.batch_size=10

args.epoch=10


Thank you.


vs code server failed to start


try to kill the server which you want access and try to access again


1 .ctrl-shift-p and choose:
2. Remote-SSH: kill VS Code Server on Host...
3. access remote again


Thank you.


9/07/2021

nginx.conf, set to accept last path and poxy it.

 


ex)

www.xxx.yy/aaa/ping

www.xxx.yy/bbb/ping

Those urls will go to /ping proxy.


location ~ ^/.*/(ping|invocations) {
rewrite ^/.*/(.*)$ /$1 last;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_redirect off;
proxy_pass http://gunicorn/$1;
}

9/01/2021

AWS dynamoDB safe searching way, search item by scan and get it by paginator.

 refer to follow code

--

import boto3

dynamo_client = boto3.client('dynamodb')

def Search_Item(TABLE_NAME, uid):
paginator = dynamo_client.get_paginator('scan')
operation_parameters = {
'TableName': TABLE_NAME,
'FilterExpression': 'UID = :search',
'ExpressionAttributeValues': {
':search': {'S': uid},
}
}
page_iterator = paginator.paginate(**operation_parameters)
return page_iterator

UID_list = []
TABLE_NAME ='example_Table'
UID ='example_uid'

page_iterator = Search_Item(TABLE_NAME, UID)
for page in page_iterator:
for item in page['Items']:
UID_list.append(item['UID']['S'])

--


Thank you.

www.marearts.com




8/27/2021

AWS, find all file list in s3 bucket

 Find all file list in s3 bucket

import boto3
BUCKET_INPUT = 'bucket_name'
PREFIX_INPUT = 'some_prefix_folder_name'
item_list_all = []
paginator = s3_client.get_paginator('list_objects')
operation_parameters = {'Bucket': BUCKET_INPUT} #,'Prefix': PREFIX_INPUT}
page_iterator = paginator.paginate(**operation_parameters)
for page in page_iterator:
item_list_all.append(page['Contents'])
key_all = []
for page in item_list_all:
for i in range(len(page)):
key_all.append(page[i]["Key"])
print("The total number of files in the bucket:", len(key_all) )

Thank you.

www.marearts.com


8/11/2021

ERROR: Could not build wheels for tokenizers which use PEP 517 and cannot be installed directly

I met this error when I install simpletransformers or transformers.

I did many try but no luck.

But this was solution to me.

Install Rust before transformer installation.

so..

curl https://sh.rustup.rs -sSf | bash -s -- -y
PATH="/root/.cargo/bin:${PATH}"

and install transformers

Thank you.

www.marearts.com



8/08/2021

unable to prepare context: unable to evaluate symlinks in Dockerfile path: lstat /var/lib/snapd/void/Dockerfile: no such file or directory

 

if you install docker using snap, it can be happen.

so remove docker from snap installation

> sudo snap remove docker


And install docker again using apt

> sudo apt update
> sudo apt upgrade
> sudo apt install docker.io
> sudo systemctl enable --now docker


Thank you.
www.marearts.com


8/04/2021

labelme2yolo, yolo2labelme format converter

 

original page is here:

https://marearts.notion.site/Labelme2Yolo-Yolo2Labelme-3564dd886ac64b5499d6f2784a8a4be8


yolo2labelme main code

if __name__ == "__main__":
#parameters
paser = argparse.ArgumentParser()
args = paser.parse_args("")
##### set params #####
#input : yolo image, label path
args.yolo_images_path = './yolo_data/images/'
args.yolo_labels = './yolo_data/labels/'
#class count
args.yolo_nc = 80
#class names
args.yolo_names = ['aeroplane', 'apple', 'backpack', 'banana', 'baseball bat', 'baseball glove', 'bear', 'bed', 'bench', 'bicycle', 'bird', 'boat', 'book', 'bottle', 'bowl', 'broccoli', 'bus', 'cake', 'car', 'carrot', 'cat', 'cell phone', 'chair', 'clock', 'cow', 'cup', 'diningtable', 'dog', 'donut', 'elephant', 'fire hydrant', 'fork', 'frisbee', 'giraffe', 'hair drier', 'handbag', 'horse', 'hot dog', 'keyboard', 'kite', 'knife', 'laptop', 'microwave', 'motorbike', 'mouse', 'orange', 'oven', 'parking meter', 'person', 'pizza', 'pottedplant', 'refrigerator', 'remote', 'sandwich', 'scissors', 'sheep', 'sink', 'skateboard', 'skis', 'snowboard', 'sofa', 'spoon', 'sports ball', 'stop sign', 'suitcase', 'surfboard', 'teddy bear', 'tennis racket', 'tie', 'toaster', 'toilet', 'toothbrush', 'traffic light', 'train', 'truck', 'tvmonitor', 'umbrella', 'vase', 'wine glass', 'zebra']

#output : labelme output path
args.labelme_path = './output_yolo2labelme/'
args.labelme_save_image = True #False #create jpg images in output path
#######################


#######################
#start processing
yolo2labelme_main(args)
#######################


labelme2yolo main code

#parameters
paser = argparse.ArgumentParser()
args = paser.parse_args("")

##### set params #####
#input : labelme output path
args.labelme_path = './output_yolo2labelme/'
args.labelme_save_image = True #False #create jpg images in output path

#output : yolo image, label path
args.yolo_images_path = './output_yolo_data/images/'
args.yolo_labels = './output_yolo_data/labels/'
#class count
args.yolo_nc = 80
#class names
args.yolo_names = ['aeroplane', 'apple', 'backpack', 'banana', 'baseball bat', 'baseball glove', 'bear', 'bed', 'bench', 'bicycle', 'bird', 'boat', 'book', 'bottle', 'bowl', 'broccoli', 'bus', 'cake', 'car', 'carrot', 'cat', 'cell phone', 'chair', 'clock', 'cow', 'cup', 'diningtable', 'dog', 'donut', 'elephant', 'fire hydrant', 'fork', 'frisbee', 'giraffe', 'hair drier', 'handbag', 'horse', 'hot dog', 'keyboard', 'kite', 'knife', 'laptop', 'microwave', 'motorbike', 'mouse', 'orange', 'oven', 'parking meter', 'person', 'pizza', 'pottedplant', 'refrigerator', 'remote', 'sandwich', 'scissors', 'sheep', 'sink', 'skateboard', 'skis', 'snowboard', 'sofa', 'spoon', 'sports ball', 'stop sign', 'suitcase', 'surfboard', 'teddy bear', 'tennis racket', 'tie', 'toaster', 'toilet', 'toothbrush', 'traffic light', 'train', 'truck', 'tvmonitor', 'umbrella', 'vase', 'wine glass', 'zebra']
#######################


#######################
#start processing
labelme2yolo_main(args)
#######################



Thank you.

www.marearts.com

πŸ™‡πŸ»‍♂️

8/03/2021

make python package zip for lambda layer using docker

 Here is good reference for this subject:

https://aws.amazon.com/premiumsupport/knowledge-center/lambda-layer-simulated-docker/

https://dev.to/matthewvielkind/creating-python-aws-lambda-layers-with-docker-4376



If we use docker then it's very simple.

Let's go through step by step


1. make Main folder & python library folder

ex) In this tutorial, we suppose to need flask package

mkdir flask-layer
cd flask-layer
mkdir -pv python/lib/python3.6/site-packages
or
mkdir -pv python/lib/python3.8/site-packages


2. make requirements.txt and put packages to install by pip

flask==1.1.1


So folder structure looks like this:

├── requirements.txt └── python/ └── lib/ ├── python3.6/ │ └── site-packages/ └── python3.8/ └── site-packages/


3. Run docker to install packages

note, change python version properly

docker run -v "$PWD":/var/task "public.ecr.aws/sam/build-python3.6" /bin/sh -c "pip install -r requirements.txt -t python/lib/python3.6/site-packages/; exit"

 

4. compress package to zip

zip -r yourpack.zip python > /dev/null


5. use zip file for your lambda

That's it all.


Thank you.

www.marearts.com

πŸ™‡πŸ»‍♂️