semantic segmentation vs instance segmentation

 

* semantic segmentation

Finding out which class each pixel belongs to

* instance segmentation

To find out the pixel position of a detected object class based on object detection(localisation)

Is that confuse? ^^

Thank you.

🙇🏻‍♂️

www.marearts.com

미리하는 그리고 재미로 하는 대선투표 http://vote.marearts.com

 

재미로 하는 대선투표 

http://vote.marearts.com

docker MongoDB pull and run

 

#pull image

docker pull mongodb

#run container

docker run --name mongodb -d -p 27017:27017 mongo

ex) access in python 

import pymongo # package for working with MongoDB

client = pymongo.MongoClient("mongodb://localhost:27017/")

Thank you.

www.marearts.com

Retrive docker container IP address

replace 'container_name_or_id' -> 'you id or container name'

docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' container_name_or_id

Thank you.

mongodb python, Table recode export, loading, save csv, load csv

 refer to below code:

.

#connet db

client = pymongo.MongoClient("mongodb://localhost:27017/")

#connet table

who_table = db["who_table"]

#delete all

who_table.drop()

#load csv to table

df = pd.read_csv('1.csv')

data = df.to_dict('records')

who_table.insert_many(data)

#show all items

for x in who_table.find():

print(x)

#save table to csv

cursor = who_table.find({})

df = pd.DataFrame(list(cursor))

df.to_csv('2.csv', index=False)

.

Thank you.🙇🏻‍♂️

www.marearts.com

IP 2 Geometry information python

 Here is official page:

https://pypi.org/project/ip2geotools/

https://github.com/tomas-net/ip2geotools

This is my try

..

# pip install ip2geotools

from ip2geotools.databases.noncommercial import DbIpCity

response = DbIpCity.get('165.132.116.105', api_key='free')

print(response)

..

The results :

165.132.116.105

Seodaemun-gu

Seoul

KR

37.5790747

126.9367861

Thank you. 🙇🏻‍♂️

www.marearts.com

increase number in dynamodb field key value

 

Get Access permission

.

import boto3

##########################################################

ACCESS_KEY = 'xxxx'

SECRET_KEY = 'xxx'

dynamo_client = boto3.client('dynamodb',

aws_access_key_id = ACCESS_KEY,

aws_secret_access_key = SECRET_KEY

)

.

sample code

.

#update when db

def update_when_db(who, when):

dynamo_client.update_item(

TableName='president_when', #table name

Key={'when': {'S': when}}, #primary key filed name, #key value, if key file not exist, it will create

UpdateExpression="ADD #counter :increment",

ExpressionAttributeNames={'#counter': who}, #key filed name that you want to increase

ExpressionAttributeValues={':increment': {'N': '1'}} #increase 1

)

.

Thank you.

🙇🏻‍♂️

www.marearts.com

Rotation(orientation) correction when uploading photos from mobile (python, pillow)

 Try to use ImageOps function.

Solution is simple refer to below code:

.

#pip install Pillow

from PIL import Image, ImageOps

image = Image.open('img.jpg') #open image

#### orient fix

image = ImageOps.exif_transpose(image)

#### go futher!

.

Thank you.

🙇🏻‍♂️

www.marearts.com

Signature & Handwritten Detection in scanned image

 

Live Demo is here:

http://signhand.marearts.com/

API description is here:

https://www.marearts.com/Signature-Handwritten-Detector-c81f875ddb66490a8db990b50ce6f665

Sample data:

https://github.com/MareArts/signature_handwritten_data

Thank you.

🙇🏻‍♂️

www.marearts.com

Signature & Handwritten Detector on document image

Handwritten & Signature Detector on Document Image

* Live Demo

✍🏼 Signature & Handwritten Detector on Document

http://signhand.marearts.com/

* API explanation

https://www.marearts.com/Signature-Handwritten-Detector-c81f875ddb66490a8db990b50ce6f665

----------------------------------------

🌷 MareArts Free API List 🌷

MareArts 🎬 Live 

http://live.marearts.com

✍🏼 Signature & Handwritten Detector on Document

http://signhand.marearts.com/

🚙 Auto Number Plate Recognition - Korean

http://redkongkr.marearts.com/

🚦 Car License Plate detector

http://lpdetector.marearts.com/

🏳️‍🌈 Representative Color Extractor

http://recolor.marearts.com/

📓 OCR Tess-v5

http://tessv5.marearts.com/

📐 Regular Expression Generator

http://regular.marearts.com/

🐠 QRcode Generator

http://qr.marearts.com/

🦧 Barcode Generator

http://barcode.marearts.com/

🌊 Elastic Image Generator

http://elastic.marearts.com/

🏁 Mosaic Image Generator

http://mosaic.marearts.com/

⚽️ PDF to IMG convertor

http://pdf2img.marearts.com/

🏀 PDF to TXT convertor

http://pdf2txt.marearts.com/

🏈 PDF to HTML convertor

http://pdf2html.marearts.com/

⚾️ PDF to JSON convertor

http://pdf2json.marearts.com/

🙇🏻‍♂️ Thank you!

https://www.marearts.com

-----------------------------------

How to combine multiple criterions to a loss function? Multiple loss function for single model.

You can simply reference below code:

ex1)

b = nn.MSELoss()(output_x, x_labels) a = nn.CrossEntropyLoss()(output_y, y_labels) loss = a + b loss.backward()

ex2)

b = nn.MSELoss() a = nn.CrossEntropyLoss() loss_a = a(output_x, x_labels) loss_b = b(output_y, y_labels) loss = loss_a + loss_b loss.backward()

And there are many opinions in here:

https://discuss.pytorch.org/t/how-to-combine-multiple-criterions-to-a-loss-function/348/27

Thank you.

www.marearts.com

🙇🏻‍♂️

pdf2img, pdf to image, python library

 way #1

--

# pip install pdf2image

from pdf2image import convert_from_path

pdffile = '2081033884.pdf'

pages = convert_from_path(pdffile, 500)

#Saving pages in jpeg format

for i, page in enumerate(pages):

page.save(f'pdf2image_{i}.jpg', 'JPEG')

--

way #2

--

#pip install pymupdf

import fitz

pdffile = '2081033884.pdf'

doc = fitz.open(pdffile)

#split pages

for i, page in enumerate(doc.pages()):

pix = page.get_pixmap()

img_filename = f"fitz_{i}.jpg"

pix.pil_save(img_filename, format="jpeg", dpi=(300,300)) #, ... more PIL parameters)

--

Thank you.

www.marearts.com

🙇🏻‍♂️

python, simple mkdir and remove folder

import os

#remove folder

os.system('rm -rf ./folder1/folder2/')

#make folder

os.system('mkdir -pv ./folder1/folder2/')

 

Thank you.

www.marearts.com

🙇🏻‍♂️

print pdf from html, python code

#pip install weasyprint

from weasyprint import HTML

#read html

HTML_path = './sample.html'

#convert html2pdf with coordinate

pdf = HTML(HTML_path).write_pdf()

#save pdf

open('sample_weasyprint.pdf', 'wb').write(pdf)

 

Thank you.

www.marearts.com

python polygon list to rectangle

 refer to below code:

--

def returnRect(ploygon):

x_list = []

y_list = []

for v in polygon:

x_list.append(v[0])

y_list.append(v[1])

left = min(x_list)

right = max(x_list)

top = min(y_list)

bottom = max(y_list)

return left, top, right, bottom

polygon = [[776.1818181818181, 221.2121212121212], [895.8787878787879, 219.6969696969697], [892.8484848484848, 198.48484848484847], [879.2121212121211, 146.96969696969697], [795.8787878787879, 150.0], [779.2121212121211, 160.6060606060606]]

x1,y1,x2,y2 = returnRect(polygon)

print(x1,y1,x2,y2)

--

Thank you.

www.marearts.com

compute_class_weight() takes 1 positional argument but 3 were given

put a explicit parameter name like:

from sklearn.utils import class_weight

class_weights=class_weight.compute_class_weight(class_weight='balanced',classes=np.unique(y),y=y)

Thank you.

www.marearts.com

.local/lib/python3.8/site-packages/kaggle/api/kaggle_api_extended.py", line 164, in authenticate raise IOError('Could not find {}. Make sure it\'s located in'

If you are using Linux:

1. pip install kaggle

2. cd ~/.kaggle

3. homepage www.kaggle.com -> Your Account -> Create New API token

4. mv ~/Downloads/kaggle.json ./

5. chmod 600 ./kaggle.json

Thank you 🙇🏻‍♂️

install awscli for linux (python)

Try to do these command


>pip install awscli --upgrade --user

>sudo apt install awscli



Thank you.

www.marearts.com

Get file list in s3 bucket folder (example python code)

 refer to source code

..

import boto3

import botocore

ACCESS_KEY = '**'

SECRET_KEY = '**'

s3_client = boto3.client(

's3',

aws_access_key_id=ACCESS_KEY,

aws_secret_access_key=SECRET_KEY,

region_name = 'eu-west-1'

)

bucket_name = '***'

folder_name = '***'

#check folder exist

try:

result = s3_client.list_objects_v2(Bucket=bucket_name, Prefix=folder_name )

for o in result['Contents']:

print(o['Key']) #file list in folder

except botocore.exceptions.ClientError as e:

print('no folder exist')

..

Thank you 

www.marearts.com

aws dynamodb scan all item using paginate

 

Refer to code, Thank you!

..

import boto3

ACCESS_KEY = '***'

SECRET_KEY = '***'

dynamo_client = boto3.client(

'dynamodb',

aws_access_key_id=ACCESS_KEY,

aws_secret_access_key=SECRET_KEY,

region_name = 'eu-west-1'

)

def Search_Item(TABLE_NAME):

paginator = dynamo_client.get_paginator('scan')

operation_parameters = {'TableName': TABLE_NAME,}

page_iterator = paginator.paginate(**operation_parameters)

return page_iterator

TABLE_NAME ='****'

page_iterator = Search_Item(TABLE_NAME)

for page in page_iterator:

for item in page['Items']:

print(item['UID']['S'])

..

www.marearts.com

🙇🏻‍♂️

Docker container stop by image name

 Stop docker container by image name:

imagename='mydockerimage'
docker stop $(docker ps | awk '{split($2,image,":"); print $1, image[1]}' | awk -v image=$imagename '$2 == image {print $1}')

Stop docker container by image name and tag:

imagename='mydockerimage:latest'
docker stop $(docker ps | awk -v image=$imagename '$2 == image {print $1}')

docker swarm simple command

*install virtualbox

-+-

#install virtualbox on terminal

#osx

brew install virtualbox

#linux

sudo apt install virtualbox

-+-

 

* install docker-machine

...

#osx

curl -L https://github.com/docker/machine/releases/download/v0.16.2/docker-machine-`uname -s`-`uname -m` >/usr/local/bin/docker-machine && chmod +x /usr/local/bin/docker-machine

#linux

curl -L https://github.com/docker/machine/releases/download/v0.16.2/docker-machine-`uname -s`-`uname -m` >/tmp/docker-machine && chmod +x /tmp/docker-machine && sudo cp /tmp/docker-machine /usr/local/bin/docker-machine

...

*docker-machine

...

#docker-machine basic cmd

docker-machine ls

docker-machine rm name

docker-machine kill name

docker-machine create name

...

python save & load argparse

 

refer to code:

..

from argparse import ArgumentParser

import json

parser = ArgumentParser()

parser.add_argument('--seed', type=int, default=8)

parser.add_argument('--resume', type=str, default='a/b/c.ckpt')

parser.add_argument('--surgery', type=str, default='190', choices=['190', '417'])

args = parser.parse_args()

with open('commandline_args.txt', 'w') as f:

json.dump(args.__dict__, f, indent=2)

parser = ArgumentParser()

args = parser.parse_args()

with open('commandline_args.txt', 'r') as f:

args.__dict__ = json.load(f)

print(args)

..

Thank you!

🙇🏻‍♂️

1D sequence (signal) data resampling (python)

 import numpy as np

import matplotlib.pyplot as plt

from scipy.signal import resample

from scipy.interpolate import interp1d

def ResampleLinear1D(original, targetLen):

original = np.array(original, dtype=np.float)

index_arr = np.linspace(0, len(original)-1, num=targetLen, dtype=np.float)

index_floor = np.array(index_arr, dtype=np.int) #Round down

index_ceil = index_floor + 1

index_rem = index_arr - index_floor #Remain

val1 = original[index_floor]

val2 = original[index_ceil % len(original)]

interp = val1 * (1.0-index_rem) + val2 * index_rem

assert(len(interp) == targetLen)

return interp

if __name__=="__main__":

original = np.sin(np.arange(256)/10.0)

targetLen = 100

# Method 1: Use scipy interp1d (linear interpolation)

# This is the simplest conceptually as it just uses linear interpolation. Scipy

# also offers a range of other interpolation methods.

f = interp1d(np.arange(256), original, 'linear')

plt.plot(np.apply_along_axis(f, 0, np.linspace(0, 255, num=targetLen)))

# Method 2: Use numpy to do linear interpolation

# If you don't have scipy, you can do it in numpy with the above function

plt.plot(ResampleLinear1D(original, targetLen))

# Method 3: Use scipy's resample

# Converts the signal to frequency space (Fourier method), then back. This

# works efficiently on periodic functions but poorly on non-periodic functions.

plt.plot(resample(original, targetLen))

plt.show()

jupyter notebook to python py

 

run this command

>jupyter nbconvert your_notebook_name.ipynb --to python

thank you.

pytorch cuda definition

Their syntax varies slightly, but they are equivalent:

⠀	.to(name)	.to(device)	.cuda()
CPU	to('cpu')	to(torch.device('cpu'))	cpu()
Current GPU	to('cuda')	to(torch.device('cuda'))	cuda()
Specific GPU	to('cuda:1')	to(torch.device('cuda:1'))	cuda(device=1)

Note: the current cuda device is 0 by default, but this can be set with torch.cuda.set_device().

write classification_report to txt file

Refer to code ^^ 

..

from sklearn.metrics import classification_report

report = classification_report(y_test_true_list, y_test_pred_list, labels=labels, target_names=target_names)

fp = open('report.txt', 'w')

fp.write(report)

fp.close()

..

Thank you.

www.marearts.com

python pil change dpi without save file

 

look at the code! ^^

..

#load image

tif_path = './input_img.tif'

#open image

pil_image = Image.open(tif_path)

#change dpi in memory

temp_mem_file = BytesIO()

page.save(temp_mem_file, format='png', dpi=(100,100))

#save it to pil

temp_mem_file.seek(0)

pil_image = Image.open(temp_mem_file)

#do it someting

#..

#save file

pil_image.save('./output.png')

#www.marearts.com

#study.marearts.com

..

Thank you.

convert numpy.ndarray object to float

*Here is object dtype numpy.ndarray

print(oneD_pt.dtype, type(oneD_pt), oneD_pt)
> object <class 'numpy.ndarray'> [3 2 1 2 2] 

* convert float dtype numpy.ndarray

oneD_pt=oneD_pt.astype('float')

print(oneD_pt.dtype, type(oneD_pt), oneD_pt)

> float64 <class 'numpy.ndarray'> [3. 2. 1. 2. 2.]

thank you.

python argparse simple example

 

It's useful to save hyper parameter for machine learning

import argparse

paser = argparse.ArgumentParser()

args = paser.parse_args("")

args.batch_size=10

args.epoch=10

Thank you.

vs code server failed to start

try to kill the server which you want access and try to access again

1 .ctrl-shift-p and choose:
2. Remote-SSH: kill VS Code Server on Host...

3. access remote again

Thank you.

nginx.conf, set to accept last path and poxy it.

 

ex)

www.xxx.yy/aaa/ping

www.xxx.yy/bbb/ping

Those urls will go to /ping proxy.

location ~ ^/.*/(ping|invocations) {

rewrite ^/.*/(.*)$ /$1 last;

proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

proxy_set_header Host $http_host;

proxy_redirect off;

proxy_pass http://gunicorn/$1;

}

TensorFlow cuda version compatible

 

AWS dynamoDB safe searching way, search item by scan and get it by paginator.

 refer to follow code

--

import boto3

dynamo_client = boto3.client('dynamodb')

def Search_Item(TABLE_NAME, uid):

paginator = dynamo_client.get_paginator('scan')

operation_parameters = {

'TableName': TABLE_NAME,

'FilterExpression': 'UID = :search',

'ExpressionAttributeValues': {

':search': {'S': uid},

}

}

page_iterator = paginator.paginate(**operation_parameters)

return page_iterator

UID_list = []

TABLE_NAME ='example_Table'

UID ='example_uid'

page_iterator = Search_Item(TABLE_NAME, UID)

for page in page_iterator:

for item in page['Items']:

UID_list.append(item['UID']['S'])

--

Thank you.

www.marearts.com

AWS, find all file list in s3 bucket

 Find all file list in s3 bucket

import boto3

BUCKET_INPUT = 'bucket_name'

PREFIX_INPUT = 'some_prefix_folder_name'

item_list_all = []

paginator = s3_client.get_paginator('list_objects')

operation_parameters = {'Bucket': BUCKET_INPUT} #,'Prefix': PREFIX_INPUT}

page_iterator = paginator.paginate(**operation_parameters)

for page in page_iterator:

item_list_all.append(page['Contents'])

key_all = []

for page in item_list_all:

for i in range(len(page)):

key_all.append(page[i]["Key"])

print("The total number of files in the bucket:", len(key_all) )

Thank you.

www.marearts.com

ERROR: Could not build wheels for tokenizers which use PEP 517 and cannot be installed directly

I met this error when I install simpletransformers or transformers.

I did many try but no luck.

But this was solution to me.

Install Rust before transformer installation.

so..

curl https://sh.rustup.rs -sSf | bash -s -- -y

PATH="/root/.cargo/bin:${PATH}"

and install transformers

Thank you.

www.marearts.com

unable to prepare context: unable to evaluate symlinks in Dockerfile path: lstat /var/lib/snapd/void/Dockerfile: no such file or directory

 

if you install docker using snap, it can be happen.

so remove docker from snap installation

> sudo snap remove docker

And install docker again using apt

> sudo apt update
> sudo apt upgrade
> sudo apt install docker.io
> sudo systemctl enable --now docker

Thank you.

www.marearts.com

labelme2yolo, yolo2labelme format converter

 

original page is here:

https://marearts.notion.site/Labelme2Yolo-Yolo2Labelme-3564dd886ac64b5499d6f2784a8a4be8

yolo2labelme main code

if __name__ == "__main__":

#parameters

paser = argparse.ArgumentParser()

args = paser.parse_args("")

##### set params #####

#input : yolo image, label path

args.yolo_images_path = './yolo_data/images/'

args.yolo_labels = './yolo_data/labels/'

#class count

args.yolo_nc = 80

#class names

args.yolo_names = ['aeroplane', 'apple', 'backpack', 'banana', 'baseball bat', 'baseball glove', 'bear', 'bed', 'bench', 'bicycle', 'bird', 'boat', 'book', 'bottle', 'bowl', 'broccoli', 'bus', 'cake', 'car', 'carrot', 'cat', 'cell phone', 'chair', 'clock', 'cow', 'cup', 'diningtable', 'dog', 'donut', 'elephant', 'fire hydrant', 'fork', 'frisbee', 'giraffe', 'hair drier', 'handbag', 'horse', 'hot dog', 'keyboard', 'kite', 'knife', 'laptop', 'microwave', 'motorbike', 'mouse', 'orange', 'oven', 'parking meter', 'person', 'pizza', 'pottedplant', 'refrigerator', 'remote', 'sandwich', 'scissors', 'sheep', 'sink', 'skateboard', 'skis', 'snowboard', 'sofa', 'spoon', 'sports ball', 'stop sign', 'suitcase', 'surfboard', 'teddy bear', 'tennis racket', 'tie', 'toaster', 'toilet', 'toothbrush', 'traffic light', 'train', 'truck', 'tvmonitor', 'umbrella', 'vase', 'wine glass', 'zebra']

#output : labelme output path

args.labelme_path = './output_yolo2labelme/'

args.labelme_save_image = True #False #create jpg images in output path

#######################

#######################

#start processing

yolo2labelme_main(args)

#######################

labelme2yolo main code

#parameters

paser = argparse.ArgumentParser()

args = paser.parse_args("")

##### set params #####

#input : labelme output path

args.labelme_path = './output_yolo2labelme/'

args.labelme_save_image = True #False #create jpg images in output path

#output : yolo image, label path

args.yolo_images_path = './output_yolo_data/images/'

args.yolo_labels = './output_yolo_data/labels/'

#class count

args.yolo_nc = 80

#class names

args.yolo_names = ['aeroplane', 'apple', 'backpack', 'banana', 'baseball bat', 'baseball glove', 'bear', 'bed', 'bench', 'bicycle', 'bird', 'boat', 'book', 'bottle', 'bowl', 'broccoli', 'bus', 'cake', 'car', 'carrot', 'cat', 'cell phone', 'chair', 'clock', 'cow', 'cup', 'diningtable', 'dog', 'donut', 'elephant', 'fire hydrant', 'fork', 'frisbee', 'giraffe', 'hair drier', 'handbag', 'horse', 'hot dog', 'keyboard', 'kite', 'knife', 'laptop', 'microwave', 'motorbike', 'mouse', 'orange', 'oven', 'parking meter', 'person', 'pizza', 'pottedplant', 'refrigerator', 'remote', 'sandwich', 'scissors', 'sheep', 'sink', 'skateboard', 'skis', 'snowboard', 'sofa', 'spoon', 'sports ball', 'stop sign', 'suitcase', 'surfboard', 'teddy bear', 'tennis racket', 'tie', 'toaster', 'toilet', 'toothbrush', 'traffic light', 'train', 'truck', 'tvmonitor', 'umbrella', 'vase', 'wine glass', 'zebra']

#######################

#######################

#start processing

labelme2yolo_main(args)

#######################

Thank you.

www.marearts.com

🙇🏻‍♂️

make python package zip for lambda layer using docker

 Here is good reference for this subject:

https://aws.amazon.com/premiumsupport/knowledge-center/lambda-layer-simulated-docker/

https://dev.to/matthewvielkind/creating-python-aws-lambda-layers-with-docker-4376

If we use docker then it's very simple.

Let's go through step by step

1. make Main folder & python library folder

ex) In this tutorial, we suppose to need flask package

mkdir flask-layer

cd flask-layer

mkdir -pv python/lib/python3.6/site-packages

or

mkdir -pv python/lib/python3.8/site-packages

2. make requirements.txt and put packages to install by pip

flask==1.1.1

So folder structure looks like this:

├── requirements.txt └── python/ └── lib/ ├── python3.6/ │ └── site-packages/ └── python3.8/ └── site-packages/

3. Run docker to install packages

note, change python version properly

docker run -v "$PWD":/var/task "public.ecr.aws/sam/build-python3.6" /bin/sh -c "pip install -r requirements.txt -t python/lib/python3.6/site-packages/; exit"

 

4. compress package to zip

zip -r yourpack.zip python > /dev/null

5. use zip file for your lambda

That's it all.

Thank you.

www.marearts.com

🙇🏻‍♂️