8/27/2021

AWS, find all file list in s3 bucket

 Find all file list in s3 bucket

import boto3
BUCKET_INPUT = 'bucket_name'
PREFIX_INPUT = 'some_prefix_folder_name'
item_list_all = []
paginator = s3_client.get_paginator('list_objects')
operation_parameters = {'Bucket': BUCKET_INPUT} #,'Prefix': PREFIX_INPUT}
page_iterator = paginator.paginate(**operation_parameters)
for page in page_iterator:
item_list_all.append(page['Contents'])
key_all = []
for page in item_list_all:
for i in range(len(page)):
key_all.append(page[i]["Key"])
print("The total number of files in the bucket:", len(key_all) )

Thank you.

www.marearts.com


8/11/2021

ERROR: Could not build wheels for tokenizers which use PEP 517 and cannot be installed directly

I met this error when I install simpletransformers or transformers.

I did many try but no luck.

But this was solution to me.

Install Rust before transformer installation.

so..

curl https://sh.rustup.rs -sSf | bash -s -- -y
PATH="/root/.cargo/bin:${PATH}"

and install transformers

Thank you.

www.marearts.com



8/08/2021

unable to prepare context: unable to evaluate symlinks in Dockerfile path: lstat /var/lib/snapd/void/Dockerfile: no such file or directory

 

if you install docker using snap, it can be happen.

so remove docker from snap installation

> sudo snap remove docker


And install docker again using apt

> sudo apt update
> sudo apt upgrade
> sudo apt install docker.io
> sudo systemctl enable --now docker


Thank you.
www.marearts.com


8/04/2021

labelme2yolo, yolo2labelme format converter

 

original page is here:

https://marearts.notion.site/Labelme2Yolo-Yolo2Labelme-3564dd886ac64b5499d6f2784a8a4be8


yolo2labelme main code

if __name__ == "__main__":
#parameters
paser = argparse.ArgumentParser()
args = paser.parse_args("")
##### set params #####
#input : yolo image, label path
args.yolo_images_path = './yolo_data/images/'
args.yolo_labels = './yolo_data/labels/'
#class count
args.yolo_nc = 80
#class names
args.yolo_names = ['aeroplane', 'apple', 'backpack', 'banana', 'baseball bat', 'baseball glove', 'bear', 'bed', 'bench', 'bicycle', 'bird', 'boat', 'book', 'bottle', 'bowl', 'broccoli', 'bus', 'cake', 'car', 'carrot', 'cat', 'cell phone', 'chair', 'clock', 'cow', 'cup', 'diningtable', 'dog', 'donut', 'elephant', 'fire hydrant', 'fork', 'frisbee', 'giraffe', 'hair drier', 'handbag', 'horse', 'hot dog', 'keyboard', 'kite', 'knife', 'laptop', 'microwave', 'motorbike', 'mouse', 'orange', 'oven', 'parking meter', 'person', 'pizza', 'pottedplant', 'refrigerator', 'remote', 'sandwich', 'scissors', 'sheep', 'sink', 'skateboard', 'skis', 'snowboard', 'sofa', 'spoon', 'sports ball', 'stop sign', 'suitcase', 'surfboard', 'teddy bear', 'tennis racket', 'tie', 'toaster', 'toilet', 'toothbrush', 'traffic light', 'train', 'truck', 'tvmonitor', 'umbrella', 'vase', 'wine glass', 'zebra']

#output : labelme output path
args.labelme_path = './output_yolo2labelme/'
args.labelme_save_image = True #False #create jpg images in output path
#######################


#######################
#start processing
yolo2labelme_main(args)
#######################


labelme2yolo main code

#parameters
paser = argparse.ArgumentParser()
args = paser.parse_args("")

##### set params #####
#input : labelme output path
args.labelme_path = './output_yolo2labelme/'
args.labelme_save_image = True #False #create jpg images in output path

#output : yolo image, label path
args.yolo_images_path = './output_yolo_data/images/'
args.yolo_labels = './output_yolo_data/labels/'
#class count
args.yolo_nc = 80
#class names
args.yolo_names = ['aeroplane', 'apple', 'backpack', 'banana', 'baseball bat', 'baseball glove', 'bear', 'bed', 'bench', 'bicycle', 'bird', 'boat', 'book', 'bottle', 'bowl', 'broccoli', 'bus', 'cake', 'car', 'carrot', 'cat', 'cell phone', 'chair', 'clock', 'cow', 'cup', 'diningtable', 'dog', 'donut', 'elephant', 'fire hydrant', 'fork', 'frisbee', 'giraffe', 'hair drier', 'handbag', 'horse', 'hot dog', 'keyboard', 'kite', 'knife', 'laptop', 'microwave', 'motorbike', 'mouse', 'orange', 'oven', 'parking meter', 'person', 'pizza', 'pottedplant', 'refrigerator', 'remote', 'sandwich', 'scissors', 'sheep', 'sink', 'skateboard', 'skis', 'snowboard', 'sofa', 'spoon', 'sports ball', 'stop sign', 'suitcase', 'surfboard', 'teddy bear', 'tennis racket', 'tie', 'toaster', 'toilet', 'toothbrush', 'traffic light', 'train', 'truck', 'tvmonitor', 'umbrella', 'vase', 'wine glass', 'zebra']
#######################


#######################
#start processing
labelme2yolo_main(args)
#######################



Thank you.

www.marearts.com

πŸ™‡πŸ»‍♂️

8/03/2021

make python package zip for lambda layer using docker

 Here is good reference for this subject:

https://aws.amazon.com/premiumsupport/knowledge-center/lambda-layer-simulated-docker/

https://dev.to/matthewvielkind/creating-python-aws-lambda-layers-with-docker-4376



If we use docker then it's very simple.

Let's go through step by step


1. make Main folder & python library folder

ex) In this tutorial, we suppose to need flask package

mkdir flask-layer
cd flask-layer
mkdir -pv python/lib/python3.6/site-packages
or
mkdir -pv python/lib/python3.8/site-packages


2. make requirements.txt and put packages to install by pip

flask==1.1.1


So folder structure looks like this:

├── requirements.txt └── python/ └── lib/ ├── python3.6/ │ └── site-packages/ └── python3.8/ └── site-packages/


3. Run docker to install packages

note, change python version properly

docker run -v "$PWD":/var/task "public.ecr.aws/sam/build-python3.6" /bin/sh -c "pip install -r requirements.txt -t python/lib/python3.6/site-packages/; exit"

 

4. compress package to zip

zip -r yourpack.zip python > /dev/null


5. use zip file for your lambda

That's it all.


Thank you.

www.marearts.com

πŸ™‡πŸ»‍♂️