1/03/2020

OpenCV Docker error "ImportError: libSM.so.6: cannot open shared object file: No such file or directory"

Try this command

apt-get update
apt-get install -y libsm6 libxext6 libxrender-dev
pip install opencv-python

12/13/2019

python OrderDict to Dict


od = OrderedDict([('name', 'signature'), ('pose', 'Unspecified'), ('bndbox', OrderedDict([('xmin', '423'), ('ymin', '1237'), ('xmax', '785'), ('ymax', '1349')]))])
print(od)
output_dict = json.loads(json.dumps(av))
print(output_dict)





reference is here : https://stackoverflow.com/questions/20166749/how-to-convert-an-ordereddict-into-a-regular-dict-in-python3

12/08/2019

keras model summary, print out for each layers property

from keras.layers import Input
from keras.applications import VGG16
vgg_model = VGG16(weights="imagenet", include_top=False, input_tensor=Input(shape=(224, 224, 3))) #the head FC layer off

#how many layers
print(len(vgg_model.layers))



#last layer
print(vgg_model.layers[-1].name)



#print whole layer
for idx, layer in enumerate(vgg_model.layers):
print(idx+1, '-----')
print(layer.output_shape)
print(layer.name)
print('--------')



#summary of model
vgg_model.summary()



12/06/2019

using specific gpu device for TensorFlow

setting first or second gpu machine
CUDA_VISIBLE_DEVICES=0 python script_one.py
CUDA_VISIBLE_DEVICES=1 python script_two.py
another way
use first (it didn't work for me)
export CUDA_VISIBLE_DEVICE=0
./train.py
use second (it didn't work for me)
export CUDA_VISIBLE_DEVICE=1
./train.py
use both (it didn't work for me)
export CUDA_VISIBLE_DEVICE=0,1
./train.py

refer to here:
https://stackoverflow.com/questions/44135538/tensorflow-using-2-gpu-at-the-same-time

12/05/2019

monitoring gpu status in command (terminal)



pip install gpustat
>gpustat -cp




monitoring continuously
>watch -n 0.5 -c gpustat -cp --color


12/03/2019

find pdf file (or some exe file) in directories and copy it to another directory, python sample code


import os
import glob
from shutil import copyfile


files = []
start_dir = '/Volumes/input/'
output_path = '/Volumes/output/'
pattern = "*.pdf"

total = 0
for dir,_,_ in os.walk(start_dir):
files.extend(glob.glob(os.path.join(dir,pattern)))
for i,v in enumerate(files):
#found pdf files
print(total,i,v)
#extract filename only
filename = v.split('/')[-1]
#make new filename and output path
output_filename = output_path + str(total) + '_' + filename
#if file exist? then no copy
exist = glob.glob(output_filename)
#if not copy
if len(exist) == 0:
copyfile(v, output_filename)
#print out copied filename
print('copy! : ', output_filename)
#increase global count
total += 1

11/29/2019

download zip file from url, python sample code




from bs4 import BeautifulSoup
import requests
import os
import sys

def downloadZip(url, prefix_url, outpath):
mbyte=1024*1024
html = requests.get(url).text
soup = BeautifulSoup(html, features='lxml')


for name in soup.findAll('a', href=True):
#find A tag
zipurl = name['href']
#find file extension
if( zipurl.endswith('.zip') ):
#make download path
outfname = outpath +'/'+ zipurl.split('/')[-1]
#make url
zipurl = prefix_url+zipurl #http://aaa.com/ + 'abc.zip'
print(zipurl)
r = requests.get(zipurl, stream=True)
if( r.status_code == requests.codes.ok ) :
fsize = int(r.headers['content-length'])
print('Downloading %s (%sMb)'%(outfname, fsize/mbyte))
with open(outfname, 'wb') as fd:
for chunk in r.iter_content(chunk_size=1024): # chuck size can be larger
if chunk: # ignore keep-alive requests
fd.write(chunk)
fd.close()

base_path = os.getcwd()
path_join = os.path.join(base_path, 'data_download_pdf')
sys.path.append(path_join)

# point to output directory
outpath = path_join
url = 'https://www.gsa.gov/real-estate/real-estate-services/leasing-policy-procedures/lease-documents/lease-documents-region-1-new-england/'
prefix_url = 'https://www.gsa.gov/cdnstatic'

downloadZip(url, prefix_url, outpath)