12/23/2020

delete over than specific megabyte in git history using bfg

firstly, install bfg


on mac

> brew install bfg

delete object if over than 50m in git commit history

> bfg --strip-blobs-bigger-than 50M

make sure & apply .gitignore





First, to check what files are you actually tracking
> git ls-tree --name-only --full-tree -r HEAD


Let say that you found unwanted files in a directory like cache/ so, it's safer to target that directory instead of all of your files.
So instead of:

> git rm -r --cached .


It's safer to target the unwanted file or directory:
> git rm -r --cached cache/


Then proceed to add all changes:
> git add .


and commit
> git commit -m ".gitignore is now working"

list up and sorting file(object) size in git / command

 use following cmd:

> git rev-list --objects --all | git cat-file --batch-check='%(objecttype) %(objectname) %(objectsize) %(rest)' | sed -n 's/^blob //p' | sort --numeric-sort --key=2 | cut -c 1-12,41- | $(command -v gnumfmt || echo numfmt) --field=2 --to=iec-i --suffix=B --padding=7 --round=nearest


12/22/2020

get YouTube video url from channel name, python youtube-dl library

 

install youtube-dl package first.

pip install YouTube-dl


<code>

import subprocess
direct_output = subprocess.check_output('youtube-dl --get-id https://www.youtube.com/channel/UCAwWYtU_DEdRFxEHl9879dRYBfQ/videos --no-check-certificate', shell=True)
Lines = direct_output.decode()
idv = Lines.split('\n')
play_url_list = []
for idv in Lines.split('\n'):
if idv == '':
continue
print(idv)
play_url_list.append( 'https://www.youtube.com/watch?v={}'.format(idv) )

</code>


Thank you.


Download or streaming YouTube video through cv2, python sample code


firstly, install pafy python package

> pip install pafy


if you some this related error "youtube_dl", install this. pafy works based on this package.

> pip install youtube-dl


you can check youtube-dl version:

> youtube-dl --version


if you also have some certification error, you can use --no-check-certification option
try this and test, this command will download video.

> sudo youtube-dl --no-check-certificate https://www.youtube.com/watch?v=N8A-BxTfTaY

ok then everything will be work using following code:

import pafy
from matplotlib import pyplot as plt
import cv2

url = "https://www.youtube.com/watch?v=TKbNDqcgjzw"
video = pafy.new(url, ydl_opts={'nocheckcertificate': True})
best = video.getbest(preftype="mp4")

cap = cv2.VideoCapture()
cap.open(best.url)

while(True):
# Capture frame-by-frame
ret, frame = cap.read()
# Display the resulting frame
cv2.imshow('frame',frame)
if cv2.waitKey(1) & 0xFF == ord('q'):
break

# When everything done, release the capture
cap.release()
cv2.destroyAllWindows()

Thank you.


12/21/2020

python notebook clear output

 

from IPython.display import clear_output

for i in range(10):
    clear_output(wait=True)
    print("Hello World!")


12/17/2020

generate .spec file with onefile option for pyinstaller



Use

pyi-makespec --onefile yourprogram.py

to generate a sample spec file for onefile mode.




reference :

https://stackoverflow.com/questions/47143315/using-onefile-with-a-spec-in-pyinstaller

12/14/2020

rsync exclude file and folder

 


command is like that:

rsync -avz --progress ./source ./destination --exclude-from './exclude-list.txt'


you can make ignore file and folder like this:

 exclude-list.txt

*.txt
*.jpg
*.jpeg
*.zip
*.ipynb
.git
*.json
*.done
*.png
*.xml
*.pdf
*.pyc
*/build
*/dist
*/experiment
*/grapher_outputs
*.pkl
*/system_evaluation


12/10/2020

linux screen command list in summary

*create screen
screen -S name

*leave with alive
Ctrl a, d

*enter screen 
screen -r name 

*kill specific screen
screen -S name -X quit

*show screen list
screen -ls

*detach specific screen

screen -d name


*kill all screen
killall screen

12/07/2020

shuffle dict in python

 

code

import random

d = {'a':[1,2], 'b':[2,4], 'c':[3,5], 'd':[2,4]}

l = list(d.items())

random.shuffle(l)

d = dict(l)

print(d)

result

{'b': [2, 4], 'd': [2, 4], 'a': [1, 2], 'c': [3, 5]}


12/01/2020

torch.nan_to_num not found error

torch.nan_to_num

This is for 1.8 version.


so use it instead of this.

temp[torch.isnan(temp)] = 0

temp is tensor