10/24/2023

hugging face onnx exporting model quantisation method

 refer to example code


.


from functools import partial
from transformers import AutoTokenizer
from optimum.onnxruntime import ORTQuantizer, ORTModelForSequenceClassification
from optimum.onnxruntime.configuration import AutoQuantizationConfig, AutoCalibrationConfig

model_id = "distilbert-base-uncased-finetuned-sst-2-english"

onnx_model = ORTModelForSequenceClassification.from_pretrained(model_id, export=True)
tokenizer = AutoTokenizer.from_pretrained(model_id)
quantizer = ORTQuantizer.from_pretrained(onnx_model)
qconfig = AutoQuantizationConfig.arm64(is_static=True, per_channel=False)

def preprocess_fn(ex, tokenizer):
return tokenizer(ex["sentence"])

calibration_dataset = quantizer.get_calibration_dataset(
"glue",
dataset_config_name="sst2",
preprocess_function=partial(preprocess_fn, tokenizer=tokenizer),
num_samples=50,
dataset_split="train",
)

calibration_config = AutoCalibrationConfig.minmax(calibration_dataset)

ranges = quantizer.fit(
dataset=calibration_dataset,
calibration_config=calibration_config,
operators_to_quantize=qconfig.operators_to_quantize,
)

model_quantized_path = quantizer.quantize(
save_dir="path/to/output/model",
calibration_tensors_range=ranges,
quantization_config=qconfig,
)

..

options for several instructions

.



optimum-cli onnxruntime quantize --help
usage: optimum-cli <command> [<args>] onnxruntime quantize [-h] --onnx_model ONNX_MODEL -o OUTPUT [--per_channel] (--arm64 | --avx2 | --avx512 | --avx512_vnni | --tensorrt | -c CONFIG)

options:
-h, --help show this help message and exit
--arm64 Quantization for the ARM64 architecture.
--avx2 Quantization with AVX-2 instructions.
--avx512 Quantization with AVX-512 instructions.
--avx512_vnni Quantization with AVX-512 and VNNI instructions.
--tensorrt Quantization for NVIDIA TensorRT optimizer.
-c CONFIG, --config CONFIG
`ORTConfig` file to use to optimize the model.

Required arguments:
--onnx_model ONNX_MODEL
Path to the repository where the ONNX models to quantize are located.
-o OUTPUT, --output OUTPUT
Path to the directory where to store generated ONNX model.

Optional arguments:
--per_channel Compute the quantization parameters on a per-channel basis.

..


refer to this page for details:

https://huggingface.co/docs/optimum/onnxruntime/usage_guides/quantization#quantize-seq2seq-models



refer to this code as well

.

you may be able to get idea. 

# Export to ONNX
model = ORTModelForSeq2SeqLM.from_pretrained(model_path, from_transformers=True, export=True, provider='CUDAExecutionProvider').to(device)
model.save_pretrained(onnx_path)

# quantization code
encoder_quantizer = ORTQuantizer.from_pretrained(onnx_path, file_name='encoder_model.onnx')
decoder_quantizer = ORTQuantizer.from_pretrained(onnx_path, file_name='decoder_model.onnx')
decoder_wp_quantizer = ORTQuantizer.from_pretrained(onnx_path, file_name='decoder_with_past_model.onnx')
quantizer = [encoder_quantizer, decoder_quantizer, decoder_wp_quantizer]
dqconfig = AutoQuantizationConfig.avx512_vnni(is_static=False, per_channel=False)
for q in quantizer:
q.quantize(save_dir=output_path, quantization_config=dqconfig)


#inference code
model = ORTModelForSeq2SeqLM.from_pretrained(
model_id=model_path,
encoder_file_name='encoder_model_quantized.onnx',
decoder_file_name='decoder_model_quantized.onnx',
decoder_with_past_file_name='decoder_with_past_model_quantized.onnx',
provider='CUDAExecutionProvider',
use_io_binding=True,
).to(self.device)
tokenizer = AutoTokenizer.from_pretrained('google/flan-t5-large')

...

dataset = self.dataset(input_dict)
dataset.set_format(type='torch', device=self.device, columns=['input_ids', 'attention_mask'])
data_loader = DataLoader(dataset, batch_size=self.batch_size, collate_fn=self.data_collator)
generated_outputs: List[OUTPUT_TYPE] = []
for i, batch in enumerate(data_loader):
_batch = {key: val.to(self.device) for key, val in batch.items()}
outputs = self.model.generate(**_batch, generation_config=self.generation_config)
decoded_outputs = self.tokenizer.batch_decode(outputs.cpu().tolist(), skip_special_tokens=True)

.

Thank you.

note! quantisation and optimise is different. 


10/23/2023

comparing custom custom_vit_image_processor vs vit_image_processor of tranformers

 check custom image process is same with origin inner processing function in transformers.

.

pixel_values1 = self.feature_extractor(images=image, return_tensors="pt").pixel_values

# Convert numpy array to PyTorch tensor
pixel_values2 = self.custom_vit_image_processor(image)
pixel_values2 = torch.tensor(pixel_values2, dtype=torch.float32).unsqueeze(0) # Add batch dimension and ensure float32 type

# 1. Shape Check
assert pixel_values1.shape == pixel_values2.shape, "The tensors have different shapes
# 2. Absolute Difference
diff = torch.abs(pixel_values1 - pixel_values2)

# 3. Summarize Discrepancies www.marearts.com
mean_diff = torch.mean(diff).item()
max_diff = torch.max(diff).item()
min_diff = torch.min(diff).item()
print(f"Mean Absolute Difference: {mean_diff}")
print(f"Maximum Absolute Difference: {max_diff}")
print(f"Minimum Absolute Difference: {min_diff}")


# Additionally, if you want to see where the maximum difference occurs:
max_diff_position = torch.where(diff == max_diff)
print(f"Position of Maximum Difference: {max_diff_position}")

..


Thank you.

Hope to helpful.


10/17/2023

nerf studio installation error

error:


. VanillaPipeline.get_train_loss_dict: 12.6875 Traceback (most recent call last): File "/home/mare/anaconda3/envs/nerfstudio/bin/ns-train", line 8, in sys.exit(entrypoint()) File "/home/mare/anaconda3/envs/nerfstudio/lib/python3.8/site-packages/nerfstudio/scripts/train.py", line 262, in entrypoint main( File "/home/mare/anaconda3/envs/nerfstudio/lib/python3.8/site-packages/nerfstudio/scripts/train.py", line 247, in main launch( File "/home/mare/anaconda3/envs/nerfstudio/lib/python3.8/site-packages/nerfstudio/scripts/train.py", line 189, in launch main_func(local_rank=0, world_size=world_size, config=config) File "/home/mare/anaconda3/envs/nerfstudio/lib/python3.8/site-packages/nerfstudio/scripts/train.py", line 100, in train_loop trainer.train() File "/home/mare/anaconda3/envs/nerfstudio/lib/python3.8/site-packages/nerfstudio/engine/trainer.py", line 259, in train loss, loss_dict, metrics_dict = self.train_iteration(step) File "/home/mare/anaconda3/envs/nerfstudio/lib/python3.8/site-packages/nerfstudio/utils/profiler.py", line 127, in inner out = func(*args, **kwargs) File "/home/mare/anaconda3/envs/nerfstudio/lib/python3.8/site-packages/nerfstudio/engine/trainer.py", line 479, in train_iteration _, loss_dict, metrics_dict = self.pipeline.get_train_loss_dict(step=step) File "/home/mare/anaconda3/envs/nerfstudio/lib/python3.8/site-packages/nerfstudio/utils/profiler.py", line 127, in inner out = func(*args, **kwargs) File "/home/mare/anaconda3/envs/nerfstudio/lib/python3.8/site-packages/nerfstudio/pipelines/base_pipeline.py", line 298, in get_train_loss_dict ray_bundle, batch = self.datamanager.next_train(step) File "/home/mare/anaconda3/envs/nerfstudio/lib/python3.8/site-packages/nerfstudio/data/datamanagers/base_datamanager.py", line 542, in next_train ray_bundle = self.train_ray_generator(ray_indices) File "/home/mare/anaconda3/envs/nerfstudio/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "/home/mare/anaconda3/envs/nerfstudio/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl return forward_call(*args, **kwargs) File "/home/mare/anaconda3/envs/nerfstudio/lib/python3.8/site-packages/nerfstudio/model_components/ray_generators.py", line 56, in forward ray_bundle = self.cameras.generate_rays( File "/home/mare/anaconda3/envs/nerfstudio/lib/python3.8/site-packages/nerfstudio/cameras/cameras.py", line 463, in generate_rays raybundle = cameras._generate_rays_from_coords( File "/home/mare/anaconda3/envs/nerfstudio/lib/python3.8/site-packages/nerfstudio/cameras/cameras.py", line 646, in _generate_rays_from_coords coord_stack[coord_mask, :] = camera_utils.radial_and_tangential_undistort( File "/home/mare/anaconda3/envs/nerfstudio/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py", line 328, in _fn return fn(*args, **kwargs) File "/home/mare/anaconda3/envs/nerfstudio/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py", line 490, in catch_errors return callback(frame, cache_entry, hooks, frame_state) File "/home/mare/anaconda3/envs/nerfstudio/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 641, in _convert_frame result = inner_convert(frame, cache_size, hooks, frame_state) File "/home/mare/anaconda3/envs/nerfstudio/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 133, in _fn return fn(*args, **kwargs) File "/home/mare/anaconda3/envs/nerfstudio/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 389, in _convert_frame_assert return _compile( File "/home/mare/anaconda3/envs/nerfstudio/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 569, in _compile guarded_code = compile_inner(code, one_graph, hooks, transform) File "/home/mare/anaconda3/envs/nerfstudio/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper r = func(*args, **kwargs) File "/home/mare/anaconda3/envs/nerfstudio/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 491, in compile_inner out_code = transform_code_object(code, transform) File "/home/mare/anaconda3/envs/nerfstudio/lib/python3.8/site-packages/torch/_dynamo/bytecode_transformation.py", line 1028, in transform_code_object transformations(instructions, code_options) File "/home/mare/anaconda3/envs/nerfstudio/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 458, in transform tracer.run() File "/home/mare/anaconda3/envs/nerfstudio/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2074, in run super().run() File "/home/mare/anaconda3/envs/nerfstudio/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 724, in run and self.step() File "/home/mare/anaconda3/envs/nerfstudio/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in step getattr(self, inst.opname)(inst) File "/home/mare/anaconda3/envs/nerfstudio/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 2162, in RETURN_VALUE self.output.compile_subgraph( File "/home/mare/anaconda3/envs/nerfstudio/lib/python3.8/site-packages/torch/_dynamo/output_graph.py", line 833, in compile_subgraph self.compile_and_call_fx_graph(tx, list(reversed(stack_values)), root) File "/home/mare/anaconda3/envs/nerfstudio/lib/python3.8/contextlib.py", line 75, in inner return func(*args, **kwds) File "/home/mare/anaconda3/envs/nerfstudio/lib/python3.8/site-packages/torch/_dynamo/output_graph.py", line 957, in compile_and_call_fx_graph compiled_fn = self.call_user_compiler(gm) File "/home/mare/anaconda3/envs/nerfstudio/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper r = func(*args, **kwargs) File "/home/mare/anaconda3/envs/nerfstudio/lib/python3.8/site-packages/torch/_dynamo/output_graph.py", line 1024, in call_user_compiler raise BackendCompilerFailed(self.compiler_fn, e).with_traceback( File "/home/mare/anaconda3/envs/nerfstudio/lib/python3.8/site-packages/torch/_dynamo/output_graph.py", line 1009, in call_user_compiler compiled_fn = compiler_fn(gm, self.example_inputs()) File "/home/mare/anaconda3/envs/nerfstudio/lib/python3.8/site-packages/torch/_dynamo/repro/after_dynamo.py", line 117, in debug_wrapper compiled_gm = compiler_fn(gm, example_inputs) File "/home/mare/anaconda3/envs/nerfstudio/lib/python3.8/site-packages/torch/__init__.py", line 1607, in __call__ return self.compiler_fn(model_, inputs_, **self.kwargs) torch._dynamo.exc.BackendCompilerFailed: backend='eager' raised: TypeError: eager() got an unexpected keyword argument 'mode' Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information You can suppress this exception and fall back to eager by setting: import torch._dynamo torch._dynamo.config.suppress_errors = True ..

Solution

* find where train.py is installed
(nerfstudio) mare@mare>NerfStudio$ find /home/mare/anaconda3/envs/nerfstudio/lib/python3.8/site-packages/ -name 'train.py'

in my case it installed in here:
/home/mare/anaconda3/envs/nerfstudio/lib/python3.8/site-packages/nerfstudio/scripts/train.py
(nerfstudio) 


* edit train.py code 
nano /home/mare/anaconda3/envs/nerfstudio/lib/python3.8/site-packages/nerfstudio/scripts/train.py

add this two line on top of code
import torch._dynamo
torch._dynamo.config.suppress_errors = True

ok it's my solution.


Thank you.
www.marearts.com




10/08/2023

round to first significant decimal

The code extract first found and convert to 1

ex)

Input: 0.02323, Output: 0.01
Input: 0.0023, Output: 0.001
Input: 23, Output: 1
Input: 0.0042323, Output: 0.001

..

 refer to this code:

def custom_marearts_round(number):
# Ensure number is a float
float_number = float(number)

# If number is zero, return it as is
if number == 0:
return 0
# If the number is an integer, return 1
elif float_number.is_integer():
return 1
# Convert the number to exponential notation
exponential_notation = f'{float_number:.1e}'
# Extract the exponent part and convert to int
exponent = int(exponential_notation.split('e')[-1])
# Convert to desired output format
result = float(f'1e{exponent}')
return result

# Examples:
input1 = 0.02323
output1 = custom_marearts_round(input1)
print(f"Input: {input1}, Output: {output1}")

input2 = 0.0023
output2 = custom_marearts_round(input2)
print(f"Input: {input2}, Output: {output2}")

input3 = 23
output3 = custom_marearts_round(input3)
print(f"Input: {input3}, Output: {output3}")

input4 = 0.0042323
output4 = custom_marearts_round(input4)
print(f"Input: {input4}, Output: {output4}")



..



๐Ÿ™‡๐Ÿป‍♂️ Thank you

www.marearts.com