Hi everyone! 😀 In the last video we've seen how to accelerate the speed of our programs with Pytorch and CUDA - today we will take it another step further with Torch-TensorRT! We will focus on a Machine Learning process called Inference (which is when the model is trained, perfected and ready to make a prediction). For this we will load a state-of-the-art artificial neural network and we will use it to classify a picture of my cat! 🙀🙀🙀 Specifically - we will borrow ResNet50 for our little Pytorch experiment! 😉 We will also run a speed test comparing Pytorch models running on CPU, on CUDA and on Torch-TensorRT - which of these do you think is faster?? ⏲️ TIMESTAMPS ⏲️ ----------------------------------- 00:00 - intro 01:05 - clone Torch-TensorRT 01:40 - install and setup Docker 03:52 - install Nvidia Container Toolkit & Nvidia Docker 2 05:02 - Torch-TensorRT container (option #1) 07:22 - Torch-TensorRT Nvidia NGC container (option #2) 09:00 - import Pytorch 09:16 - load ResNet50 10:25 - load sample image 11:45 - sample image transforms 14:48 - batch size 16:19 - prediction with ResNet50 17:12 - softmax function 18:07 - ImageNet class number to name mapping 20:10 - predict top 5 classes of sample image (topk) 23:33 - speed test benchmark function 27:33 - CPU benchmarks 28:13 - CUDA benchmarks 30:09 - trace model 31:20 - convert traced model into a Torch-TensorRT model 33:02 - TensorRT benchmarks 34:32 - download Jupyter Notebook 34:50 - HOW DID I MISS THIS??? 35:31 - thanks for watching! 🛑 REFERENCED TUTORIALS 🛑 ---------------------------------------------------------------------- ⭐ CUDA Parallel Computing for beginners: https://www.youtube.com/watch?v=r9IqwpMR9TE ⭐ Neural Networks for beginners: https://www.youtube.com/watch?v=i1AqHG4k8mE ⭐ Machine Learning Databases: https://www.youtube.com/watch?v=8z2oLfK2sIc ⭐ Gradient Descent: https://www.youtube.com/watch?v=jwStsp8JUPU ⭐ INSTALLATION LINKS AND BASH COMMANDS ⭐ -------------------------------------------------------------------------------------- 1. Clone Torch-TensorRT and change directory: $ git clone https://github.com/NVIDIA/Torch-TensorRT $ cd Torch-TensorRT 2. Docker installation guide: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html#docker $ curl https://get.docker.com/ | sh \ && sudo systemctl --now enable docker $ sudo groupadd docker $ sudo usermod -aG docker $USER $ newgrp docker $ docker run hello-world 3. Nvidia Docker 2 installation: $ distribution=$(. /etc/os-release;echo $ID$VERSION_ID) \ && curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add - \ && curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-dock er.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list $ sudo apt-get update $ sudo apt-get install -y nvidia-docker2 $ sudo systemctl restart docker $ sudo docker run --rm --gpus all nvidia/cuda:11.0-base nvidia-smi 4. Official Torch TensorRT Container: $ docker build -t torch_tensorrt -f ./docker/Dockerfile . $ docker run --gpus=all --rm -it -v $PWD:/Torch-TensorRT --net=host --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 torch_tensorrt:latest bash $ cd /Torch-TensorRT/notebooks $ jupyter notebook --allow-root --ip 0.0.0.0 --port 8888 5. Nvidia NGC Container (alternative to 4): https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorch $ docker pull nvcr.io/nvidia/pytorch:21.12-py3 $ docker run --net=host --gpus all -it --rm -v local_dir:/container_dir nvcr.io/nvidia/pytorch:21.12-py3 $ jupyter notebook --allow-root --ip 0.0.0.0 --port 8888 6. My cats picture: https://github.com/MariyaSha/Inference_withTorchTensorRT/blob/main/img1.jpg 7. Transforms Normalize - Pytorch Documentation: https://pytorch.org/vision/stable/transforms.html 8. ImageNet class to name mapping: https://github.com/pytorch/hub/blob/master/imagenet_classes.txt 9. Complete Notebook on Github: https://github.com/MariyaSha/Inference_withTorchTensorRT --------------------------------------------------------------------------- 💗 THANK YOU SO MUCH FOR WATCHING! 💗 - Sound effect by: https://www.zapsplat.com/ - Icons by: https://www.flaticon.com/ - Resnet50 image by: https://commons.wikimedia.org/wiki/File:ResNet50.png