I created this project as part of my efforts to benchmark Tensorflow Lite on the MaaXBoard. I wanted to compare the MaaXBoard's performance running inference to similar developer boards, so I chose the Google Coral and Raspberry Pi 3 Model B+.
- Instructions for benchmarking the Raspberry Pi 3 Model B+ are here.
- Instructions for benchmarking the Coral dev board are here.
- Final benchmarking article is here.
- Alasdair Allan's original big benchmarking roundup is here.
MobileNet V.1 and V.2
Luckily for us, there are pre-converted models that meet all of the requirements, like these versions of MobileNet v1 and MobileNet v2. Even luckier - Alasdair Allan created a script to benchmark MobileNet last year, so I can run the exact same script and compare my results to his.
This benchmark uses the MobileNet v2 SSD model trained on the COCO dataset.
I wanted to try a couple other benchmarks, so I browsed through tfhub.com, where Google hosts hundreds of pre-trained Tensorflow models available on tfhub.dev. I was interested in image classification and detection models that were precompiled to Tensorflow Lite, optimized for mobile and edge devices, and had a version precompiled for the Google Coral as well.
I decided to try EfficientNet-Lite as well, since it's so new and shiny. I ended up benchmarking two different versions because the first version I benchmarked was the original version that was optimized for Coral Dev Board's Edge TPU and it didn't perform so well.INSTALL PREREQUISITES
- First, set the MaaXBoard up headlessly.
- I recommend installing full Tensorflow as well first so that you're familiar with that process.
Create a new virtual environment called 'tflite.'
Get your python version:
- Set up Tensorflow Lite. You'll have to download the appropriate version of aarch64 linux and whatever python version you have. My Python is 3.7.3, so the flags are
sudo pip3 install --upgrade tflite_runtime-2.1.0-cp37-cp37m-linux_aarch64.whl
- Set up OpenCV. I already have OpenCV installed on my other virtualenv, so I can just link to it:
ln -s /usr/lib/python3.7/site-packages/cv2/python-3.7/cv2.cpython-37m-aarch64-linux-gnu.so ./cv2.so
- Install pre-requisites for Tensorflow Lite:
sudo apt install swig libjpeg-dev zlib1g-dev
Check to make sure you have tensorflow installed on your virtual environment:
pip3 show tensorflow
Enter python interactive mode so we can check the opencv version:
python3 -c 'import cv2; print(cv2.__version__)'
Download model and benchmarking script
pip3 install Pillow
Install curl to be able to retrieve the files from dropbox:
sudo apt-get update
sudo apt install curl
We're using the same files on the MaaXBoard that were used on the Raspberry Pi.
The original files Alasdair created were hosted on dropbox here. Because of changes to Tensorflow v.2, I've updated the code and also included code to benchmark image classification. The only thing it doesn't contain is the EfficientNet-Lite model, but that we can download from the Google Coral github.
You can download the files to the MaaXBoard like this:
RUN THE BENCHMARKS
curl -L https://hacksterio.s3.amazonaws.com/uploads/attachments/1143280/RASPBERRYPI.zip > files.zip
Run the benchmark for MobileNet v.2 on MaaXBoard:
python3 benchmark_tf_lite.py --model mobilenet_v2.tflite --label coco_labels.txt --input fruit.jpg --output out.jpg --runs 1000
Elapsed time is 364.2792464489976 ms
apple score = 0.953125
banana score = 0.83984375
Run the benchmark for MobileNet v.1 on MaaXBoard:
python3 benchmark_tf_lite.py --model tflite_for_rpi_V1/mobilenet_v1.tflite --label tflite_for_rpi_V1/coco_labels.txt --input fruit.jpg --output out.jpg --runs 1000
Elapsed time is 282.3099827770002 ms
apple score = 0.80078125
apple score = 0.515625
MobileNet v.1 is faster, but just like Raspberry Pi and Coral Dev Board, it thinks the banana is an apple.
Download model from here: https://coral.ai/models/
I downloaded it to my host computer and copied it over to the board:
scp efficientnet-edgetpu-L_quant.tflite ebv@[IP ADDRESS]:
curl -OL https://raw.githubusercontent.com/google-coral/edgetpu/master/test_data/imagenet_labels.txt > imagenet_labels.txt
Benchmark for EfficientNet L on MaaXBoard:
python3 benchmark_classification.py --model efficientnet-edgetpu-L_quant.tflite --label imagenet_labels.txt --input fruit.jpg --runs 1000
Elapsed time is 3342.868333061997 ms
0.125490: pineapple, ananas
As you can see, it's painfully slow and not very accurate.Benchmark the lite0 model
There is a version of EfficientNet-Lite that's not specialized for the edge TPU, so I decided to try running this one. It's quite a bit smaller than the other version - only 5.17MB: https://tfhub.dev/tensorflow/lite-model/efficientnet/lite0/int8/1
Benchmark for EfficientNet v0 on MaaXBoard:
python3 benchmark_classification.py --model efficientnet_lite0_int8_1.tflite --label effiecientnet/imagenet_labels.txt --input fruit.jpg --runs 1000
Elapsed time is 180.288102787 ms
banana score = 0.72265625
Wow, this model is way faster and way more accurate!FINAL RESULTS
If you want to know how the MaaXBoard performed vs the other hardware, the full benchmarking article is here.