Saturday, June 22, 2024
HomeIoTBenchmarking TensorFlow and TensorFlow Lite on Raspberry Pi 5

Benchmarking TensorFlow and TensorFlow Lite on Raspberry Pi 5

All the way in which again in 2019 I spent plenty of time machine studying on the sting. Over the course of about six months I printed greater than a dozen articles on benchmarking the then new technology of machine studying accelerator {hardware} that was solely simply beginning to seem in the marketplace, and gave a collection of talks round the findings.

So much has modified within the intervening years, however after a getting a current nudge I returned to my benchmark code and — after fixing a few of the inevitable bit rot — I ran it on the new Raspberry Pi 5.

Headline outcomes from benchmarking

Working the benchmarks on the new Raspberry Pi 5 we see important enhancements in inferencing pace, with full TensorFlow fashions operating virtually ×5 sooner than on they did on Raspberry Pi 4. We see an identical improve in inferencing pace when utilizing TensorFlow Lite, with fashions once more operating virtually ×5 sooner than on the Raspberry Pi 4.

Nonetheless maybe the extra spectacular result’s that, whereas inferencing on Coral accelerator {hardware} remains to be sooner than utilizing full TensorFlow fashions on the Raspberry Pi 5, the brand new Raspberry Pi 5 has comparable efficiency when utilizing TensorFlow Lite to the Coral TPU, displaying basically the identical inferencing speeds.

ℹ️ Info As per our earlier outcomes with the Raspberry Pi 4 we used lively cooling with the Raspberry Pi 5 to CPU temperature secure and stop thermal throttling of the CPU throughout inferencing.

The conclusion is that customized accelerator {hardware} could now not be wanted for some inferencing duties on the edge, as inferencing instantly on the Raspberry Pi 5 CPU — with no GPU acceleration — is now on a par with the efficiency of the Coral TPU.

ℹ️ Info The Coral {hardware} makes use of quantization the identical means TensorFlow Lite does to cut back the dimensions of fashions. Nonetheless to make use of a TensorFlow Lite mannequin with Edge TPU {hardware} there are a couple of further steps concerned. First you want to convert your TensorFlow mannequin to the optimized FlatBuffer format to characterize graphs utilized by TensorFlow Lite. However moreover you additionally have to compile your TensorFlow Lite mannequin for compatibility with the Edge TPU utilizing Google’s compiler.


Inferencing speeds with TensorFlow and TensorFlow Lite on the Raspberry Pi 5 are considerably improved over Raspberry Pi 4. Moreover, the Raspberry Pi 5 now presents comparable efficiency to the Coral TPU.

Half I — Benchmarking

A extra in-depth evaluation of the outcomes

In our unique benchmarks present we noticed that the 2 devoted boards, the Coral Dev Board from Google and the JetsonNano Developer Package from NVIDIA, have been the very best performing out of our surveyed platforms. Of those two boards the Coral Dev Board ran considerably sooner, with inferencing instances round ×4 shorter than the Jetson Nano for a similar machine studying mannequin.

Nonetheless, on the time the benchmarking outcomes made me wonder if we had gone forward and began to optimize issues in {hardware} just a bit too quickly.

The considerably sooner inferencing instances we noticed then from fashions which making use of quantization, and the dominance of the Coral platform which additionally relied quantization to extend its efficiency, instructed that we should always nonetheless be exploring software program methods earlier than persevering with to optimize accelerator {hardware} any additional.

These outcomes from benchmarking on the Raspberry Pi 5 appear to bear my unique doubts out. It has taken 4 years for basic CPUs to meet up with what was then the very best in school accelerator silicon. Whereas a new technology of accelerator {hardware} is now obtainable which can be extra performant — and sure, I will be that once I can — the Coral TPU remains to be seen by many as “finest in school” and is in widespread use regardless of an absence of help from Google for their accelerator platform.

The Raspberry Pi 5 is now performant sufficient to maintain up with inferencing in real-time video and performs on a par with the Coral TPU, and the outcomes suggest that for a lot of use circumstances Coral {hardware} could possibly be changed for a big price saving by a Raspberry Pi 5 with none efficiency degradation.


Because of the lack of help from Google for the pycoral library — updates appears to have stopped in 2021 and the library now not works with fashionable Python distributions — together with the problem in getting Coral TPU {hardware} to work with fashionable working techniques the numerous discount in inferencing instances we see on the brand new Raspberry Pi 5 could be very welcome.

Half II — Methodology

Concerning the benchmarking code

Benchmarking was accomplished utilizing TensorFlow, or for the {hardware} accelerated platforms that don’t help TensorFlow their native framework, utilizing the identical fashions used on the opposite platforms transformed to the suitable native framework.

For the Coral EdgeTPU-based {hardware} we used TensorFlow Lite, and for Intel’s Movidius-based {hardware} we used their OpenVINO toolkit. Benchmarks have been carried out twice on the NVIDIA Jetson Nano, first utilizing vanilla TensorFlow fashions, and a second time utilizing these fashions after optimization utilizing NVIDIA’s TensorFlow with TensorRT library.

Inferencing was carried out with the MobileNet v2 SSD and MobileNet v1 0.75 depth SSD fashions, each fashions educated on the Frequent Objects in Context (COCO) dataset. The 3888×2916 pixel take a look at picture was used containing two recognizable objects within the body, a banana🍌 and an apple🍎. The picture was resized all the way down to 300×300 pixels earlier than presenting it to the mannequin, and every mannequin was run 10,000 instances earlier than a median inferencing time was taken.

ℹ️ Info The primary inferencing run, which might take as much as ten instances longer on account of loading overheads, is discarded from the calculation of the typical inferencing time.

Whereas within the intervening years different benchmark frameworks have emerged which are arguably extra rigorous, the benchmarks introduced listed below are meant to replicate actual world efficiency. A variety of the opposite newer benchmarks measure the time to finish solely the inferencing stage. Whereas that’s a a lot cleaner (and shorter) operation than the timings measured right here — which embrace arrange time — most individuals aren’t actually enthusiastic about simply the time it takes between passing a tensor to the mannequin and getting a consequence. As a substitute they need end-to-end timings.

One of many issues that these benchmarks do not do is optimization. They take a picture, go it to a mannequin, and measure the consequence. The code is straightforward, and what it measures is akin to the efficiency a median developer doing the identical activity would possibly get, quite than an skilled machine studying researcher than understands the complexities and limitations of the fashions, and the way to adapt them to particular person platforms and conditions.

Establishing your Raspberry Pi

Go forward and obtain the newest launch of Raspberry Pi OS and arrange your Raspberry Pi. Except you’re utilizing wired networking, or have a show and keyboard hooked up to the Raspberry Pi, at a minimal you’ll have to put the Raspberry Pi on to your wi-fi community, and allow SSH.

When you’ve arrange your Raspberry Pi go forward and energy it on, after which open up a Terminal window in your laptop computer and SSH into the Raspberry Pi.

ssh pi@raspberrypi.native

When you’ve logged in you possibly can set up TensorFlow and TensorFlow Lite.

⚠️Warning Beginning in Raspberry Pi OS Bookworm, packages put in by way of pip have to be put in right into a Python digital atmosphere. A digital atmosphere is a container the place you possibly can safely set up third-party modules in order that they received’t intervene along with your system Python.

Putting in TensorFlow on Raspberry Pi 5

Putting in TensorFlow on the Raspberry Pi is much more sophisticated than it was, as there isn’t any longer an official bundle obtainable. Nonetheless thankfully there’s nonetheless an unofficial distribution, which a minimum of means we do not have to resort to constructing and putting in from supply.

sudo apt set up -y libhdf5-dev unzip pkg-config python3-pip cmake make git python-is-python3 wget patchelf
python -m venv --system-site-packages ~/.python-tf
supply ~/.python-tf/bin/activate
pip set up numpy==1.26.2
pip set up keras_applications==1.0.8 --no-deps
pip set up keras_preprocessing==1.1.2 --no-deps
pip set up h5py==3.10.0
pip set up pybind11==2.9.2
pip set up packaging
pip set up protobuf==3.20.3
pip set up six wheel mock gdown
pip set up opencv-python
ARCH=`python -c 'import platform; print(platform.machine())'`
pip set up --no-cache-dir${TFVER}/tensorflow-${TFVER}-cp${PYVER}-none-linux_${ARCH}.whl

Putting in TensorFlow Lite on Raspberry Pi 5

There may be nonetheless an official TensorFlow Lite runtime bundle obtainable for Raspberry Pi, so set up is far more easy than for full TensorFlow the place that possibility is now not obtainable.

python -m venv --system-site-packages ~/.python-tflite
supply ~/.python-tflite/bin/activate
pip set up opencv-python
pip set up tflite-runtime

Working the benchmarks

The script is used to run TensorFlow benchmarks on Linux (together with Raspberry Pi) and macOS. This script may used — with a TensorFlow set up which incorporates GPU help — on NVIDIA Jetson {hardware}.

supply ~/.python-tf/bin/activate
./ --model PATH_TO_MODEL_FILE --label PATH_TO_LABEL_FILE --input INPUT_IMAGE --output LABELLED_OUTPUT_IMAGE --runs 10000

For instance on a Raspberry Pi, benchmarking with the MobileNet v2 mannequin for 10,000 inference runs the invocation could be:

./ --model ssd_mobilenet_v2/tf_for_linux_and_macos/frozen_inference_graph.pb --label ssd_mobilenet_v2/tf_for_linux_and_macos/coco_labels.txt --input fruit.jpg --output output.jpg --runs 10000

This may output an output.jpg picture with the 2 objects (the banana and the apple) labelled.

The script is used to run TensorFlow Lite benchmarks on Linux (together with Raspberry Pi) and macOS.

supply ~/.python-tf-lite/bin/activate
./ --model PATH_TO_MODEL_FILE --label PATH_TO_LABEL_FILE --input INPUT_IMAGE --output LABELLED_OUTPUT_IMAGE --runs 10000

⚠️Warning Fashions handed to TensorFlow Lite have to be quantized. To take action the mannequin have to be transformed to TensorFlow Lite format.

Getting the benchmark code

The benchmark code is now obtainable on GitHub. The repository contains all of the sources wanted to breed the benchmarking outcomes, together with fashions, code for all of the examined platforms, and the take a look at imagery used. There may be additionally an ongoing dialogue about the way to enhance the benchmark to make it extra simply run on new {hardware}.



Please enter your comment!
Please enter your name here

Most Popular

Recent Comments