Home / News

Blog - News

Low Power Deep Learning on the OpenMV Cam powered by the Cortex-M7 Processor

Edge computing helps reduce latency and power consumption by performing signal processing at the source of the data. Recently ARM released the CMSIS-NN library, an efficient neural network library optimized for the Cortex-M based microcontrollers. The CMSIS-NN library (pronounced Sim-Sis) brings deep learning to low-power microcontrollers such as the Cortex-M7 based OpenMV camera. In this blog post we'll go through training a custom neural network using Caffe on a PC and deploying the network on the OpenMV Cam.

CMSIS-NN:

The CMSIS-NN library consists of a number of optimized neural network functions using SIMD and DSP instructions, separable convolutions, etc... and most importantly, it supports 8-bit fixed point representation. Using fixed-point avoids costly floating point operations, reduces the memory footprint, and uses less power when running inference. However, this means that models have to be quantized first before using them with CMSIS-NN.

Quantization, simply put, is the mapping of a range of numbers to a more compact range of numbers, or in this case the mapping of 32-bits floats to 8-bit fixed-point numbers. The hardest part about quantizing models is finding the min and max ranges of layer inputs/outputs to evenly distribute the floating point values across an 8-bit representation. Fortunately, ARM also provides a script to quantize Caffe model weights and activations. If you're interested in more details about the CMSIS-NN library and the quantization process please see this paper published by ARM researchers.

The default CMSIS-NN library comes with a CNN example trained on the CIFAR-10 dataset. However, this example is hard-coded, meaning it must compiled and linked with the main application. So, we extended the CMSIS-NN library and supporting scripts. Our code allows users to convert Caffe models to a quantized binary format which can be loaded from the file-system (SD Card or internal flash) at run-time. Additionally, our code takes care of preprocessing the input image, subtracting the mean, and scaling the data if required.

In the next sections of this blog post I'll demonstrate how to use the CMSIS-NN library with the OpenMV camera to train a simple CNN model on a smile detection dataset. The model achieves ~93% accuracy and the camera consumes about 150mA @ 3.3V while running the network.

Training a network with Caffe:

First, if you're just getting started with neural networks and Caffe I highly recommend this tutorial. Note that you should keep in mind the CMSIS-NN library and conversion scripts have limitations on the number and types of supported layers - so your model should be simple.

Dataset:

The smile dataset that we used can be found here. The dataset consists of ~3000 positive images and ~9000 negative images. We need the number of positive and negative images to be close otherwise the network will biased to a class (class imbalance). To fix this we can augment the dataset using this augmentation script on the positive images to increase the number of positive examples by 4x. The image augmentation script can be used like this:

python2 augment_images.py --input images/train/ --output images/train_aug/ --count 4

Training the network:

You can use any deep learning library to train the network. However, if you're not using Caffe you need to convert the network output to a Caffe format to work with the ARM scripts. In the future ARM will provide more conversion scripts to accept models from TensorFlow, etc.

Quantizing the model:

The first step after training the network is to use the quantization script provided by ARM to convert the Caffe model weights and activations from floating point to fixed point format. As mentioned before, quantization is performed to reduce the size of the network and avoid floating point computations.

The nn_quantizer script works by testing the network and figuring out the best format for the dynamic fixed point representation. The output of this script is a serialized Python (.pkl) file which includes the network's model, quantized weights and activations, and the quantization format of each layer. Running this command generates the quantized model:

python2 nn_quantizer.py --model models/smile/smile_train_test.prototxt --weights models/smile/smile_iter_*.caffemodel --save models/smile/smile.pkl

Converting the model to binary:

The next step is to use our NN converter script to convert the model into a binary format runnable by the OpenMV Cam. The converter script outputs a code for each layer type followed by the layer's dimensions and weights (if any).

On the OpenMV Cam our firmware reads the binary file and builds the network in memory using a linked-list like structure.

Running this command generates the binary model:

python2 nn_convert.py --model models/smile/smile.pkl --mean /path/to/mean.binaryproto --output smile.network

Deployment on an OpenMV camera:

While it's possible to slide the detection window over the entire image doing so would be very slow. Instead we use the built-in Haar cascade face detector to extract faces from the image, and then pass the region of interest (ROI) to the CNN to detect smiles. The first part of the smile detection code loads the network into memory and loads the built-in face detection Haar Cascade.

# Load Smile Detection network
net = nn.load('/smile.network')

# Load Face Detection Haar Cascade
face_cascade = image.HaarCascade("frontalface", stages=25)
print(face_cascade)

The next step is capturing a snapshot and finding all the faces.

# Capture snapshot
img = sensor.snapshot()
# Find faces.
objects = img.find_features(face_cascade, threshold=0.75, scale_factor=1.25)

Finally, for each detected face, the region of interest is slightly cropped and passed to the neural network. Note that the smile detection network is trained on tightly cropped faces so we have to reduce the size of the ROI.

# Detect smiles
for r in objects:
# Resize and center detection area
r = [r[0]+10, r[1]+25, int(r[2]*0.70), int(r[2]*0.70)]
out = net.forward(img, roi=r, softmax=True)
img.draw_string(r[0], r[1], ':)' if (out[0] > 0.8) else ':(', color=0, scale=2)

Smile Detection

Moving forwards:

The OpenMV Cam uses the Cortex-M7 processor without any external DRAM attached and only uses the internal SRAM. At any point in time the processor can go into low-power mode drawing about 50 uA while maintaining all state, then wake-up again on an interrupt, take a picture, and run the neural network before turning off again.

For example, on the upcoming OpenMV Cam H7 we're able to run a Lenet-6 CNN trained on the MNIST data set at 50 FPS while using only 3mA @ 3.3V per inference. With a 1Ah 3.7V Lipo battery you can deploy a CNN in the field running every minute that would last for over a year.

In particular, CNN support on Cortex-M7 processors lines up particularly well with deploying smart thermal vision based cameras that are able to detect people accurately from low-resolution thermal vision images. Smart sensors powered by CMSIS-NN and Cortex-M7 processors are coming soon!

For a MNIST demonstration checkout the video below:

Firmware Release:

You can grab the latest firmware (v3.0.0) from our GitHub here. Example scripts are here. And the models are here. The OpenMV IDE release to package everything up is coming soon.

Introducing the OpenMV Cam H7

Hi everyone,

We've got a lot of news today! But, first, checkout the the upcoming OpenMV Cam H7!

The OpenMV Cam H7 is powered by ST Microelectronics new STM32H7 microcontroller which features a 400 MHz ARM Cortex M7 CPU capable of executing 2 instructions per clock. The STM32H7 achieves a coremark score of 2020 which is comparable to the 2060 coremark score of the 1 GHz Raspberry Pi 0 for general purpose instruction execution performance (the ARM Cortex M7 core is a lot more efficient than the ARM11 core the Raspberry Pi Zero uses). Additionally, the OpenMV Cam H7 now features removable camera modules which allow you to use the OpenMV Cam with Global Shutter and FLIR Lepton sensors for serious computer vision applications.

Here's a short list of the product enhancements:

  • 2x Better Performance than the OpenMV Cam M7
  • High quality JPEG image compression without much overhead thanks to JPEG encoder hardware on chip.
  • Much lower power consumption (40 nm tech versus 90 nm tech for the OpenMV Cam M7) - approximately ~80 mA less.
  • An on board switching power supply capable of supplying 1A.
  • A new LiPo Battery Connector for JST compatible LiPo batteries.
  • Removable Camera Modules!
    • A FLIR Lepton 1/2/3 Carrier Board for plug and play thermal computer vision.
    • An MT9V034 RGB Global Shutter for blur free computer vision capable of running at up to 240 FPS @ QQVGA.

We're almost done with the development of the OpenMV Cam H7, everything is working great so far with the OV7725 camera (which comes with the OpenMV Cam H7 by default). We've only got the FLIR Lepton board and the MT9V034 board left to complete before we're ready to KickStart the new product. If you'd like to help us out, please spread the word and help get more people on our email list and following us on Twitter/YouTube.

OpenMV IDE v1.9.0 Released

The key new feature of OpenMV IDE v1.9.0 is mult-language support. The IDE has now been translated into Spanish, French, German, Italian, Russian, Chinese, and Japanese!

Other than the new-multi language support features... we fixed bugs related to the OpenMV Cam working on Macs, the video recorder not working on some linux machines, and various other things.

OpenMV Cam Firmware v2.9.0 Released

With the latest OpenMV Cam firmware, we massively improved a lot of the basic image manipulation methods, drawing features, filtering, and argument parsing. Tons of things were improved in the image module.

In particular:

  • Drawing methods all now have everything you'd ever want and need. Thickness options, filling, text scaling and kerning support, etc.
    • New methods for drawing arrows, flood filling, drawing images, etc.
  • Much improved binary image manipulation (support for masks, etc).
  • All the standard image math ops (add, mul, div, min, max, etc).
  • Adaptive Contrast Limited Adaptive Histogram Equalization Support
    • It's fast and massively improves image quality for folks using the OpenMV Cam outside in dynamically lit environments.
  • Better image filtering options (adaptive thresholding on everything, more obvious sharpening support, bilateral image filtering, cartoon image filtering, etc.).
  • And finally, a fun CIFAR-10 CNN Built-In!

ARM recently released a new Convolution Neural Network Library for Cortex M4/M7 microcontrollers that lets them run small networks trained on your desktop/laptop or the cloud on your OpenMV Cam. The built-in CIFAR-10 CNN shows off this kind of functionality. That said, coming soon in our next firmware release we'll have support for running ANY CNN (that's small enough) on your OpenMV Cam using a conversion script that will convert a network trained using Caffe to a binary blob your OpenMV Cam can load and run. And... thanks to the deep learning revolution there's tons of data sets out there for all kinds of things.

So, we'll try to distribute OpenMV IDE in the future with pre-converted CNNs for all sorts of things you'd like to detect with your OpenMV Cam. We're serious about supporting this new functionality for the OpenMV Cam M7 and the OpenMV Cam H7. With smaller networks looking for only one class or not you should be able to easily hit 30+ FPS CNN performance on the OpenMV Cam H7 for detecting things like people entering and exiting doorways (a very common request).

Anyway, enjoy OpenMV IDE v1.9.0 and Firmware v2.9.0!

OpenMV Cams in Stock Again!

Hi all,

We've finally finished getting the next batch of 2.5K OpenMV Cam M7s ready for sale! Even better, for this production run only we're offering a free OpenMV Cam M7 Case with every OpenMV Cam M7 sold! Finally, new OpenMV Cams will now come pre-focused with no microscopic dirt on the sensor or lens! Woot!

New Products

Moving on, we've got a few new products for sale. At the top of the list is our new OpenMV Cam M7 Case:

The case is made to perfectly fit around your OpenMV Cam M7 and is backwards compatible with the OpenMV Cam M4 V1/2. We had to iterate quite a few times to get the case right for injection molding.

Next up is the Motor Shield:

The shield features a 2-channel 2A per-channel motor driver chip and a beefy 5V regulator to power your OpenMV Cam. It works with any battery that outputs between 6-9VDC. With the motor shield building a robot with your OpenMV Cam is now much easier! Best of all, you can use it with the Servo Shield too.

Finally, we've got a new lens:

The new Super Telephoto Lens above has some serious 8X zoom. It's a 25mm lens that gives you a lot of detail at a distance. Note however, that it does not come with an IR Cut filter for outdoor use.

New Products Coming Soon

Now, for something we're really excited about - polarizing filters!

The polarizing filter is a simple but incredibly useful piece of optical tech that can make the difference between success and failure for computer vision in the presence of the sun. In particular, for DIY Robocar racing its absolutely necessary to be able to follow a line on a shiny floor when the sun is overhead. Best of all, the filter fits right under your M12 lens mount.

Now, no-one sells 14x14mm polarizing lens filters for M12 lens mounts. We tried to source cheap ones from quite a few places. But, we couldn't find any. So, we went to Edmund's Optics for custom samples to test out. Here are the results:

Without a filter

w/ the default 2.8mm lens
w/ IR cut filter
Without Filter
With a filter

w/ the default 2.8mm lens
w/ IR cut filter
With Filter

Night and day difference right?

Anyway, we haven't yet purchased a production run of polarizing filters. But, if you're interested in helping us out please place a pre-order for some polarizing lens filters. The filters are not cheap to buy and have to be ordered in bulk (i.e. thousands) to get a decent price per unit. If you do plan to pre-order some filters please don't put anything else in your cart. We don't expect to stock this product for a few months.

Store Upgrades

In addition to the above new products we've also upgraded our online store! All products now have bulk discount pricing! OpenMV will now offer 10% off purchases of 10 or more items, 20% off purchases of 50 or more items, and 30% off purchases of 100 or more items.

Moving on, we've also finished moving our inventory to an integrated shipping solutions provider. We can now offer USPS, FedEx, UPS, and DHL (soon) shipping to all customers. Additionally, once we've finished working out all the bugs we'll be able to offer same day shipping for all orders using our new platform - the store will finally be able to fulfill your order automatically!

And a shout out to MacroFab

Integrating our Shopify store with our new fulfillment backend was not... painless. In the last few weeks we lost several thousand dollars to miscalculated shipping costs, stale already completed orders being automatically shipped again (which is a shipping cost and a loss of merchandise), and surprisingly expensive kitting fees from things not arriving just right from our manufactures. That said, we expected it to be a little bumpy moving to a professional drop shipping company. So... we're fine. But, we'd just like to give a shout out to MacroFab who helped OpenMV get started. Their integrated manufacturing and fulfillment service allowed us to build products, get them kitted, and then shipped to customers all from one place (albeit you'll be clearing orders manually). But, with MacroFab, we never had to deal with any "surprise" shipping/kitting costs. If you're thinking about building an electronic product and selling it we highly recommend you start your business with MacroFab! It's a pleasure working with them.