Thursday, December 5, 2013

OpenMV Update: 25FPS Face Detection, USB Support and More

So I've been working on OpenMV for the past week and this is what I have so far:

USB Support:
The camera now supports USB OTG full speed, I've also written a small userspace tool with libusb/SDL to interface with the camera and view the frame buffer, this makes it really easy to debug the image processing code, and it also lets you change the sensor's registers while watching the results in realtime.


I've mentioned building the STM32F4xx libraries in a previous post, you can checkout the repo linked there if you want to build the libraries.

Face Detection:
Many were very interested in this feature, well I've managed to get the viola-jones face detector working on the camera, and it's working fine.. For those of you familiar with the detector, the haar cascade is exported as a C header which is linked to the binary and loaded into the CCM (Core Coupled Memory) a 64KB memory block connected directly to the core. Only one integral image is pre-computed and allocated on the heap, the other one, the squared integral image, which is used for computing the standard deviation, can't fit into memory for the QQVGA resoultion, and so, instead, the standard deviation is computed on the fly for every detection scale using some SIMD instructions to speed it up a bit.


The memory can hold up to 23 stages, however, using only 12 stages and with a relatively large scale step, the detector is working great, with occasional false detections of course, more stages can be used if greater accuracy is required, but not without some performance penalty...As for the numbers, the camera can process 7-8FPS QQVGA, and for QQCIF (88x72) I get 25FPS

Here's a video of the face detector in action running at 25FPS:


Here's another video of color tracking running at 30FPS:


Other Updates:
I've been doing some general fixes here and there, mainly to improve the image quality, in addition to that, I've compiled all the libraries and code with optimizations (-O2) and I've seen great improvements in speed, there's also a new pixel format, grayscale, which is basically just the Y channel extracted from the YUV422 to avoid doing that every time a grayscale image is required.

The QCIF/QQCIF are working now (the sensor can output 60FPS when using QQCIF ) and through some other register probing, I've removed a few useless registers and discovered that the sensor has digital zoom, cool!

There's also simple motion detection code in progress, it's based on frame differencing and using the first frame as the background, more work will be done here as soon as I get around to it. And I will probably try template matching next.


I've also just finished a new hardware revision, it has a tiny uSD socket, which I imagine can be used for anything from storing haar cascades, snapshots or video to buffering larger frames, the new revision is also a bit smaller. 

21 comments:

  1. wow the project is amazing. Can you post some more info on how you used ccm?

    Did you upload this small aplication to the repo?

    I really want to play with it :) I would like to try it on my quad for auto take off. Do you have any news on manufacter it? do you have any extra board and camera that you can sell?

    ReplyDelete
    Replies
    1. Thanks, I've used the CCM exactly as described in my previous post :) and you can checkout the code if you like, the repo has everything including the userspace code (the source of libraries and USB stuff are kept in a different repo though).
      I'm working with someone on getting it manufactured, hopefully soon you could get one, I have some extra PCBs but there's a new revision with and uSD socket, so those are useless now it's better to wait for the new ones.

      Delete
  2. Can you creakout pins PB10 to PB15, I2C2 and SPI2, that way we can interface it with other projects :)

    ReplyDelete
    Replies
    1. No, there's not much space left, but it has serial broken out already.

      Delete
  3. You can breakout unused gpio to small testpoint pads. I do that for all my mcu pcbs. Its better to solder to testpoints than the lqfp pins directly. It would add some value to your board because it will be more hackable.

    ReplyDelete
    Replies
    1. true, but I was thinking you wouldn't need that because you have serial and can have the other board do all the control stuff, plus, you can't draw to much power out it if anyway.

      Delete
    2. I don't think that people will need that GPIOs to source power. The main thing is that it costs nothing extra on PCB ("just" a time which not everybody has :) ) to breakout the most interesting unused GPIOs to small testpoints. I guess that if someone would like to use only your camera module and no ten times slower arduino then your board will be the winner.
      Just like the metRo_ in the comment above talked about I2C2 and SPI2. I would imagine that SPI2 would be great for connection to some cheap Nokia SPI LCD.
      Anyway, your project is really great and you've done lot of work. I'll keep an eye on this.

      Delete
    3. yes that would be nice, okay, I will add a few test points on the next revision, and see if I can find a way to fit a header there too :)

      Delete
  4. Hello Ibrahim
    This is a cool project. Do you have a paypal account or something similar where one can donate a small amount to your project?

    HC

    ReplyDelete
    Replies
    1. Hi, yes I have paypal, i.abdalkader@gmail.com
      thanks :)

      Delete
  5. Do you know if the OV9650 has an IR cut-off filter that is possible to remove? If so, it would be very easy to replace the LED with an IR LED or two, which would be very interesting for me.

    ReplyDelete
    Replies
    1. no, I have no idea, how do I check out ?

      Delete
    2. It would be between the lens and the sensor, but I'm not sure how to get access to it. I'm going to order one right now so I can take a look at it—I'll post another comment after I get it.

      Delete
    3. I took one apart before, and I still have the lens, I will take a picture and upload now..

      Delete
    4. http://i.imgur.com/P7t3xHs.jpg

      Delete
  6. The micro python KS now knows about your board. (see comments page)
    Damien would like more info.
    Please contact him
    http://www.kickstarter.com/projects/214379695/micro-python-python-for-microcontrollers/comments

    ReplyDelete
    Replies
    1. Hi, yes I already contacted Damien, I'm also a supporter of Micro Python, so I think I will get early access to the code, and as soon as it's released I will try porting it STM32F407 to run it on OpenMV and a new project I'm working on, if no one else writes a port... Thanks anyway :)

      Delete
    2. You probably already have this sorted but I saw this post on DMA and eth 407 Camera interface:
      http://blog.frankvh.com/2011/08/19/stm32f2xx-digital-camera-interface-dcmi/
      In case its useful.

      Delete
  7. Sweet! I am glad I didn't order a board yet. Has the new revision been tested?

    ReplyDelete
    Replies
    1. No not yet, I'm still waiting for the boards, but you should definitely order the new one, it hasn't changed much, so everything should still be working :)

      Delete
  8. Hi,

    Awesome work! I am currently working on developing a ultra low cost pick & place solution that requires basic vision such as template matching (rectangular outlines, to determine size, center point and rotation angle). I am currently planning to use a 1.3MP CMOS sensor (like the OV9640) and the STM32F407 (or the STM32F429 for higher speed). How far have you progressed with the template matching code on your project ? I look forward to hearing back from you.

    Regards,

    Raj

    ReplyDelete