Dear Pixy Team,
Still Pixy addicted: The driver I wrote for the fischertechnik-Robo-TX toy controller and a first test robot is up and running. Thanks again for the quick fix for the i2c write problem I have posted some time ago.
Now I’m facing quite frustrating problems, when I’m trying to train Pixy to recognize more than two or three color signatures. After some first tries I have reduced my color set to red, green, blue and yellow legos. Nicely disjunctive colors, no problems for Pixy, one might think. But either the yellow one is overexposed or the blue and the green one are too dark to be recognized. At the same time I am fighting with false positives in the darker regions. Pixy’s color detection performance seems to be quite sensible to changes in the light conditions.
Thus I made some tests and delved a bit into Pixy’s color detection code. I think that I have identified some weaknesses and that fixing them could improve Pixies color classification performance.
Please have a look at the attached document.
Cheers,
Helmut
Hi Helmut,
This is fairly extensive. Thank you for sharing.
I’m not sure if you are saying this or not, but the lookup that occurs on the M0 side, the one that ignores Y, isn’t the end of color filtering.
The divide by Y takes place on the M4 side in blobs.cpp, runlengthAnalysis(), and then the comparison the the bounds takes place, which is the filtering step.
The lookup simply eliminates pixels to reduce the processing load on the M4.
Edward
Hi Helmut,
Wow – this is cool that you prepared this! I like your approach with the color bars – and I’m surprised by the results. At first blush the color gains for white balance seem off. I enabled auto white balance and things improved a little – the bars became taller at least. I need to look at this some more — you’ve definitely piqued my interest though.
In general if you train Pixy on an object and then reduce the exposure (brightness setting), you’ll notice that Pixy is good at detecting the object even at very low exposures. This is an unbiased way to affect the brightness. (and you might have to set min brightness to 0 to see it completely).
I agree with much of what you’re saying — bear in mind that we chose the color space and method because it was accurate and could be done at frame-rate. There are more accurate methods. We’ve done a fair amount of testing with HSV, but we couldn’t find a way to represent it efficiently in an LUT. We could do an RGB 555 lookup where the index is 5 bits of red, 5 bits of green and 5 bits of blue, but losing the lower order bits creates quantization noise and limits the precision (we found).
Maybe frame-rate is less important to some users.
So here’s the challenge:
Take some representative pixels from something you want to detect – we’ll call it the training set.
Create a “model” representation using the training set.
Using the model and a color filtering algorithm, reliably detect that object under varying exposure levels and lighting conditions.
Bonus points for efficiency.
Bonus points for adaptability. ie, can your model adapt, track the object (in a sense).
This was our first problem to solve with Pixy and we solved it using the current scheme, but other schemes are interesting as well.
In PixyMon if you uncomment line 23 in cblobmodule.cpp
MON_MODULE(CBlobModule);
and recompile, you will reveal a new properties tab (see attached pic) and several new menu options. This was our testbed for developing the current algorithm. Using the mon-module interface, new algorithms can be tested fairly easily using real pixels from Pixy. It would be cool if you (or someone) could improve on our existing algorithm, and we’d be able to see it in action and compare. It’s kind of fun to work with pixels at this level! (I think)
Happy to discuss more…
Hi Edward and Rich,
Many thanks for your responses and hints. Maybe my quite lengthy (sorry) pamphlet creates the impression that the color LUT is the root cause for all kinds of color detection problems Pixy might show under certain conditions. Actually I don’t know this. But I believe that a better color model for the lookup would be beneficial.
I agree with that “the lookup that occurs on the M0 side, the one that ignores Y, isn’t the end of color filtering.” But I think that it is the weak point of the filtering chain.
“The lookup simply eliminates pixels to reduce the processing load on the M4.”: This actually applies only, if there is only one signature defined. In fact the lookup does a bit more: It already assigns the signature to a pixel and the final filtering on the M4 just checks the compatibility to this preselected signature. It does not check if the pixel would better fit to another signature. Thus, if the lookup classifies a pixel due to the suboptimal color model incorrectly, the final filtering step won’t fix it.
This is illustrated by a little experiment in the attached pdf.
Many thanks for the MON_MODULE(CBlobModule) hint. I’m looking forward to check this out. … But first I have to get PixyMon built correctly … it builds, OK some warnings, but when I start it, it crashes immediately. So maybe I will bother you with some silly build questions. First I will try to get a clean installation. I just installed the open-source-mingw-qt on top of an already existing msvs installation. Maybe this was a bad idea.
Cheers,
Helmut
Hello Helmut,
The LUT has some constraints. It’s possible (conceptually) for a pixel to satisfy all 7 color signatures if the signatures overlap in hue-space, but the LUT only returns 1 signature. To deal with the potential conflict, the LUT is prioritized, with signature 1 getting priority and any LUT entries that are left over are available for signatures 2 through 7. This can result in some strangeness if you have two signatures that are close to each other in the hue space, but we did it this way for efficiency. It’s expected/assumed that most signatures occupy their own niche in hue-space and don’t have to share with other signatures. This assumption sometimes breaks though.
Regarding the crash, try copying the libusb-1.0.dll into the same directory as the PixyMon executable.
Edward