<< return to Pixycam.com

Question about segmentation part of code

Pixy Software
#1

Hello
I’m trying to understand how exactly You are segmenting the image. I have seen description of CMVision algorithm which is similar to Yours but unfortunately I can’t understand Your implementation. First question, which format do You obtain from the camera (RGB565 or YUV422)? And second question how do You comparise it with LUT table? I think it is made in assembler on pixy but I found second version of it too. It is written in C and I think it is a part of pixymon app. Name of this file is cccmodule.cpp. And this is the part which I don’t understand:

void CccModule::handleLine(uint8_t *line, uint16_t width)
{
    uint32_t index, sig, sig2, usum, vsum, ysum;
    int32_t x, r, g1, g2, b, u, v, u0, v0;
    Qval newline;
    // new line
    m_qq->enqueue(&newline);
    x = 1;

next:
    usum = vsum = ysum = 0;
    r = line[x];
    g1 = line[x-1];
    g2 = line[x-width];
    b = line[x-width-1];
    u = r-g1;
    v = b-g2;
    ysum += r + (g1+g2)/2 + b;
    usum += u;
    vsum += v;

    u0 = u>>(9-CL_LUT_COMPONENT_SCALE);
    v0 = v>>(9-CL_LUT_COMPONENT_SCALE);
    u0 &= (1<<CL_LUT_COMPONENT_SCALE)-1;
    v0 &= (1<<CL_LUT_COMPONENT_SCALE)-1;
    index = (u0<=width)
        return;

    if (sig==0)
        goto next;

    r = line[x];
    g1 = line[x-1];
    g2 = line[x-width];
    b = line[x-width-1];
    u = r-g1;
    v = b-g2;
    ysum += r + (g1+g2)/2 + b;
    usum += u;
    vsum += v;

    u0 = u>>(9-CL_LUT_COMPONENT_SCALE);
    v0 = v>>(9-CL_LUT_COMPONENT_SCALE);
    u0 &= (1<<CL_LUT_COMPONENT_SCALE)-1;
    v0 &=(1<<CL_LUT_COMPONENT_SCALE)-1;
    index = (u0<=width)
        return;

    if (sig==sig2)
        goto save;

    goto next;

save:
    Qval qval(usum, vsum, ysum, (x/2<enqueue(&qval);
    x += 2;
    if (x>=width)
        return;
    goto next;
}

If You could tell me more about this part I would be really very grateful.
Thank You
Greetings
JB

#2

Hello JB,
The image format is 8-bit RGB raw Bayer pattern. The code above is used for cooked mode. It’s essentially a simulation of what is happening on the M0 core of Pixy. There is a lookup operation to determine if the pixel is “interesting” and if there are two pixels that match in a row, it is saved in a queue for the M4 to process later.

hope this helps!

Edward

#3

Thank You for Your reply. Yes, it helped me a lot, but if You could write somethning about line variable. How are the data placed there? I don’t understand how do You unpack the data to separate colors: r , g1, g2, b. And why do You have g1 and g2 since You have 8-bit RGB.
Greetings
JB

#4

Hello JB,
Unpacking the separate colors follows the Bayer pattern. In the Bayer pattern there are two times the number of green pixels than red and blue pixels (that’s why there is the g1 and g2 pixels.)

Edward

#5

Hello again,
Could You explain me these rows:

    u0 = u>>(9-CL_LUT_COMPONENT_SCALE);
    v0 = v>>(9-CL_LUT_COMPONENT_SCALE);
    u0 &= (1<<CL_LUT_COMPONENT_SCALE)-1;
    v0 &= (1<<CL_LUT_COMPONENT_SCALE)-1;
    index = (u0<<CL_LUT_COMPONENT_SCALE) | v0;

Why it can’t be just something like this:

index = (u0<<8) | v0;

u and v are two chrominance channels from YUV color space? For example from YUV422?

Greetings
JB

#6

Hello J B,
The u and v in the code aren’t the same as the U and V in YUV. u and v in the code are convenient chrominance channels, right. Your code: index = (u0<<8) | v0; is pretty close to what we’re doing, except we’re forming a 12-bit index instead of a 16-bit index.

Edward

#7

Thank You for response. I’m trying to implement Yours algorithm with another camera OV2640. I can’t obtain raw Bayer format. Can I use YUV422 image format as input to Yours algorithm? Maybe I should only change some parameters like minSat, etc. How do You think? If You use only 12 bit index so why Your LUT has 0x10000 elements? Maybe do You have some description of this algorithm. It is awesome but I can’t find any additional information about it. I only know that it is similar to CMVision but there is any information about how to generate LUT.

#8

Hello J B,
We need to write a white paper or something about the algorithm, but to be honest, it’s way down on our list! Using YUV422 is fairly different than raw Bayer but it could probably be adapted. I’m guessing it would take some background in computer vision though. I’m not too familiar with these things though. (sorry!)

Edward

#9

Thank You, so I am trying to implement Yours algorithm with 8-bit raw Bayer image. But still I don’t understand why You are comparing two pixels with the same signatures in a row. And why You increase x (in handleLine) with 2?

#10

Hello J B,
Some of the choices are in the interest of accuracy and efficiency. We look for adjacent pixels that satisfy the color model to reduce noise (increase accuracy).

Edward

#11

Thank You. Could You tell me if I can use colorlut code which is running on Pixy with color segmentation code which is placed in pixymon and works in cooked mode? Color segmentation code for Pixy is written in assembler and I don’t understand it. I’m trying to skip queue part of code. Below I show a part of code. Unfortunately after segmentation it can’t recognize appropriate object. It ideally omits the desired object and finds some invalid colors of background.

uint32_t i = 0, ii;
uint8_t xline[320];
#define CL_LUT_COMPONENT_SCALE	6
void handleLine(uint8_t *line, uint16_t width)
{
    uint32_t index;
    int32_t x, r, g1, g2, b, u, v, u0, v0;
    uint8_t sig1, sig2;


    x = 1; i = 0;
while(1) {
	sig1 = 0; sig2 = 0;

    r = line[x];
    g1 = line[x-1];
    g2 = line[x-width];
    b = line[x-width-1];
    u = r-g1;
    v = b-g2;

    u0 = u>>(9-CL_LUT_COMPONENT_SCALE);
    v0 = v>>(9-CL_LUT_COMPONENT_SCALE);
    u0 &= (1<<CL_LUT_COMPONENT_SCALE)-1;
    v0 &= (1<<CL_LUT_COMPONENT_SCALE)-1;
    index = (u0<>(9-CL_LUT_COMPONENT_SCALE);
//    v0 = v>>(9-CL_LUT_COMPONENT_SCALE);
//    u0 &= (1<<CL_LUT_COMPONENT_SCALE)-1;
//    v0 &= (1<<CL_LUT_COMPONENT_SCALE)-1;
//    index = (u0<=width)
        break;
}
}

void img_segm1(image_t *frame)
{
	uint32_t y;

	UB_LCD_SetCursor2Draw(0, 0);

	for (y = 1; y < (uint32_t)frame->h; y += 2) {
		handleLine(frame->pixels + (y * frame->w), frame->w);

		for (ii = 0; ii < 320; ii++) {
			if (xline[ii] == 1)
				LCD_RAM = RGB_COL_MAGENTA;
			else
				LCD_RAM = RGB_COL_YELLOW;
		}
		for (ii = 0; ii < 320; ii++) {
			if (xline[ii] == 1)
				LCD_RAM = RGB_COL_MAGENTA;
			else
				LCD_RAM = RGB_COL_YELLOW;
		}
	}
}

LUT generating code is Yours code. img_segm1 function is called in main loop.

#12

Hello J B,
I don’t see why not. It will be slower, but if you’re looking for something that’s easier to modify, then that sounds like a reasonable approach.

Edward