<< return to Pixycam.com

cam_getFrame via USB question

Hi,

I am trying to add the ‘cam_getFrame’ request message to the hello_pixy sample application.

From searching the code base the API appears to be:
“cam_getFrame”,
(ProcPtr)cam_getFrameChirp,
{CRP_INT8, CRP_INT16, CRP_INT16, CRP_INT16, CRP_INT16, END},
“Get a frame from the camera”
@p mode one of the following CAM_GRAB_M0R0 (0x00), CAM_GRAB_M1R1 (0x11), CAM_GRAB_M1R2 (0x21)”
@p xOffset x offset counting from left”
@p yOffset y offset counting from top”
@p width width of frame”
@p height height of frame”
@r 0 if success, negative if error”
@r BA81 formatted data”

I have extended the libpixyusb example application ‘hello_pixy’ by adding the following code (also see attached file):

{
unsigned char pixels[307200];
uint8_t *frame = (uint8_t *)&pixels[0];

int32_t response;^M
int     return_value, res;

response = 0;
return_value = pixy_command("cam_getFrame",  // String id for remote procedure
                             0x01, 0x21,     // mode
                             0x02, 0,        // xoffset
                             0x02,0,         // yoffset
                             0x02,640,       // width
                             0x02,480,       // height
                             0,              // separator
                             &response,      // pointer to mem address for return value
                             &pixels,        // pointer to mem address for returned frame
                             0);

fprintf(stderr,"getFrame returned: %d - %d\n", return_value, response);

This code returns the following:

getFrame returned: -3 - -2

In other words, it fails. I have had success with other commands (cam_getMode, cam_setMode) that I have implemented in this manner.
I believe it has something to do with the returned memory buffer in this message …
Do I need to initialize a setting on the camera prior to getting frames?
Am I allocating memory for the image map correctly?
Am I passing the pointer to the image map properly?

In other words what am I doing wrong with ‘cam_getFrame’?

Thanks,

Chris

Hi,

I am bumping this question as I have had people contact me via email about using a python wrapper for pixy (see http://www.cmucam.org/boards/9/topics/4448) since I posted about having one that is somewhat functional. Besides, I would really like to get this working for my own use!

However, this getFrame issue is preventing me from finishing an initial iteration of the wrapper. I am hoping to get the attention of one of the CMU PixyCam5 coders who may be able to comment on what I am doing wrong?

Thanks again,

Chris

Chris,

From the pixy code, cam_getFrame() returns a (-3) when an unsupported capture size (width and height) is specified.

From the code in “cameravals.h” it looks like the following capture sizes are supported:

[ width, height ]

  1. [ 1280, 800 ]
  2. [ 640, 400 ]
  3. [ 320, 200 ]

In your case, in the call to pixy_command(), try changing the height parameter to 400 instead of 480.

Thanks,
John

Hi John,

This change seemed to make a difference but I am still getting an error from pixy_command; the response from the pixy is now set to 0 but the return value from the call to pixy_command is still a negative number … I am assuming this still means it failed?

I changed the width and height as follows:

return_value = pixy_command("cam_getFrame",  // String id for remote procedure
                             0x01, 0x21,     // mode
                             0x02, 0,        // xoffset
                             0x02,0,         // yoffset
                             0x02,320,       // width
                             0x02,200,       // height
                             0,              // separator
                             presponse,      // pointer to mem address for return value
                             pframe,         // pointer to mem address for returned frame
                             0);

The printf statement now returns:

getFrame returned: -3 - 0

This means that the return code sent from the pixy board (the pixy interpreter?) is now 0 … which I take it means the pixy successfully processed the request?
But the return code from pixy_command() is still an error code (-3) … meaning there was a communication failure over the USB connection?

I am going to try and dig more deeply into this … I am guessing that:

  • The command was successfully parsed and processed by the camera board,
  • The failure code is occurring when the board sends the frame back to the connected computer (RPi in this case)?
  • Given that all the other commands sent to the board are working properly and they involve much smaller responses (in terms of volume of data send) the problem is due to the size of the returned message?
  • A timeout?

Do you have any additional thoughts on this, or a suggestion on how I can get a better understand of what is going on?

Again, thanks for your help,

Chris

Chris,

The -3 is a chirp parse error: CRP_RES_ERROR_PARSE. This is returned when the chirp protocol does not have enough arguments to put the returned data into.

The remote procedure call cam_getFrame returns a BA81 fourcc type, from which renderflags, width, and height may be ignored.

BA81 data is a Bayer pattern. Refer to http://en.wikipedia.org/wiki/Bayer_filter for details.

Here’s what the remote procedure call should look like:

int32_t response;
uint32_t fourcc;
int8_t renderflags;
uint16_t width, height;
uint32_t numPixels;
uint8_t * pixels;

pixy_command(“cam_getFrame”, // String id for remote procedure
0x01, 0x21, // mode
0x02, 0, // X-offset
0x02, 0, // Y-offset
0x02, 320, // width
0x02, 200, // height
END, // separator
&response, // pointer to the memory address for return value
&fourcc,
&renderflags,
&width,
&height
&numpixels,
&pixels, // pointer to memory address for returned frame
END);

Thanks,
John

John,

I believe this completely explains my problem!

Thank you for taking the time to post such a detailed a response … hopefully I will get a chance to try it out this weekend.

Very excited now,

Chris

Hi,
I tried John’s code, as I was trying to do this as well. The good news is that I got a zero for the return_value as well as the correct values for width/height and numpixels and the pixel data is the right size. Unfortunately I got a -2 for the “response”. Not sure if this is expected or not. Also the image is garbage, even after demosaicing. Will continue to look further. Looking forward to hearing if this is working for you Chris.

Thanks,
Shari

Actually I was able to get a 0 for response. I had been using the wrong mode for the width/height combo I used. Still trying to decipher the data.

Shari

Hi Shari,

The code snippet at the end of this post worked for me in terms of getting a frame.
Both the return code from ‘pixy_command’ and the ‘response’ field from the pixy are set to 0 … indicating success.

In terms of doing something with the returned frame (only starting to look at this) …

As Indicated by John (see above) the returned frame is a ‘BA81 fourcc type’ image … implying that it has to be demosaiced. I have only just started looking at this … my starting point is to go through the code for the ‘pixymon’ app and see what functions are called. Have you made any progress on this yourself? (and if so what are you using to process the image?)

Chris

Start of code snippet:

//unsigned char pixels[907200];
//uint8_t *frame = (uint8_t *)&pixels[0];
uint8_t *frame;
int32_t response;
uint32_t fourcc;
int8_t renderflags;
uint16_t width, height;
uint32_t numPixels;

response = 0;
return_value=0;
//printf("Before call: %p\n", (void *)frame);
return_value = pixy_command("cam_getFrame",  // String id for remote procedure
                             0x01,0x21,      // mode
                             0x02,0,         // xoffset
                             0x02,0,         // yoffset
                             0x02,320,       // width
                             0x02,200,       // height
                             0,              // separator
                             &response,      // pointer to mem address for return value
                             &fourcc,
                             &renderflags,
                             &width,
                             &height,
                             &numPixels,
                             &frame,     // pointer to mem address for returned frame
                             0);^M
printf("Returned -  [%d] - %d - %d - %d - %d %p\n",
         return_value, response ,
         int(width), (int)height, (int)numPixels, (void *)frame);
fprintf(stderr,"getFrame returned: %d - %d\n", return_value, response);
pixy_error(return_value);
//exit(return_value);

Hi,
When I look at my raw data before even attempting to Demosaic, it is obvious it is bad. It is mostly 0’s with some data interspersed. For demosaicing I was testing with https://github.com/jdthomas/bayer2rgb, but it’s not really useful with bad data. What does your data look like?

Note that I’m grabbing a frame when in the default program, not sure if need to run a different program first.

Thanks,
Shari

Initially, I was seeing the same thing … mostly ones and zeros … highest it was going was 5 I think.

I then went through the default program (I am talking about the hello_pixy app) and commented out anything that performed a set. I believe this means I am using whatever ‘default’ settings are initialized when the camera powers up.

after this the data looked ‘good’. Good is very subjective at this point … if I block the lens I get all low numbers (1-4 as before) … if I uncover the lens and let it take a ‘normal’ image I get a range of numbers from low up to 255.

I am now trying to demosaic the returned image using the code from the pixymon app.

One problem … my first call to cam_getFrame returns a valid result … all subsequent calls fail until I re-start the app. Not sure if this means an acknowledgement message needs to be sent prior to grabbing the next frame?

Chris

When I cover my lens I get the same results I do when it is off, so I don’t think I’m actually getting captured data. I also have the issue that after my first capture subsequent captures don’t work. When this happens the pixy LED flashes a new color green/blue/yellow/red - I am only using signature 1, so I typically only see a red LED. I’m guessing I’m in a strange mode. BTW, when I was referring to default program I was talking about the default program in Firmware (program 0 - This is the nomenclature used in pixymon.)

Shari

Ok … got this to work …

Running on a Raspberry Pi using a python shell I can:

  • Connect to the pixy camera,
  • Get pixy configuration settings,
  • Set pixy Configuration settings,
  • Download an imageFrame (currently only 320x200),
  • Display the downloaded image using simplecv.

I’ve checked the working code into github but it is a bit of a mess … it needs to be cleaned up.

Some problems I am having:

  • The initial call to cam_getFrame works as expected but subsequent calls fail with a return code of -1 or -3.

John, is there an acknowledgement message that needs to be sent back to the camera board after receiving a frame or can you think of something else that is going on to cause this?

Thanks,

Chris

Hi,
That’s great! Can you send me a pointer to your code on github. Hopefully that will help me figure out my issue.

Thanks,
Shari

Found your code. Will let you know when I get it working. Need a bigger SD card first…

Hi Shari,

At work today … will try to do a bit of code cleanup this evening.

Just some thoughts that occur to me:

  • You will need to install simplecv for python (this was a bit of a struggle) … I actually re-started the process with a clean, current version of Raspbian and the process went very smoothly.
  • The simple CV package installs the PIL (python image processing library) … I would upgrade this to the PILLOW distribution … it is current and works better
  • I should have kept a log of what packages I installed on the Pi to get this all working … I am uploading my RPi image for another project and will link to it here … that might help.
  • I modified the CMU cmake files for libpixyusb to compile a shared image of libpixyusb … this is required to make a python wrapper … you will need to set LD_LIBRARY_PATH to point to /usr/local/lib to make this work when running python

Some questions for the CMU developers (John, I am hoping you are still following this thread):

  • I am now able to process an image from the pixy using ‘cam_getFrame’ but any subsequent call after the first one fails with either a -1 or -3 return code. I am thinking an acknowledgement needs to be sent or a cleanup function needs to be called? I am not really sure and will try digging into the USB communications … do you have any insight into this?
  • Are there any diagnostics in the library that can be turned on so that I can get a feel for the communication between the pixy board and a client … either in pixymon or libpixyusb?
  • Are there any written spec’s on the communication protocol or on the different commands that make up the communication protocol?

Chris

Chris,

Could you send me the link to your github pixy project?

Currently libpixyusb does not have logging built-in, although this shouldn’t be difficult to add in the future. I don’t know if there is a written specification for Chirp, but I will check.

Thanks,
John

John,

The code is at https://github.com/DarfieldChris/pixy.git (in the pixython branch).

I cleaned the code up a bit … I am still trying to figure out why multiple calls fail.

Chris

I have not made much progress here but this is what I have found:

The second call to cam_getFrame returns with an error code of -1 …
subsequent calls to cam_getFrame after the failed second call return -3.
I am still guessing this has something to do with sending some sort of acknowledgement (that is not being sent) but have not managed to parse much out of the libpixyusb api so far.

I then thought of trying to close and then re-open the pixy connection.
In other words … I tried the following in the code:

pixy_command("cam_getFrame", ...

pixy_close();
...
pixy_init();
...
pixy_command("cam_getFrame", ...

Unfortunately, the second call to pixy_init() fails … I have not had a chance to look into this in too much detail but I am guessing the pixy_close() call does not effectively terminate/cleanup the pixy connection such that a subsequent call to pixy_init() would work properly?

Not sure if anybody (John?) can shed any light on this?

Thanks for all of the feedback to-date,

Chris

Chris,

By default, Pixy runs a program to grab and process frames (for block detection). If we try to manually grab frames (using pixy_command(“cam_getFrame”,…)) while this program is running there could be contention for hardware resources.

As a workaround, we could stop the program running on pixy using the “stop” chirp Remote Procedure Call (RPC). To start the default program on Pixy use the “start” chirp RPC. (see below) In order to detect blocks and grab frames simultaneously, we could try wrapping calls to the “cam_getFrame” chirp RPC with calls to “stop” and “start” respectively.

{
int return_value;
int chirp_response;
// Stop running a program on Pixy //
return_value = pixy_command(“stop”, END_OUT_ARGS, &chirp_response, END_IN_ARGS);
// Start last running program on Pixy //
return_value = pixy_command(“start”, END_OUT_ARGS, &chirp_response, END_IN_ARGS);
}

pixy_init() initializes libUSB and creates a thread which makes calls to Chirp::service() (effectively polling for events) and caches received block data from Pixy.

Thanks,
John