<< return to Pixycam.com

Snapshot with libpixyusb?

Hi Jim,
You should call stop() before grabbing frames by using the command below because Pixy is running it’s blob detection program by default and you’ll step on it’s internal frame grabs:

return_value = pixy_command(“stop”, END_OUT_ARGS, &chirp_response, END_IN_ARGS);

When you’re done you can start the program again:

return_value = pixy_command(“start”, END_OUT_ARGS, &chirp_response, END_IN_ARGS);

But this isn’t totally necessary because Pixy will resume the program when it loses connection with your program.

thanks!

Thanks for the hint! I added a stop, which appeared to succeed. However, I still get all zeros in the frame. Lens cap is off and PixyMon works. snippet:

    return_value = pixy_command("stop", END_OUT_ARGS, &response, END_IN_ARGS);   
    printf(" STOP returned %d response %d\n", return_value, response);
    response = 0;
    return_value = pixy_command("cam_getFrame",  // String id for remote proced$

program output:

pi@raspberrypi:~/pixy/build/hello_pixy$ sudo ./hello_pixy
initialized Pixy - 0
 STOP returned 0 response 0
getFrame return value 0 response 0
returned w 320 h 200 npix 64000
 average pixel value 0
 00 00 00 00 00 00 00 00
 00 00 00 00 00 00 00 00
 00 00 00 00 00 00 00 00
 00 00 00 00 00 00 00 00
 00 00 00 00 00 00 00 00
 00 00 00 00 00 00 00 00
 00 00 00 00 00 00 00 00
 00 00 00 00 00 00 00 00

Ah, sorry, you need to change your line

unsigned char pixels[72000]={0xff};

to

unsigned char *pixels;

and then pass the pointer to this pointer into pixy_command (like you have now). Libpixyusb will just give you the pointer to the video frame memory to avoid the copy.

Great, that works fine! I assumed that the space I had allocated would be used.
Where and how does libpixyusb allocate frame memory? I looked through the source code, and can find nothing that is obviously related to cam_getFrame.

libpixyusb uses a middleware called “chirp” to communicate over USB. Chirp can look at the message size coming from Pixy and allocate its buffer memory accordingly.

You’ll lose the frame data once another chirp message comes in, so it’s up to you to copy the data — but if you stop pixy’s program and you don’t issue any more commands, no messages will come in and your data will remain intact.

Rich,
Thanks very much for your help! Everything seems to be working well. For future reference, here is a working bit of code to grab a 1280x40 block. No doubt it could be improved. Next stop: spectral data acquisition.


//example of using libpixyusb to grab a 1280x40 block (maximum camera resolution)

#include 
#include 
#include 
#include 
#include 
#include "pixy.h"

void handle_SIGINT(int unused)
{
  // On CTRL+C - abort //
  printf("\nBye!\n");
  exit(0);
}

int main(int argc, char * argv[])
{
  int      pixy_init_status;

  // Catch CTRL+C (SIGINT) signals //
  signal(SIGINT, handle_SIGINT);

  // Connect to Pixy //
  pixy_init_status = pixy_init();
  printf("initialized Pixy - %d\n", pixy_init_status);
  if(pixy_init_status != 0)
  {
    // Error initializing Pixy
    pixy_error(pixy_init_status);
    return pixy_init_status;
  }

  // getFrame Example //
  { //local
    unsigned char current_frame[72000]; // ~largest possible given current hardware
    unsigned char *pixels;  //returned pointer to video frame buffer
    int32_t response, fourcc;
    int8_t renderflags;
    int return_value, res;
    uint16_t width, height;
    uint32_t  numPixels;
	
//  stop blob processing	
    return_value = pixy_command("stop", END_OUT_ARGS, &response, END_IN_ARGS);
    printf("STOP returned %d response %d\n", return_value, response);

    response = 0;
    return_value = pixy_command("cam_getFrame",  // String id for remote procedure
                                 0x01, 0x00,      // mode 0 = 1280x800 25 fps
                                 0x02,   0,        // xoffset
                                 0x02,   0,         // yoffset
                                 0x02, 1280,       // width
                                 0x02, 40,       // height (56 max @ 1280 w)
                                 0,              // separator
                                 &response,      // pointer to mem address for return value
                                &fourcc,         //contrary to docs, the next 5 args are needed
                                &renderflags,
                                &width,
                                &height,
                                &numPixels,
                                 &pixels,        // pointer to mem address for returned frame
                                 0);

    printf("getFrame returned %d response %d\n", return_value, response);
    printf("returned w %d h %d npix %d\n",width,height,numPixels);

// quit now if not successful:
    if(return_value != 0) return return_value;

// save this block
   memcpy(&current_frame, pixels,numPixels);

   // display average and 8x8 pixel dump

   unsigned int i,j,ind,start;
   unsigned long avg=0;

   for(i=0; i<numPixels; i++) avg += current_frame[i];
   avg = avg/numPixels;
   printf("average pixel value %d\n",avg);

// dump a few raw pixels

   start=(height/2)*width+width/2; //roughly in middle of frame
   for (i=0; i<8; i++) {
        for (j=0; j<8; j++) {
			ind = i*width + j + start;
			printf(" %02x",current_frame[ind]);
        }
		printf("\n");
    }
    // Sleep for 1/10 sec //
   while(1) usleep(100000); //(exit on ^C)
  } //end local
}

Nice work — thanks Jim!

has anyone worked on it further I wanna save it as an image file…

Hello Hermanand,
You can save as an image through PixyMon:

http://cmucam.org/projects/cmucam5/wiki/How_to_Grab_a_Frame

Are you using a Raspberry Pi?

Edward

You can grab and then write out a raw frame and, using imagemagick, convert that into a “pseudo-gray” .PNG file.
It won’t be proper RGB format. To do that, the frame needs to be Bayer-interpolated in the proper way. I started working on that and got distracted by other projects, so if anyone wants to take up the challenge, others (including me) would appreciate it! Look into the PixyMon source code for the Bayer interpolation procedure.

The imagemagick command to convert a raw 320x200 frame to .PNG format is:

$ convert -size 320x200 -depth 8 gray:dump.img dump.png

Here is the RPi code that writes out a 320x200 frame as dump.img:

#include 
#include 
#include 
#include 
#include 
#include "pixy.h"


void handle_SIGINT(int unused)
{
  // On CTRL+C - abort //

  printf("\nBye!\n");
  exit(0);
}

int main(int argc, char * argv[])
{

    unsigned char current_frame[72000]; //largest possible given hardware
    unsigned char *pixels;  //returned pointer to internal frame buffer
    int32_t response, fourcc;
    int8_t renderflags;

    uint8_t gain;
    uint16_t compensation;

    uint8_t* pgain;
    uint16_t* pcomp;

    int return_value, res;
    uint16_t width, height;
    uint32_t  numPixels;

    unsigned int pass,i,j,ind,start;
    FILE *fp;

  // Catch CTRL+C (SIGINT) signals
  signal(SIGINT, handle_SIGINT);

  printf("hello Pixy\n");

  // Connect to Pixy //
  return_value = pixy_init();
  printf("initialized Pixy - %d\n", return_value);
  if(return_value != 0)
  {
    // Error initializing Pixy //
    printf("pixy_init(): ");
    pixy_error(return_value);
    return return_value;
  }
    return_value = pixy_command("stop", END_OUT_ARGS, &response, END_IN_ARGS);
    printf(" STOP returned %d response %d\n", return_value, response);

    return_value = pixy_cam_get_exposure_compensation(&gain, &compensation);
    printf("getECV returned %d values: 0, 0x%02x, 0x%04x\n",return_value,gain, compensation);

    for (pass=1; pass<3; pass++){
        printf("Pass %d\n",pass);

    response = 0;
    return_value = pixy_command("cam_getFrame",  // String id for remote procedure
				 0x01, 0x21,      // mode 0 = 1280x800
                                 0x02,   0,        // xoffset
                                 0x02,   0,         // yoffset
                                 0x02,  320,       // width
                                 0x02,  200,       // height (56 max @ 1280 w)
                                 0,              // separator
                                 &response,      // pointer to mem address for return value
			        &fourcc,
    				&renderflags,
     				&width,
     				&height,
     				&numPixels,
                                 &pixels,        // pointer to mem address for returned frame
                                 0);

    printf("getFrame returned %d response %d\n", return_value, response);
    printf("returned w %d h %d npix %d\n",width,height,numPixels);

// quit now if not successful:
    if(return_value != 0) return return_value;

    return_value = pixy_cam_get_exposure_compensation(&gain, &compensation);
    printf("getECV returned %d values: 0, 0x%02x, 0x%04x\n",return_value,gain, compensation);

// save this block
// display average

   memcpy(&current_frame, pixels,numPixels);

   unsigned long avg=0;
   for(i=0; i<numPixels; i++) avg += current_frame[i];
   avg = avg/numPixels;
   printf("average pixel value %d\n",avg);

}

// dump the block as a file

    fp=fopen("dump.img","w");
        if(fp != NULL) {
        fwrite(current_frame, 1, numPixels, fp);
        fclose(fp);
        printf("wrote dump.img\n Done!\n");
        }
        else perror("file allocation failure");
   exit(0);
 }

I have done the Bayer Interpolation based on the code used in PixyMon and this the result I got. The colors are not identifiable.

It looks like the interpolation was not done correctly. Post the code.

#include 

#include 
#include 
#include 
#include 
#include 
#include "pixy.h" 

void interpolateBayer(unsigned int width, unsigned int x, unsigned int y, unsigned char *pixel, unsigned int &r, unsigned int &g, unsigned int &b);

void handle_SIGINT(int unused)
{
    printf("\nBye!\n");
  exit(0);
}

int main(int argc, char * argv[])
{
  int      index;
  int      blocks_copied;
  int      pixy_init_status;

  signal(SIGINT, handle_SIGINT);

  pixy_init_status = pixy_init();
  printf("initialized Pixy - %d\n", pixy_init_status);

  if(!pixy_init_status == 0)
  {
    printf("pixy_init(): ");
    pixy_error(pixy_init_status);
    return pixy_init_status;
  }

  {
    unsigned char *pixels;
    uint32_t frame[200][320];
    uint32_t r[200][320];
    uint32_t g[200][320];
    uint32_t b[200][320];
    char rgb[200][320];
    int32_t response, fourcc;
    int8_t renderflags;
    int return_value, res;
    uint16_t width, height;
    uint32_t  numPixels;
    unsigned char current_frame[72000];

     uint8_t gain;
    uint16_t compensation;

    uint8_t* pgain;
    uint16_t* pcomp;
    
    return_value = pixy_command("stop", END_OUT_ARGS, &response, END_IN_ARGS);   
    printf(" STOP returned %d response %d\n", return_value, response);

    return_value = pixy_cam_get_exposure_compensation(&gain, &compensation);
    printf("getECV returned %d values: 0, 0x%02x, 0x%04x\n",return_value,gain, compensation);

    pixy_command("cam_setAEC", UINT8(0x01), END_OUT_ARGS,  &response, END_IN_ARGS);
    printf("response %d",response);
    pixy_command("cam_setAWB", UINT8(0x01), END_OUT_ARGS,  &response, END_IN_ARGS);
    printf("response %d",response);
    
    response = 0;
    return_value = pixy_command("cam_getFrame",  // String id for remote procedure
                                 0x01, 0x21,      // mode
                                 0x02,   0,        // xoffset
                                 0x02,   0,         // yoffset
                                 0x02, 320,       // width
                                 0x02, 200,       // height
                                 0,              // separator
                                 &response,      // pointer to mem address for return value
                                &fourcc,  //for some reason these 5 args are needed, contrary to the docs
                                &renderflags,
                                &width,
                                &height,
                                &numPixels,
                                 &pixels,        // pointer to mem address for returned frame
                                 0);

    fprintf(stderr,"getFrame return value %d response %d\n", return_value, response);
    printf("returned w %d h %d npix %d \n",width,height,numPixels);

    if(return_value != 0) return return_value;

    return_value = pixy_cam_get_exposure_compensation(&gain, &compensation);
    printf("getECV returned %d values: 0, 0x%02x, 0x%04x\n",return_value,gain, compensation);
    
   unsigned int i,j,ind,start;

   unsigned long avg=0;
   for(i=0; i> 2;
      b = (*(pixel - width - 1) + *(pixel - width + 1) + *(pixel + width - 1) + *(pixel + width + 1)) >> 2;
    }
    else
    {
      r = (*(pixel - 1) + *(pixel + 1)) >> 1;
      g = *pixel;
      b = (*(pixel - width) + *(pixel + width)) >> 1;
    }
  }
  else
  {
    if (x & 1)
    {
      r = (*(pixel - width) + *(pixel + width)) >> 1;
      g = *pixel;
      b = (*(pixel - 1) + *(pixel + 1)) >> 1;
    }
    else
    {
      r = (*(pixel - width - 1) + *(pixel - width + 1) + *(pixel + width - 1) + *(pixel + width + 1)) >> 2;
      g = (*(pixel - 1) + *(pixel + 1) + *(pixel + width) + *(pixel - width)) >> 2;
      b = *pixel;
    }
  }
}



I took those RGB arrays and used

            Bitmap bmp = new Bitmap(width, height);

            for (int y = 0; y < height; y++)
            {
                for (int x = 0; x < width; x++)
                {

                    System.Drawing.Color color = System.Drawing.Color.FromArgb(rx[y, x],gx[y, x],bx[y, x]);
                    
                    bmp.SetPixel(x, y, color);
                    

                }
            }

I believe you are on the right track. I got as far as you did with the interpolation, but did not try to write an RGB file. It appeared there was an indexing error, (as it does in the .bmp file you posted). If you are off by 1 in either direction, the result is hopeless.

I suggest to take a snapshot of a white card through a red, green or blue filter and check the indexing by printing out some raw pixel values, before and after interpolation.

On posting code, the “pre” (preformatted) button to encapsulate the code seems to work best. The “Inline Code” button does not work. Note that the posted code in the previous post has some characters reinterpreted as html. Can you edit it to change that?

Sorry about the code my first time… Alright I ll work on the part you told me .

I am sorry for such a delay I resolved the issue and everything works perfectly I have added with this the working code.

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <signal.h>
#include <string.h>
#include <math.h>  
#include "pixy.h" 
#include <opencv2/core/core.hpp>
#include <opencv2/highgui/highgui.hpp>
#include <opencv2/opencv.hpp>
#include <opencv/cv.h>
#include <opencv/highgui.h>
#include <ctime>

using namespace cv;
using namespace std;

void interpolateBayer(unsigned int width, unsigned int x, unsigned int y, unsigned char *pixel, unsigned int &r, unsigned int &g, unsigned int &b);
Mat renderBA81(uint8_t renderFlags, uint16_t width, uint16_t height, uint32_t frameLen, uint8_t *frame);
Mat getImage();

int main(int argc, char * argv[])
{
    int pixy_init_status;
    int return_value;
    int32_t response;

    pixy_init_status = pixy_init();
  
    if(!pixy_init_status == 0)
    {
        printf("pixy_init(): ");
        pixy_error(pixy_init_status);
        return pixy_init_status;
    }

    return_value = pixy_command("stop", END_OUT_ARGS, &response, END_IN_ARGS); 
    return_value = pixy_rcs_set_position(1, 900);
    return_value = pixy_rcs_set_position(0, 500);

    Mat image = getImage();
}

Mat getImage()
{
    unsigned char *pixels;
    int32_t response, fourcc;
    int8_t renderflags;
    int return_value, res;
    uint16_t rwidth, rheight;
    uint32_t  numPixels;
    uint16_t height,width;
    uint16_t mode;
    
    return_value = pixy_command("run", END_OUT_ARGS, &response, END_IN_ARGS);   
    return_value = pixy_command("stop", END_OUT_ARGS, &response, END_IN_ARGS);

    return_value = pixy_command("cam_getFrame",  // String id for remote procedure
                                 0x01, 0x21,      // mode
                                 0x02,   0,        // xoffset
                                 0x02,   0,         // yoffset
                                 0x02, 320,       // width
                                 0x02, 200,       // height
                                 0,            // separator
                                 &response, &fourcc, &renderflags, &rwidth, &rheight, &numPixels, &pixels, 0);

    return renderBA81(renderflags,rwidth,rheight,numPixels,pixels);
}

inline void interpolateBayer(uint16_t width, uint16_t x, uint16_t y, uint8_t *pixel, uint8_t* r, uint8_t* g, uint8_t* b)
{
    if (y&1)
    {
        if (x&1)
        {
            *r = *pixel;
            *g = (*(pixel-1)+*(pixel+1)+*(pixel+width)+*(pixel-width))>>2;
            *b = (*(pixel-width-1)+*(pixel-width+1)+*(pixel+width-1)+*(pixel+width+1))>>2;
        }
        else
        {
            *r = (*(pixel-1)+*(pixel+1))>>1;
            *g = *pixel;
            *b = (*(pixel-width)+*(pixel+width))>>1;
        }
    }
    else
    {
        if (x&1)
        {
            *r = (*(pixel-width)+*(pixel+width))>>1;
            *g = *pixel;
            *b = (*(pixel-1)+*(pixel+1))>>1;
        }
        else
        {
            *r = (*(pixel-width-1)+*(pixel-width+1)+*(pixel+width-1)+*(pixel+width+1))>>2;
            *g = (*(pixel-1)+*(pixel+1)+*(pixel+width)+*(pixel-width))>>2;
            *b = *pixel;
        }
    }

}


Mat renderBA81(uint8_t renderFlags, uint16_t width, uint16_t height, uint32_t frameLen, uint8_t *frame)
{
    uint16_t x, y;
    uint8_t r, g, b;
    Mat imageRGB;

    frame += width;
    uchar data[3*((height-2)*(width-2))];

    uint m = 0;
    for (y=1; y<height-1; y++)
    {
        frame++;
        for (x=1; x<width-1; x++, frame++)
        {
            interpolateBayer(width, x, y, frame, &r, &g, &b);
            data[m++] = b;
            data[m++] = g;
            data[m++] = r;
        }
        frame++;
    }

    imageRGB =  Mat(height - 2,width -2, CV_8UC3, data);
   
    return imageRGB;
}

Hi all!
I know that this post is an old discussion, but I’d like to know where I have to write this code to try it.
I read that a program is needed (QT Project, I think) to compile libpixyusb and other. It is the program to use?

Hello Elia,
This page explains how to get started with libpixyusb (it’s what’s being used in this topic discussion.)

http://cmucam.org/projects/cmucam5/wiki/Building_the_libpixyusb_example_on_Linux

Hope this helps!

Edward

Hi guys!

First of all I want to thank all the people posting in this thread, you really helped me.
I am working on a small application in order to get pictures from the Pixy. The goal is to use these picture to get the calibration matrix of the camera using the openCV library.
To do so I took Hemanand Ramasamy’s code (thanks a lot for this work). Took me a while to understand how to compile it (“g++ -I/usr/include/libusb-1.0/ -I/usr/local/include -L/usr/local/lib/ pkg-config --cflags --libs opencv snapshot.cpp -o snapshot -Wall -lpixyusb -lusb-1.0 -lboost_system -lboost_thread -lboost_date_time -lboost_chrono” if can help someone).
However I get some weird things inside my image. Anyone having an idea of why this is happening? Am I missing something big? Can it be related to the camera mode? Or its software revision?
As you will see, the image is not complete as if the program was not able to recover the whole picture.
I also add the code allowing me to save the image.

Thanks for your consideration and help.

Maxime

#include 
#include 
#include 
#include 
#include 
#include   
#include 
#include 
#include 
#include 
#include 
#include 
#include 

using namespace cv;
using namespace std;

void interpolateBayer(unsigned int width, unsigned int x, unsigned int y, unsigned char *pixel, unsigned int &r, unsigned int &g, unsigned int &b);
Mat renderBA81(uint8_t renderFlags, uint16_t width, uint16_t height, uint32_t frameLen, uint8_t *frame);
Mat getImage();

int main(int argc, char * argv[])
{
    int pixy_init_status;
    int return_value;
    int32_t response;

    pixy_init_status = pixy_init();

    if(!pixy_init_status == 0)
    {
        printf("pixy_init(): ");
        pixy_error(pixy_init_status);
        return pixy_init_status;
    }
	
    return_value = pixy_command("stop", END_OUT_ARGS, &response, END_IN_ARGS); 
    //return_value = pixy_rcs_set_position(1, 900);
    //return_value = pixy_rcs_set_position(0, 500);

    Mat image = getImage();
    
    // Display and save the image
    namedWindow( "Snapshot", WINDOW_AUTOSIZE );
    imshow("Snapshot", image);
    waitKey(0);
    imwrite("/home/pi/pixy/calib/snapshot.jpg", image);
}

Mat getImage()
{
    unsigned char *pixels;
    int32_t response, fourcc;
    int8_t renderflags;
    //int return_value, res;
    int return_value;
    uint16_t rwidth, rheight;
    uint32_t  numPixels;
    //uint16_t height,width;
    //uint16_t mode;

    return_value = pixy_command("run", END_OUT_ARGS, &response, END_IN_ARGS);
    return_value = pixy_command("stop", END_OUT_ARGS, &response, END_IN_ARGS);

    return_value = pixy_command("cam_getFrame",  // String id for remote procedure
                                 0x01, 0x21,      // mode
                                 0x02,   0,        // xoffset
                                 0x02,   0,         // yoffset
                                 0x02, 320,       // width
                                 0x02, 200,       // height
                                 0,            // separator
                                 &response, &fourcc, &renderflags, &rwidth, &rheight, &numPixels, &pixels, 0);

    return renderBA81(renderflags,rwidth,rheight,numPixels,pixels);
}

inline void interpolateBayer(uint16_t width, uint16_t x, uint16_t y, uint8_t *pixel, uint8_t* r, uint8_t* g, uint8_t* b)
{
    if (y&1)
    {
        if (x&1)
        {
            *r = *pixel;
            *g = (*(pixel-1)+*(pixel+1)+*(pixel+width)+*(pixel-width))>>2;
            *b = (*(pixel-width-1)+*(pixel-width+1)+*(pixel+width-1)+*(pixel+width+1))>>2;
        }
        else
        {
            *r = (*(pixel-1)+*(pixel+1))>>1;
            *g = *pixel;
            *b = (*(pixel-width)+*(pixel+width))>>1;
        }
    }
    else
    {
        if (x&1)
        {
            *r = (*(pixel-width)+*(pixel+width))>>1;
            *g = *pixel;
            *b = (*(pixel-1)+*(pixel+1))>>1;
        }
        else
        {
            *r = (*(pixel-width-1)+*(pixel-width+1)+*(pixel+width-1)+*(pixel+width+1))>>2;
            *g = (*(pixel-1)+*(pixel+1)+*(pixel+width)+*(pixel-width))>>2;
            *b = *pixel;
        }
    }

}

Mat renderBA81(uint8_t renderFlags, uint16_t width, uint16_t height, uint32_t frameLen, uint8_t *frame)
{
    uint16_t x, y;
    uint8_t r, g, b;
    Mat imageRGB;

    frame += width;
    uchar data[3*((height-2)*(width-2))];

    uint m = 0;
    for (y=1; y<height-1; y++)
    {
        frame++;
        for (x=1; x<width-1; x++, frame++)
        {
            interpolateBayer(width, x, y, frame, &r, &g, &b);
            data[m++] = b;
            data[m++] = g;
            data[m++] = r;
        }
        frame++;
    }

    imageRGB =  Mat(height - 2,width -2, CV_8UC3, data);

    return imageRGB;
}

Hello Maxime,
Hmm, that’s odd. Does PixyMon seem to work fine on this machine with the USB cable you are using? Also, I’m not sure if it’s the acquisition of the image that’s the problem or the JPEG encoding. (although I suspect the acquisition.)

It might be that Pixy is stepping on your frame. Pixy runs the color connected components algorithm by default, and I assume it’s running it while you are trying to grab a frame. You should stop the program on Pixy by calling the stop() command on Pixy:

return_value = pixy_command(“stop”, END_OUT_ARGS, &chirp_response, END_IN_ARGS);

When you’re done grabbing frames you can start the program again by running:

return_value = pixy_command(“start”, END_OUT_ARGS, &chirp_response, END_IN_ARGS);

Hope this helps!

Edward