My initial tests of the Pixy2's object recognition were disconcerting. The camera's color-based object recognition seemed to be very light dependent, and prone to false positives.

I had hoped that with more even lighting, and the consistent background of the arena, the recognition would be more consistent.

The Pixy2's sensitivity can be set, and I tried adjusting that from its default value.

This seemed to produce more accurate results without the false positives I had seen previously. The object was still recognized at the maximum 1.5 meter distance of the arena, but the recognition was lost beyond that distance.

In Tidy Up the Toys, the colored blocks start at a distance of 1.1 meters from the robot. At this distance, the object was reliably recognized.

In lower light conditions, the object was still recognized, although sporadic false positives appeared.

There were no false positives with similar objects of different colors.

The Pixy2 can store 7 different objects. After training the camera on each of the three different colored objects, it was able to recognize all three.

I found that it was necessary to train the Pixy2 in a well lit environment. This seemed to give the camera a better understanding of the objects' shapes, allowing the Pixy2 to still recognize the objects when the lighting was worse.

I also found that different colors required different sensitivities to avoid false positives.

-Colin

The Pixy2 uses color to recognize objects, and returns the coordinates of a bounding box around those objects, similar to the TensorFlow algorithms. In the Pi Wars challenges, the toy blocks and fish tank should be distinctly colored objects against a non-colored arena wall, so the Pixy2 seems like a promising option.

The Pixy2 is trained by holding a colored object in front of the camera.

The camera uses a region growing algorithm to find the connected pixels in the image that make up the object in front of the camera. It seems to use the dominant color of the object to decide which pixels are part of the object. The more pixels that the Pixy2 is able to detect, the more accurate its understanding of the object's shape will be.

Once the Pixy2 has been "trained" to detect the object, it can then identify the same object in front of it.

This is where the big limitation of the Pixy2 appears. Color is inherently tied to light. The Pixy2 seems to have poor light sensitivity in general, which makes it difficult to use in rooms that aren't well lit.

Even when the object is still clearly visible in the camera's image, it's not always detected. Why? Because the difference in lighting has changed the saturation of the object's color, such that the object no longer matches the color that the Pixy2 was trained to recognize.

Adding more light can cause a similar problem.

Will this make the Pixy2 unusable? Possibly. We plan to run the challenges in a well lit space, with relatively consistent lighting. Hopefully the combination of consistent lighting and consistent arena walls will mitigate these problems.

​-Colin

As a potentially easier alternative to TensorFlow, we decided to try out the Pixy2 camera. This camera has built-in object recognition, similar to TensorFlow, using color. The PiWars "toys" and "fish tank" can be colored objects, set inside a white arena, so identifying them should be doable.

I attached the Pixy2 PCB to the front of the Dexter robot, and plugged it into the Pi via USB. The LED on the Pixy2 turned on, and I followed the provided instructions to install the Pixy2 driver libraries and dependencies.

I tried running the provided test program, but pixy.init() failed, suggesting that the camera wasn't connected properly.

The provided PixyMon application can be used to auto-detect the USB Pixy2. I tried connecting the Pixy2 to PixyMon on my computer, and it still failed. I then tried swapping out the USB wire, and the camera connected!

-Colin

​How can our robot understand what it's looking at? How can it search for, say, a colored block, or the "fish tank"? One possibility is to use a camera with some sort of object recognition. I currently have a Pi Camera Module connected to my test robot, and I was wondering if there's a simple way to use that to identify objects. I searched around a bit on YouTube, and found some examples of object detection, using neural networks to classify objects in the camera's image frame.

Most of the examples I found used the open source library TensorFlow Light to run an object classifier. TensorFlow Light is a less resource-intensive version of the TensorFlow library, and is more suitable for devices like the Pi. This method of object detection supposedly works on any camera with adequate resolution.

But, how do these neural network classifiers work? As I understand it, TensorFlow uses a particular classification algorithm, called a Deep Neural Network object detection algorithm. This is a module that can be trained to search for a particular type of object, and then draw boxes around each instance of that object it sees in the image.

The algorithm is trained by giving it a large collection of training pictures, and another collection of test pictures. For each training picture, you have to manually draw a box around the object in that picture, and feed that information into the algorithm, along with the image. The neural network will then incrementally adjust itself, over and over, to find patterns between the training images. It will then look at the test images, and try to find that same object in those. You also draw identifying boxes in the test images, so the algorithm can see how accurate it was. You keep running the algorithm over the training and test images until its test-image accuracy reaches an adequate level. At that point, the values of the neural network's connections can be saved as a classifier file, and loaded onto the Raspberry Pi for use in the robot.

In our case, if the arena floor and walls are a uniform color that's distinct from the objects, then it should be relatively easy for the classifier to identify the objects. We could place the blocks in various places in the arena, take a bunch of training pictures of them, and then train the classifier.

It looks like there are multiple DNN classifer algorithms that TensorFlow can use. The SSD Mobilenet algorithm seems to run faster than the others, and is designed for low-powered devices, so that's probably a good place to start.

Once the classifier file is loaded onto the Pi, actually using it seems fairly simple. You can load it using the TensorFlow Python library, then take an image from the camera, and pass that to the classifier. The classifier will then return an array of box coordinates, drawn around each instance of the object it found in that frame, along with a confidence percentage.

If we want the robot to turn toward an object, we could turn the robot in a circle a few degrees at a time, and take a camera picture after each turn, then run the classifier on that picture. If the object is found in the circle, we could keep turning toward it until the robot is facing directly toward the object. Then, we could determine the distance to the object by the relative size of its bounding box.

I'm not sure if we could determine the orientation to the object easily using these tools. For example, I don't know if these classification algorithms could tell us whether the robot is facing the corner of the green box, or its side.

​-Colin