Nokia Bell Labs

2018-09-10

When I was at FRC Team 3637, one of the mentors was a researcher at Nokia Bell Labs. The same Bell Labs that the transistor, C, UNIX, and many other important things came out of, but now run by Nokia. It turns out that they do internships, and I was able to get a spot in their newly-starting robotics department.

One of the big things that Nokia was pushing at Bell Labs with their new research in 5G wireless was edge computing. The idea is that time-sensive computation is done in servers in the cloud, but physically close to where the data needs to go. This gives lower latency that traditional cloud computing, but still alows end devices to be rather lean. One application of this is in robotics: there are a lot of sensor-packed mobile devices, that need a lot of compute, yet stay small and power friendly. The bots need to get their sensor data processed very quickly in order to know where they are and avoid hitting things. However, sticking a rack of GPUs on a vacuum cleaner for vision processing doesn't always work that well.

This kind of thing really needs to work with a variety of robots. Each robot can't have it's own edge servers, so there needs to be some compatibility. This is what I was working on. The idea was that each bot has similar enough hardware, so the same software stack can be run on both the bot and the cloud. The researchers there had things going for one bot, using Python and ROS. It was my job to make it work with another, different bot. While they were both differential drive with the similar control electronics and sensors, the similarities end there.

The bot I was working on was an old FIRST Robotics bot from Team 3637. This was intended as a step-up from the small research bot they were using, and into something that was more useful. There were two main obstacles: using the FRC motor drivers, and dealing with a much bigger bot.

The motor drivers were standard FRC Talon SRX's. These are pretty darn robust drivers, that handle encoder feedback, PID control, and are controlled over CAN. However, they are designed to work only with the FRC robot controller, known as a RoboRIO. This was not an option for us. The CAN protocol is not exactly published, although there is a bit of source code floating around. Using a USB-CAN converter, I was able to intercept the packets between the RoboRIO and the Talons. I checked this with the bits of code I could find to verify that things mathced, at least mostly. There were a few fustrating differences resolved with a bit of trail-and-error, but I got the bits of the protocol that we needed figured out. The result was a large spreadsheet that included what every bit of every type of packet did. Finally, I was able to work this into useable Python code that got integrated into the main codebase. This included interfacing with the encoders, and using the Talon's PID controllers to regulate speed.

The next step was to get the whole bot up and running smoothly. The PID speed control algorithm on the Talon motor drivers needed to be tuned. This was my first main introduction into PID, or Proportional, Integral, Derivative control. It turns out that by summing up different amounts of the derivative and integral of the error signal with the error itself could get the speed pretty darn close to a set target. The key is to figure out the right mix. I learned the process of graphing the error from the speed it is going to the target speed, and how to incrementally adjust your P, I, and D constants to get pretty close. I was able to get this tuned pretty well.

However, the gearboxes on this bot had quite a bit of slack between the encoders and the motors. This meant that when the bot slowed down or stopped suddenly, there would be a large delay between the motors slowing down and the encoders registering it. This caused the PID to quickly spike the output to try to get it to respond, which would often cause unstable oscilating to ensue. Not good on a large bot. I tried adding a bit of huristics to try and take up the gear slack slowly before going straight into PID control, but that never really worked too great. Ultimately, I ended up adding F to the PID, as a Feed-forward. It turns out that motor will generally go a constant velocity when given a constant voltage and load. Knowing this, I could figure out a rough multiplier on the target speed to give a rough starting point, giving the PID less work and less drastic changes. Even better, the Talon motor drivers even supported this. After figuring out a reasonable F value, this went much better. There was no more chattering on sudden stops.

While working on all this and running the bot around the lab, I noticed that it would occasionally stop responding for a fraction of a second, then continue as normal. When you are working on a small, slow bot, this is not a problem. However, on the FRC bot, this would be running someone over or not. It was particularly noticable if it happened while the bot was stopping. While it wasn't responding, the bot would continue forward for a bit. Once it came back, it would realize that it went too far and go back. After measuring and graphing the main loop times over and over, I eventually found that the main loop including some debug reporting over the network. While this was fast most of the time, once and a while the network would be slow and it would hang for a fraction of a second. I ended up moving this networking to a seperate (python) thread, and the problem went away.

Overall, this provided a nice introduction to many things, including ROS, PID, CAN, and control loop timing.