Inhand Manipulation with Dexterous Hand
As part of my assistive robotics course final project, I worked with my peer to develop a force control strategy for a 4-finger dexterous hand. We used a combination of impedance control, QP, and nullspace control to hold on to the object and maintain its goal pose.
We wanted to understand how force control works for manipulators, since grasping is a very important part in humanoid robotics. It also is especially useful for assistive robots that can help people without fingers in everyday tasks. To begin with, we model our system as below. The state x, contact positions c, joint positions and velocities of the hand q, the grasp matrix G, Jacobian J, the wrench on the object w, forces applied at the fingertips f, and the mass, coriolis, and gravity matrices M, C, g. These allow us to define the system linear and angular accelerations through physics.
The contact velocities can be defined as products of the grasp matrix and desired wrench, or the Jacobian and joint velocities.
The wrench on the object is the product of the forces applied and grasp matrix. The torque on the joints of the hand is found using the Jacobian and applied forces.
We basically satisfy the system of equations subject to the force and friction constraints to prevent breaking the object held and slipping of the fingers.
We use the Allegro 4 fingered dexterous hand for our control. This is a very widely used hand in robotics that can easily be incorporated into MuJoCo simulator, and we control the hand using torque control. We also use a cube with different sides on it so we can see the pose being changed.
We use impedance control for this delicate task to manipulate the pose of the object. Impedance control is preferred over position control because it directly deals with torques that will not break the object or command some unachievable position goals. The equations used for impedance control and a diagram is shown.
An additional change in contact position is added into the nullspace of the system, so that when a finger slips, it can regain its position on the object again, without affecting its pose. The nullspace basically means applying torques where the pose of the object does not change. So like when you hold a bottle but just move your fingers for good grip but don't rotate the bottle.
But what about actually holding the object at the fingertips? We can move its pose and keep the fingers in contact, but impedance control alone does not apply enough force to hold the object. Here is where Quadratic Programming (QP) comes in. Because this is a nonlinear system, we need to be nonlinear programming to solve it. Shown to the left is a diagram of what we want to achieve along with the equations.
Basically we want to apply an additional force at the fingertips to hold the object, without applying too much force or at too steep of an angle that is just slips. So we can model the force and second order dynamics using the grasp matrix.
Finally the only thing left to deal with is nullspace. In a manipulator like this, it is possible to have singularities in the Jacobian of the system. Basically this means we lose a degree of freedom while calculating the torques from the forces because two joints are perfectly lined up. Now we cannot move that joint, or the singularity may generate huge torque and break the system.
As a result, torques can be applied in the nullspace, so that it doesn't affect the goal of the system, but always avoids singularities in the Jacobian. A simple approach is shown to the right, where we just match the third and fourth joint positions such that they will never be totally aligned.
So once we put it all together, we were able to hold the cube at a given position! But, our impedance controller did not keep all the fingers on the cube. So we were holding the cube with only 2 of 4 fingers. Also, since we did not have time in the semester to create a contact schedule, we couldn't actively manipulate the pose of the object. But in this project, I learned how to design QP problem for a manipulation task, understand the math behind impedance control, and also nullspace control. It taught me the basics of the control problem, and all the intricate things that go into it.
I then designed the RL environment for this hand (it's a little different because here I added visuals for the fingertips in IsaacLab). Now, instead of doing torque control, I train an RL policy to do position control and move the object to the desired pose. The policy observes the pose of the object, the desired pose, and joint positions and velocities. These observations are sufficient for the policy to learn how to use the entire hand to move the cube.
