I am using the world to cast a ray.
Is there a way for rigidbodies to ignore that raycast?
You can get a raycast to ignore a particular set of rigid bodies using the shape collection filter in the raycast input. To use the filter, you'd first need to divide your rigid bodies into collision groups. You can then use the shape collection filter in your raycast input to specify which collision group(s) to consider. Please look at the Physics->API->Collide->Ray Casting->Ray Cast Filtering demos and at the Havok Physics Help under Interacting with a Simulation->Collision Filtering for some more details.