Forcibly halt movement of slow rigidbodies, but not falling ones. (2D)

First, some background:
I’m trying to optimise the performance of a large number of dynamic rigidbodies. There are too many to simulate them all at once, so I want as many of them slept (or better yet, unsimulated) as possible, without it looking weird.
Therefore, I want to check whether each rigidbody is moving, and deactivate it if not. This doesn’t seem to work if I just check the velocity, because in a pile of rigidbodies, colliding ones have some velocity (often quite large, strangely) but they are not visibly moving.
Current solution:
Every X frames store the position of each body, and check whether it’s displacement from it’s position X frames ago is small enough to be considered stationary.

Problem:
The current solution causes some falling objects to stop in mid air, especially if they collide with others and are therefore moving slower than usual. Increasing X could help this, but then means that the other objects are not “cleaned up” as often.
Any suggestions to detect whether the object is actually “at rest” or falling would be most welcome! Or alternatively, if you have a better Idea as to how to detect “not visibly moving” objects that’d be great too!

Well, the problem was actually caused by a different part of the code!
As for anyone experiencing the problem I thought I had, I would suggest increasing X or decreasing the minimum displacement to be considered stationary.
Sorry to have wasted anyone’s time.