How to use layer-based collision on a parent-child system to make the parent pass through and the child collide?

I have a parent gameobject called Avatar and it's children, Capsule and Cylinder. The Avatar gameobject is merely a container gameobject for Capsule and Cylinder. Cylinder is bigger than Capsule and completely surrounds it. I would like Capsule to collide with walls and Cylinder to pass through them.
Let's say the Walls are in Layer1, I assigned Layer2 to Cylinder and Avatar and assigned Layer3 to Capsule. In the collision matrix, I unchecked Layer1 Layer2 while keeping Layer1 Layer3 checked.
I still find that the Capsule passes through the Walls. Is this because it's parent object, Avatar, is in Layer2? How can I achieve the effect I'm looking for?