Horizontal/Vertical Layout Group forcing control?

Hey, i am using Vertical/Horizontal layout groups in some places in my project, i have noticed that whenever i disable or remove the component, i cannot use scroll at all if the item was scroll-able inside a viewport.

for example i have made a Scroll View, gave it my content and that content has Vertical/Horizontal layout group, everything works fine i can scroll to where ever i intended to but once i remove the Vertical/Horizontal layout group then the scrolling stops functioning like Vertical/Horizontal layout group was the controller over the scrolling. anyone knows why its like that?

i want to know if there is any way to avoid that whenever i disable or remove the component it will not remove my scrolling functionality as far as i understood Vertical/Horizontal layout group are very expensive components.