Sharing Compute Buffers between multiple Compute Shader instances

I am currently working on some voxel generation using Compute Shaders, but am having a performance issue when it comes to multiple compute shaders sharing a buffer. Essentially I have a compute shader instance for each chunk, but would like to share data between chunks.

Given two chunks, I would like for each chunk to have a reference to the other chunks primary compute buffer in order to perform culling on which faces to draw. I've tried creating a second compute buffer of the same type in each shader, and then assigning that to the neighboring chunk's primary buffer using shader.SetBuffer(handle, name, buffer). When I do this I get huge slowdowns from Gfx.GetComputeBufferData_Request, and within is Semaphore.WaitForSignal. This happens when rendering using Graphics.DrawProceduralNow() in the OnRenderObject() function.

I could probably create separate compute buffers for each neighboring chunk but I'd really like to save the memory by using the same buffer.

EDIT I tried the using a separate compute buffer instance for each neighboring chunk, and it seems that the slowdown is still present so it must not come from sharing the buffers. Is there a limit to how many compute buffers you can have? I could potentially collapse all the neighbor buffers into a single one but then writing to it becomes a huge pain.