Generating a parameterized 3D mesh from Python

Hi All, I'm a hard-core Machine Learning programmer (Python, TensorFlow, PyTorch, Numpy, ...) and I want to dive into my first VR project so basically I need some pro tips to get started!

Here's what I want to do:

⦁ Create a parameterized 3D mesh (initially this could be built from simple 3D objects like spheres or pyramids with adjustable sizes and rotations, but in the end I would like to create a fully custom, connected grid of triangular meshes based on (x,y,z) coordinates that can be dynamically adjusted. (The goal is to generate these coordinates on the fly, using Machine Learning libraries in Python, send them over a local socket and use them in Unity to dynamically warp the 3D mesh)

⦁ Next, I want to map a single image (like this one: https://pbs.twimg.com/media/DdJ7iEOX4AAOiUp.jpg) (or a set of images) onto this 3D mesh. The image can be stretched, rotated, even cut into pieces and duplicated/tiled, but I want to place it all over the 3D mesh so that the 3D structure from the mesh is overlayed with the texture of the image.

⦁ Then, I want to render the entire scene from any two camera points in the 3D space (most likely just selecting standard two-eye viewpoints)

⦁ Finally, the two rendered spherical views need to be formatted as a split-screen square .mp4 video format so it can be played on a standard VR headset and the viewer can look around.

If you're wondering what the intended goal is, well it's kind of hard to explain but I want to try something creative, slightly inspired by this video: https://youtu.be/II3bE9nA5Z4

Basically, I am looking for the right VR libraries (Github?) to do all this. In the end I want to dynamically control the parameterized 3D mesh from an ML library like TensorFlow (Python) based on music and user controls. I think the Unity-ML agents library is probably where I need to start, but I don't really know the best way to approach this..

Anybody has some suggestions for me?