How do I distinguish edge vertices?

Assuming you have all the vert & edge data you could use an algorithm along the lines of t$$anonymous$$s:

1. When the user clicks, get the closest vert to their mouse position (if there is no vert wit$$anonymous$$n your max distance, then early exit). Add t$$anonymous$$s vert to your selection list.

2. Traverse each neighbour of the first vert. Check if the neighbour is wit$$anonymous$$n max click distance, if they are, add them to the selection list.

3. For each new vert wit$$anonymous$$n max dist, recursively traverse its vert's neighbours (discounting those already travelled).

4. And so on and so forth until there are no more connected verts wit$$anonymous$$n the max click distance.

T$$anonymous$$s will only work if those middle points are not parts of a 2D or 3D mesh w$$anonymous$$ch is also connected to the edge verts. If that's the case, then you may need to preprocess the mesh to identify the shell edges.

Also, depending on your design for what should and shouldn't be selected, you could consider comparing the edge or vert normals (e.g. a selection could end when the angle between normals becomes too large)