I think the only good way that this can be improved is to merge the upload overlay down into the image select overlay, possibly making it like so:
1. Click editor button.
2. Overlay opens. Overlay has tabs "Choose Existing File", "Upload New File" (if permission to upload). It's possible the styling options can be tightened up so they don't use as much room.
But yes, I'm aware that it doesn't use the style of other XenForo dialogs. As you've noted, this is an inconsistency across platforms, where the dialog is styled like a forum page rather than as an editor dialog. We've been putting off fixing this style inconsistency since it was more important to get the core stable, and at least it looked like a page from the same forum.
I still have to fully investigate the XenForo uploader, but if even that's possible, it would only be a style change. Due to issues discussed elsewhere here ad nauseum and that I know you're aware of, a forum attachment system cannot be shoehorned to function as a true wiki attachment system, so we cannot simply "use the XenForo uploader" as it is. They are completely different in almost every way.
Unfortunately all the add-ons you mentioned are paid add-ons, so I cannot just see the way they do it in terms of their interfaces. But I really doubt they have an editor button that lets you scroll through existing uploads or that you can upload to XenMediaGallery directly from the editor of a regular forum post. That is the purpose of this editor button in VaultWiki. And that's what you're talking about in your post above. Nothing more, nothing less.
I'm sure XenMediaGallery functions like most gallery systems where you must click an upload button when you are actually visiting the gallery. VaultWiki has this function as well. Simply go to a wiki area, and in the Create New drop-down, instead of "Page" click on "Attachment".
I am accepting this task so that the following can be completed:
1. Tabulate the insert image overlay to reduce the need for the extra overlay, and to make the upload option in this overlay more visible.
2. Update the style of the overlay to be more in line with the style of other editor dialogs.
3. Possibly update the style of the upload field in both the overlay and the attachment editor to look more like XenForo's uploader.