To answer your last question, you cannot set a thumbnail size for an image that is originally smaller than the thumbnail size, because this would make the image larger and be pixelated.
What is the filetype of the image that shrank too small?
For example, I have strange effects with PNG on my server, and had to stick with using only JPG. Weird effects have also occurred if a JPG had certain chroma subsampling settings. This has always seemed to me to be a limitation of whatever software (Magick, GD, etc) the server used.
It also seems that the 70 x 63 is similar to a regular vBulletin thumbnail size. It's possible that the 180 x 179 image was not changed because it met the wiki's requirements already, and was unaffected, but your 566 x 506 image should have also used the 180 width for its thumbnail.
Was the image page recently moved INTO the Image namespace (via a thread move)? Have you tried rebuilding thumbnails after doing so?
Are you really running 1.0.0 Beta 3? Or are you running 1.0.0 Beta 2? In Beta 2, there was a bug in which rebuilding thumbnails would only rebuild the first wiki image it found using the wiki thumbnail settings, and all other images using the vB settings.