I always thought UHD used a different laser than standard blu-ray, but only just found out it was a trick of h265 encoding and triple layer discs.

Based on the mini-BD format, assuming triple layer, the upper limit would have been around 24GB.

Now you’ve got me curious what capacity a UMD form factor could achieve with a UHD Blu-ray laser.

Yes/No. Both Sony and Microsoft have quality control processes to ensure that whatever is published is going to play on first entry of the disc.

That said, publishers use A LOT of workarounds. Day 1 patches to “finish” the game. Download code inserts. And as of recent, mandatory online server check-ins. As far as I’m aware, Nintendo is the only one who allows publishing half the product with required download.