If you set a page as not published, you can still reach it if you know the direct URL. For example:
/content/example-page.php
Could this be fixed by setting the file permission as protected in some way? Then unprotect it when it's republished through the CMS?