(note: I suggested this simple fix should maybe become part of the WP code because it's the best place to implement it, but here it is in the meantime so you can use it)
When an external user accesses a site set up for /?page=123 style URLs, they can force the display of the site's underlying web engine 404 page by going to a page that lies "outside" WP, say site.com/abcdefg. Not only is that untidy, such default pages can sometimes contain information that can assist in working out how to hack a site.
Add the following to the root .htaccess file will force such an attempt into the WordPress generated 404 page (most themes have them in varying degrees of politeness - you can find it as 404.php in the theme directory).
# BEGIN WordPress
# Force 404s into WordPress by pointing at non-existing page reference.
ErrorDocument 404 /?page_id=999999
# END WordPressIf you run a different URL style, adjust accordingly - just make sure it points to a WordPress page that doesn't exist.
(a quick thank you to whoever runs http://www.htaccessbasics.com which made this easy to find :) ).