The process for creating a custom error page for SharePoint is different from how you would normally set one up for a website hosted by IIS.
First create your custom error page as an html document.
Save the custom error page to the following location:
C:\Program Files\Common Files\Microsoft Shared\web server extensions\12\TEMPLATE\LAYOUTS\1033
The error page then needs to be assigned to the web appication it is to be used for. Because the custom error page is assigned at the web application level you can create a different page per web application if required.
The assignment is done with the following code:
Microsoft.SharePoint.Administration.SPWebApplication webapp =
Microsoft.SharePoint.Administration.SPWebApplication.Lookup(new Uri([site url]));
webapp.FileNotFoundPage = [errorpage.html];
webapp.Update();
Replace [site url] and [errorpage.html] with the relevant values for your site and your error page.
To test that this has worked try to navigate to a page on your site that does not exist. Rather than seeing the standard page not found error you should now see your custom page.