PWA Eventing Service. Exception : System.Runtime.Remoting.RemotingException: Failed to connect to an IPC Port: The system cannot find the file specified.

SCENARIO:
When a custom event handler is configured on PWA server the eventing service fails and timesheet events go into error state.

I developed Server-side event handlers for Project Server Timesheet Approval Cycle to send emails to timesheet writer and approvers. This included an assembly that calls PSI web services to interact with PWA that is registered in GAC and basicHttpBinding timeout changes to cater timout issues occuring while reading web service response.

The solution  I provided was successfully tested and verified on UAT environment and deployed to Production server and here starts the issue...

ISSUE:

    After registering the Event handlers to PWA the following error starts prompting. All the timesheets started giving error  and queued.   When cancelled from the queue the timesheet showed submitted state because they update the Published DB but didnot reflect in Reporting DB.

1.   “Couldn't validate configuration on eventing service: Exception: System.Runtime.Remoting.RemotingException: Failed to connect to an IPC Port: The system cannot find the file specified.  

2.   “Standard Information:PSI Entry Point:   Project User: DOMAIN\User  Correlation Id: 1dcd87ae-9345-4733-b950-f8443ebb3fcb  PWA Site URL: http://site/PWA  SSP Name: Project Server PWA_ServiceApp  PSError: NoError (0) Queue unable to interact with SQL. Queue type (Project Queue, Timesheet Queue etc): TimesheetQ Exception: Microsoft.Office.Project.Server.BusinessLayer.Queue.QueueSqlException: CompleteGroup failed ---> System.Data.SqlClient.SqlException: Could not allocate space for object 'dbo.MSP_QUEUE_TIMESHEET_GROUP_ARCHIVE'.'CL_MSP_QUEUE_TIMESHEET_GROUP_ARCHIVE' in database 'ProjectServer_Published' because the 'PRIMARY' filegroup is full. Create disk space by deleting unneeded files, dropping objects in the filegroup, adding additional files to the filegroup, or setting autogrowth on for existing files in the filegroup.   

3.   Error is: GeneralActionCanceledBecauseServerEventServiceProblem. Details: Attributes:  TimesheetReviewed  . Standard Information: PSI Entry Point:   Project User:DOMAIN\User  Correlation Id: 443945d0-741a-4eb2-99bf-4c2ba50e6365  PWA Site URL: http://site/PWA   SSP Name: Project Server PWA_ServiceApp  PSError: GeneralActionCanceledBecauseServerEventServiceProblem (22002)

After rolling back all the changes and unregistering the assembly from PWA   the IPC remote call error still prompted on every event call. Like if I delete , recall ,submit, approve or reject a timesheet

In the logs it shows in following order.

1.       Attempts to call IPC

2.       Throws error for number of retries attempted with following logs.

1.   ServerEventManager: Reconfiguring events 

2.   This occurs on every timesheet event i.e: submit, recall, delete, reviewed even after rolling back the deployment from Production server As shown in the log  below.  “Error during farm configuration change. Exception : System.Runtime.Remoting.RemotingException: Failed to connect to an IPC Port: The system cannot find the file specified. 

Resolution: 

 Most of the issues like thi are due to Firewall blocking IPC connections.

to resolve this issue you need to change the User account settings to minimum.

• Go to Start   ---------->  Control Panel  ---------->  User Accounts  
•  Change User Account Setting ----------------> Set the settings to never notify ----> Press OK

• • Restart the server if prompted.
  • Deploy the provided assembly and register the event handlers to PWA
  • Test It
  Issue Resolved :)