PWA Eventing Service. Exception : System.Runtime.Remoting.RemotingException: Failed to connect to an IPC Port: The system cannot find the file specified.

SCENARIO:
When a custom event handler is configured on PWA server the eventing service fails and timesheet events go into error state.

I developed Server-side event handlers for Project Server Timesheet Approval Cycle to send emails to timesheet writer and approvers. This included an assembly that calls PSI web services to interact with PWA that is registered in GAC and basicHttpBinding timeout changes to cater timout issues occuring while reading web service response.

The solution  I provided was successfully tested and verified on UAT environment and deployed to Production server and here starts the issue...

ISSUE:

    After registering the Event handlers to PWA the following error starts prompting. All the timesheets started giving error  and queued.   When cancelled from the queue the timesheet showed submitted state because they update the Published DB but didnot reflect in Reporting DB.

1.   “Couldn't validate configuration on eventing service: Exception: System.Runtime.Remoting.RemotingException: Failed to connect to an IPC Port: The system cannot find the file specified.  

2.   “Standard Information:PSI Entry Point:   Project User: DOMAIN\User  Correlation Id: 1dcd87ae-9345-4733-b950-f8443ebb3fcb  PWA Site URL: http://site/PWA  SSP Name: Project Server PWA_ServiceApp  PSError: NoError (0) Queue unable to interact with SQL. Queue type (Project Queue, Timesheet Queue etc): TimesheetQ Exception: Microsoft.Office.Project.Server.BusinessLayer.Queue.QueueSqlException: CompleteGroup failed ---> System.Data.SqlClient.SqlException: Could not allocate space for object 'dbo.MSP_QUEUE_TIMESHEET_GROUP_ARCHIVE'.'CL_MSP_QUEUE_TIMESHEET_GROUP_ARCHIVE' in database 'ProjectServer_Published' because the 'PRIMARY' filegroup is full. Create disk space by deleting unneeded files, dropping objects in the filegroup, adding additional files to the filegroup, or setting autogrowth on for existing files in the filegroup.   

3.   Error is: GeneralActionCanceledBecauseServerEventServiceProblem. Details: Attributes:  TimesheetReviewed  . Standard Information: PSI Entry Point:   Project User:DOMAIN\User  Correlation Id: 443945d0-741a-4eb2-99bf-4c2ba50e6365  PWA Site URL: http://site/PWA   SSP Name: Project Server PWA_ServiceApp  PSError: GeneralActionCanceledBecauseServerEventServiceProblem (22002)

After rolling back all the changes and unregistering the assembly from PWA   the IPC remote call error still prompted on every event call. Like if I delete , recall ,submit, approve or reject a timesheet

In the logs it shows in following order.

1.       Attempts to call IPC

2.       Throws error for number of retries attempted with following logs.

1.   ServerEventManager: Reconfiguring events 

2.   This occurs on every timesheet event i.e: submit, recall, delete, reviewed even after rolling back the deployment from Production server As shown in the log  below.  “Error during farm configuration change. Exception : System.Runtime.Remoting.RemotingException: Failed to connect to an IPC Port: The system cannot find the file specified. 

Resolution: 

 Most of the issues like thi are due to Firewall blocking IPC connections.

to resolve this issue you need to change the User account settings to minimum.

• Go to Start   ---------->  Control Panel  ---------->  User Accounts  
•  Change User Account Setting ----------------> Set the settings to never notify ----> Press OK

• • Restart the server if prompted.
  • Deploy the provided assembly and register the event handlers to PWA
  • Test It
  Issue Resolved :)

Reset SharePoint Config Cache

  The   path for the Configuration Cache under Windows Server 2008 is: C:\ProgramData\Microsoft\SharePoint\Config\<GUID>

Stop the Timer service. To do this, follow these steps:
1.      Click Start, point to Administrative Tools, and then click Services.
2.      Right-click SharePoint 2010 Timer, and then click Stop.
3.      Close the Services console.
4.      On the computer that is running Microsoft SharePoint Server 2010 and on which the Central Administration site is hosted, click Start, click Run, type explorer, and then press ENTER.
5.      In Windows Explorer, locate and then double-click the following folder: C:\ProgramData\Microsoft\SharePoint\Config\GUID
Notes
1.      The GUID placeholder specifies the GUID folder. There may be more than one of these.
2.      The ProgramData folder may be hidden. To view the hidden folder, follow these steps:
·         On the Tools menu, click Folder Options.
·         Click the View tab.
·         In the Advanced settings list, click Show hidden files and folders under Hidden files and folders and then click OK.
3.      (You can also simply type this directly in the path if you do not want to show hidden files and folders.)
4.      Back up the Cache.ini file. (Make a copy of it. DO NOT DELETE THIS FILE, Only the XML files in the next step)
5.      Delete all the XML configuration files in the GUID folder (DO NOT DELETE THE FOLDER). Do this so that you can verify that the GUID folders content is replaced by new XML configuration files when the cache is rebuilt.
Note When you empty the configuration cache in the GUID folder, make sure that you DO NOT DELETE THE GUID FOLDER AND THE CACHE.ini file that is located in the GUID folder.
If Cache .ini is Missing
6.      Double-click the Cache.ini file.
7.      On the Edit menu, click Select All.
8.      On the Edit menu, click Delete.
9.      Type 1, and then click Save on the File menu. (Basically, when you are done, the only text in the config.ini file should be the number 1)
10.  On the File menu, click Exit.
11.  Start the Timer service. To do this, follow these steps:
·         Click Start, point to Administrative Tools, and then click Services.
·         Right-click SharePoint 2010 Timer, and then click Start.
·         Close the Services console.
12.  Note The file system cache is re-created after you perform this procedure. Make sure that you perform this procedure on all servers in the server farm.
13.  Make sure that the Cache.ini file in the GUID folder now contains its previous value. For example, make sure that the value of the Cache.ini file is not 1.
14.  Check in the GUID folder to make sure that the XML files are repopulating. This may take a bit of time.


  If Cache.ini is missing under   “C:\ProgramData\Microsoft\SharePoint\Config\” or is deleted by mistake, perform the following Steps:
Now the mission is to get my Timer Job up and functioning. So I took the back-door approach to create GUID folder with Cache.ini file. Here are the steps I performed to get the config GUID of SharePoint.
1.      Stop the SharePoint 2010 Timer process.
1.      Open Windows Registry Editor
2.      Locate the ConfigDB key. Usually, it must be in this place if you are using SharePoint 2010.[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Shared Tools\Web Server Extensions\14.0\Secure\ConfigDB]
3.      You will see String Values entries for
4.      DSN: Connection string for your configuration database.
5.      Id: GUID of SharePoint configuration database. Copy this GUID.
6.      Goto C:\ProgramData\Microsoft\SharePoint\Config\ folder. (ProgramData folder is hidden, make sure change windows folder options to make it visible to you)
7.      Create a folder and name it as the GUID (that you just copied from Registry).
8.      Go into this folder, create a text file and name it as Cache.ini.
9.      Now edit this Cache.ini, key in value 1.  Save and close the file.
10.  Start the SharePoint 2010 Timer process.
11.  Do an IIS RESET
12.  Now check all your application pools are up and running. Otherwise, start them manually.