Bug 5338

PB 2019 Application crashing with the exception code 0xc0000374 21 October, 2020

Ravindran Naarayanan
28 September, 2020
Product: PowerBuilder Category: Installation
Version: 2019 R2 Build: 2170
Classification: Publishing: Public
Priority: P1
Status: Reproducing Reason:
Mark Lee @Appeon 21 October, 2020
#19
Hi Ravi,

1. I have transferred the attachment to our Development Team for analysis. 
But they find that this log didn’t help locate or resolve the issue.
 
2. Could you please further locate which line of code is causing the crash via gradually commenting out a part of code?

3. It would be very helpful if you could provide a small reproducible case for us for further analysis.

4. If you can’t provide a small case for us, it is suggested you provide a machine that can reproduce the issue and with the remote tool installed (e.g., TeamViewer) for us so that we could remotely debug it.

Regards,
Mark Lee
Ravindran Naarayanan 14 October, 2020
#18
MemoryLeakDumpAnalysisFile.mht (1371KB)

Hi,

We recorded memory leaks from DebugDiag tool and I have attached the analysis file for reference. It is crashing due to access violation exception on PBDW190.DLL.

Regards,
Ravi
Mark Lee @Appeon 29 September, 2020
#17
Hi Ravi,

Thanks for your feedback.
As we said before, using the PFC version that is not compatible with the PB version will cause some unknown crashing issues and will make the PB application unstable so it’s necessary to upgrade PFC. But of course, this is your decision to make.
 
The issue you reported is not a consistently reproducible issue for all customers.  
So for such an issue, we would require that you provide a test case so that we can help you.  

Regards,
Mark Lee
Ravindran Naarayanan 29 September, 2020
#16
Hi Mark,

Our vendor confirmed that the PFC version is 5, however they also mentioned that the crashing modules or windows do not use PFC.

Regards,
Ravi
Mark Lee @Appeon 29 September, 2020
#15
Hi Ravi,

1. According to our meeting, we found that your application was upgraded from PB 7 to PB 2019 directly and now contains the old version of PFC Framework. And regarding the feedback from other customers, if the PFC is not compatible with PB version, it would causes some crashing issues. So we suggest that you can first upgrade your PFC Framework to 2019 and see if it solves the issue.

Please note that:
The PFC libraries are now "Open Source" since PB 10.0. They have been moved however in recent years over to GitHub. You can find the PFC Framework in the link below:
https://github.com/OpenSourcePFCLibraries 

2. If upgrading PFC doesn't solve the issue. We notice that the crashing issue always happens on a specific place in your application (the dropdown window of the machine you used in the meeting). It would be great if you can write a small case and make sure it can reproduce the issue on your side first and then provide it for us. Thanks!

Regards,
Mark Lee
Mark Lee @Appeon 29 September, 2020
#14
Hi Ravi,

Please suggest the Meeting ID. Thanks!

Regards,
Mark Lee
Ravindran Naarayanan 29 September, 2020
#13
Hi Mark,

I am available now. I have opened the below meeting invite for remote troubleshooting. Please join the web meeting

https://meeting.windstream.com/my/nravi
Mark Lee @Appeon 29 September, 2020
#12
Hi Ravi,

Thanks for your feedback.

1. Can you providing a VM for us to reproduce this issue?
 
2. If not then may I ask if you are available now. If so, we can schedule a meeting to remotely debug it.

Please let me know soon and which tool you'd like to use for this session (TeamViewer or GoToMeeting).

Best regards,
Mark Lee
Ravindran Naarayanan 29 September, 2020
#11
Hi Mark,

Please find the responses.

2. The application is 32 bit and it was compiled in Windows 2016 64 bit OS.
3. We have tried rebooting it didn't work.
4. We have tried the same in different Windows 2016 & 2012 VMs, having the same issue
5. Performed this step, but still crashing
6. I believe, this may not be applicable to us as we are using VMs. 

Regards,
Ravi
Mark Lee @Appeon 29 September, 2020
#10
Hi Ravi,

1. Currently we can’t find out the cause of it from the dump file. In the Dump file it shows that it is Windows’ DLL that caused the error:
"NTDLL!RTLREPORTCRITICALFAILURE+83In tbs__PID__7288__Date__09_28_2020__Time_07_46_58AM__951__Second_Chance_Exception_C0000374.dmp the assembly instruction at ntdll!RtlReportCriticalFailure+83 in C:\Windows\System32\ntdll.dll from Microsoft Corporation has caused a corrupted heap exception (0xC0000374) when trying to perform an unknown operation on memory location 0x00000000 on thread 0
Heap corruption was detected in this dump"

2. Would you let us know which platform is your app compiled, is it 32-bit or 64-bit? 

3. According to other customer experiences, I suggest you can reboot the client machine and try it again.

4. Please test on another machine to see if the same issue exists.

5. Please copy the below 3 DLLs to the target folder where the MSI is to be installed, but not the folder of the MSI file or the executable folder. (Not the folder: "C:\Windows\System32")
 atl100.dll
 msvcr100.dll
 msvcp100.dll
You can find these DLL files in the PB install folder:
For 32bit dll, go to C:\Program Files (x86)\Appeon\Shared\PowerBuilder
For 64bit dll, go to C:\Program Files (x86)\Appeon\Shared\PowerBuilder\x64

6. I’m not sure if this will work for you as well, but please take the example of how the customer solved the issue in the link below for your reference:
https://community.appeon.com/index.php/qna/q-a/pb-r3-application-instability-crashes-in-ntdll-dll-on-windows-10 

Regards,
Mark Lee
Chris Pollach @Appeon 28 September, 2020
#9
Hi Ravi;

  The other VC++ redistribution libraries are OK to leave on the deployment machine.

  At this time, I will now transfer this ticket over to the main Support / Engineering team for their review of your situation. They might then ask you to download & run some more low level diagnostics to help understand what is going on with these crashes in the MS-Windows DLL (ntdll).

Regards ... Chris
Ravindran Naarayanan 28 September, 2020
#8
Hi Chris,

The Microsoft 2010 Visual C++ Redistributable runtime library has been installed, but the app still crashed

Note: All the other runtime library files like 2015, 2017 & 2019 are still available, they have not been uninstalled.

Regards,
Ravi
Chris Pollach @Appeon 28 September, 2020
#7
Hi Ravi;

  Sorry, I should have been more specific ... 

2) Microsoft Visual C++ 2010 runtime libraries (2010 redistributable).


Regards ... Chris
Ravindran Naarayanan 28 September, 2020
#6
Hi Chris,

Please find my responses
1. msvcr100.dll and msvcp100.dll are available on the deployment PC
2. Microsoft Visual 2015-2019 C++ runtime library installed verified
3. atl100.dl is also available on the deployment PC

gdiplus.dll is available in the Windows directory and I also copied over the gdiplus.dll to the working folder, it still did not resolve the crashing issue
Chris Pollach @Appeon 28 September, 2020
#5
Hi Ravi;

  Also, you must have the VS2010 redistributable installed on the deployment PC as well and then manually copy the following DLL files from the PB Developer PC to the deployment PC if these are not there, as follows:

1) msvcr100.dll and msvcp100.dll
2) Microsoft Visual C++ runtime libraries
3) Microsoft .NET Active Template Library (ATL) module atl100.dll

  (if the above are not present on the deployment computer).

Note: Some fies like "" are also dependent on the "gdiplus.dll" as well. If that DLL is missing, you can get these type of issues.

For more information on the above, please use the PB Help & search for the keyword "runtime". From there, take the "PowerBuilder Runtime Files" link and carefully read through the Microsoft files section prerequisites.

HTH
Regards ... Chris
Ravindran Naarayanan 28 September, 2020
#4
Yes Chris, we followed the same process
Chris Pollach @Appeon 28 September, 2020
#3
Hi Ravi;

  Did you use the PB Packager to create an install MSI for the PB Runt-time, then run that MSI on the MS-Windows server where the PB App(s) is being deployed?

Regards ... Chris
Ravindran Naarayanan 28 September, 2020
#2
tbs__Windows2012__Second_Chance_Exception_C0000374_CrashHangAnalysis.mht (624KB)

Also adding the mhtml file for the same issue from Windows 2012 machine
Ravindran Naarayanan 28 September, 2020
#1
tbs__Windows2016__Second_Chance_Exception_C0000374_CrashHangAnalysis.mht (632KB)

*Phenomenon:
The Powerbuilder application is crashing with the below error message

Faulting application name: tbs.exe, version: 1.0.0.1, time stamp: 0x5e03ac98
Faulting module name: ntdll.dll, version: 10.0.14393.3630, time stamp: 0x5e8d42d1
Exception code: 0xc0000374
Fault offset: 0x000d9661
Faulting process id: 0xd90
Faulting application start time: 0x01d695ac032660eb
Faulting application path: D:\tbs.exe
Faulting module path: C:\Windows\SYSTEM32\ntdll.dll
Report Id: bd735446-9c24-4462-aea6-35a708754c72
Faulting package full name: 
Faulting package-relative application ID: 

We were initially getting this error at will and after enabling the compatibility mode to set to Windows 8 it worked. However after the application was deployed into production it is crashing in various windows of the application and we are unable to reproduce it at will. This is a work stoppage at this moment, any help would be greatly appreciated.
*Reproduce Steps:


Remarks:
I have attached debug mhtml file for analysis. We have also followed the steps provided in the link https://www.appeon.com/standardsupport/search/view?id=4572 but it has not helped.
OS:
Windows Server 2016
Platform:
64-bit
Database Type:
Oracle
Database Version:
19C