Bug 3736

pbdwe1xx.dll 64Bit truncates addresses causing crashes due to the use of Get/SetWindowLong instead of Get/SetWindowLongPtr 06 July, 2020

Rudolf Alexander Kerbitz
05 December, 2019
Product: PowerBuilder Category: DataWindow
Version: 2017 R3 Build:
Classification: Sybase (legacy) bug Publishing: Public
Priority: P2
Status: Scheduling Reason:
Mark Lee @Appeon 06 July, 2020
#23
Hello Rudolf,

Thanks for your feedback and for the confirmation.
I will transfer this info to our product team for further analysis.
And I will keep you posted if any progress we would make.
Thanks you for so much.

Best regards,
Mark Lee
Rudolf Alexander Kerbitz 01 July, 2020
#22
Hello Mark,

thank you for prividing build 2378. I performed regression tests on it and want to share my findings in return. 
1. The number of calls to GetWindowsLong for accessing pointers stored in a window's data has decreased drastically! Apparently your colleagues fixed the window procedure "dwWinProc", located in PBDWE.DLL virtual address 0x00262f40 and the routines that it calls. It does not show up in my trace-logs anymore. 
2. PBSNC190.DLL still needs to be corected. The hotfix I described in message from June 23rd is still applicable, the text segment of the binary in build 2378 apprears to be identical to 2357.

The following window procedures still call GetWindowsLong for accessing pointers:
 - FN_DataWindowWnd (hint: while processing a WM_KEYDOWN message)
 - FN_WindowWnd (hint: while processing a WM_QUERYENDSESSION message)
 
Best regards

Rudolf Alexander Kerbitz
Mark Lee @Appeon 29 June, 2020
#21
Hello Rudolf,

Thanks for providing the new test case and for your detailed analysis.
I found that this issue may be related to the issue in the second case you provided, and I also found the two issues are fixed in our internal version which is Build 2378.
So I have uploaded the new  PB 2019 R3 standalone beta to the FTP server.

Please use the same FTP account which we provided before to download the new package (PB2019R3_2378.zip) and retest it again.
FTP server: download.appeon.com
Username: rudolf.kerbitz@copasysteme.de
We suggest that you use an FTP Tool (like the free FileZilla client) to connect to the FTP using the settings in comment 13.

Regards,
Mark Lee
Rudolf Alexander Kerbitz 23 June, 2020
#20
Hello Mark

regarding the access violation caused within PBSNC190.DLL (x64) wich was described in my message from June 22nd I wanted to share the hotfix, that I applied in order to be able to continue testing until it will be fixed by your dev-team. Two assembly instructions have to be replaced in PBSNC190.DLL (x64). Fortunately the replacement instructions have identical op-code lengths, which makes applying the hotifx as easy as:

Navigate to file offset 0x2128:
	replace 4 bytes	"44 8D 5B 34"	(lea r11d, [rbx+0x34])
	with 			"4C 8D 5B 34"	(lea r11, [rbx+0x34]).

File offset 0x212C, 3 bytes:
	replace	3 bytes	"49 63 CB"		(movsxd rcx,r11d)
	with			"4c 89 d9"		(mov    rcx,r11).

Best regards

Rudolf Alexander Kerbitz
Rudolf Alexander Kerbitz 22 June, 2020
#19
Bug3736-3.zip (471KB)

Hello Mark,

during the last week I tracked another access violation which turned out to be caused by simple address truncation within PBSNC190.DLL. The instruction  " "44 8D 5B 34" (i.e. LEA r11d,[rbx+34h]) at PBSNC190.DLL+0x2d28 only loads the lower 32 bit (note the suffix 'd' on the register argument "r11d") of the pointer to the string generated by the previous call to PBSHR190.DLL!shformatDateTime. This either causes an access violation or undefined behaviour in case the generated string was allocated above the 4GB boundary. 

Please find a minimum example workspace and executables attached. 

In order to force this truncation error to become apparent immediately I enabled high entropy address layout randomization for the executable such that the image is loaded at a base address way above the 4GB boundary and all allocations (stack + heap) also are also located above 4GB (use: "editbin /HIGHENTROPYVA exampleapp.exe after deploying the example application"). 

Reprocucing:
0. Create MSSQL database with a single Table Test0000 containing only a column "somedate" of type "datetime" (see. createtable.sql) and adjust SQLCA settings in the exampleapp open event accordingly. Deploy and enable high entropy address layout randomization for the deployed executable.
1. run run_example_with_hooks.bat to start the example with API-Hooks attached, debug output will be written to "debugoutput.txt"
2. A message box opens that asks you to attach the debugger. Attach VisualStudio-debugger to exampleapp.exe and acknowlede the message box by pressing "OK".
3. Shortly before the actual access violation occurs a debug breakpoint will be triggered that I used to locate the origin of the address truncation.
4. Step out (Shift+F11) and view the disassembly, the top of the callstack should now look something like this:
>	PBSHR190.DLL!000000007237d9d5()
 	PBSHR190.DLL!000000007237dd0a() <---- this is PBSHR190.DLL!shformatDateTime
 	PBSNC190.DLL!0000000071c72d25() <---- this is the culprit
 	PBSNC190.DLL!0000000071c71609()
 	PBSHR190.DLL!00000000723ab377()
 	PBDWE190.DLL!0000000071fce0e1()
 	[...]
5. Navigate to the topmost PBSNC190.DLL entry within the callstack (marked with "this is the culprit). The disassembly looks something like this:
 	call 	0000000071C9E9B6     
 	mov 	rax,qword ptr [rbx] 	<------- this is the instruction right after the call to PBSHR190.DLL!shformatDateTime
 	lea		r11d,[rbx+34h] 	 	<-------- this is the truncating instruction
6. Continue execution until the truncating instruction is reached. Note the address pointed to at by [rbx+34h] (e.g 0x00000215c3c76060 L"'2020-06-22 13:15:14.")
7. Continue execution. The Debugger will popup an exception message: "Exception thrown at 0x0000000071FDC0EF (PBDWE190.DLL) in exampleapp.exe: 0xC0000005: Access violation reading location 0xFFFFFFFFC3C76060." 
8. As you can see in debugoutput.txt line 7035ff the PBVM was in the midst of executing object function UPDATE for class DATASTORE, e.g. PBVM190.DLL!fnUpdate) and it failed because the address of the string containing the timestamp was truncated.

Best regards
Rudolf Alexander Kerbitz
Mark Lee @Appeon 15 June, 2020
#18
Hello Rudolf,

Thanks for providing the test case.
I can reproduce it on our side and will escalate this problem to our development team for further analysis.
We will need some to figure it out and will get back to you if any progress we would make.
Thanks for your understanding.

Best regards,
Mark Lee
Rudolf Alexander Kerbitz 10 June, 2020
#17
Bug3736-2.zip (306KB)

Hello Mark,

please find a minimum example project attached. In addition to the PB 2019 R3 workspace I added my API-Hook injection tool "Inject.exe", the hooks library "APIHooks.dll" and a batch to run the example with the hooks attached. Click on run_example.bat to start "example2.exe" with injected API-hooks. A message box will ask you to attach a debugger and press OK to continue. When you attach VisualStudio to it, a combination of the output generated by the "/pbdebug" and messages about allocations, windowcreation and Get/SetWindowLong(Ptr)W calls will be printed to VisualStudio's debug output window in chronological order for the application's main thread. Example output can be found in "example_debug-output.txt". APIHooks.dll is build to trigger a breakpoint when GetWindowLongW is called on the same data that was previously set with SetWindowLongPtr (see example_debug-output.txt lines 191-193). 

Best regards

Rudolf Alexander Kerbitz
Mark Lee @Appeon 09 June, 2020
#16
Hello Rudolf,
 
Thanks for your feedback and for providing the log info.  
I have transferred it to our development team but after the analysis, they still want you to provide a reproducible sample test case (PBT / PBL) for us to reproduce it locally. Thanks in advance.

In fact, we need to change more than thousands Get/SetWindowLong, if you could provide the test case it would help us address it more clearly and avoid causing other issues. And we need to be more careful.
 
BTW: Thanks for taking your time & effort to test out the PB2019 R3 standalone beta and providing us your valuable feedback. Appeon really appreciates it!

Regards,
Mark Lee
Rudolf Alexander Kerbitz 08 June, 2020
#15
Hello Mark,

thank you for providing the download. I wanted to share my findings that I got so far. 

The test-case which I provided appears to be fixed using the 2019 R3 build 2357. Moreover I made a test build of our application and checked the actual usage of Get/SetWindowLong(Ptr)W by hooking these API functions using DLL-injection. According to the collected runtime information at least one call to GetWindowsLongW was missed to be replaced in build 2357. These particular calls access data on windows of class "pbdw190" at nIndex=0 that was previously set using SetWindowsLongPtrW. 

These calls happen during the call of CreateWindowExW from dwCreateWindowEx which is located located at PBDWE190.dll + 0x00022edb . The excerpt of the corresponding callstack is shown below. The given offsets within PBDWE190.DLL refer to virtual addresses within the DLL:

>	APIHooks.dll!HookGetWindowLongW(HWND__ * hWnd=0x0000000000250d82, int nIndex=0x00000000)
>	PBDWE190.DLL!0000000000260590()
>	PBDWE190.DLL!0000000000264542()
>	user32.dll!UserCallWinProcCheckWow()
>	user32.dll!DispatchClientMessage()
>	user32.dll!__fnINLPCREATESTRUCT()
>	ntdll.dll!KiUserCallbackDispatcherContinue()
>	win32u.dll!NtUserCreateWindowEx()
>	user32.dll!VerNtUserCreateWindowEx()
>	user32.dll!CreateWindowInternal()
>	user32.dll!CreateWindowExW()
>	APIHooks.dll!HookCreateWindowExW(unsigned long dwExStyle=0x00000000, const wchar_t * lpClassName=0x000000006f63e438, const wchar_t * lpWindowName=0x000000000305e6e4, unsigned long dwStyle=0x46000400, int X=0x00000000, int Y=0x00000000, int nWidth=0x00000000, int nHeight=0x00000000, HWND__ * hWndParent=0x0000000000010010, HMENU__ * hMenu=0x0000000000000000, HINSTANCE__ * hInstance=0x000000006f2e0000, void * lpParam=0x00000000030845bc)
>	PBDWE190.DLL!0000000000022edb()    <------ this is dwCreateWindowEx + 0x13b
>	[...]

Best regards

Rudolf Alexander Kerbitz
Mark Lee @Appeon 27 May, 2020
#14
Hello Rudolf,
 
Just to let you know that the PB 2019 R3 standalone beta is ready.
1. Please remember, this version is our internal test version and it may not be very stable.
2. Before you do the test, please backup your PB source code first. And then when you finish the testing, you can recover the PB source code again.
You can download the PB install package from our FTP server.
3. We also suggest you use a clean machine to test which doesn't have any PowerBuild version installed.

FTP server: download.appeon.com
Username: rudolf.kerbitz@copasysteme.de
Password: It will be sent in a separate email. 
 
We suggest that you use an FTP Tool (like the free FileZilla client) to connect to the FTP using the settings in comment 13.

Regards,
Mark Lee
Mark Lee @Appeon 27 May, 2020
#13
ftp settings
Mark Lee @Appeon 23 April, 2020
#12
Hello Rudolf,

Thanks for the reminder.
We think it would be better if we could provide a new standalone beta PB version for you by the end of May. The beta version now is not stable and easy to crash because of the new feature functionalities added.
And we just released R2 GA, so please wait for another several weeks until we test and think the beta version is stable enough. 
Thanks for your understanding and patience.

Regards,
Mark Lee
Rudolf Alexander Kerbitz 21 April, 2020
#11
(In reply to Mark Lee @Appeon from comment #10)
Dear Rudolf,

Thanks for the quick update. We really appreciate your patience and
understanding!
After we release PB 2019 R2 GA, our development team will have enough time
to fix the bug and build a new standalone beta PB version for you.
And then we can send you the download link maybe in April or May.
Thanks for your understanding again.

Regards,
Mark Lee
Hello Mark, since April is drawing to an end it's likely that you are now able to refine your estimate about when you will be able to provide us with the standalone beta. Is there any update on the progress that you would like to share with us? Best regards, Rudolf Alexander Kerbitz
Mark Lee @Appeon 15 January, 2020
#10
Dear Rudolf,

Thanks for the quick update. We really appreciate your patience and understanding!
After we release PB 2019 R2 GA, our development team will have enough time to fix the bug and build a new standalone beta PB version for you.
And then we can send you the download link maybe in April or May.
Thanks for your understanding again.

Regards,
Mark Lee
Rudolf Alexander Kerbitz 14 January, 2020
#9
Dear Mark,
 
we were glad to read that your development team successfully fixed the issue for which we provided the test case. We understand that fixing all of this issue requires time, after all, at least seven modules need to be revised. Naturally this is not done by solely replacing the mentioned accessor functions. It's clear that also the related code needs to be adjusted too and in doing so it's likely that your team will spot additional places in the code in which pointers are accidentally being truncated.

Due to this extensive impact we would like to discuss the opportunity to provide our assistance for quality assurance by testing new builds containing this bugfix with our applications under NDA.

Best regards
Rudolf Alexander Kerbitz
Mark Lee @Appeon 07 January, 2020
#8
Dear Rudolf,

Thanks for your update.
Our developer team made a deeper analysis of it and we confirm that it works well in your test case when we use the GetWindowLongPtr instead of the GetWindowLong function in the PB internal code.
However, if we fix this issue, that means we need to change many DLLs and this will also refer to many code.
So in this case, in order to keep quality, we need more time to test it. According to our developer team’s schedule, it will be fixed in the PB 2019 R3 version but the release date is not decided yet.
I appreciate your understanding.

Regards,
Mark Lee
Rudolf Alexander Kerbitz 06 January, 2020
#7
Dear Appeon Support, 

due to this issue 64-bit builds of our products cannot work reliably. 

We were forced to suspend shipping this version of our products and 
have to appease our customers to be patient and use the 32-bit build 
until further notice. 

So far approximately one month has passed without any progress
and the bug - which is critical to us - is not even scheduled yet.

With all due respect, when are you going to take action?

Yours sincerely 
Rudolf Alexander Kerbitz
Mark Lee @Appeon 09 December, 2019
#6
Hi Rudolf,

Thanks for your information.
And we'll transfer it to our development team for consideration. 

Regards,
Mark Lee
Rudolf Alexander Kerbitz 06 December, 2019
#5
pardon the typo, the sentence below was supposed to read "[...] the following library-files of the 2017 and 2019 runtime also import Get/SetWindowLong[...]"
Rudolf Alexander Kerbitz 06 December, 2019
#4
Thank you for your quick response, we appreciate your efforts.

In addition it is worth mentioning that besides pbdwe1xx.dll the following library-files of the 2017 and 2019 runtime also import Get/SetWindowLongPtr and likewise should be adjusted in order to rule out that they do not cause random crashes:

pbdpl170.dll
pbodb170.dll
pbrtc170.dll
pbshr170.dll
pbUIS170.dll
pbvm170.dll

------------

pbvm190.dll
pbdpl190.dll
pbodb190.dll
pbrtc190.dll
pbshr190.dll
pbtheme190.dll
pbUIS190.dll

Best regards
Rudolf Alexander Kerbitz
Mark Lee @Appeon 06 December, 2019
#3
Hi Rudolf,

Thanks for report this problem.
We will escalate this problem to our development team for further analysis.
We will need some time to figure it out, and we will get back to you if any progress we would make.

BTW, we've found that PB 12.6 has the same behavior as well.

Regards,
Mark Lee
Chris Pollach @Appeon 05 December, 2019
#2
Hi Rudolf;

   Thank you for bringing this issue to Appeon's attention. I was able to recreate this issue in both PB2017R3 and PB2019 running on W10 build 18363. 

   We will need some time to figure out this issue and will get back to you as soon as we know more.

Regards ... Chris
Rudolf Alexander Kerbitz 05 December, 2019
#1
pbdwe1xx.dll 64Bit truncates addresses causing crashes.zip (306KB)

*Phenomenon:
64Bit-Application crashes due to access violation caused by dereferencing truncated addresses.
Applies to Powerbuilder version 17 and 19.
Modules that causes the error: pbdwe170.dll, pbdwe190.dll 

*Reproduce Steps:
We procuded a minimum code example for reproducing the phenomenon in folder PBCrash2
1. Start PBCrash2\pbcrash.exe
2. Attach debugger to the running process (optional)
3. Click "Allocate" Button
4. Click "Open" Button
5. SEH-Exception: Access violation is thrown in PBDWE1xx.DLL!dwWinProcStatic

Remarks:
Phenomenon occurs on high memory pressure, i.e. when allocated memory exceeds 32bit address space.
During WM_CREATE message the lpParam argument passed via the LPCREATESTRUCT to the window 
with class name "pbdwst170" is truncated to 32Bit. Subsequent events that require accessing the data
of the window cause an access violation (see WindowEvents.log for details).
Although the address is already truncated at creation
PBDWE1xx.DLL!dwWinProcStatic would truncate the address too because it uses GetWindowLongW for 
accessing the data (see "callstack screenshot.png").

For compatibility with 64Bit-Applications one should use Get/SetWindowLongPtr instead of Get/SetWindowLong.
see: https://docs.microsoft.com/en-us/windows/win32/api/winuser/nf-winuser-getwindowlongptrw

Please assure to replace all references to Get/SetWindowLong with Get/SetWindowLongPtr
OS:
All
Platform:
64-bit
Database Type:
not related to database
Database Version: