Bug 4479

memory leak 14 October, 2020

S&F Datentechnik
22 April, 2020
Product: PowerBuilder Category: PowerBuilder IDE
Version: 2019 R2 Build:
Classification: Sybase (legacy) bug Publishing: Public
Priority: P2
Status: Scheduling Reason:
S&F Datentechnik 14 October, 2020
#25
PB2019_19.1.0.2328.zip (964KB)

Hi Mark,
attached is a file with the results of some testruns of the genapp and the genapp itself.
The results are from tests with the "singletest" option enabled. 
You should be able to reproduce the results with the genapp on your side.
The used PB Version is 19.0.1.2328 on Windows 10.

With PB 19.1.0.2353, it seems the Memory Problem is gone.
Did you change something to specifically alter the Memory usage or was that unintentionally fixed when you changed something else?

Regards,
Gerrit
Mark Lee @Appeon 27 September, 2020
#24
Hi Gerrit,

Sorry for the late reply. Currently, it’s quite difficult for our development team to handle this issue.
 
1.May I ask if we can have a remote session to debug this issue?
We would recommend you make the remote session during our working time which is 9:00 ~ 17:00 (we are in UTC+8 ) thus we can get immediate help from the other team if necessary. Please let us know when you can make the session and which tool you'd like to use for this session (TeamViewer or GoToMeeting).
 
2.If it not convenient to have this remote session, can you provide a video to demonstrate this memory leak issue for us?

Regards,
Mark Lee
S&F Datentechnik 16 July, 2020
#23
Hi Mark,
we got the results regarding PB 2019 R2,  PB 12.6.4058 and PB 2019 on the same machine (Windows 10).
We did also some testing on another machine running Windows 8.1. The results where a little different, but the problem with the increasing memory was the same.

After some additional testing it seems the Memory usage is strange.
It seems the increased use of Memory stops at some point. 
Which means open and closing the same window leads to an increasing use of Memory until it reaches a maximum. 
After that it is not increasing while you use the same window. 
It seems the increasing Memory stops after a few 100 times opening the same window.
Open another type of window leads again to an increasing use of Memory everytime until that also reaches a limit, and so on.

It looks like PB is alloccating Memory each time a visual object is created, which is normal.
After closing/destroying visual objects, the Memory is not relesead, which is understandable, since it can be used the next time more Memory is needed without the need of allocating additional Memory.
But after that there is something that is not like it should be:
Following Scenario:
1. An app opened a window.
2. The used Memory is increasing
3. Close the opened window, the used Memory is the same as if the window is opened
4. Now open the same window again, the used Memory is increasing again. This should not happen, because there should be enough allocated and free Memory available, since the app didn't release the Memory it used after Opening the window the first time.
5. Close the window again and the used Memory stays the same as in 4.
6. Repeat 4. to 5. a few hundred times and the Memory is increasing each time the window is opened.
7. After a few hundred times it is not increasing anymore and stays the same every time the window is opened.
8. Open another window-type and the memory is increasing again

The occupied memory is never released again as long as the app is running. 
It seems it is only partially reused when the same visual object type is used again after a few 100 times. 
This leads to an increasing use of memory over time especially when many different visual-objects are used.
That, in addition to the overall higher memory usage in PB since PB 2019 R2 leads to problems in larger applications, like ours.

It is understandable that occupied memory is not released directly after the objects, which used it, are destroyed.
PB should reuse that memory directly, since it is already occupied but not used, but it is only doing it for the same object-type after it already allocated way more memory for that type than it needs for that specific object-type.
Since garbageCollect () doesn't free memory which was used for visual objects, that memory is lost for the lifetime of the app.

Btw., it seems it is reusing GDI-Objects. They are only increasing when more GDI-Objects are used than before.
Opening the same or another visual object-type (which does not need more GDI-Objects) doesn't increase it every time.

Regards,
Gerrit
Mark Lee @Appeon 15 July, 2020
#22
Hi Marco,

Just to let you know, for the memory leak issue, our development team is still analyzing it.
 
We have done some testing with several machines of the same Windows version. The growth of the memory is not obvious in some machines but not in the rest of the machines. But currently, we are still not sure if this issue is related to the Windows version.
 
One more question. You mentioned that the issue only occurs in PB 2019 R2 but not PB 12.6.4058 and PB 2019. Do you get this result by testing on the same machine?
 
We still need more time to analyze this issue and will keep you updated. Thanks for your understanding!

Regards,
Mark Lee
Mark Lee @Appeon 10 July, 2020
#21
Hi Marco,

I checked with our development team about the status, however, due to some reasons, we can’t get back to you right now but will update you at least until next Wednesday.
Thanks for your understanding.

Regards,
Mark Lee
S&F Datentechnik 09 July, 2020
#20
Hi,

whats the status here? When will a fix be available?
!!! We still need this bug to be fixed ASAP !!!

Regards,
Marco Diers
S&F Datentechnik 15 June, 2020
#19
Hi,

Ticket moved from private to public.

It is not possible for us to use this runtime because of this bug. We think that that no other customer can use this runtime which has also a "fat" app like our apps.
Is there any other Appeon customer which can't also use this runtime for production?

!!! We still need this bug to be fixed ASAP !!!

Regards,
Marco Diers
Chris Pollach @Appeon 04 June, 2020
#18
Hi Mark;

  Its both the IDE and the PB run-time that have the memory leaks. Its been reported by many customers over the past while. Especially, starting in the PB 12.x era code line. You used to be able to force a free memory when you used the SEND() command to send a Minimize/Maximize to the PB App's MDI Frame but, that trick no longer works in Appeon PB releases. No memory is released in PB 2019/2019R2 using that old approach.

  There was a small increase in resource consumption in PB 2017 but then a definite major increase in PB 2019 and now again some more in PB 2019 R2 from my observations. The changes to the STD framework in PB 2019 GA allowed me to track these areas and so I shared them with you to pass over to engineering. I have "tweaked" this resource tracking in my STD Framework (beta) for 2019R2.

  App crashing will occur under two scenarios: A) App that runs consistently performing endless processing or B) An App run from the IDE over and over again (ie: unit testing), where the IDE never releases all App memory after it gets back control. Each time you run from the IDE, the App's available address space memory continues downward (as per my screen capture attachment).

  Under W10 with PB 2019 R2 in a 4G address space, I find on my W10 test PC's that only 2.1G of free App memory is available when running from the IDE (1st time). Each time you run this available memory quickly drops to 1.5G and then lower on slower scale. However after many App runs, the free memory in the IDE's address space drops very low. I suspect then that a PB APP and/or IDE crashing scenario  just becomes inevitable. Note: Restarting the IDE every once in a while completely releases all PB IDE memory back to the O/S and upon restart, your back to a free 2.1G.

  I am not sure what else to tell you other than if the Engineering Team can run tests with my STD Demo App or with customer App's like Gerrit's and monitor what is going on within the PBVM - tha might expose some areas of memory optimization code changes.

Note1: Found out under PB2019/2019R2 that if a PB App issues multiple GarbageCollect() commands in a relatively short order (ie: Timer) - that the PB App will randomly crash.

Note2: Issuing a GarbageCollect() under PB2019R2 in my STD Demo App actually shows the "available" memory going down - and *not* up (as I would have expected).

HTH
Regards ... Chris
Mark Lee @Appeon 04 June, 2020
#17
Hi Gerrit,
 
Thanks for your confirmation.
Please let us know how to test the genapp so that the memory will go up to 50MB, 100MB, or 1.2GiB. I used the Task Manager to monitor the genapp, and found that the memory is always under 10MB, even though we spent more than five hours performing 100000 test-runs.
 
Hi Chris,
 
Thanks for your feedback and for providing the test case.
Can you elaborate on how to reproduce the memory leak or crash step by step?
I believe maybe what you and Gerrit said is right.  But I also need to prove it and let our development team believe it, so they could do what they need to do it.
Thanks for your understanding.
 
Regards,
Mark Lee
Chris Pollach @Appeon 03 June, 2020
#16
Hi Mark / Gerrit;

  FYI: Since PB 2019 GA I have changed my STD Framework to track Application, Device and Paging memory used as well as overall DataWindow performance. The Framework in its newer PB 2019 build 2170 and latest R2 build 2328 allows you to also graph your application's usage of the above. I have attached a screen capture of a test that I ran today using R2 build 2328.

  I have noticed in my STD Framework as it is moving from PB 2017 to PB 2019R2 that the App's resources were only partially released in 2017 whereas in PB 2019 MR 2170 through R2 build 2328 that virtually no resources are being released until of course, the App completely closes. 

  The PB IDE as well is being affected by this change in memory management as each time you perform a RUN from the IDE, the App's "usable" memory diminishes. So after many test runs from the IDE, your PB App will get into an unusable state and possibly crash like Gerrit states.

  The latest *beta* STD framework has been changed to send the PB App user W10 notifications if the above resources get too low (a threshold you can set in the INI file). You can see this notification in today's test in the screen capture attachment from the LOG image that I also added to the screen capture.

  So I would concur with Gerrit that "used" memory always seems to creep upwards or in my case (same thing) "available" memory always decreases over time (or # runs from the IDE).

  If you would like see the above STD framework in action around tracking these resources, please download the "Demo" OrderEntry App from here:

Build 2170:  https://sourceforge.net/projects/stdfndclass/files/Applications/PowerBuilder/OrderEntry/

Build 2328(beta):  https://sourceforge.net/projects/stdfndclass/files/Applications/PowerBuilder/OrderEntry/Beta/PB2019R2/


HTH
Regards ... Chris
Chris Pollach @Appeon 03 June, 2020
#15
PB2019R2 Memory Test (by Chris)
S&F Datentechnik 03 June, 2020
#14
Hi Mark,

to 1. like you mentioned it is the same problem with 2328.

to 2. the previous Reply

Regards,
Gerrit
S&F Datentechnik 03 June, 2020
#13
Hi Mark,

1. I will try it.

2. the used Memory in the genapp isn't going up to 1.2 GiB with that amount of test, thats right.
But it is increasing, that's the problem. The genapp is only a very little app with only a few controls.
That's why it isn't going up to 1.2 GiB. If you run it long enough it will go there.
The genapp only shows that there is a problem with increasing Memory where it should not increase.
In PB 2019 R2 the Memory is going up all the time (except nonvisual), with PB 2019 it is not going up all the time.

Our main applications are far bigger than the genapp and THEY reach the 1.2 GiB with PB 2019 R2 and don't with PB 2019. Which leads to crashes.

Regards,
Gerrit
Mark Lee @Appeon 03 June, 2020
#12
Hi Gerrit,

Thanks for your quick feedback.
1. Can you upgrade your PB 2019 R2 Build 2323 to PB 2019 R2 Build 2328 first and then try and see if it works?
2. I have tested the case you provided on our side more than 20 times but the memory it used never went beyond 1,2 GiB (I tested using PB 2019 R2 Build 2328 in Win10 1909).
Even though we performed 10000 test-runs in the genapp, the test results are similar.
And in fact, you can also check the test result (protokoll 10000.csv) which you provided before, the memory only increases from 24 to 29, not very obvious. So it also confuses me. And my test results are the same as yours.

Regards,
Mark Lee
S&F Datentechnik 02 June, 2020
#11
Hi Mark,
the problem is the increasing use of Memory, not the GDI Objects.
As you can see in the genapp, the GDI Objects only increase a little and then stay the same the whole time the app is running.
The increasing memory over time is the main problem. Since the genapp is doing the same things all the time, the used Memory should not increase. But it does, even so the genapp is using GarbageCollect() after each test.
There is a far greater problem with Memory usage in our applications, since they use a lot more objects.
In our Tests we have a Memory problem with the PB 2019 R2, which leads to crashes because the used Memory went beyond 1,2 GiB. There is no problem with PB 2019 and the same applications.

The second problem is the instability caused by using GarbaceCollect(), which Chris Pollach mentioned in the first Reply to this ticket. This may be the cause of some crashes, where we were aunable to locate the cause. And again, there is no problem with PB 2019.

Regards,
Gerrit
Mark Lee @Appeon 01 June, 2020
#10
Hi Marco,

Just to let you know that after our development team further analyzed this issue, they think we cannot to fix it by now.
When you start your test app, the GDI will increase almost 8-10, the reason is that the Global GDI source will be created and this will occupy the resources.
And these GDI resources will not be released until the app is closed. 
The benefit is that when it loops next time, the second or third time it spends is always less than the first time.  

Regards,
Mark Lee
Mark Lee @Appeon 21 May, 2020
#9
Hi Marco,

Thanks for your quick update.
I checked your PB 12.6 test result and found that the GDI number doesn’t increase so much (from 85 to 87), but the Memory increases from 15 to 55.  
However, the memory leak issue is not only a important but also a difficult problem. 
We fully understand your situation. And we've submitted this issue to our development team. 
They will determine whether they can fix it based on their analysis of the issue and their schedule.
Currently, our development team has further analyzed this issue, but they don't have any solutions yet.
We will update you on the status next week about the progress.
Thanks for your patience and understanding.

Regards,
Mark Lee
S&F Datentechnik 20 May, 2020
#8
PB126.7z (13KB)

Hi Mark,

hmm ... that's not the expected info. You wrote on April 30 "... will update this next week". We thought we will get the fix then. Maybe you only meant to update the ticket status!?!

Now you said "We will need some time to figure it...". What does it mean in more detail? Please give us a timeframe when we will **get the fix**. As already written we need it ASAP.

BTW: Our test show's that there's no memory leak in PB 12.6.4058. See our attached test results. But it's very independently from that! We need a solution in PB 2019 R2 ASAP because this issue is a k.o. criteria to update to PB 2019 R2.

Regards,
Marco Diers
Mark Lee @Appeon 20 May, 2020
#7
Hi Marco & Christian,

Sorry for the late reply.
Thanks for providing the excellent test case.
We will escalate this problem to our development team for further analysis.
We will need some time to figure it out and will get back to you if any progress we would make.
 
BTW, we found that PB 12.6 has the same behavior as well.
And for the performance issue, we will always treat it as a high priority.

Regards,
Mark Lee
Mark Lee @Appeon 30 April, 2020
#6
Hi Christian,
 
We understand your situation and will update this next week.

Regards,
Mark Lee
S&F Datentechnik 30 April, 2020
#5
Hi,Mark Lee,

please give us a timeframe when a fix will be available. We need it ASAP!
Because of this issue it's not possible for us to update to PB 2019 R2.
But we need to update e. g. to support Windows Server 2019 for our customers.

Regards,
Marco Diers
S&F Datentechnik 30 April, 2020
#4
Please push this to prio 1. We cannot use it with this bug.

Regards, Christian
Mark Lee @Appeon 23 April, 2020
#3
Hi,

Thanks for reporting this problem.
We will work on this issue and get back to you soon.
Thanks for your patience and understanding.

Regards,
Mark Lee
Chris Pollach @Appeon 22 April, 2020
#2
Hi Bug;

  Thank you for that excellent Test Case App!

  FWIW: I just added this type of memory tracking to my latest framework. It can now even give you a graphic output as your App executes. 

FYI:  http://chrispollach.blogspot.com/2020/04/std-web-service-framework-2020r1.html

     Check out the memory graph. Even after closes, destroys and a GarbageCollect about 3/4 of the way through my App. Available memory never goes back up.


In my PB2019 MR2170 and R2 testing this week, I have noticed similar issues ...

1) When you run any App from the PB2019Rx IDE - the best usable memory is 2.167G out of 4G address space in W10.

2) As Windows are closed and NVUO's destroyed - the PBVM does not recover the memory

3) The more you run an App from the IDE, the lower the available address space memory is each run you re-run. For example, Run#1 - 2.167G, Run#2 - 1.98G, Run#3 - 1.84G, Run#4 - 1.67G ... etc. the only way to free-up available memory was to recycle the complete PB2019Rx IDE.

4) If your App issues a GarbageCollect() command - not only does that not clean up closed windows and destroyed NVUOs, etc memory ... the memory available even falls further down(????) not up as I expected.

5) If an App issues repeated GarbageCollect() commands within a short period of time, the App will become unstable. Usually ending in an uncontrolled crash.


  I will now transfer this ticket over to the main Support / Engineering team to review how memory GDI and NVUO is managed by the PBVM. There certainly seems to be issues in this process.

Regards ... Chris
S&F Datentechnik 22 April, 2020
#1
Genapp.zip (247KB)

*Phenomenon:
There is a Memory leak in PB 19.1.0.2323.
PB is not releasing all Memory after a visual object is destroyed at runtime.

*Reproduce Steps:
See attached genapp.
It also has a Protokoll 10000.csv with the result of 10000 test-runs of the genapp in a row.

Remarks:
OS:
Windows 10
Platform:
All
Database Type:
Database Version: