Request your free trial!


Fill out the two fields below and we will contact you to schedule your free trial.

Thank you! We will contact you to schedule your trial.

Geplaatst: 7 August 2018

ADF Performance Tuning: A Field Report

Last week I was doing an extensive performance analysis / health check on a large ADF project, with the newest version of our ADF Performance Monitor product. In this performance assessment/analysis I have focused high-level on the most important performance bottlenecks. We could see in the ADF Performance Monitor that end-users experience very slow page load times, they were waiting much more than needed. This ADF application needed attention; it could run more efficient like nearly all ADF applications can. In this blog I describe some of my findings, maybe interesting for other ADF projects as well.

Complete overview

The first thing I always do is configuring the ADF Performance Monitor on all WebLogic managed servers (in this case 4) to have a complete overview of the performance:

In this case a typical daily performance summary was (top left section):

What already is strange here is that the AVG total time end-users needs to wait (0,57 Sec) is more than double the time the AVG process time by the application server (0,25 Sec)!

Problem 1: Very Slow Browser Load Time

On the chart at the right bottom we can see the explanation for this. In this chart we see in a glance in which layer processing time has been spent; database (yellow), webservice (pink), application server (blue), network (purple), and browser load time (grey).

More than one third of the time spent in is grey, meaning that more than one third of the process time is spent in the browser! This is the time spent by the browser, after receiving the response from the server to build the DOM-tree, and rendering/loading the content. Also, as we can see the purple color representing the time spent in the network (HTTP request network time, HTTP response network time) is relatively high: around 1/6th of a request on average. This is far more compared to other ADF projects.

This is the biggest ‘bottleneck’. It turned out that Explorer 11 was the current (and the only installed browser) of all their end-users, installed on its Citrix workstations. Web applications in general, and ADF applications in particular, perform very poor in Internet Explorer (regardless of the version). This is because of the inefficient JavaScript engine and the very slow browser load time. My first recommendation: install and use Google Chrome or Firefox as web browser as these have a very fast browser load time, and very performance friendly browsers. My estimation is that this will improve the performance in general with at least 25% or more for this project. Read here more on browser load time in ADF apps. In generally it is a good practice to stimulate end-users to install and use use Google Chrome or Firefox for ADF applications.

Click Actions Analysis

The next analysis was an ADF click action analysis. A click action is the start trigger event of an HTTP request by the browser, by an action that a user takes within the UI. These are most often physical clicks of end-users on UI elements such as buttons, links, icons, charts, and tabs. But it can also be scrolling and selection events on tables, rendering of charts, polling events, auto-submits of input fields and much more. With monitoring by click action you get insight in the click actions that have the worst performance.

I go very frequently to this overview to see what click action has the worst performance (is responsible for the most total processing time, and thus where we can win the most in terms of performance):

We see here that a poll event (ADF Faces component of type oracle.adf.RichPoll, with id ‘p1′) is responsible by far for the most total processing time (!). On this day there were in total 106.855 poll requests. That is more than one third of all the HTTP requests (296.435)!

Problem 2: Far Too Often Polling

There was a mechanism implemented in the application to force an end-user to be logged in at maximum one time. The way this was implemented was very bad for the server load; every minute a poll (HTTP request) was send to the WebLogic server that called Java code that updated a database table. It had also a side-effect that many end-user sessions were kept alive on the server for many hours (even for the many inactive users that never closed their browser window). The poll was responsible for the most time-consuming action in the application in terms of serving processing time. For now, as we couldn’t change this whole functionality quickly, we reduced the number of calls to three times less (one third now); we kept the same polling mechanism but now every three minutes (to avoid 2/3 of all the polling and to reduce the server load as well). Of course, later we should find an alternative solution.

We saw that the poll caused many very slow HTTP requests that included very slow database queries, frequent expensive ApplicationModule pooling, and other slow executions because it was restoring pages after passivation. It was responsible for 1/3 of all the processing time of the most frequent actions:

Problem 3: Memory Overconsumption

The third – a typical bottleneck in ADF – was an increase in response time (and decline in performance) because of the huge memory usage. The cause of this huge memory usage is that the application data which is retrieved from the database into memory is not properly limited; too many rows (thousands). To make matters worse, these rows and their attributes were retained in the session for an unnecessary period of time (by very frequent expensive ApplicationModule pooling). We can see in the ADFBC Memory Analyzer the total number of rows fetched by ViewObjects at runtime, and the maximum fetched rows. In this case we saw many ViewObjects fetching thousands of rows during an HTTP request:

The solution of this main problem is found in reducing the size of sessions by decreasing of the amount of data loaded and held in the session (setting maximum fetchsizes, adding bind params, fixing ViewCriterias). We have already identified all the locations in the source code and solved the most important of this list. Read more on this subject here.

Problem 4: Too Frequent ‘Expensive’ ApplicationModule Passivations & Activations.

As you know ApplicationModule pooling is a mechanism in ADF that enables multiple users to share several application module instances. It involves saving and retrieving session state data from the database or file. This mechanism is provided to make the application scalable and becomes very important under high load with many concurrent users. The default values of ApplicationModule pools are far too small; especially if you have more than 10 end-users.

I think this is one of the most important things to ‘tune’ in general in ADF applications. Activations and passivations are the root cause of many very slow click actions. It is the root cause of errors after incomplete activations. In general, in my opinion it is better to try to turn off the whole ApplicationModule pooling mechanism – for so far as it is possible.

To do this we increased the size of all the ApplicationModule pools. In this way we make the application more scalable and avoid very expensive passivations and activations. We increased the following parameters – depending of the usage.

Further:

If you want to know more on ApplicationModule pools and tuning watch the video I made on ADF Performance tuning a few years ago (a big part of the video is on pooling parameters).

Problem 5: Too many UIShell Tabs Could be Opened Simultaneously

This was a UIShell application. To avoid resource (but also memory) overconsumption we reduced the maximum number of opened tabs from 10 to 5. This will reduce the resource and memory consumption of the server as well and force the end-user to close unused tabs (and free up resources).

Conclusion

We have found many other bottlenecks as well. But already addressing/resolving these 5 big bottlenecks we have put already a smile on the face of many end-users!


Tags: , , , , , , ,

Geplaatst: 9 July 2018

ADF Performance Monitor: Monitoring with Percentiles

What is best metric in performance monitoring – averages or percentiles? Statistically speaking there are many methods to determine just how good of an overall experience your application is providing. Averages are used widely. They are easy to understand and calculate – however they can be misleading.

This blog is on percentiles. Percentiles are part of our recent new 7.0 version of the ADF Performance Monitor. I will explain what percentiles are and how they can be used to understand your ADF application performance better. Percentiles, when compared with averages, tell us how consistent our application response times are. Percentiles make good approximations and can be used for trend analysis, SLA agreement monitoring and daily to evaluate/troubleshoot the performance. (Lees meer..)


Tags: , , , , , , , , ,

Geplaatst: 21 May 2018

ADF Performance Tuning: Manage Your Fetched Data

In this blog I want to stress how important it is to manage the data that you fetch and load into your ADF application. I blogged on this subject earlier. It is still underestimated in my opinion. Recently I was involved in troubleshooting the performance in two different ADF projects. They had one thing in common: their servers became frequently unavailable, and they fetched far too many rows from the database. This will likely lead to memory over-consumption, ‘stop the world’ garbage collections that can run far too long, a much slower application, or in the worst case even servers that run into an OutOfMemoryError and become unavailable.

Developing a plan to manage and monitor fetched data during the whole lifetime of your ADF application is an absolute must. Keeping your sessions small is indispensable to your performance success. This blog shows a few examples of what can happen if you do not do that.

(Lees meer..)


Tags: , , , , , ,

Geplaatst: 23 April 2018

ADF Performance Monitor – Major New Version 7.0

We are very happy to announce that a major new version 7.0 of the ADF Performance Monitor will be available from May 2018. There are many improvements and major new features. This blog describes one of the new features; on usage statistics and performance metrics of end-user click actions.

A click action is the start trigger event of an HTTP request by the browser, by an action that a user takes within the UI. These are most often physical clicks of end-users on UI elements such as buttons, links, icons, charts, and tabs. But it can also be scrolling and selection events on tables, rendering of charts, polling events, auto-submits of input fields and much more. With monitoring by click action you get insight in the click actions that have the worst performance, that cause most errors, that are used most frequently, e.g. You can see in which layer (database, webservice, application server, network, browser) the total execution time has been spent. You can SLA monitor the business functions that are behind the click actions – from the perspective of the end-user. (Lees meer..)


Tags: , , , , ,

Geplaatst: 18 February 2018

ADF Performance Tuning: Avoid a Long Browser Load Time

It is not always easy to troubleshoot ADF performance problems – it is often complicated. Many parts needs to be measured, analyzed and considered. While looking for performance problems at the usual suspects (ADF application, database, network), the real problem can also be found in the often overlooked browser load time. The browser load time is just an important part of the HTTP request and response handling as is the time spent in the applicationserver, database and network. The browser load time can take a few seconds extra time on top of the server and network process time before the end-user receives the HTTP response and can continue with his work. Especially if the browser needs to build a very very ‘rich’ ADF page – the browser needs to build and process the very large DOM-tree. The end-user needs to wait then for seconds, even in modern browsers as Google Chrome, Firefox and Microsoft Edge. Often this is caused by a ‘bad’ page design where too much ADF components are rendered and displayed at the same time; too many table columns and rows, but also too many other components can cause a slow browser load time. This blog shows an example, analyses the browser load time in the ADF Performance Monitor, and suggest simple page design considerations to prevent a large browser load time.

(Lees meer..)


Tags: , ,

Geplaatst: 23 October 2017

ADFPM is a great product that’s helped our group tremendously in improving the performance of our application.

The ADF Performance Monitor is a great product that’s helped our group tremendously in improving the performance of our application. It has especially helped identify better practices and consistency in the way we code.
Kenton Ho, President at Perfit – Computer Systems Group




Geplaatst:

The visual user interface of the Performance Monitor makes it easy for developers to spot performance bottlenecks

The AMIS ADF Performance Monitor product is an excellent tool for collecting and aggregating fine    grained information about ADF application performance during development, testing and in   production. The visual user interface of the Performance Monitor makes it easy for developers to spot performance bottlenecks in the view, the controller and the business component layer.
Frank Nimphius, Senior Principal Product Manager at Oracle




Geplaatst: 17 March 2017

Measuring Network Time to and from the Browser

We added a great new feature to the ADF Performance Monitor: network and browser load time information. Now you know exactly every end-user experience of your ADF application, in real-time. You can quickly resolve any performance bottlenecks with this end-to-end visibility. You can even drill down into an individual user to analyze the experience – to understand the ADF application behavior. The dashboard is improved with several overview and detail graphs that shows the layer (database, webservice, application server, network, and browser load-time) where the time is spent of your application. This is very useful to troubleshoot problems. This blog is about network time. This blog is on browser load time.

The ADF Performance Monitor is an advanced tool specifically build for ADF applications and is aware of the intricacies of the ADF framework. It traces key ADF actions across tiers and services to provide end-to-end visibility and automatically maps each tier to easily visualize the relationship between them. This Tracing provides deep visibility into the cause of application performance issues down to the tiniest detail.  (Lees meer..)




Geplaatst: 28 February 2017

Measuring Slow Network Time Between Application Server and Database

Recently I was implementing the ADF Performance Monitor at a customer site and doing a detailed performance analysis. This customer had severe (and less severe) performance problems but was unable to pinpoint the exact pain points. Frequently, end-users were complaining because they were experiencing first a very good performance (response times less than a second), and then suddenly experiencing a drop in performance for a certain period of time (an hour or so) with response times between five to twenty seconds.

In this particular case, at the hour overview of the ADF Performance Monitor (from 14:00 to 15:00) we can see the sudden start – and end of this drop in performance. In the top right section HTTP response times can be viewed over the selected time range. This graph makes visible when the load is high (and how high), how the response times are distributed over the categories (very slow, slow and normal) and when there are performance problems (more red and yellow colored parts of the bars).Visible is that from 14:10 to 14:40 the response times have increased significantly. This should be a trigger to drill down to this period in the monitor – for further analysis in order to find the root cause.

 

We can drill down from an hour to a 5 minute range (14:25 – 14:30) by clicking on the graph in the menu. Now we see an an overview of all individual HTTP request in this 5 minute range on the top graph – with time in WebLogic (blue) and time in database (yellow). We see many long response times between 5 and 30 seconds. We see that most is consumed in WebLogic but we still don’t know where the time is spent exactly. Also we can see from that the JVM (bottom graph) appears to be ‘happy’; the JVM can easily manage the load so the JVM is not the problem in this case:

 

The next step is to drill down to so called ‘ADF request call stacks’ in this time range. An ADF request call stack of the ADF Performance Monitor gives visibility into which ADF method caused other methods to execute, organized by the sequence of their execution. A complete breakdown of the HTTP request is shown by actions in the ADF framework (lifecycle phases, model and ADF BC executions, ApplicationModule pooling, ViewObject query executions,  time to fetch database rows into the ADF app, e.g.), with elapsed times and a view of what happened when. Call stacks are a very useful help to understand specific situations, it shows bottlenecks and where we can avoid bad ADF practices and write more efficient ADF code. The parts of the ADF Request that consume a lot of time are highlighted and indicated with an alert signal.

 

After investigating many ADF request callstacks, we we able to detect that the time that was needed to fetch database records by ADF ViewObjects was extremely long. For example, the time shown in this callstack (see image above) to fetch only 173 records into the ViewRowCache of this ViewObject took more than 5,3 seconds and later in the same HTTP request 6,2 seconds (!). And that while the ViewObject query time – method executeQueryForCollection() – took only around 50 milliseconds. Normally, ViewObjects should be able to fetch this number of records in easily only a few milliseconds. ADF always executes this in the following order: first the ViewObject query is executed by the executeQueryForCollection() method. Then, when this method is completed, the database rows of this ViewObject are being fetched. This action and how long this takes is shown in the callstack as ‘Fetching, creating rows for <viewobject instance name>, fetched <xxx rows> ‘.

When the time to fetch rows takes long – and the number of rows fetched database rows is still relatively low – this is a very strong hint that during this request the network between the production WebLogic server and the database seemed to be the bottleneck. This can be the case when they are very far from each other and/or the internet connection is very slow. Or that the network is already very busy.

We knew there was a firewall between the database and the Weblogic server that definitely did not help the communication between these two. But, it turned out that the firewall was not the main problem. It was the periodically database batchjobs (unfortunately executed by the operation team during daytime) that were taking very very many network resources. See the following screenshot from the database Enterprise Manager with metrics from exactly the same time period:

 

We can clearly see in this picture too that from 14:10 to 14:40 there is many network traffic – caused by a costly database job that was executed by the operation team on that particular moment. The solution is to not execute these batch-jobs anymore during daytime :). Now end-users do not have to suffer anymore from these jobs that could be executed in the night as well.

 




Geplaatst: 15 February 2017

Detecting and Analyzing a High ADF BC Memory Consumption

Like other web applications, Oracle ADF applications potentially use a lot of JVM memory. Many times, the root cause of a high memory usage is that application data retrieved from the database into memory is not properly limited; hundreds or thousands of rows (with too many attributes) are fetched and held in ADF BC memory. This can lead to memory over-consumption, very long running JVM garbage collections, a freeze of all current requests or even OutOfMemoryErrors. To make matters worse, these rows and their attributes are frequently retained (passivated and activated) in the session for an unnecessary long period of time. The solution to this problem can be found in reducing the size of sessions by decreasing of the amount of data loaded and held in the session. With a low memory consumption, a more responsive, stable and scalable ADF application can be delivered.

This blog describes one of the new features of the ADF Performance Monitor; a kind of ADF BC memory recorder and analyzer. It detects and warns when too many rows are being fetched (from the database or webservice) and held in ADF BC memory. (Lees meer..)