Geplaatst: 25 August 2022

Frequently companies ask for our seven-day trial – to try out the capabilities of the ADF Performance Monitor on their ADF application. This trial is meant only for companies that are interested in purchasing an ADFPM license. Often, we have had trials where everything that can go wrong went wrong. With disappointing results. In this blog I will describe what frequently went wrong and how to prevent it.

(Lees meer..)

Geplaatst: 21 June 2022

We are happy to announce that we have a new whitepaper on the ADF Performance Monitor. This blog publishes a new whitepaper that gives more information about the architecture, features and implementation of the ADF Performance Monitor. It is updated with the many features of our new major version 9.5. Recently we also made also a quick introduction video on the product.

(Lees meer..)

Geplaatst: 29 March 2022

We have a new introduction video of the ADF Performance Monitor (3:40 minutes) ! It gives a quick introduction on the product.

Geplaatst: 29 September 2021

We have again a major new version of the ADF Performance Monitor available – version 9.5 ! We have added many new valuable features and improvements. Many overview screens have got a facelift and new charts. In several blogs I will write on them.

This blog is on one of those new features, automatic SLA and health KPI warnings. The monitor will automatically interpret the metrics and will show warnings if the ADF application is not meeting the configured SLA thresholds (KPIs). Or if configured JVM and system health thresholds are not met – like JVM garbage collection, JVM CPU load, system CPU load, OS memory, database, webservice, application server, network, and browser. From now on it will be even more fast and simple to interpret the metrics. You do not have to be a performance expert/engineer, the monitor will already show the (type of) problems!
(Lees meer..)

Geplaatst: 15 July 2020

Last week I blogged in part 1 on our major new version of the ADF Performance Monitor – version 9.0. It was about monitoring the CPU load of the JVM process and of the whole underlying operating system. It was also about the total used and free physical (RAM) memory of the whole system, and the Linux load averages that provides an excellent view on the system load.

This blog (part 2) describes more new features. The CPU execution time of individual HTTP requests and click actions is now available. “What request/click action in the application is responsible for burning that CPU ? ” That question you can now answer with the monitor. The monitor gives a clear indication how expensive certain HTTP requests and click actions are in terms of CPU cost. Further we added browser (user-agent) metrics for each request. We also improved the ADF callstacks (snapshot that gives visibility into which ADF method caused other methods to execute, organized by the sequence of their execution and execution times).
(Lees meer..)

Geplaatst: 8 July 2020

I’m very excited to announce that we have a major new version of the ADF Performance Monitor – version 9.0 !

We have added many valuable new features; new metrics that can detect and help explain poor performance, disruptions, hiccups, and help troubleshooting ADF applications. Like operating system metrics: the CPU usage of the ADF application, the total CPU usage of the whole underlying operating system, the total used and free physical (RAM) memory of the whole system, and the Linux load averages. A high CPU usage rate and memory usage may indicate a poorly tuned or designed application. Optimizing the application can lower CPU utilization. Generic APM tools have these kinds of metrics too in some way, but the combination of system metrics with ADF specific metrics of the ADF Performance Monitor makes it even more possible to relate performance problems.

Another reason to pay attention to system metrics is that nowadays more and more applications are deployed on the cloud. Very likely there will be shared virtual machines and resources (CPU, memory, network). Applications and processes could influence each other if frequently other processes have a very high usage of the available CPU or memory capacity.

This blog (part 1) describes the first part of these new features. Part 2 describes the CPU execution time of individual HTTP requests and click actions. It answers the question: “What request/click action in the application is responsible for burning that CPU ? (Lees meer..)

Geplaatst: 2 April 2020

Application errors are often hard to retrieve, or take a lot of time to resolve. When you are suffering from errors, and have a lack of clarity when errors happen, you would like to have useful error diagnostics for analysis.

The ADF Performance Monitor automatically captures detailed diagnostics for each and every error/exception occurrence. You can view your errors to see the highest priority issues your team should focus on. This blog shows the renewed error overview of our newest version of the ADF Performance Monitor – with real production metrics.

(Lees meer..)

Geplaatst: 19 March 2019

Recently I was analyzing and troubleshooting the performance of an ADF application. Much was already improved before I came. Due to a very recent new hardware/infrastructure environment, the server and database process time was nearly 50% faster after migration. In this blog I want to show you the impact it had on the total server process time of HTTP requests. Such a sudden improvement is visible in the ADF Performance Monitor in a glance, and in Week and Month trend analysis overviews. Maybe you need to investigate your hardware/infrastructure as well ,and consider an upgrade; if your hardware/infrastructure is relatively old, if your machines are full, or if virtualization software is not implemented efficiently.

(Lees meer..)

Geplaatst: 7 January 2019

Intris is the leading Belgian provider of freight forwarding, customs and warehousing management solutions. Headquartered in Antwerp, Intris provides its integrated software and cloud-based solutions to logistics services providers in Belgium and the Netherlands.

Ben Rombouts is Chief Operating Officer at Intris. Recently he has written a detailed review on the ADF Performance Monitor – a tool Intris uses for monitoring the performance of their large Oracle ADF application.

For what and how the ADF Performance Monitor is used

The ADF Performance Monitor is used within our development team as an extra quality check when building new functionalities. After developing the code, the developers carry out their test scenarios and check the results based on the metrics generated by the tool. With this, non-performing queries are instantly removed, and we get a better insight where we need to work on additional performance improvements.

Since our standard application consists of several modules, our customers do not use all functionalities in the same way, or equally frequently. That is why the tool is also used for many LIVE customers in production, considering the following parameters:

• Number of users
• Type of hardware that the application runs on
• Functionalities used by the customer

Our account managers use the data in different ways and base themselves mainly on the dashboard:

1 – Average Response Time

General information about the average response time to give a correct indication of the performance during steering committee meetings. Previously there was much more subjectivity here (type “Every action takes seconds in your application”). This has ensured that these discussions are over now and that we can focus on the real issues.

2 – Errors that are reported

These are split into effective technical errors and errors that are of a more functional nature. This also gives a good impression of the fact that some users make the same mistakes and so there is a need for additional training. The technical errors are made into issues that are passed on to the development team and, depending on the importance, included in new releases.

3 – Discussions about what the performance issues are related to

Since we enable the tool at different clients on different platforms, we can also compare this over the environments. For example, we can see that the database time is always a constant, but that there are variations in network and browser time. This can then be addressed to the system administrator of the customer.

4 – Click Actions

From the ADF Click Actions overview we also get very useful information about the specific use of our application:

• which actions last the longest?
• which actions are carried out most frequently?
• which users experience the most problems?

This makes it much more convenient to focus on the real problems and clearly report to the customer why we focus on certain matters, and why we give other things a lower priority.

5 – Addressing Technical Problems

At frequent intervals we also try to go through several environments with a senior developer to check more technical problems that can be improved in the application. Sometimes, for example, if we notice things at a customer where certain actions take longer and longer, so there is a problem in the queries. Other customers do not have any problems with this now, for example because they have less data, but in the future, they will not run into this type of problem because we can take them pro-actively from the application.

How the ADF Performance Monitor helped

1 – View things in an objective way

The tool mainly helped us to view things in an objective way. For example, some actions in the application can take quite a long time for an end-user but are only executed 2-3 times a week. If we put this in perspective in relation to actions that are carried out 100 times a day, it is already much clearer where you need to focus.

2 – Quickly Troubleshoot problems

When customers report certain errors via our support, we can consult logging much faster because we can see very quickly which actions were performed by which user at that specific moment.

3 – ADFBC Memory Overview

From the overview ADFBC memory overview you can quickly find out where there are any problems in queries. These are issues that are sometimes not noticed by customers, but where you can prevent problems in a proactive way.

4 – Objective Insight in Use of the Application

The tool also gave us a much clearer and more objective insight into the use of the application. This is rather a ‘side effect’ of using the tool, but it gives a quick and clear overview to prepare steering committees for reporting.

How the ADF Performance Monitor saved much time (and money)

To express this in time/money is quite difficult, but you can safely say that you can win a lot of time in the following areas:

• 50-60% time savings for researching performance-related issues. Because developers really get a very low-level insight into the framework, it is much easier to tackle performance issues and generate large performance profits with limited actions
• 5-10% time savings by incorporating extra quality checks in development cycle (time gains come from avoiding hotfixes)
• 20-30% time saving when investigating errors reported by customers
• Great time gain during steering committees to keep the subjectivity out of the discussions so that there can be focused on the actions that really matter to end users. For example, a manager can complain that he does a certain action 2 or 3 times a week which takes him about 5 minutes per action. If you put a workload of 1 day in improving this specific action you can decrease the time to 2 minutes, which is a 60% time gain for this specific action. This will save 6-9 minutes each week, but just for 1 person. Compare this to an action that 20 end users are executing each day for more than 100 times. They don’t complain about it, but with the same effort of 1 day workload we can improve this action with “only” 2 seconds. This will gain you more than 20.000 seconds or 333 minutes each week!! In this way you can put things in a broader context and convince customers where they will really gain time.

Read all our customer reviews on our reviews page.

Geplaatst: 1 October 2018

In this blog I will discuss the top 10 typical performance problems I see in general in Oracle ADF projects – and will discuss solutions.

Top 10 Typical Bottlenecks – Illustrated by ADF Callstacks

I will illustrate the top 10 typical bottlenecks with ‘ADF callstacks’ – a feature of the ADF Performance Monitor. An ADF callstack, a kind of snapshot of the ADF framework, gives visibility into which ADF methods caused other methods/operations to be executed, organized by the sequence of their execution. A complete breakdown of the HTTP request is shown by actions in the ADF framework (Fusion lifecycle phases, model (BindingContainer) and ADF BC executions, start & end of taskflows, e.g.), with elapsed times in milliseconds and a view of what happened when. The parts of the ADF Request that consume a lot of time are highlighted and indicated with an alert signal.

Nr 1: Slow ViewObject SQL Queries

The number one bottleneck is – as is in many web applications – SQL queries of ViewObjects to the database. This can be caused by many things: the SQL query is written in a suboptimal way, the data model is not efficient, the datasets in the database are far too high, indexes are lacking, indexes are not working as expected, e.g.

The first step is to get visibility and see which SQL queries are slow – and with what runtime parameter values:

In this blog is described in detail how you can instrument your ViewObect to get visibility into slow ViewObject queries.

It is always god to be able to analyze the runtime generated SQL (including applied ViewCriteria, and runtime bind parameter values) that is executed in the database. This to be able to reproduce problematic slow queries:

Nr 2: Too Frequent and Slow ApplicationModule Pooling

A very typical and big bottleneck in ADF is too frequent and too slow ApplicationModule Pooling. ApplicationModule pooling is a mechanism in ADF that enables multiple users to share several application module instances. It involves saving and retrieving session state data from the database or file. This mechanism is provided to make the application scalable and becomes very important under high load with many concurrent users. The default values of ApplicationModule pools are far too small; especially if you have more than 10 end-users.

I think this is one of the most important things to tune in general in ADF applications. Activations and passivations are the root cause of many very slow click actions. It is the root cause of errors after incomplete activations. In general, in my experience it is better to try to turn off the whole ApplicationModule pooling mechanism – for so far as it is possible.

Look at an example – a callstack showing what can happen when passivation/activation is not configured well. When there are too many ViewObject rows and attributes in memory, and passivated/activated:

We can see in the callstack:

• A slow activation of the ApplicationModule called HRService (4030 milliseconds).
• Later in the HTTP callstack we see a more slow activation of the different HRService instance (9460 miliseconds), caused by a slow activation of transient attributes in activateTransients() of ViewObject HRService.LocationViewRO (8281 milliseconds).
• We can see that ther are 4095 rows of LocationViewRO fetched (!)

To avoid the expensive passivations/activations we can increase the size of the ApplicationModule pools. In this way we make the application more scalable and avoid the passivations/activations. I usually increase the following parameters – depending of the usage:

• jbo.ampool.maxavailablesize
• jbo.recyclethreshold
• jbo.ampool.minavailablesize

Further:

• jbo.ampool.timetolive = -1
• jbo.doconnectionpooling=true
• jbo.txn.disconnect_level=1
• jbo.ampool.doampooling=true (default)

If you want to know more on ApplicationModule pools and tuning watch the video I made on ADF Performance tuning a few years ago (a big part of the video is on pooling parameters).

Nr 3: Redundant ViewAccessor Processing

The default value of the property of Row Level Bind Values on a ViewObject ViewAccessor is true. Set this value explicitly to false if you have no bind variables that really depend other attribute values in the row. This property is meant to set if there are bind variables in the lookup view object that can have a different value for each row. The first time it really executes the query, the second time the framework still calls the method executeQueryForCollection() on the ViewObject. Internally, in the ADF framework, it recognizes that the query already has been executed, and does not execute the query itself to the database again.

However, it is still a big inefficiency that for all rows the executeQueryForCollection() method is executed, and internally processed in the ADF framework. As we can see in the example below (callstack metrics of the ADF Performance Monitor in JDeveloper console log), this whole process still takes around 200 milliseconds extra for 35 rows (check the execution timeline on the left).

Now we set it to false:

After we set it to false there will be far less processing internally in the ADF framework. The query is executed once, and not any more for each row the whole ViewAccessor is evaluated. The whole HTTP request is around 200 milliseconds faster:

Nr 4: Memory Overconsumption (Fetching Too Many Database Rows and Attributes)

ADF applications potentially use a lot of JVM memory. Many times, the root cause of a high memory usage is that application data retrieved from the database into memory is not properly bounded; too many rows are fetched and held in ADF BC memory. This can lead to memory over-consumption, very long running JVM garbage collections, a freeze of all current requests or even OutOfMemoryErrors. To make matters worse, these rows and their attributes are frequently passivated and activated for an unnecessary long period of time. The solution to this problem can be found in reducing the size of sessions by decreasing of the amount of data loaded and held in the session.

Developing a plan to manage and monitor this fetched data during the whole lifetime of an application is an absolute must. It is indispensable to your performance success. The first step is to measure the current number of rows fetched by the ADF application and determine the appropriate maximum fetch sizes for your data collections. You can measure real-time and historically how many rows the ViewObjects load into Java memory with the ADFBC Memory Analyzer. Limit all Viewobjects, that frequently fetch more than 500 rows on a regular basis, with a maximum fetchsize. Or use extra bind variables, add extra ViewCriteria to limit the number of rows.

Read more in two of my previous blog on managing fetched data from the database, and on limiting the memory consumption of an ADF application:

In the screenshot we can see that nearly two billion rows (!), to be precise 1.855.223, were loaded in the JVM memory of our server. This proces of loading took 186.538 milliseconds (186 seconds).

Nr 5: Slow PL/SQL Executions executed from ADF application

Just like SQL queries from ViewObjects, PLSQL procedures/functions executed from the ADF application be very slow if not implemented well:

In the screenshot we see a PLSQL procedure call hr_main.sleep() with a bind param value of 4. The whole execution takes 4004 milliseconds (more than 4 seconds).

In this case the PLSQL procedure/function is the root cause, not the ADF application code in the ADF application. The problems must be resolved in the database, and can be anything; suboptimal written SQL queries in the procedure/function, the data model is not efficient, the datasets in the procedure/function are far too high, e.g.

Nr 6: ‘Too Rich’ Pages Result in Slow Browser Load Time

We shouldn’t make our ADF pages too ‘rich’ – we should limit the number of ADF Faces components to some extent. Especially the content of ADF Faces container components. Rendering too many table columns – in combination with sending too many rows to the browser – causes a long browser load time. The same applies to af:listView, af:tree and af:treeTable components with hundreds of rows. But also, other container components like af:iterator – when too many child components are rendered – the browser needs seconds to do the hard work.

For example: the loading of a table component (lazy loading of an af:table component showing Locations) takes around 1 to 1.5 seconds (grey color represents browser load time):

A page should spent not more than half a second or less in the browser to load.

Some general tips:

• Don’t set the fetchsize property of an af:table, af:treetable or af:listView to more than the number of rows displayed on the browser (set it to less than 40)
• Don’t use more than 20 columns on a table if possible – display only the most important columns. Do not display/render data that is not immediately needed. Move extra (detail) columns to detail pages or detail popups
• Expand af:tree and af:treeTable components to at maximum one or two levels deep if you have hundreds of nodes or more
• Read-only tables load faster than editable tables. Use EditingMode=”clickToEdit” instead of always editable
• Another way how you can ‘lazy’ load extra columns in a very simple way – and improve the browser load time – is to render columns on demand, for example by showing a show/hide checkbox.
• Use rendered instead of visible to display components conditionally
• Several properties like af:popup have the property contentDelivery. set this to lazy or lazyUncached
• Make IDs of the following ADF faces container components as small as possible (max 2 characters):

-af:pageTemplate
-af:region
-af:panelCollection
-af:table
-af:treetable
-af:tree
-af:ListView
-af:iterator

Read more on this subject in one of my previous blogs on slow browser load time.

Nr 7: End-Users on Slow Internet Explorer

Many companies and organizations do have the complete control over which type of browser end-users are using. They have control over it because they have a Citrix environment or internal network where end-users work on. Many times, still Internet Explorer is the only browser that is installed and allowed to use.

Web applications in general, and ADF applications in particular, perform very poor in Internet Explorer (regardless of the version). This is because of the inefficient JavaScript engine and the very slow browser load time. Google Chrome and Firefox are currently the most performance friendly browsers. This is very well known under developers. Still many companies, that do have the possibility to change to Chrome/Firefox, insist on Internet Explorer/Edge. My estimation is that changing to Chrome/Firefox will improve the performance in general with 20% or more. In general, it is a good practice to stimulate end-users to install and use Google Chrome or Firefox for ADF applications.

On the next screenshot we can see in which layer processing time has been spent; database (yellow), webservice (pink), application server (blue), network (purple), and browser load time (grey).

In this example from a field report, more than one third of the time spent in is grey, meaning that more than one third of the process time is spent in the browser (!). This is the process time spent by the browser, after receiving the response from the server, to build the DOM-tree  and to render/load the content. It turned out that all end-users were using Internet Explorer 11.

Nr 8: Slow SOAP/REST Webservice Calls

SOAP/REST Webservice calls can be consume a lot of time as well – depending on the network time, and the response time of the webservice request. It is good to have insight into this, especially if the response time takes very long:

In the above screenshot we can see three very slow SOAP webservice calls (executed in 4085, 5347, and 15321 milliseconds(!) ).

Nr 9: Inefficient ViewObject fetchsize

Many developers do not set the fetchSize every time on a ViewObject when they create a ViewObject. The default size of 1 is very inefficient. For example, a page with a table of only the EmployeesViewObject. The fetchsize is 1; this means that for each row the ADF application makes a complete roundtrip to the database (very inefficient):

The time to fetch the 107 employee rows takes 56 milliseconds. Not that super much, but it can be far more efficient, for example with a fetchsize of 120 (using 1 roundtrip to the database):

Now the fetch action takes only 11 milliseconds (we win 45 milliseconds). This seems to be minimal, but if you do this on each ViewObject this will be much better the for the performance!

Nr 10: ChangeEventPolicy is set to PPR

From ADF Release 11G R2 and onwards, the default value is of the global ChangeEventPolicy property is ppr. This is not good for performance, as there is a lot of performance overhead in this property. Too many iterators and components are automatically refreshed on each HTTP request.

In my opinion it is better to change it to none:

You better use manual partialTriggers in the page to refresh components. Be aware that if you change it in your current application, you should add the manual partialTriggers and test if everything works.

Conclusion

Of course, there are many, many more possible causes of poor performance (not well configured JVM, JVM heap too small, long running JVM garbage collections, slow network, hardware, e.g.), but these were our top ten typical bottlenecks we see frequently at companies that use the ADF Performance Monitor.