The next part will follow shortly, and will discuss the flexfield datamodel.

Even though the article is not yet finished, I thought the part about ‘SPECIAL’ and ‘PAIR’ validation types might be interesting enough. Many people seem to think they can only use the seeded validation sets. However, you can also create your own validation sets. And their options are very powerful. So I wanted to publish this part of the article as a prelude to the full story.

Special Validation

Special validation is used to provide flexfield functionality for a single value. What that means is that you can have for example a concurrent program parameter that will be filled with a Key flexfield value, or a range of flexfield values.

Let’s go back to the Key Flexfield. We know that they are combinations of different segment values that are stored in a separate combination table.

When you want to submit a key-flexfield combination as a parameter to a concurrent program, you can code your own validation for the separate values. But you’ll be missing the nice functionality that gives you pop-ups, a validation over the resulting combination and if needed the ID-value for the flexfield combination.

That is possible with a ‘Special’ validation type.

The special validation uses a number of user exits to enter, validate and query keyflex segments. With special validation, you will be able to enter one or more segment values for a key flexfield. To enter these segment values, 3 user exits can be used. They are: ‘POPID’, ‘VALID’ and ‘LOADID’.

POPID is used to enable the user to enter the flexfield segment value. It is called when the users cursor enters the segment value field. With this user exit, you decide which segment values should be shown, and how they should be shown.

 VALID is called when the user exits the segment value, or confirms the chosen flexfield combination. It validates the entered value against the values existing in the key flexfield table.

LOADID is optional, and it can be used to choose which information will be returned as flexfield value. This can be the concatenated segments, or the id-value for the flexfield combination or segment values.

These 3 user exits can be assigned to 3 ‘events’. There are more events possible, but they are either not yet in use, or their use is not yet supported. So we will only use ‘Validate’, ‘Edit’ and ‘Load’.

Sounds complicated, so far? Don’t worry; this is not an easy validation. But we’ll build some examples to give you an idea. First we start with building a very easy special validation. This will be built on our Code Combination key flexfield. We’ll be using a concurrent program ‘Test Flex Validation’ program to see our different options.

This program is based on the following procedure:

CREATE OR REPLACE PROCEDURE XXX_TEST_FLEXFIELD_PARAMS

( errbuf   out varchar2

, retcode  out varchar2

, p_flex   in  varchar2

, p_flex2  in  varchar2 := ‘XXX’

, p_flex3  in  varchar2 := ‘XXX’

, p_flex4  in  varchar2 := ‘XXX’

, p_flex5  in  varchar2 := ‘XXX’

) IS

BEGIN

   FND_FILE.PUT_LINE(FND_FILE.OUTPUT,p_flex);

   FND_FILE.PUT_LINE(FND_FILE.OUTPUT,p_flex2);

   FND_FILE.PUT_LINE(FND_FILE.OUTPUT,p_flex3);

   FND_FILE.PUT_LINE(FND_FILE.OUTPUT,p_flex4);

   FND_FILE.PUT_LINE(FND_FILE.OUTPUT,p_flex5);

END;

This will only write the parameter value to the output of the request. To use flexfields as parameters for concurrent programs, we need to define a value set based on them.

We will start with the barest setup to enter a key-flexfield combination. For this article, we use the accounting flexfield, with code ‘GL#’  and id-num ‘101’.

In this case, we have the following definition:

So what does this mean?

The first box is for the edit event. This will be triggered when the user enters the cursor into the field with this value set.

FND POPID         This is the user exit to pop up a flexfield screen, and let the user enter the flexfield values.

CODE="GL#"     This is the flexfield code for the key flexfield that we will be using.

APPL_SHORT_NAME="SQLGL" The short name for the application the flexfield belongs too. Together with ‘Code’, this will identify the flexfield itself.

NUM="101"       The id-number for the flexfield structure. If you have only a single structure flexfield, it is optional. For flexfields enabled for multiple structures, you need to enter the id-number.

VALIDATE="PARTIAL"   Validate can be ‘None’, ‘Partial’ or ‘Full’. None means the combination is not validated. Partial means that the separate segments are validated, there is no validation if the combination exists. Full means that segments and combination will be checked, and if a new value is entered, this will be inserted into the key flexfield table.

SEG=":!VALUE"                This is the forms field that will be used to store the value of the segments.

The second box is for the ‘Validation’ event. This code will be called when the user navigates out of the field, or submits the entire combination.

Now when we set this value set as a parameter for our concurrent program, we can see how the validation works:

Now when we run the program, we get this pop-up:

We have all the functionality of the key flexfield. We can use the ‘Combinations’ button to search for existing combinations, and all separate segments will be validated, as will be the final combination.

When we submit a value to our program, it will show the concatenated segments as the value of our parameter:

Now let’s see some more features of this validation. For example, we’d like to have the value of the combination id. (CODE_COMBINATION_ID in our case, since we use the Accounting Flexfield).

To get that, we need to add the LOADID user exit:

The ‘Load’ event will get the combination-id from the flexfield table. This is only possible for the ‘VALIDATE=”FULL”, since it will validate the whole combination. Also we need to set the ID=”:!ID”. This will populate the :!ID column with the ID value of the combination.

Finally, I added the ‘DINSERT=”NO” ‘, because we don’t want to allow insertion of new code combinations from this value set. (And Validation=”FULL” by default inserts new combinations into the flexfield column).

Now when we run the concurrent request, we see that the parameter value is the code_combination_id instead of the concatenated segments:

With these user exits it is also possible to select just a number of segments, instead of the whole combination. For this we remove the ‘Load’ / ‘LOADID’ part again.

Then we add a ‘DISPLAY=”x” ‘ to the ‘Edit’ and ‘Validate’ user exits. The “display” parameter is defaulting to ‘ALL’. But you can also specify separate segments by their sequence number or names. In our case, we display the first 2 segments:

Now when we run the concurrent program, we get a pop-up for only the first 2 values:

A very nice feature (at least as far as I’m concerned) is the use of a where clause on the combination values. Consider the following ‘Enter’ code:

FND POPID

CODE="GL#"

NUM="101"

APPL_SHORT_NAME="SQLGL"

VALIDATE="FULL"

TITLE="Special Validation Key"

ID=":!ID"

SEG=":!VALUE"

DESC=":!MEANING"

WHERE="segment2 not like '1%' "

The “WHERE” clause prevents us from choosing combinations that have a segment2 starting with ‘1’. When we run our concurrent program with this, and choose the combinations:

There is no Dpt starting with 1.

When we add the “WHERE”-clause to the validation event too, it will prevent us from entering the values manually:

The last feature that we’ll look into is the use of a pl/sql validation through the special validation routines. By using the user-exit PLSQL, we can call an anonymous PL/SQL block in our ‘Validation’ event. I created a value set with the following function for the ‘Validation’ event:

FND PLSQL " declare

  v_value varchar2( 10 ) := :!value ;

  v_sum number;

  v_valid boolean;

begin

   v_sum:=0;

   for i in 1..length(v_value) loop

    v_sum :=v_sum+(length(v_value)+1-i)*substr(v_value,i,1);

  end loop;

  if mod(v_sum,11)=0 then

     v_valid := TRUE;

  else

     v_valid:=FALSE;

  end if;

  if not v_valid then

      fnd_message.set_name('FND','FND_GENERIC_MESSAGE' );

      fnd_message.set_token('MESSAGE','This is not a valid bank account');

      fnd_message.raise_error;

  end if;

END; "

This PL/SQL procedure validates a (Dutch) bank account number. If it does need pass the test, a message will be displayed. This gives you almost unlimited possibilities for validating entered data.

As you can see, it is only a ‘Validate’ event. Because we don’t need any special functionality for entering the data. We can limit the entry to numbers only on the ‘Validation Set’ main page.

Now when we use this value set for our concurrent program, we can only enter valid dutch bank accounts:

And

The list of parameters for the user exits is longer than this. So we won’t be going through all the possibilities. You can check the Developers Guide and the Flexfield guide for a complete listing of options. (Did you notice the flexfield title that I sneaked into the pop-up? Try and find the option for that!)

Please try the different options for yourself, and realize the possibilities of the special validation.

Pair Validation

Meanwhile, we’ll continue to the ‘Pair’ validation. The pair validation is very much like the ‘special’ validation. It uses the same kind of user exits, but this time, a range of segment values or combinations is selected.

Let’s first create a range of the account segment. Instead of using POPID and VALID, we use POPIDR and VALIDR. The R-version of the user-exits automatically create a range.

Of course we need 2 parameters to set the range. However, we need only one validation set.

I created the validation set ‘XXX_PAIR_VAL’. I entered only the edit and validate events:

The next step is to set the parameters for both the low and high value. Both parameters have the validation set ‘XXX_PAIR_VAL’.

Now when we run the program, we can enter a range. This includes validation that the high value is indeed higher or equal to the low value.

Of course the concurrent program will receive the values for 2 parameters.

When we use the full validation we can enter a range of the whole account combination. Note that we cannot use the FULL validation for pair-validation. Because that would mean the use of the combination-id from the flexfield table and based on the combination-id’s you cannot build a range.

So we can only use PARTIAL and NONE for the validation. For that same reason, I have not yet had a reason to use a LOAD event for PAIR validation. It is however allowed to use one.

I created a PAIR validation for the whole accounting range as follows:

When used in the concurrent program, it will indeed allow us to enter a range of all segments:

That completes the chapter on PAIR validation too.

In this article we will describe the infrastructure of Oracle eBusiness Suite (eBS). In its simplest form, eBS is a 3-tier application with a client tier, Application-tier and DB-tier.

Database-Tier

Let’s start with the DB-Tier. Surprisingly, the database tier has only very little eBS specific features.

Of course we need a database (instance) and therefore an ORACLE_HOME. But the database can either be a single-instance or a RAC-installation and all Oracle RDBMS features are transparently available for eBS.

The management of the RDBMS Installation is also independent of eBS.

DB-Tier filesystem

Let’s start with the filesystem on the DB-Tier. Of course there is an Oracle_home installation needed, for the RDBMS-Instance. This will be installed during installation of eBS. But also a fresh installed ORACLE_HOME can be used, with an eBS database.

In the ORACLE_HOME, an extra directory is added. The Appsutil directory. This directory contains the software and data needed for running Autoconfig and Rapidclone.

All other directories are at the discretion of the eBS DBA.

Oracle Instance

When we look at the instance to run eBS, we find a number of mandatory parameters for eBS. These are found in Metalink notes 216205.1 and 396009.1 (At the time of writing. Please verify these notes for yourself).

These parameters are recommended or mandatory based on testing by Oracle Corp. They will automatically be set by the eBS installer. But you should take note of them when you use a fresh installed ORACLE_HOME.

Then we finally come to the contents of the database.

The eBS Database

Let’s start with the schemas in the database. Oracle eBS creates a separate schema for every module. The schema is named as the short_name of the module, for example AP (Oracle Payables / Accounts Payable), AR (Oracle Receivables / Oracle Receivables).

There is a separate schema for the Application owner APPS.

The Application schemas contain the tables, indexes and sequences for the different applications. All objects in these schemas (except indexes, of course) have a synonym in the APPS Schema. In the APPS Schema we also find all PL/SQL objects, views and Materialized Views.

A major part of eBS is written in PL/SQL. All PL/SQL objects are also installed in the APPS Schema.

User sessions within eBS will usually run in the APPS Schema as well.

That brings us to an extra schema in the database: APPLSYSPUB. This schema has access to some of the eBS tables and packages, that allow it to validate eBS logins and start an APPS-session based on that login information. We will see the details of this later on.

Before release 11.5.2 every schema had its own tablespace. However, the number of modules for eBS (and with that the number of schemas) is ever increasing. So managing the database became more and more complex. In 11.5.2 Oracle introduced the Oracle Applications Tablespace Model (OATM). Within this model, the tablespaces in eBS are based on functionality, rather than schemas.

In this model, we see the following tablespaces:

APPS_TS_TX_DATA – Containing all transaction tables, Materialized Views and IOT’s

APPS_TS_TX_IDX – Containing all indexes for transaction tables

APPS_TS_SEED – Containing the tables and indexes with seeded data (as opposed to transaction data).

APPS_TS_INTERFACE – For Open Interface tables

APPS_TS_SUMMARY – Contains summary tables for several modules (AR, PA, BIM, etc)

APPS_TS_NOLOGGING – For tables and objects that are created with the NOLOGGING option

APPS_TS_ARCHIVE – Containing archive and history tables (and indexes)

APPS_TS_QUEUES – Containing the AQ (Advanced Queuing) objects

APPS_TS_MEDIA – Containing tables with LOB’s. For media objects or documents.

The Undo and Temp tablespaces are not part of the tablespace model.

Application Tier

Now it’s time to look at the Application Tier. In fact the Application Tier consists of 3 different services: Web service, Forms service and Concurrent Processing. In 11i installations, there is also an Administration service.

The Application Tier significantly changed from R11i to R12. We’ll discuss the 11i Apps Tier shortly, and then discuss the R12 tier in more detail.

The 11i infrastructure

Both the R11 and R12 infrastructure consist of a Web-service, a Forms Service and a Concurrent Processing part.

We will be discussing the different services here. The following picture shows all components and their communications. You might want to keep it for reference during this article.

R11i Web Service

For 11i, the web tier is built on Oracle iAS 9i. The iAS installation provides webservices (Apache HTTP/HTTPS), a Java Runtime Engine (JSERV) and a PL/SQL engine (modplsql).

The web service also acts a gateway for the Concurrent Request log and output files. And it is the first point of access when starting a forms session. (When using a socket forms connection, when using a forms servlet the web tier will host the forms process).

A detail from the picture above shows the iAS structure.

The core of iAS is the web server. This is the front-end for the client. Requests can also be forwarded to and from the forms server and the concurrent processing. We’ll see that in the next paragraphs.

Within the iAS Jserv and modplsql are plugins. They are the only components that communicate with the database. When they are called, they execute java (Jserv) or PL/SQL (modplsql) and return an html page. This page is then sent to the client through the HTTP service.

The Jserv delivers a Java Runtime Environment. In the Jserv, java servlets can be run. Also the JSP-files are executed in the Jserv. A JSP-file (Java Server Page) is a page with java code that returns an html-page (similar to the way scripting languages like PhP work). The java part is executed in Jserv, which returns the html to the webserver. The webserver redirects the html to the client.

Let’s take a closer look at the components and their executions:

Webserver

The webserver is based on the regular Apache 2 webserver. The configuration file is also equal to the Apache config file. The configuration is set in httpd.conf (or httpds.conf for SSL).

Instead of starting the webserver through $APACHE_HOME/bin/httpdctl, we start through $APACHE_HOME/apachectl. The default port number used for eBS 11i is 8000. This is part of port-pool 0. For different port-pools, the port number is increased. So for port-pool 1 the webserver runs on port 8001.

The root directory for the webserver is set to $OA_HTML, which is by default $COMMON_TOP/html. This directory contains all *.html files for eBS.

A number of virtual directories are set up within eBS.

JServ

As mentioned before, java code is executed by Jserv. Jserv is a java servlet engine. That means that it can run both servlets and jsp-files.

These servlets are mostly located in the $JAVA_TOP. The *.jsp files are located in the $OA_HTML directory.

One of the options of Jserv is to create a database connection to the rdbms database. This is done by a JDBC Thin Client connection.

Before we look at the configuration for Jserv, examine the following picture.

Within Jserv, we can define different java environments, called zones. These zones are configured with different servlets or java archives (jar-files). Each zone is configured with its own configuration file. Within the zone the startup parameters (initargs) for the servlet are defined.

On the other side of the picture, you see a group. All java processes within Jserv are grouped together. You must define at least one group. The default group is ‘OACoreGroup’. Within each group, we create one or more processes that will be mapped to our zones.

This mapping is done by mounting the zones and the groups to different logical directories. In the picture, a mountpoint is created: /oa_servlets/. It refers to the group ‘OACoreGroup’, which holds 3 java processes. And it is mapped to zone ‘root’, which includes the servlet ‘dummy’ with a set of startup parameters.

When iAS receives a call to the virtual directory /oa_servlets/ it will be recognized as a Jserv mount point and the request will be forwarded to Jserv. In this example Jserv has 3 java processes in the group for this mount point. And they will be able to run all the servlets in the zone.

Sounds complicated? Take a look at the following configurations:

Jserv.conf:

ApJServGroup OACoreGroup 3 1 /etc/oracle/iAS/Jserv/etc/jserv.properties

ApJServGroupMount /oa_servlets balance://OACoreGroup/root

Jserv.properties:

wrapper.bin=/opt/oracle/iAS/Apache/Apache/bin/java.sh

zones=root<host>

root<host>.properties = /etc/oracle/iAS/Jserv/etc/zone.properties

Zone.properties

servlet.Dummy.initArgs=message=I'm a dummy servlet

Within jserv.conf we define a group called OACoreGroup. This group is running 3 Java processes. And the definition of the group is in the jserv.properties file. The 1 indicates the weight for load-balancing with multiple groups.

Then we mount the zone ‘root’ to the group ‘OACoreGroup’. This mount point is linked to the virtual directory /oa_servlets/.

The virtual directory is used for redirection to the JServ. When a request is made to the virtual directory Jserv will be called. The part of the URL after the virtual directory is the path to the servlet. This path will be searched for in the $CLASSPATH.

When the java servlets need to connect to the database, they can build a connection using JDBC. The access information is stored in a *.dbc file in $FND_TOP/secure. The dbc-file is referred to in the parameters for the zone.

Modplsql

Let’s take a look at the modplsql module. This module is designed to run pl/sql procedures within the database. The connection is based on the wdbsvr.app file. This file contains the DAD (Database Access Descriptor), including the access data to the eBS database.

The module is also called through a virtual directory. For example http://<host>:<port>/pls/TESTDB/ dummy. /pls/ is the virtual directory that refers to modplsql. TESTDB is the name of the DAD and dummy is the name of a pl/sql procedure accessible for the db-user from the DAD.

That concludes the 9i iAS module for now.

Formsserver

Let’s take a look at the forms server.

Oracle forms can be set up in two ways, socket connection and servlet. The default is socket connection. With a socket connection, a separate forms server and dedicated forms processes are used. For the servlet connection, a java servlet is called within the iAS.

The formsserver itself is installed in the 8.0.9 ORACLE_HOME. Forms has a forms server, and one or more client processes. The forms server is started with f60svr. It will spawn a f60webmx process for every client session connecting.

On the server side, forms are run in the forms client processes. On the client side, they are run in a java applet. When the client clicks a forms based function in the Personal Home Page, it calls an URL that refers to the forms client executable in the 8.0.6 ORACLE_HOME.

This URL is taken from the profile option ‘ICX: Forms Launcher’, and the default value is like ‘http://<server>:<port>/dev60cgi/f60cgi’. The parameters referring to the function being clicked are added to this URL as parameters. (i.e. the name of the form to be started)

When this URL is called, the webserver will execute the executable f60cgi. This executable returns a HTML page to the client. This page is called the ‘Base HTML’ for this forms server. (by default this is $OA_HTML/US/appsbase.htm)

This HTML page calls the J-initiator plugin (or the native JVM when configured). It also includes the parameters to connect to the forms server and the name of the form to start.

The J-initiator will start an applet on the client, which connects to the forms listener process. The forms listener process then assigns a dedicated forms client process.

At this point the whole chain looks like this:

The configuration for the forms server is in the appsweb.cfg file in ($OA_HTML/bin). This file contains the basic coloring scheme for the forms server, the forms settings and the referral information to the J-Initiator plugin. The plugin on the client side is called through its class-id, which is also set in the appsweb.cfg.

Concurrent Managers

The last part of the application tier is the concurrent processing part. The concurrent managers are used to execute background and batch processes.

Different executables including host-command files, pl/sql procedures, Oracle Reports, SQL*Loader control files and Binary executables can be defined to be run as concurrent programs. Parameters are also optionally defined with the concurrent programs.

The executable files are defined separate from the concurrent programs. So an executable can be run as different programs with different parameters.

The programs are executed through ‘Requests’. A request is started as a concurrent program and the values of its parameters. It can be scheduled to start at a specific time, or in a specific schedule. The output of the program can be sent to a printer. It is also available through the application.

This picture shows the relation between programs, requests and managers.

The managers

We’ll take a closer look at the concurrent programs later. Let’s first look at the concurrent managers. There are concurrent managers and transaction managers. Also a number of control managers are defined.

We’ll start with the ‘Internal Manager’. This is the first manager to be started. Its purpose is to control the stopping and (re-)starting of the other managers. When Generic Service Management is enabled (default as of 11.5.7), it delegates to the ‘Service Managers’. On every node where concurrent processing is enabled, a ‘Service Manager’ is started. However, only one Internal Manager is running at any time.

The other concurrent managers are defined with a work shift that controls how many processes a concurrent manager should have at certain times. The work shifts consist of a time-range and a number of processes. The Service Managers (or Internal Manager) will start and stop processes according to these work shifts.

Another part of the setup of concurrent managers is their specialization rules. The specialization rules indicate which programs are valid for a concurrent manager, or are excluded for that manager. They work on an include/exclude principle. When programs are included for that manager, the manager can only run those programs. When programs are excluded, the manager can run any program except the excluded ones.

When a request (to run a program) is submitted from eBS, it will be placed in FND_CONCURRENT_REQUESTS with a status_code ‘I’ (The eBS forms have fewer statuses than the codes in the table). The manager processes will query this table for requests that they are eligible to run.

Once a manager process finds a request with status_code ‘I’, which it is eligible to run then it will put the request on its own queue. It will then run the executable with the defined parameters. The logfile and outputfile are written to the filesystem in $APPLCSF/$APPLLOG resp. $APPLCSF/$APPLOUT.

There are some special cases that need to be discussed. The first is the incompatibility. Concurrent Programs can be made incompatible with each other. That means that they cannot run at the same time. Once a program is started that is defined as incompatible with another, it will be automatically put on the queue for the ‘Conflict Resolution Manager’. This special manager will check if any incompatible program is running or ‘Pending’ with code ‘I’. If so, it will hold the request on its own queue. If no incompatible program is running or ‘Pending’, then it will set the status of the request to ‘Pending’.

Another special case are the ‘Transaction Managers’. They are started and stopped the same way as the other concurrent managers. But they do not use the request queue. Transaction managers are called online from the eBS forms. And they execute a limited number of programs. These programs defined within their executables. They are called through the ‘FND_TRANSACTION.SYNCHRONOUS’ procedure, which uses the ‘DBMS_PIPE’ package.

The programs

It’s time to look at the concurrent programs. As mentioned before, a concurrent program is an instantiation of an executable. The executable is defined with a short name, an application (module), a filename and a method.

The short name will uniquely identify the executable. The other data is needed to determine what should be run for this executable. If the executable is an OS-based program, the application will be used to derive the directory on the file system where the executable is found.

When the executable is defined as PL/SQL, the filename will contain the procedure that needs to be run.

The concurrent program is defined as the executable with an (optional) set of parameters. It also has some properties for the printing of the output (print-style, pre-defined printer, size of the output).

Depending on the type of executable that needs to be run, the parameters will be sent to the executable ordered or named. For PL/SQL and host files, the parameters are ordered. And the order in which they are defined in the form defines how they will be sent to the executable. For reports the parameters are named, which means they are sent as <parameter>=<value>, ….

After the request has finished, the request table ‘FND_CONCURRENT_REQUESTS’ will be updated with the status information and a reference to the log- and output file. During the execution of the request, a status_code and phase_code are updated. The exact values of these fields are described in one of the next articles. That will go deeper into concurrent processing.

The output

The log- and output file from the requests are written to the directories $APPLCSF/$APPLLOG and $APPLCSF/$APPLOUT. But they of course also need to be made available to the user. This is done through the ‘Applications Report Review Agent’.

There is quite a lot of setup that can be done for the whole process. But within the scope of this article, we’ll only look at the basic architecture.

From the eBS form the log and output file are available from 2 buttons. These buttons call the web server for ‘FNDWRR.exe’ (.exe on both Windows and Unix).

FNDWRR.exe is a cgi-executable that will call the ‘FNDFS listener’.

This is an 8.0.6 TNS-listener in the eBS ORACLE_HOME on the eBS application tier. One of the less known features of the TNS-listeners is that they can do more than create database connections.

In the listener.ora, you can define a program to be called when a connection is made on a certain tns-entry. That feature is used for the FNDFS listener. When it is called, it will redirect traffic to the ‘FNDFS’ (FND File System) executable. This executable will read the requested file from the file-system and send it to FNDWRR.exe.

Again, we have a schema to show the whole flow:

This complex retrieval is of course needed because the concurrent processing tier can be separate from the forms and web-tiers.

A customer asked me to do a presentation about AQ (Advanced Queuing). This article is a rewrite of that presentation. We’ll be getting some hands-on experience with AQ, and then some tips on the issues that might occur.

As always, your comments to this article are more than welcome. If you enjoy this article, find it useful or maybe not at all, please let me know by leaving a comment on the site.

The article is also available in PDF format from here.

Queuing: I don’t want to queue.

The title might be true in many cases, but there are also situations where a queue is very convenient. For example in the case of batch processing where a batch process handles multiple incoming messages from an online process. Or when 2 processes need inter-process communication, but still need to function independently of each other.

In eBS we use queues for the workflow system. (Deferred items, notifications for the workflow mailer and the Business Event System). Some more queues are found for concurrent processing and SFM.

So whether you like it or not, you’ll have to queue. The trick is to manage these queues to get optimal performance for your system.

(Advanced) Queue design

Before we can start building queues, there are some things to consider.

AQ supports both point-to-point queues and publish-subscribe-queues (also called multi-consumer queues).

Point-to-point queues are emptied by only one specific process. One or more processes can enqueue messages on the queue, but only one process can dequeue them.

In contrast, a publish-subscribe queue can have many processes reading the messages in the queue. Either the messages are broadcasted, or the receivers have to subscribe to a certain kind of messages.

Of course the publish-subscribe queue has some very interesting properties. But we’ll start our item with the point-to-point queue.

So you’ll first have to decide who the senders and receivers of the queue data will be. In this article, we start with using a point-to-point queue. After that we start using multi-consumer queues.

Another thing to consider is the payload of the message. Of course, the messages will need some content to give it a meaning to the receiver. This content is called the payload. And you can either use a custom type (including XML), or a raw type.

During this article, we’ll see some more features of AQ. But when we decide on the type of queue and the payload type, we can build our own queues. All queues are built on queue-tables. These tables hold the data in the queue. On top of these tables, the actual queue and some management views are built.

To build a queue-table, we use the dbms_aqadm package:

dbms_aqadm.create_queue_table(queue_table =>’<table_name>’

                          ,queue_payload_type => [‘RAW’|<custom_type>]);

This creates the queue table including a LOB segment for the payload, some indexes, and an ‘Error queue’:

Begin

dbms_aqadm.create_queue_table(queue_table=>’xxx_test’

                           ,queue_payload_type=>’RAW’);

End;

Select object_name,object_type from dba_objects where created>sysdate-1/24;

OBJECT_NAME                 OBJECT_TYPE

—————————–         ———–

SYS_C0011768                INDEX

XXX_TEST                    TABLE

SYS_LOB0000073754C00029$$   LOB

SYS_LOB0000073754C00028$$   LOB

AQ$_XXX_TEST_T              INDEX

AQ$_XXX_TEST_I              INDEX

AQ$_XXX_TEST_E              QUEUE

AQ$_XXX_TEST_F              VIEW

AQ$XXX_TEST                 VIEW

This created the base-table for a point-to-point queue. The table is a regular heap-oriented table. And you are free to create extra indexes on it, if you feel the urge. The necessary indexes have been created already.

The queue that is created now is the default error queue. Messages that failed dequeuing will be set on this queue.

Now it’s time to create the actual queue. The queue-tables are the infrastructure for storing the messages and related information. The queue can now be created to control the queuing and dequeuing of messages.

For both point-to-point as publish-subscriber queues, the command is:

dbms_aqadm.create_queue (queue_table =>’<table_name>’

                      queue_name => ‘<queue_name>’);

So for us we run:

Begin

dbms_aqadm.create_queue (queue_name => 'xxx_test_q'

  ,queue_table => 'xxx_mc_test');

End;

This creates an object of type QUEUE. This is the object that will control the contents of the underlying tables / IOT’s.

Before we can start using our queues, we also have to ‘start’ them. On starting, we indicate whether the queue is available for queuing, dequeuing or both:

Begin

 dbms_aqadm.start_queue(queue_name=>’xxx_test_q’

 ,enqueue=>TRUE

 ,dequeue=>TRUE);

End;

Our queue is now enabled for both queueing and dequeuing. Let’s first verify if things are working correctly.

To enqueue (or dequeue) a message, we use the dbms_aq package. It has an enqueue and dequeue procedure. Both with their own parameters. The parameters include en-/dequeue options, message properties, a message_id and of course the message itself:

Declare

V_payload Raw(200);

V_msgid Raw(200);

V_enq_options Dbms_aq.Enqueue_options_t;

v_msg_properties dbms_aq.message_properties_t;

Begin

V_payload:=Utl_raw.Cast_to_raw('Hello world!');

Dbms_aq.Enqueue(Queue_name=>'xxx_test_q'

,Message_properties=>V_msg_properties

,Enqueue_options=>V_enq_options

,Payload=>V_payload

,Msgid=>V_msgid);

Dbms_output.Put_line(rawtohex(V_msgid));

end;

This enqueues a ‘Hello world!’ message, and returns the message id. If everything works correctly, you’ll see the msgid as a hexadecimal string. (Don’t forget to set serveroutput on).

We created 2 extra parameters: v_enq_options with the options used to enqueue this message. And v_msg_properties to set additional information about the message.

V_enq_options is of type ‘dbms_aq.enqueue_options_t’. This is a record of:

Visibility   BINARY_INTEGER  –Options are: dbms_aq.on_commit and dbms_aq.immediate. This indicates whether the enqueue is part of the current transaction, or done autonomously.

Relative_msgid      RAW(16)             –If the message needs to be enqueued at a specific position, it will be relative to this msgid.

Sequence_deviation BINARY_INTEGER –-Options are: DBMS_AQ.BEFORE, DBMS_AQ.TOP or NULL (default). If before then the message is before the relative_msgid. If top, the message will be the first to be dequeued.

V_msg_properties is of type ‘dbms_aq.message_properties_t’. This is a record of:

priority        BINARY_INTEGER  – Any integer, to set the priority. Smaller is higher priority. The default is 1.

delay           BINARY_INTEGER  – If the message needs to be delayed before it can be dequeued, set the time in seconds here. The default is dbms_aq.no_delay.

expiration      BINARY_INTEGER  – For messages that need to expire after a certain time, set the expiration time in seconds. (Offset from the delay). The default is dbms_aq.never.

correlation     VARCHAR2(128)   – A free text field that can be used to identify groups of messages.

attempts        BINARY_INTEGER  – Number of failed attempts to dequeue, before the message will be failed and marked as expired.

recipient_list  DBMS_AQ.AQ$_RECIPIENT_LIST_T –- Only valid for multi-consumer queues. Sets the designated recipients.

exception_queue VARCHAR2(51)    – The exception queue to use, when it is not the default.

enqueue_time    DATE                – Set automatically during enqueue

state           BINARY_INTEGER  – Automatically maintained by AQ, to indicate the status of the message.

Let’s see if the dequeue also works. For this the procedure dequeue is used, with similar parameters.

Declare

V_payload Raw(200);

V_msgid Raw(200);

V_deq_options Dbms_aq.dequeue_options_t;

v_msg_properties dbms_aq.message_properties_t;

Begin

Dbms_aq.dequeue(Queue_name=>'xxx_test_q'

,Message_properties=>V_msg_properties

              ,dequeue_options=>V_deq_options

              ,Payload=>V_payload

             ,Msgid=>V_msgid);

Dbms_output.Put_line(utl_raw.cast_to_varchar2(V_payload));

End;

This time, our message should be displayed.

For the dequeue, we used v_deq_options of type ‘dbms_aq.dequeue_options_t’. This is a record of:

consumer_name  VARCHAR2(30)       – Indicates the consumer for multi-consumer queues.

dequeue_mode   BINARY_INTEGER –- How to dequeue the messages. Either leave it on the queue, or remove it. Either dbms_aq.browse and dbms_aq.remove (default).

navigation     BINARY_INTEGER –- Indicate where to start dequeuing. Dbms_aq.next_message (default), to continue from the previous dequeue. Dbms_aq.first_message to start at the top of the queue. Dbms_aq.next_transaction to skip the rest of this message group.

visibility     BINARY_INTEGER –- same as dbms_aq.enqueue_options_t.visibility.

wait           BINARY_INTEGER –- The time (in seconds) the package should wait if no message is available. Default is dbms_aq.forever.

msgid          RAW(16)     – When specified, only the message with this msgid will be dequeued.

correlation    VARCHAR2(128)  – Only messages with this correlation will be dequeued (may include wildcards).

Note how message_properties and payload are now out-parameters.

This is probably the simplest queue possible. We enqueued and dequeued a raw message. We didn’t specify the visibility. So your session still needs to commit these actions.

Now let’s enqueue our message again, and see how it works behind the curtain.

Declare

V_payload Raw(200);

V_msgid Raw(200);

V_enq_options Dbms_aq.Enqueue_options_t;

v_msg_properties dbms_aq.message_properties_t;

Begin

V_payload:=Utl_raw.Cast_to_raw('Hello world!');

Dbms_aq.Enqueue(Queue_name=>'xxx_test_q'

,Message_properties=>V_msg_properties

,Enqueue_options=>V_enq_options

,Payload=>V_payload

,Msgid=>V_msgid);

Dbms_output.Put_line('Msg_id: '||rawtohex(V_msgid));

end;

Msg_id: 499CE4809F2641E1BFBC8AFBC8DB5AFA

The queue table is an ordinary heap-table, so we can query it.

select q_name, rawtohex(msgid) msg_id, priority, state, enq_time, enq_uid

from   xxx_test;

q_name     msg_id                         priority state enq_time                   enq_uid

XXX_TEST_Q  499CE4809F2641E1BFBC8AFBC8DB5AFA     1     0 21-03-10 17:24:01,876000000 SYSTEM

We see our msgid again. A priority flag. A state flag, the time of enqueueing the message, and the user that enqueued the message. The message is also in the table, but since it is a blob, we won’t bother selecting from it yet.

There are more columns in the table, that control the order and by who the messages are dequeued. Most of them are still null, so we will see them when needed.

A useful alternative to the table is to query the queue-view aq$<table_name>. This will show the translated values of the state. (0 = READY). And especially when using multi-consumer queues, it will use a join to select a more complete picture of the queue.

When we dequeue the message, it will disappear from the queue. (And be deleted from the queue table). However, this can be controlled by the retention parameter of the queue.

Let’s set this parameter, so we can check the data after the dequeue.

We set the retention time to 240 (seconds):

begin

DBMS_AQADM.ALTER_QUEUE(queue_name =>'xxx_test_q'

,retention_time => 240);

end;

Now when we dequeue the message, it will remain in the queue:

Declare

V_payload Raw(200);

V_msgid Raw(200);

V_deq_options Dbms_aq.dequeue_options_t;

v_msg_properties dbms_aq.message_properties_t;

Begin

Dbms_aq.dequeue(Queue_name=>'xxx_test_q'

,Message_properties=>V_msg_properties

              ,dequeue_options=>V_deq_options

              ,Payload=>V_payload

,Msgid=>V_msgid);

Dbms_output.Put_line(utl_raw.cast_to_varchar2(V_payload));

End;

select       queue

,      rawtohex(msg_id)msg_id

,      msg_priority

,      msg_state

,      enq_timestamp

,      enq_user_id

,      deq_timestamp

,      deq_user_id

from   aq$xxx_test

QUEU     MSG_ID                         MSG_PRIO MSG_STATE ENQ_TIMESTAMP  ENQ_USER DEQ_TIMESTAMP     DEQ_USER_ID

XXX_TEST_Q AEC2CD2E34514363B6739969E8E8D353    1 PROCESSED 19-03-10 18:31:40 SYSTEM 19-03-10 21:26:45 SYSTEM

Now the message has been set to state ‘PROCESSED’, and some dequeue information has been added.

It’s time to start navigating queues when there are multiple messages in the queue.

Messages are by default dequeued in the order in which they are enqueued. On creation of the queue table, you can set other dequeue orders. But it is also possible to dequeue messages in a different order by navigating the queues, or using filter-criteria.

To show the dequeueing order we enqueue 10 different messages.

Declare

V_payload Raw(200);

V_msgid Raw(200);

V_enq_options Dbms_aq.Enqueue_options_t;

v_msg_properties dbms_aq.message_properties_t;

Begin

V_enq_options.visibility := dbms_aq.immediate;

for i in 1..10 loop

    V_payload:=Utl_raw.Cast_to_raw('This is the '||to_char(to_date(i,'J'),'jspth')||' message');

    Dbms_aq.Enqueue(Queue_name=>'xxx_test_q'

    ,Message_properties=>V_msg_properties

                  ,Enqueue_options=>V_enq_options

                  ,Payload=>V_payload

                        ,Msgid=>V_msgid);

    Dbms_output.Put_line(rawtohex(V_msgid));

end loop;

end;

This enqueues the text ‘This is the first message’ till ‘This is the tenth message’. On dequeuing, the messages come out in the same order:

Declare

V_payload Raw(200);

V_msgid Raw(200);

V_deq_options Dbms_aq.dequeue_options_t;

v_msg_properties dbms_aq.message_properties_t;

Begin

for i in 1..10 loop

    Dbms_aq.dequeue(Queue_name=>'xxx_test_q'

                 ,Message_properties=>V_msg_properties

                  ,dequeue_options=>V_deq_options

                   ,Payload=>V_payload

                 ,Msgid=>V_msgid);

    Dbms_output.Put_line(utl_raw.cast_to_varchar2(V_payload));

end loop;

End;

This is the first message

This is the second message

………

This is the tenth message

When we created the queue table, we choose the default sort order. This is by enqueue_time. We can also build a queue that uses priority dequeuing. First we create a queue:

begin

dbms_aqadm.create_queue_table(queue_table=>'xxx_test_prio'

                             ,sort_list => 'PRIORITY,ENQ_TIME'

                             ,queue_payload_type=>'RAW');

dbms_aqadm.create_queue(queue_name=>'xxx_test_prio_q'

                       ,queue_table=>'xxx_test_prio');

dbms_aqadm.start_queue(queue_name=>'xxx_test_prio_q');

end;

We indicated a sort_list now. The options are ‘ENQ_TIME’ (default), ‘ENQ_TIME,PRIORITY’,‘PRIORITY’,’PRIORITY,ENQ_TIME‘. Now we enqueue some messages with reversed priorities:

Declare

V_payload Raw(200);

V_msgid Raw(200);

V_enq_options Dbms_aq.Enqueue_options_t;

v_msg_properties dbms_aq.message_properties_t;

Begin

V_enq_options.visibility := dbms_aq.immediate;

for i in 1..10 loop

    V_payload:=Utl_raw.Cast_to_raw('This is the '||to_char(to_date(i,'J'),'jspth')||' message');

    v_msg_properties.priority:=11-i;

    Dbms_aq.Enqueue(Queue_name=>'xxx_test_prio_q'

    ,Message_properties=>V_msg_properties

                  ,Enqueue_options=>V_enq_options

                  ,Payload=>V_payload

                 ,Msgid=>V_msgid);

    Dbms_output.Put_line(rawtohex(V_msgid));

end loop;

end;

And we dequeue them again:

Declare

V_payload Raw(200);

V_msgid Raw(200);

V_deq_options Dbms_aq.dequeue_options_t;

v_msg_properties dbms_aq.message_properties_t;

Begin

for i in 1..10 loop

    Dbms_aq.dequeue(Queue_name=>'xxx_test_prio_q'

                 ,Message_properties=>V_msg_properties

                  ,dequeue_options=>V_deq_options

                  ,Payload=>V_payload

                 ,Msgid=>V_msgid);

    Dbms_output.Put_line(utl_raw.cast_to_varchar2(V_payload));

end loop;

End;

This is the tenth message

This is the ninth message

…………

This is the second message

This is the first message

Now it’s time to look at queueing navigation. It is possible to dequeue specific messages from the queue. You can select messages with a specific msg_id, correlation or recipient_list (for mc-queueus).

We’ll first search for a specific correlation and then a message_id. We enqueue ten messages, with different correlations:

Declare

V_payload Raw(200);

V_msgid Raw(200);

V_enq_options Dbms_aq.Enqueue_options_t;

v_msg_properties dbms_aq.message_properties_t;

Begin

V_enq_options.visibility := dbms_aq.immediate;

for i in 1..10 loop

    V_payload:=Utl_raw.Cast_to_raw('This is the '||to_char(to_date(i,'J'),'jspth')||' message');

    v_msg_properties.correlation:=to_char('Corr'||i);

    Dbms_aq.Enqueue(Queue_name=>'xxx_test_q'

                 ,Message_properties=>V_msg_properties

                  ,Enqueue_options=>V_enq_options

                  ,Payload=>V_payload

                 ,Msgid=>V_msgid);

    dbms_output.Put_line('Msg_id: '||rawtohex(V_msgid)||' Correlation: Corr'||i);

end loop;

end;

Msg_id: E8BE83A2A2A04F1EA74863B4A7C78DAF Correlation: Corr1

Msg_id: 7159B80BC3194C7AAA6910AB10E753C5 Correlation: Corr2

Msg_id: 4AF3693CF7EE4994B0F78830371437B9 Correlation: Corr3

Msg_id: 44DBC0CB09C94BB98DF2D7E48971849C Correlation: Corr4

Msg_id: 98F3E119041E47F5BF46604E014120BF Correlation: Corr5

Msg_id: B71B7F097A9E4EDBA696958326BF6300 Correlation: Corr6

Msg_id: C4F5050B02904EEEAD2842405A0BDE2A Correlation: Corr7

Msg_id: E4D923A4CB4B4DF2B64B8421A88FFC42 Correlation: Corr8

Msg_id: BE199053188648AE8FA238A01A5C9CD1 Correlation: Corr9

Msg_id: 8991E793D2DB41F5B3F9D00D283B6F6D Correlation: Corr10

Now we can dequeue the 5th (correlation) and 8th (msg_id) message:

Declare

V_payload Raw(200);

V_msgid Raw(200);

V_deq_options Dbms_aq.dequeue_options_t;

v_msg_properties dbms_aq.message_properties_t;

Begin

v_deq_options.correlation:='Corr5';

Dbms_aq.dequeue(Queue_name=>'xxx_test_q'

,Message_properties=>V_msg_properties

,dequeue_options=>V_deq_options

,Payload=>V_payload

,Msgid=>V_msgid);

Dbms_output.Put_line(utl_raw.cast_to_varchar2(V_payload));

v_deq_options.correlation:=NULL;

v_deq_options.msgid:='E4D923A4CB4B4DF2B64B8421A88FFC42';

Dbms_aq.dequeue(Queue_name=>'xxx_test_q'

,Message_properties=>V_msg_properties

,dequeue_options=>V_deq_options

              ,Payload=>V_payload

,Msgid=>V_msgid);

Dbms_output.Put_line(utl_raw.cast_to_varchar2(V_payload));

End;

This is the fifth message

This is the eighth message

Note how we have to set the correlation back to NULL for the second dequeue. Otherwise we would be trying to dequeue a message with correlation ‘Corr5’ and the specified msg_id. Since that message does not exist, our procedure will just wait for the message to appear.

By default when you dequeue from an empty queue, or try to dequeue a non-available message, the dequeue will wait indefinitely for a message to appear. You can control this behavior with the dequeue options.

V_deq_options.wait := 10; — to wait 10 seconds. Any number of 0 or higher is allowed.

V_deq_options.wait := dbms_aq.no_wait; — not waiting for the message.

V_deq_options.wait := dbms_aq.forever; — wait indefinitely 

Do note that when the time-out is reached an ’ORA-25228: timeout in dequeue from <queue> while waiting for a message’ raised. So you will need to handle the exception.

One more feature to consider is the browsing mode. So far we have seen the messages that we dequeued were removed from the queue (or at least got status ‘Processed’). By setting the dequeue options, we can first inspect messages before dequeuing them. Consider the following. We have 8 messages left in our queue:

Declare

V_payload Raw(200);

V_msgid Raw(200);

V_deq_options Dbms_aq.dequeue_options_t;

v_deq_options_rm dbms_aq.dequeue_options_t;

v_msg_properties dbms_aq.message_properties_t;

Begin

v_deq_options.wait:=dbms_aq.no_wait;

v_deq_options.dequeue_mode:=DBMS_AQ.BROWSE;

for i in 1..10 loop

begin

   Dbms_aq.dequeue(Queue_name=>'xxx_test_q'

   ,Message_properties=>V_msg_properties

   ,dequeue_options=>V_deq_options

   ,Payload=>V_payload

   ,Msgid=>V_msgid);

   Dbms_output.Put_line(utl_raw.cast_to_varchar2(V_payload));

   dbms_output.put_line(v_msg_properties.correlation);

   if v_msg_properties.correlation='Corr6' then

       v_deq_options.dequeue_mode:=DBMS_AQ.REMOVE;

       v_deq_options.msgid:=v_msgid;

       Dbms_aq.dequeue(Queue_name=>'xxx_test_q'

                      ,Message_properties=>V_msg_properties

                      ,dequeue_options=>V_deq_options

                     ,Payload=>V_payload

                      ,Msgid=>V_msgid);

      Dbms_output.Put_line(utl_raw.cast_to_varchar2(V_payload));

      v_deq_options.dequeue_mode:=DBMS_AQ.BROWSE;

      v_deq_options.msgid:=NULL;

   end if;

   exception

    when others then

      null;

   end;

end loop;

End;

This removed only the 6^th message from the queue, and left the others intact.

There are more options to the queuing / dequeuing like retrying failed attempts (rollback after a dequeue is considered a failed attempt) and queuing with a delay or an expiration time. But I think the information so far will allow you to test these options on a need-by basis.

Multi-consumer or publish-subscribe queues

Both ‘publish-subscribe’ and ‘multi-consumer’ are used for these queues. I think ‘multi-consumer’ is most often used informally. That will also be the one I will use in this article (even though ‘publish-subscribe’ is more accurate).

We build multi-consumer queues with dbms_aqadm again. But on creating the queue-table, we say that it has to be a multi-consumer queue-table:

Begin

dbms_aqadm.create_queue_table (queue_table=>’xxx_mc_test’

                           ,multiple_consumers=>TRUE

                           ,queue_payload_type=>’RAW’);

End;

Now we see more objects being created. The most important ones are:

Xxx_mc_test                      The queue table itself.

Table aq$_xxx_mc_test_s with information about the subscribers to the queue

Table aq$_xxx_mc_test_r with information about the rules for the subscriptions

IOT aq$_xxx_mc_test_h with historic information about dequeuing

IOT aq$_xxx_mc_test_i with dequeuing information

As you can see, a lot more information is stored for multi-consumer queues. In part this information has to do with the subscription and subscriber mechanism. But there is also the need to keep a history of the dequeuing, to know when a message has been dequeued by all subscribers.

We will be seeing the use of all the objects in a few minutes, when we start queuing and dequeuing messages.

When we try to enqueue messages on this queue now, we receive an ORA-24033: no recipients for message. This means we need to set up subscribers first. If we enqueue without a recipient list, the message will be made available for all subscribers.

To add a subscriber, we use the dbms_aqadm package and a new object_type: sys.aq$_agent.

This type is defined as an object of name , address and protocol. The last 2 are used in inter-system communication only.

We can just call the following procedure:

DECLARE

V_agent sys.aq$_agent;

BEGIN

   V_agent:= sys.aq$_agent('Agent1',NULL,NULL);

   DBMS_AQADM.ADD_SUBSCRIBER(queue_name=>'xxx_mc_test_q'

      ,subscriber=>v_agent);

END;

We can see the subscribers from the view aq$xxx_mc_test_s (or the underlying table: aq$_xxx_mc_test_s):

select * from aq$xxx_mc_test_s;

QUEUE        NAME   ADDRESS      PROTOCOL TRANSFORMATION

————– —— ——-     ——– ————–

XXX_MC_TEST_Q AGENT1              0    

Now let’s enqueue a message:

Declare

V_payload Raw(200);

V_msgid Raw(200);

V_enq_options Dbms_aq.enqueue_options_t;

v_msg_properties dbms_aq.message_properties_t;

Begin

v_payload := utl_raw.cast_to_raw('Hello world, again!');

Dbms_aq.enqueue(Queue_name=>'xxx_mc_test_q'

             ,Message_properties=>V_msg_properties

              ,enqueue_options=>V_enq_options

,Payload=>V_payload

             ,Msgid=>V_msgid);

Dbms_output.Put_line(utl_raw.cast_to_varchar2(V_msgid));

End;

Now when we look at the queue-view, we can see that a subscriber has been selected:

select queue,rawtohex(msg_id) msg_id,msg_state,consumer_name from aq$xxx_mc_test;

QUEUE        MSG_ID                        MSG_STATE CONSUMER_NAME

————– ——————————– ——— ————-

XXX_MC_TEST_Q BC4C48AC659946428F38F8BC3AB02184 READY     AGENT1

Now to dequeue the message, we also need to set the consumer_name in the dequeue_options. When enqueuing a message without a subscriber_name, it can be dequeued by all subscribers. But on dequeueing, the subscriber needs to identify itself.

Declare

V_payload Raw(200);

V_msgid Raw(200);

V_deq_options Dbms_aq.dequeue_options_t;

v_msg_properties dbms_aq.message_properties_t;

Begin

v_deq_options.consumer_name:='Agent1';

Dbms_aq.dequeue(Queue_name=>'xxx_mc_test_q'

,Message_properties=>V_msg_properties

,dequeue_options=>V_deq_options

,Payload=>V_payload

,Msgid=>V_msgid);

Dbms_output.Put_line(utl_raw.cast_to_varchar2(V_payload));

End;

Hello world, again!

Now when we check the queue contents, we see that the message is still there. Even after a commit, the message has been retained. Maybe you won’t see it on your system immediately. But then run:

Begin

Dbms_aqadm.stop_time_manager;

End;

And enqueue/dequeue a message again. Now when you look in xxx_mc_test or aq$xxx_mc_test, you will see the message being retained (with status ‘PROCESSED’). When you start the time_manager again, the message will disappear after some time.

The reason for this, is that Oracle enhances concurrency by using a separate table (IOT) for the dequeuing. When we enqueue a message again:

Declare

V_payload Raw(200);

V_msgid Raw(200);

V_enq_options Dbms_aq.enqueue_options_t;

v_msg_properties dbms_aq.message_properties_t;

Begin

v_payload := utl_raw.cast_to_raw('Hello world');

Dbms_aq.enqueue(Queue_name=>'xxx_mc_test_q'

,Message_properties=>V_msg_properties

,enqueue_options=>V_enq_options

,Payload=>V_payload

,Msgid=>V_msgid);

Dbms_output.Put_line(utl_raw.cast_to_varchar2(V_msgid));

End;

We can see the data in the dequeue-IOT:

select subscriber#, queue#, msg_enq_time, msgid from Aq$_xxx_mc_test_i;

SUBSCRIBER# QUEUE# MSG_ENQ_TIME                MSG_ID

———– —— ————————— ——————————–

          1      0 21-03-10 14:16:00,252000000 75E41875D957455B84D80B55AE06F81C       

Here the basic information about our message is recorded. After a subscriber dequeues the message it’s version of the record is deleted only from this table (Please try this yourself, to confirm). The queue-monitors are responsible for cleaning up the queue-table after all subscribers have dequeued the message.

Now let’s see what happens when we add a second subscriber for our queue:

DECLARE

V_agent sys.aq$_agent;

BEGIN

   V_agent:= sys.aq$_agent('Agent2',NULL,NULL);

   DBMS_AQADM.ADD_SUBSCRIBER(queue_name=>'xxx_mc_test_q'

      ,subscriber=>v_agent);

END;

Any messages that were enqueued already, won’t be available for this new subscriber. It can only dequeue messages enqueued after the subscriber was added.

Also you can’t just change subscribers in an existing session. If you try, you will get an ORA-25242: Cannot change subscriber name from string to string without FIRST_MESSAGE option.

As the message describes further, you need to change the navigation of the dequeue. The default navigation is next_message, which means that Oracle will read the queue in a read-consistent and ordered way. It will take a snapshot of the queue when the first message is dequeued, and will dequeue the messages in that order. Messages that were enqueued after the first dequeue, will be read after reading all the messages in the queue. Even if priority ordering means they are enqueued earlier.

An alternative navigation is ‘first_message’. When the navigation is set to ‘first_message’, Oracle will take a new snapshot before every dequeue, and start with the first message eligible for dequeuing.

Because we change subscribers, we need to set navigation to ‘First_message’, to force Oracle to take a new snapshot.

(Btw. If you would try ‘first_message’ with dequeue_mode ‘Browse’, you would never get beyond the first message. Try it!)

(Btw2. The same goes for changing the filter options like correlation.)

Let’s start a new session, and enqueue a new message:

Declare

V_payload Raw(200);

V_msgid Raw(200);

V_enq_options Dbms_aq.enqueue_options_t;

v_msg_properties dbms_aq.message_properties_t;

Begin

v_payload := utl_raw.cast_to_raw('Hello agents!');

Dbms_aq.enqueue(Queue_name=>'xxx_mc_test_q'

             ,Message_properties=>V_msg_properties

,enqueue_options=>V_enq_options

,Payload=>V_payload

,Msgid=>V_msgid);

Dbms_output.Put_line(utl_raw.cast_to_varchar2(V_msgid));

End;

The message is still the same in xxx_mc_test:

select q_name, rawtohex(msgid) msg_id,state,enq_time,enq_uid from xxx_mc_test

Q_NAME        MSG_ID                           STATE ENQ_TIME                    ENQ_UID             

————- ——————————– —– ————————— ——-

XXX_MC_TEST_Q 45F11423444747B99600BCD8E9B3141E     0 21-03-10 14:33:23,783000000 SYSTEM

But in the queue view, we now see 2 records:

select queue,msg_id,msg_state,enq_time,enq_user_id,consumer_name from aq$xxx_mc_test;

QUEUE          MSG_ID                           STATE ENQ_TIME          ENQ_USER_ID CONSUMER_NAME

————– ——————————– —– —————– ———– ————-

XXX_MC_TEST_Q 45F11423444747B99600BCD8E9B3141E READY 21-03-10 14:33:24 SYSTEM      AGENT1

XXX_MC_TEST_Q 45F11423444747B99600BCD8E9B3141E READY 21-03-10 14:33:24 SYSTEM      AGENT2

One record for each subscriber. We can see the same in the dequeue_iot and in the history table:

select subscriber#,queue#,msg_enq_time,msgid from aq$_xxx_mc_test_i;

SUBSCRIBER# QUEUE# MSG_ENQ_TIME                MSGID

———– —— ————————— ——————————–

          1      0 21-03-10 14:33:23,783000000 45F11423444747B99600BCD8E9B3141E

         21      0 21-03-10 14:33:23,783000000 45F11423444747B99600BCD8E9B3141E

Select Msgid,Subscriber#,Name,Dequeue_time,Dequeue_user From Aq$_xxx_mc_test_h;

MSGID                            SUBSCRIBER# NAME DEQUEUE_TIME DEQUEUE_USER

——————————– ———– —- ———— ————

45F11423444747B99600BCD8E9B3141E           1    0          

45F11423444747B99600BCD8E9B3141E          21    0          

Now when we dequeue the message, the queue table is not updated:

Select Rawtohex(Msgid) Msg_id,State,Enq_time,Enq_uid,deq_time,deq_uid From Xxx_mc_test;

MSG_ID                           STATE ENQ_TIME                    ENQ_UID DEQ_TIME DEQ_UID             

——————————– —– ————————— ——- ——– ——-

45F11423444747B99600BCD8E9B3141E     0 21-03-10 14:33:23,783000000 SYSTEM

However, the queue view reflects that the message has been dequeued by one subscriber.

Select Queue,Msg_id,Msg_state,Enq_time,Enq_user_id,Consumer_name From Aq$xxx_mc_test;

QUEUE         MSG_ID                           MSG_STATE ENQ_TIME          ENQ_USER CONSUMER_NAME

————- ——————————– ——— —————– ——– ————-

XXX_MC_TEST_Q 45F11423444747B99600BCD8E9B3141E PROCESSED 21-03-10 14:33:24 SYSTEM   AGENT1

XXX_MC_TEST_Q 45F11423444747B99600BCD8E9B3141E READY     21-03-10 14:33:24 SYSTEM   AGENT2

The record for Agent1 has been deleted from the dequeue-IOT:

select subscriber#,queue#,msg_enq_time,msgid from aq$_xxx_mc_test_i;

SUBSCRIBER# QUEUE# MSG_ENQ_TIME                MSGID

———– —— ————————— ——————————–

         21      0 21-03-10 14:33:23,783000000 45F11423444747B99600BCD8E9B3141E

And the history table also shows the dequeue:

Select msgid, subscriber#, Dequeue_time,Dequeue_user From Aq$_xxx_mc_test_h;

MSGID                            SUBSCRIBER# DEQUEUE_TIME                DEQUEUE_USER

45F11423444747B99600BCD8E9B3141E          21   21-03-10 14:33:23,783000000 SYSTEM

45F11423444747B99600BCD8E9B3141E           1

To dequeue the message for ‘Agent2’. We of course need to set the navigation to ‘First_message’:

Declare

V_payload Raw(200);

V_msgid Raw(200);

V_deq_options Dbms_aq.dequeue_options_t;

v_msg_properties dbms_aq.message_properties_t;

Begin

V_deq_options.Wait := Dbms_aq.No_wait;

V_deq_options.Navigation:=Dbms_aq.First_message;

v_deq_options.consumer_name:='Agent2';

Dbms_aq.dequeue(Queue_name=>'xxx_mc_test_q'

,Message_properties=>V_msg_properties

,dequeue_options=>V_deq_options

,Payload=>V_payload

,Msgid=>V_msgid);

Dbms_output.Put_line(utl_raw.cast_to_varchar2(V_payload));

End;

Now after the QMON has processed the queue, the records will be deleted from all queues. (When a retention time has been set, the records will of course be retained for that time).

Rules for multi-consumer queues

So far we have seen different kinds of filtering for dequeuing messages. A new option comes with multi-consumer queues, where different subscribers can put a filter on their subscriptions. These filters (rules) can take the form of (complex) predicates that return a Boolean value. The rule can reference both message_properties as payload. To reference the payload, use a qualifier of ‘tab.user_data’.

Let’s build a new queue. To make optimal use of the ‘rule’-functionality we’ll use a custom type that can be referred to in the ‘rules’. The type that we’ll use is loosely based on the emp table.

create type t_emp as object

(empno   number

,ename   varchar2(10)

,job     varchar2(9)

);

BEGIN

DBMS_AQADM.CREATE_QUEUE_TABLE (queue_table => 'xxx_rule_test'

,queue_payload_type => 't_emp');

DBMS_AQADM.CREATE_QUEUE(queue_name => 'xxx_rule_test_q'

 ,queue_table => 'xxx_rule_test');

DBMS_AQADM.START_QUEUE (queue_name => 'xxx_rule_test_q');

END;

We add 2 subscribers to this queue.

DECLARE

V_agent sys.aq$_agent;

BEGIN

V_agent:= sys.aq$_agent('HR_President',NULL,NULL);

DBMS_AQADM.ADD_SUBSCRIBER(queue_name=>'xxx_rule_test_q'

   ,subscriber=>v_agent

   ,rule=>'tab.user_data.job=''President''');

V_agent:= sys.aq$_agent('HR_Employee',NULL,NULL);

DBMS_AQADM.ADD_SUBSCRIBER(queue_name=>'xxx_rule_test_q'

   ,subscriber=>v_agent);

END;

Note how the agent ‘HR_President’ has a rule added to its subscription. Only messages where the job attribute of the payload is ‘President’ are eligible for dequeuing by this agent. Let’s enqueue some messages on this queue.

Declare

V_payload t_emp;

V_msgid Raw(200);

V_enq_options Dbms_aq.enqueue_options_t;

v_msg_properties dbms_aq.message_properties_t;

Begin

v_payload := t_emp(1,'Jones','Manager');

Dbms_aq.enqueue(Queue_name=>'xxx_rule_test_q'

             ,Message_properties=>V_msg_properties

,enqueue_options=>V_enq_options

,Payload=>V_payload

,Msgid=>V_msgid);

Dbms_output.Put_line(utl_raw.cast_to_varchar2(V_msgid));

v_payload := t_emp(2,'King','President');

Dbms_aq.enqueue(Queue_name=>'xxx_rule_test_q'

             ,Message_properties=>V_msg_properties

,enqueue_options=>V_enq_options

,Payload=>V_payload

,Msgid=>V_msgid);

Dbms_output.Put_line(utl_raw.cast_to_varchar2(V_msgid));

End;

Now we have 2 messages. Only one of which matches the rule for the ‘HR_President’ subscriber. The ‘HR_Employee’ subscriber does not have any rule, and is thus eligible for all messages. We can see this when we query the queue-view:

select queue,rawtohex(msg_id) msg_id,msg_state,consumer_name from aq$xxx_rule_test;

QUEUE        MSG_ID                              MSG_STATE CONSUMER_NAME

————— ——————————– ——— ————-

XXX_RULE_TEST_Q 4D0FF7A800834559809AD90AFCA81444 READY     HR_EMPLOYEE

XXX_RULE_TEST_Q E5A2FDFD8EE942349E9BC9DEE88CEB10 READY     HR_EMPLOYEE

XXX_RULE_TEST_Q E5A2FDFD8EE942349E9BC9DEE88CEB10 READY     HR_PRESIDENT

We see that both messages are enqueued for the ‘HR_Employee’. But only the message with the job ‘President’ is enqueued for the the ‘HR_President’.

Let’s dequeue the messages as ‘HR_President’ first, then as ‘HR_Employee’.

Declare

V_payload t_emp;

V_msgid Raw(200);

V_deq_options Dbms_aq.dequeue_options_t;

v_msg_properties dbms_aq.message_properties_t;

Begin

v_deq_options.wait := dbms_aq.no_wait;

v_deq_options.consumer_name:='HR_President';

Dbms_aq.dequeue(Queue_name=>'xxx_rule_test_q'

,Message_properties=>V_msg_properties

,dequeue_options=>V_deq_options

,Payload=>V_payload

,Msgid=>V_msgid);

Dbms_output.Put_line(v_payload.empno||' '||v_payload.ename);

end;

2 King

This time the first message enqueued was ignored for this subscriber. Only the message that met its rule was dequeued. When dequeuing as the ‘HR_Employee’ both messages will be dequeued.

Declare

V_payload t_emp;

V_msgid Raw(200);

V_deq_options Dbms_aq.dequeue_options_t;

v_msg_properties dbms_aq.message_properties_t;

Begin

v_deq_options.wait := dbms_aq.no_wait;

v_deq_options.navigation := dbms_aq.first_message;

v_deq_options.consumer_name:='HR_Employee';

Dbms_aq.dequeue(Queue_name=>'xxx_rule_test_q'

,Message_properties=>V_msg_properties

,dequeue_options=>V_deq_options

,Payload=>V_payload

,Msgid=>V_msgid);

Dbms_output.Put_line(v_payload.empno||' '||v_payload.ename);

v_deq_options.navigation := dbms_aq.next_message;

Dbms_aq.dequeue(Queue_name=>'xxx_rule_test_q'

,Message_properties=>V_msg_properties

,dequeue_options=>V_deq_options

,Payload=>V_payload

,Msgid=>V_msgid);

Dbms_output.Put_line(v_payload.empno||' '||v_payload.ename);

end;

1 Jones

2 King

After these dequeues, the queue is empty for these subscribers. The only message eligible for ‘HR_President’ was the message with ‘2,King,President’. ‘HR_Employee’ was eligible for both messages.

Remember that the ‘Rule’ must evaluate to a Boolean value. Valid references are to ‘tab.user_data.’, for object_type payloads. Also columns like ‘priority’ or ‘correlation’ from the message properties can be referenced in the rule.

Common issues with queues and troubleshooting

Above we already saw several error messages related to queues. Most of them can be expected, and should be handled in the code.

The most common issues with queues are from queues not started, or not started for enqueuing or dequeuing. The error messages for this should be quite clear, and you can just start the queue with the ‘dbms_aqadm.start_queue’ package. Note that when the queue is started for enqueuing or dequeuing only, you need to stop it first, then start again with the correct options enabled.

Another issue may occur because of the AQ error handling system. A dequeue with dequeue_mode ‘REMOVE’ that needs to roll back afterwards, is considered a failed attempt. When the number of failed attempts exceed the retry count of the queue, the message will be moved to the Exception queue. The message remains in the queue table, but with status 3: Expired. The exception_queue field will be set to the name of the exception queue.

These messages are not available for dequeuing anymore. They must be dequeued from the exception queue.

To dequeue from an exception queue, it first needs to be enabled for dequeuing. (It cannot be enabled for enqueuing). Also no subscriber_name is allowed for the dequeue.

Begin

Dbms_aqadm.Start_queue(Queue_name=>'aq$_xxx_mc_test_e',Enqueue=>False,Dequeue=>True);

end;

Declare

V_payload Raw(200);

V_msgid Raw(200);

V_deq_options Dbms_aq.dequeue_options_t;

v_msg_properties dbms_aq.message_properties_t;

Begin

V_deq_options.Wait := Dbms_aq.No_wait;

Dbms_aq.dequeue(Queue_name=>'AQ$_XXX_MC_TEST_E'

,Message_properties=>V_msg_properties

,dequeue_options=>V_deq_options

       ,Payload=>V_payload

,Msgid=>V_msgid);

Dbms_output.Put_line(Utl_raw.Cast_to_varchar2(V_payload));

End;

The last issue to note with queues (especially in eBS systems) is described in Metalink note  267137.1. If multi-consumer queues are created in an ASSM tablespace, or when using freelist groups, QMON will not perform space management on the IOT’s. This will result in ever growing IOT’s and eventually in deteriorating performance.