Monday, November 28, 2011

Implementing Oracle parallel shared server process in Java inside the Database

Behind the implementation of latest LDI open source project and the OLS products there is a functionality not well know by Oracle Java database developers, I called it Parallel Shared Server process.
The idea is to have an Oracle shared server process running during the  life-time of the instance, which means a process automatically started during database startup and stopped during database shutdown.
So which functionality this process can implement?, on LDI is an RMI server, on OLS is lightweight HTTP server, but basically you can implement anything you need for example getting information from another process and fill some table, getting statistical, consuming web services, etc. etc.
Let see in some example how it works.
We will create a TEST user and creates some Java classes running a simple Hello World RMI server.
SQL> conn / as sysdba
SQL> create user test identified by test
  2  default tablespace users
  3  temporary tablespace temp
  4  quota unlimited on users;
SQL> grant connect,resource,create any job to TEST;
SQL> exec dbms_java.grant_permission( 'TEST', 'SYS:java.net.SocketPermission', 'localhost:1024-', 'listen,resolve');
SQL> exec dbms_java.grant_permission( '
TEST', 'SYS:java.net.SocketPermission', 'localhost:1024-', 'accept, resolve');
SQL> exec dbms_java.grant_permission( '
TEST', 'SYS:java.net.SocketPermission', 'localhost:1024-', 'connect, resolve');
SQL> exec dbms_java.grant_permission( '
TEST', 'SYS:java.lang.RuntimePermission', 'setContextClassLoader', '' );
The RMI interface and server implementation running on TEST user.
SQL> conn test/test
SQL> create or replace and compile java source named "mytest.Hello" as
package mytest;
import java.rmi.Remote;
import java.rmi.RemoteException;
public interface Hello extends Remote {
    String sayHello() throws RemoteException;
    int nextCount() throws RemoteException;
}
/
SQL> create or replace and compile java source named "mytest.HelloImpl" as
package mytest;
import java.rmi.Naming;
import java.rmi.RemoteException;
import java.rmi.RMISecurityManager;
import java.rmi.registry.LocateRegistry;
import java.rmi.server.UnicastRemoteObject;
public class HelloImpl extends UnicastRemoteObject implements Hello {
    static int counter = 0;
   
    public HelloImpl() throws RemoteException {
        super();
    }
    public String sayHello() {
        return "Hello World!";
    }
    public static void main(String[] args) {
        // Create and install a security manager
        if (System.getSecurityManager() == null) {
            System.setSecurityManager(new RMISecurityManager());
        }
        try {
            HelloImpl obj = new HelloImpl();
            LocateRegistry.createRegistry(1099);
            // Bind this object instance to the name "HelloServer"
            Naming.rebind("//localhost/HelloServer", obj);
            System.out.println("HelloServer bound in registry");
        } catch (Exception e) {
            System.out.println("HelloImpl err: " + e.getMessage());
            e.printStackTrace();
        }
    }
    public synchronized int nextCount() {
        return counter++;
    }
}
/
SQL> create or replace procedure HelloServ(srvName IN VARCHAR2) as LANGUAGE JAVA NAME
        'mytest.HelloImpl.main(java.lang.String [])';
/
SQL> begin
  -- Start a Cron like process (DBMS_SCHEDULER)
  DBMS_SCHEDULER.CREATE_JOB(
   job_name          =>  'HelloServJob',
   job_type          =>  'PLSQL_BLOCK',
   job_action        =>  'begin
     HelloServ(''HelloServer'');
     exception when others then
        null;
     end;',
   start_date        =>  SYSDATE,
   enabled           => false,
   auto_drop         => false);
  DBMS_SCHEDULER.SET_ATTRIBUTE_NULL (
   name           =>   'HelloServJob',
   attribute      =>   'MAX_FAILURES');
end;
/
commit;
Now we can register two database instance trigger to automatically start and stop the job.
SQL> conn / as sysdba
SQL> CREATE OR REPLACE TRIGGER start_test_srv
  AFTER STARTUP ON DATABASE
BEGIN
  -- Start a Cron like process (DBMS_SCHEDULER)
  DBMS_SCHEDULER.ENABLE('TEST.HelloServJob');
END;
/
SQL> CREATE OR REPLACE TRIGGER stop_test_srv
  BEFORE SHUTDOWN ON DATABASE
BEGIN
  -- Start a Cron like process (DBMS_SCHEDULER)
  DBMS_SCHEDULER.STOP_JOB('TEST.HelloServJob',force=>true);
EXCEPTION WHEN OTHERS THEN
  null;
END;
/
If we process to do a shutdown/startup sequence the server will up and running, also we can start the server manually by executing:
SQL> conn / as sysdba
SQL> exec DBMS_SCHEDULER.ENABLE('TEST.HelloServJob');
SQL> commit;
after doing that we can see at $ORACLE_BASE/diag/rdbms/orcl/orcl/trace a .trc file associated with the parallel shared server process which is up and running:
-bash-4.2$ cat orcl_j000_10411.trc
Trace file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_j000_10411.trc
Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
ORACLE_HOME = /u01/app/oracle/product/11_2_0_2_0/dbhome_1
System name:    Linux
Node name:      localhost.localdomain
Release:        2.6.38.7-server-1mnb2
Version:        #1 SMP Sun May 22 22:59:25 UTC 2011
Machine:        i686
Instance name: orcl
Redo thread mounted by this instance: 1
Oracle process number: 25
Unix process pid: 10411, image: oracle@localhost.localdomain (J000)

*** 2011-11-28 18:05:41.091
*** SESSION ID:(151.35) 2011-11-28 18:05:41.091
*** CLIENT ID:() 2011-11-28 18:05:41.091
*** SERVICE NAME:(SYS$USERS) 2011-11-28 18:05:41.091
*** MODULE NAME:(DBMS_SCHEDULER) 2011-11-28 18:05:41.091
*** ACTION NAME:(HELLOSERVJOB) 2011-11-28 18:05:41.091

HelloServer bound in registry
and this process is listening into the default RMI port 1099, we can see that using:
-bash-4.2$ netstat -anp|grep ora_j0
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp        0      0 :::19189                    :::*                        LISTEN      10411/ora_j000_orcl
tcp        0      0 :::1099                     :::*                        LISTEN      10411/ora_j000_orcl 
and that's all, we can connect using an RMI client from another client session, for example:
SQL> create or replace and compile java source named "mytest.HelloClient" as
package mytest;
import java.rmi.Naming;
import java.rmi.RemoteException;
public class HelloClient {
    Hello obj = null;
    public HelloClient() {
        try {
            obj = (Hello)Naming.lookup("//localhost/HelloServer");
        } catch (Exception e) {
            System.out.println("HelloApplet exception: " + e.getMessage());
            e.printStackTrace();
        }
    }
    public String sayHello() throws RemoteException {
        return obj.sayHello();
    }
    public int nextCount() throws RemoteException {
        return obj.nextCount();
    }
    public static void main(String[] args) throws RemoteException {
        HelloClient helloClient = new HelloClient();
        System.out.println(helloClient.sayHello());
        System.out.println(helloClient.nextCount());
    }
}
/
SQL> create or replace procedure HelloClient(srvName IN VARCHAR2) as LANGUAGE JAVA NAME
'mytest.HelloClient.main(java.lang.String [])';
/
SQL> set define ?
SQL> set serverout on
SQL> exec dbms_java.set_output(32000);
SQL> exec HelloClient('HelloServer');
0
SQL> exec HelloClient('HelloServer');
1
Note that the server is state-full which means that preserve the state across calls, if we exit from above SQLPlus and connect again we will see that:

SQL> set define ?
SQL> set serverout on
SQL> exec dbms_java.set_output(32000);
SQL> exec HelloClient('HelloServer');
2
SQL> exec HelloClient('HelloServer');
3

Concluding this post I would like to remark that this parallel shared server process is running into RDBMS space and is not like starting an RMI server into the middle tier. the big difference is that all SQL access is implemented accessing directly to the RDBMS structures into the SGA because is using the internal JDBC driver.

Friday, November 25, 2011

New release of Lucene Domain Index based on 3.4.0 code

This new release of Lucene Domain Index (LDI) has been in the new SVN for a long time, but due a lot of works with the commercial version Scotas never went public in binary installation :(
Several thing happen during this time:

  • New web site (thanks a lot to Peter Wehner for the conversion from the Google docs)
  • New hosting at SF.net (now is separate SVN from the previous one CVS at DBPrism)

The change log of this version is:


  • Use latest merge policy implementation TieredMergePolicy
  • Use total RAM reported by getJavaPoolSize() when setting MaxBufferedDocs
  • Better error reporting when an Analyzer is not found.
  • Replaced execute immediate with open-fech-close functionality to avoid core dump on 10g when double check for deleted rowid
  • Included a back-port version of JUnit4 to jdk1.4 version for 10g releases
  • Added a parallel updater process, when working in OnLine mode this process do write operations on LDI structure on behalf of the AQ process
  • Delete do not longer required a write exclusive lock on index storage, now deletes are also en-queued as inserts or updates
  • Updated source to Lucene 3.4.0 code, removed some deprecated API

Download latest binary distribution at 3.4.0 directory of SF.net download area (tested with 10g/11gR2).
The addition of a new parallel shared server process is the major change which speed up a lot DML operations, I'll write in a new post on how this parallel shared server technique works.
Please report any issue during the installation or bugs at the Support Area of the project.