ElasticSearch Server

Continuing my previous post, second book that I was working for PacktPub as technical reviewer was "Elasticsearch Server".
A very good book for this development based on the library Lucene, with many examples and concepts in order to exploit the full potential of free text searches using ElasticSearch.
As with the book described in the previous post (Apache Solr 4 Cookbook) this book is a perfect resource used during the development of "Scotas's Push Connector" because it allows me to integrate and exploit their full potential.

Overview

• Learn the basics of ElasticSearch like data indexing, analysis, and dynamic mapping
• Query and filter ElasticSearch for more accurate and precise search results
• Learn how to monitor and manage ElasticSearch clusters and troubleshoot any problems that arise

Table of Contents

• Chapter 1: Getting Started with ElasticSearch Cluster
• Chapter 2: Searching Your Data
• Chapter 3: Extending Your Structure and Search
• Chapter 4: Make Your Search Better
• Chapter 5: Combining Indexing, Analysis, and Search
• Chapter 6: Beyond Searching
• Chapter 7: Administrating Your Cluster
• Chapter 8: Dealing with Problems

Authors

Rafał Kuć

Rafał Kuć is a born team leader and software developer. Currently working as a Consultant and a Software Engineer at Sematext Inc, where he concentrates on open source technologies such as Apache Lucene and Solr, ElasticSearch, and Hadoop stack. He has more than 10 years of experience in various software branches, from banking software to e-commerce products. He is mainly focused on Java, but open to every tool and programming language that will make the achievement of his goal easier and faster. Rafał is also one of the founders of the solr.pl site, where he tries to share his knowledge and help people with their problems with Solr and Lucene. He is also a speaker for various conferences around the world such as Lucene Eurocon, Berlin Buzzwords, and ApacheCon. Rafał began his journey with Lucene in 2002 and it wasn't love at first sight. When he came back to Lucene later in 2003, he revised his thoughts about the framework and saw the potential in search technologies. Then Solr came and that was it. From then on, Rafał has concentrated on search technologies and data analysis. Right now Lucene, Solr, and ElasticSearch are his main points of interest. Rafał is also the author of Apache Solr 3.1 Cookbook and the update to it—Apache Solr 4 Cookbook—published by Packt Publishing.

Marek Rogoziński

Marek Rogoziński is a software architect and consultant with more than 10 years of experience. His specialization concerns solutions based on open source projects such as Solr and ElasticSearch. He is also the co-funder of the solr.pl site, publishing information and tutorials about the Solr and Lucene library. He currently holds the position of Chief Technology Officer in Smartupz, the vendor of the Discourse™ social collaboration software.

Conclusion

Many developers know Lucene, but do not know the product ElasticSearch, to know or want to know this book is going to enter in the product and the potential of this.

Apache Solr 4 Cookbook

During my summer I had the chance to work for Packtpub as technical reviewer of two great books.
First "Apache Solr 4 Cookbook" is a very good book to entered into the world of free-text search integrated to any development or portal with real and practical examples using the latest version of the Apache Solr search.

Overview

• Learn how to make Apache Solr search faster, more complete, and comprehensively scalable
• Solve performance, setup, configuration, analysis, and query problems in no time
• Get to grips with, and master, the new exciting features of Apache Solr 4

Table of Contents

• Chapter 1: Apache Solr Configuration
• Chapter 2: Indexing Your Data
• Chapter 3: Analyzing Your Text Data
• Chapter 4: Querying Solr
• Chapter 5: Using the Faceting Mechanism
• Chapter 6: Improving Solr Performance
• Chapter 7: In the Cloud
• Chapter 8: Using Additional Solr Functionalities
• Chapter 9: Dealing with Problems
• Appendix: Real-life Situations

Author

Rafał Kuć is a born team leader and software developer. Currently working as a Consultant and a Software Engineer at Sematext Inc, where he concentrates on open source technologies such as Apache Lucene and Solr, ElasticSearch, and Hadoop stack. He has more than 10 years of experience in various software branches, from banking software to e-commerce products. He is mainly focused on Java, but open to every tool and programming language that will make the achievement of his goal easier and faster. Rafał is also one of the founders of the solr.pl site, where he tries to share his knowledge and help people with their problems with Solr and Lucene. He is also a speaker for various conferences around the world such as Lucene Eurocon, Berlin Buzzwords, and ApacheCon. Rafał began his journey with Lucene in 2002 and it wasn't love at first sight. When he came back to Lucene later in 2003, he revised his thoughts about the framework and saw the potential in search technologies. Then Solr came and that was it. From then on, Rafał has concentrated on search technologies and data analysis. Right now Lucene, Solr, and ElasticSearch are his main points of interest. Rafał is also the author of Apache Solr 3.1 Cookbook and the update to it—Apache Solr 4 Cookbook—published by Packt Publishing.

Conclusion

If you are about to start a development or entered into the world of free text this book is a very good investment in time and resources, practical examples really serve to acquire new concepts in a simple and practical with minimal effort.

Implementing Oracle parallel shared server process in Java inside the Database

Behind the implementation of latest LDI open source project and the OLS products there is a functionality not well know by Oracle Java database developers, I called it Parallel Shared Server process.

The idea is to have an Oracle shared server process running during the  life-time of the instance, which means a process automatically started during database startup and stopped during database shutdown.

So which functionality this process can implement?, on LDI is an RMI server, on OLS is lightweight HTTP server, but basically you can implement anything you need for example getting information from another process and fill some table, getting statistical, consuming web services, etc. etc.

Let see in some example how it works.

We will create a TEST user and creates some Java classes running a simple Hello World RMI server.

SQL> conn / as sysdba

SQL> create user test identified by test
  2  default tablespace users
  3  temporary tablespace temp
  4  quota unlimited on users;

SQL> grant connect,resource,create any job to TEST;

SQL> exec dbms_java.grant_permission( 'TEST', 'SYS:java.net.SocketPermission', 'localhost:1024-', 'listen,resolve');
SQL> exec dbms_java.grant_permission( 'TEST', 'SYS:java.net.SocketPermission', 'localhost:1024-', 'accept, resolve');
SQL> exec dbms_java.grant_permission( 'TEST', 'SYS:java.net.SocketPermission', 'localhost:1024-', 'connect, resolve');
SQL> exec dbms_java.grant_permission( 'TEST', 'SYS:java.lang.RuntimePermission', 'setContextClassLoader', '' );

The RMI interface and server implementation running on TEST user.

SQL> conn test/test
SQL> create or replace and compile java source named "mytest.Hello" as
package mytest;
import java.rmi.Remote;
import java.rmi.RemoteException;
public interface Hello extends Remote {
    String sayHello() throws RemoteException;
    int nextCount() throws RemoteException;
}
/
SQL> create or replace and compile java source named "mytest.HelloImpl" as
package mytest;
import java.rmi.Naming;
import java.rmi.RemoteException;
import java.rmi.RMISecurityManager;
import java.rmi.registry.LocateRegistry;
import java.rmi.server.UnicastRemoteObject;
public class HelloImpl extends UnicastRemoteObject implements Hello {
    static int counter = 0;
   
    public HelloImpl() throws RemoteException {
        super();
    }
    public String sayHello() {
        return "Hello World!";
    }
    public static void main(String[] args) {
        // Create and install a security manager
        if (System.getSecurityManager() == null) {
            System.setSecurityManager(new RMISecurityManager());
        }
        try {
            HelloImpl obj = new HelloImpl();
            LocateRegistry.createRegistry(1099);
            // Bind this object instance to the name "HelloServer"
            Naming.rebind("//localhost/HelloServer", obj);
            System.out.println("HelloServer bound in registry");
        } catch (Exception e) {
            System.out.println("HelloImpl err: " + e.getMessage());
            e.printStackTrace();
        }
    }
    public synchronized int nextCount() {
        return counter++;
    }
}
/
SQL> create or replace procedure HelloServ(srvName IN VARCHAR2) as LANGUAGE JAVA NAME
        'mytest.HelloImpl.main(java.lang.String [])';
/

SQL> begin
  -- Start a Cron like process (DBMS_SCHEDULER)
  DBMS_SCHEDULER.CREATE_JOB(
   job_name          =>  'HelloServJob',
   job_type          =>  'PLSQL_BLOCK',
   job_action        =>  'begin
     HelloServ(''HelloServer'');
     exception when others then
        null;
     end;',
   start_date        =>  SYSDATE,
   enabled           => false,
   auto_drop         => false);
  DBMS_SCHEDULER.SET_ATTRIBUTE_NULL (
   name           =>   'HelloServJob',
   attribute      =>   'MAX_FAILURES');
end;
/
commit;

Now we can register two database instance trigger to automatically start and stop the job.

SQL> conn / as sysdba
SQL> CREATE OR REPLACE TRIGGER start_test_srv
  AFTER STARTUP ON DATABASE
BEGIN
  -- Start a Cron like process (DBMS_SCHEDULER)
  DBMS_SCHEDULER.ENABLE('TEST.HelloServJob');
END;
/
SQL> CREATE OR REPLACE TRIGGER stop_test_srv
  BEFORE SHUTDOWN ON DATABASE
BEGIN
  -- Start a Cron like process (DBMS_SCHEDULER)
  DBMS_SCHEDULER.STOP_JOB('TEST.HelloServJob',force=>true);
EXCEPTION WHEN OTHERS THEN
  null;
END;
/

If we process to do a shutdown/startup sequence the server will up and running, also we can start the server manually by executing:

SQL> conn / as sysdba
SQL> exec DBMS_SCHEDULER.ENABLE('TEST.HelloServJob');
SQL> commit;

after doing that we can see at $ORACLE_BASE/diag/rdbms/orcl/orcl/trace a .trc file associated with the parallel shared server process which is up and running:

-bash-4.2$ cat orcl_j000_10411.trc
Trace file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_j000_10411.trc
Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
ORACLE_HOME = /u01/app/oracle/product/11_2_0_2_0/dbhome_1
System name:    Linux
Node name:      localhost.localdomain
Release:        2.6.38.7-server-1mnb2
Version:        #1 SMP Sun May 22 22:59:25 UTC 2011
Machine:        i686
Instance name: orcl
Redo thread mounted by this instance: 1
Oracle process number: 25
Unix process pid: 10411, image: oracle@localhost.localdomain (J000)

*** 2011-11-28 18:05:41.091
*** SESSION ID:(151.35) 2011-11-28 18:05:41.091
*** CLIENT ID:() 2011-11-28 18:05:41.091
*** SERVICE NAME:(SYS$USERS) 2011-11-28 18:05:41.091
*** MODULE NAME:(DBMS_SCHEDULER) 2011-11-28 18:05:41.091
*** ACTION NAME:(HELLOSERVJOB) 2011-11-28 18:05:41.091

HelloServer bound in registry

and this process is listening into the default RMI port 1099, we can see that using:

-bash-4.2$ netstat -anp|grep ora_j0
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp        0      0 :::19189                    :::*                        LISTEN      10411/ora_j000_orcl
tcp        0      0 :::1099                     :::*                        LISTEN      10411/ora_j000_orcl 

and that's all, we can connect using an RMI client from another client session, for example:

SQL> create or replace and compile java source named "mytest.HelloClient" as
package mytest;
import java.rmi.Naming;
import java.rmi.RemoteException;
public class HelloClient {
    Hello obj = null;
    public HelloClient() {
        try {
            obj = (Hello)Naming.lookup("//localhost/HelloServer");
        } catch (Exception e) {
            System.out.println("HelloApplet exception: " + e.getMessage());
            e.printStackTrace();
        }
    }
    public String sayHello() throws RemoteException {
        return obj.sayHello();
    }
    public int nextCount() throws RemoteException {
        return obj.nextCount();
    }
    public static void main(String[] args) throws RemoteException {
        HelloClient helloClient = new HelloClient();
        System.out.println(helloClient.sayHello());
        System.out.println(helloClient.nextCount());
    }
}
/
SQL> create or replace procedure HelloClient(srvName IN VARCHAR2) as LANGUAGE JAVA NAME
'mytest.HelloClient.main(java.lang.String [])';
/

SQL> set define ?
SQL> set serverout on
SQL> exec dbms_java.set_output(32000);
SQL> exec HelloClient('HelloServer');
0
SQL> exec HelloClient('HelloServer');
1

Note that the server is state-full which means that preserve the state across calls, if we exit from above SQLPlus and connect again we will see that:

SQL> set define ?
SQL> set serverout on
SQL> exec dbms_java.set_output(32000);
SQL> exec HelloClient('HelloServer');
2
SQL> exec HelloClient('HelloServer');
3

Concluding this post I would like to remark that this parallel shared server process is running into RDBMS space and is not like starting an RMI server into the middle tier. the big difference is that all SQL access is implemented accessing directly to the RDBMS structures into the SGA because is using the internal JDBC driver.

New release of Lucene Domain Index based on 3.4.0 code

This new release of Lucene Domain Index (LDI) has been in the new SVN for a long time, but due a lot of works with the commercial version Scotas never went public in binary installation :(
Several thing happen during this time:

• New web site (thanks a lot to Peter Wehner for the conversion from the Google docs)
• New hosting at SF.net (now is separate SVN from the previous one CVS at DBPrism)

The change log of this version is:

• Use latest merge policy implementation TieredMergePolicy
• Use total RAM reported by getJavaPoolSize() when setting MaxBufferedDocs
• Better error reporting when an Analyzer is not found.
• Replaced execute immediate with open-fech-close functionality to avoid core dump on 10g when double check for deleted rowid
• Included a back-port version of JUnit4 to jdk1.4 version for 10g releases
• Added a parallel updater process, when working in OnLine mode this process do write operations on LDI structure on behalf of the AQ process
• Delete do not longer required a write exclusive lock on index storage, now deletes are also en-queued as inserts or updates
• Updated source to Lucene 3.4.0 code, removed some deprecated API

Download latest binary distribution at 3.4.0 directory of SF.net download area (tested with 10g/11gR2).
The addition of a new parallel shared server process is the major change which speed up a lot DML operations, I'll write in a new post on how this parallel shared server technique works.
Please report any issue during the installation or bugs at the Support Area of the project.

Long time no post

I found that my last post was on Sep. 14 2010, that's too bad.
The reason of that is the startup called Scotas I worked on that almost since December of 2010 to engineering a next level of Oracle and Lucene project integration, this is by adding the Solr stack.
Its a natural evolution, Solr provides a lot of common functionality working on top of  Lucene required for the enterprise level solution.
But the integration required a set of implementation challenges to work, once of them is the implementation of long live shared server process running in Java and which implements a new dedicated HTTP listener working as an Oracle slave process, I will explain this topic on another technical post.
Another additions included in a set of products are the push technology spread on the integration with Solr running externally, ElasticSearch, Cassandra and HBase.
Behind this technology we extended the idea of Oracle/Solr integration, the Oracle ODCI API,. which enabled a NRT (Near Real Time) synchronization of the enterprise data with NoSQL layer for example.
Near Real Time means that once you commit the changes on the RDBMS layer thy are propagated automatically to the other side practically with 0 delay, and for the two way replicator such as Solr PC and ElasticSearch deletions are in real-time avoiding the false positive hits of deleted rows.
Near Real Time synchronization is declarative, no programming effort is required and allows multiples columns of one table, or multiples columns of many tables connected by a foreign key.
Well next week is the OOW 2011, I'll Be there and I Hope You Will, Too and for anybody that are coming to the event and are interested on that technology we could meet there, just drop me an email to mochoa-at-scotas.com

New Oracle Lucene Domain Index release based on Lucene 3.0.2

Just a few words to announce a new release of Oracle Lucene Domain Index, this zip is valid for 10g and 11g database version (10g using back-ported classes from 1.5 to 1.4)
This release is compiled using Lucene 3.0.2 version and incorporates a set of new features added, here the list:

• Added a long awaited functionality, a parallel/shared/slave search process used during a start-fetch-close and CountHits function
• Added lfreqterms ancillary operator returning the freq terms array of rows visited
• Added lsimilarity ancillary operator returning a computed Levenshtein distance of the row visited
• Added a ldidyoumean pipeline table function using DidYouMean.indexDictionary storage
• Added test using SQLUnit

The bigger addition is the Parallel-Shared-Slave search process, this architectural change was in my to-do list for a long time and finally I added in this release :)
The idea behind this is to have a new Oracle process started by the DBMS_SCHEDULER sub-system during the database startup process and stopped immediately before shutdown.
Now this process is responsible for implementing the ODCI methods start-fetch-close/count-hit on behalf of the client process (process associated to an specific user session) which connect to the shared-slave process by using RMI.
With this new architecture we have two principal benefits:

• Reduce memory consumption
• Increase Lucene Cache Hits

Less memory consumption because the internal OJVM implementation is attached to a client session, so the Java space used by Lucene structures is isolated and independent from another concurrent session, now all Lucene memory structures used during index scan process are created in a shared process and then not replicated.
Also if one session submits a Lucene search, this search is cached for subsequent queries, all subsequent queries coming from the same client session or any other which are associated to the same index and with the same Query string implies a hit.
I'll explain more in detail this new architecture in another post also showing how many parallel process can work together when using Parallel Indexing and Searching.
On the other hand next week I'll be at the Oracle OpenWorld 2010 in SFO presenting the session:

Schedule: Tuesday: 09:30AM
Session ID: S315660
Title: Database Applications Lifecycle Management
Event: JavaOne and Oracle Develop
Stream(s): ORACLE DEVELOP
Track(s): Database Development
Abstract: Complex applications, such as Java running inside the database, require an application lifecycle management to develop and delivery good code. This session will cover some best practices, tools, and experience managing and delivering code for running inside the database, including tools for debugging, automatic test, packaging, deployment, and release management. Some of the tools presented will include Apache Maven, JUnit, log4j, Oracle JDeveloper, and others integrated into the Oracle Java Virtual Machine (JVM) development environment.

See you there or at any of networking planned events :)

Near-shore development in Tandil-Argentina

This post is bit off topic about Oracle, but not at all.
I remember read by a first time the term near-shore development during OOW08 in a Information Week newsletter free at OTN Lounge.
Two years ago I see the increasing on development using this model in my City Tandil, 400Km far from Buenos Aires in Argentina.
Specially, I am working as external consultant with the company Temperies which was founded and growing in Tandil unlike other companies which are part of the Technology Park and came from Buenos Aires.
But why this market still growing in Tandil with well know economy changes?
First Why Tandil?

• Is a very nice small city (130000 habitants) surrounded by hills where the quality of life makes the difference in term of productivity of the development team, this is one of the bigger difference against other Argentine cities such as Buenos Aires, Cordoba or Rosario. Any worker can get his workplace in less than five minutes walking.
• Has a recognized University in Computer Science.
• Is cheap to live here, so salaries are not so high.

Second Why Argentina?
  - Is a country with a big difference in term of US$/Euro exchange rate compared with another countries like Chile for example, and believe me when an near-shore project is evaluated this point is very important because with the same money you can do the project several times :)
  - Most of the computer science graduated and under-graduated students have very good knowledge of English.
  - The time zone of Argentine is in middle of Europe and US West coast, so projects can be perfectly managed around the team synchronization, if we work with San Francisco, we work during our afternoon, if we work with Germany, we work during our morning, and thats all.
Finally I want to remark that there are plenty of offers in term of Software Factories in Tandil, from small companies with 10 to 20 employees to big factories with more than one hundred employees to fit with every need.
As I set at the beginning of this post I am working with Temperies a mid-range Software Factory which, in addition to the list of goodness exposed previously, have an strong commitment with Agile practices, Oracle and a widely experience with success stories in near-shoring projects around the world, and obviously the CIOs are good friends of mine :)
Well if anybody want to go deeper in this area just drop me an email to marcelo.ochoa gmail.com or see you at the Oracle Open World in September.