Takeaway from Evolving for the Cloud

Original presentation:

http://www.infoq.com/presentations/Evolving-Programming-for-the-Cloud

Universal Scalability Law:

Throughput limited by two factors:

    1. contention - bottleneck on shared resources
    2. coherency - communication and coordination among multiple nodes

Programing in the cloud:

Parallelism

routing requests
aggregating services
queuing work
Async I/O and Futures
workload/data/processing partition
pipelines, map-reduce

Layering
strict layered system
common code in frameworks
 -- configuration
 -- instrumentation
 -- Failure management

Services
composition
dependency
addressing
persistence
deployment

State management
Stateless instances
Durable state in persistent storage

Data model
Distributed key-value stores
Constrained by design (no ER, limited schema)
plan to shard
plan to de-normalize
  - optimize for reads
  - pre-calculate joins/aggregations/sorts
  - use async queue to update
  - tolerate transient inconsistency

Failure handling
Expect failures and handle them
 - Graceful degradation
 - timeouts and retries
 - throttling and back-off

Testing
Test early and often (Test-driven and Test-first)
Test end-to-end

Operation in the cloud:

DevOps mindset
 - automate, automate, automate
 - If you do it more than twice, script it

Configuration Injection
 - Inject at boot time and run-time
 - Separate code and configuration

Instrumentation
 - Fully instrument all components
 - Remotely attach/profile/debug components
 - Logging is insufficient
 - Use a framework

Monitoring
 - monitor requests end-to-end
 - monitor activity and performance
 - metrics, metrics, metrics
 - understand your typical system behavior

Metering
 - Variable cost
 - Efficiency matters (inefficiency multiplied by many instances)

Nearby restaurants for group dinner

Gochi Japanese Fusion Tapas

19980 E Homestead Rd
Sunnyvale, CA 95014
(408) 725-0542

Hours: Weekdays 11:30am-1:30pm, 6pm-10pm; Sat 5:30pm-10pm
Price: ~$15 - $20

Jang Su Jang (Korean)

3561 El Camino Real #10
Santa Clara, CA 95051
(408) 246-1212

Hours: Open Daily 11am-10pm
Price: ~$20 - $25

Maggiano (Italian)

3055 Olin Avenue #1000
San Jose, CA 95128
(408) 423-8973

Hours: Mon-Thu 11am-10pm; Fri-Sat 11am-11pm; Sun 12pm-10pm
Price: ~$30

Joyluck Place (Chinese Dim Sum)
10911 N Wolfe Rd
Cupertino, CA 95014
(408) 255-6988

Hours: Mon-Sun 11 am - 2:30 pm Mon-Sun 5:30 pm - 9:30 pm
Price: ~$20

Cheesecake Factory (American Chain)
3041 Stevens Creek Boulevard
Santa Clara
(408) 246-0092

Hours:
Price: ~$30 - $35

Strait Café (Singaporean)
333 Santana Row
San Jose
(408) 246-6320

Hours:
Price: ~$50

Banana Leaf (Malaysian)
182 Ranch Drive
Milpitas, CA 95035
(408) 719-9811

Hours: Close on Sunday
Price: ~$25

Layang Layang (Malaysian)
181 W Calaveras Blvd
Milpitas, CA 95035
(408) 263-6788

Hours: everyday
Price: ~$20

Sakoon (Indian)
357 Castro Street
Mountain View, CA 94041
(650) 965-2000

Hours: Monday to Sunday
Price: ~$20 (or $30 w/ private room)

Athena Grill (Mediterranean)
1505 Space Park Dr
Santa Clara, CA 95054 
(408) 567-9144 

www.athenagrill.com/

Hours: Monday to Saturday
Price: ~$15

unmappable character for encoding UTF8

Maven daily build starts to throw error for many Java source files with such an error "unmappable character for encoding UTF8". The message itself is complaining some characters are not good with javac -encoding UTF-8 which is the default encoding for our Linux build server.

Here is one example of these characters:
 * <p/>
 * <!¡ª0 indicate success ,1- indicate failure¨¤
 * <p/>
 * <status> 0 or 1<status>
 * <p/>
 * <!¡ªif status is 1 ,then have the following node¨¤

Here are a couple of options to resolve/work-around this issue:

1. Remove these characters from the source codes
2. Use utf-8 representations like copyright sign '\u00a9'
3. Add -encoding ISO-8859-1 parameter to javac command (cp1252, Latin-1 are equivalent encoding)
4. Maven: add property  <project.build.sourceEncoding>ISO-8859-1</project.build.sourceEncoding> 
5. Ant: Add encoding to build script
<javac srcdir="src" destdir="classes"   encoding="ISO-8859-1" debug="true" />     
6. Save java source file as UTF-8 (Developer can configure this in Eclipse IDE like below)

P.S.
Found the change caused the issue, we use maven-compiler-plugin, previously it was 1.5 version which was ok (with cp1252), but after change to 1.6, it will result in compilation error. We can add <encoding> to fix that because different JDK version might have different default encoding.
           <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <configuration>
                    <source>1.6</source>
                    <target>1.6</target>
                    <encoding>ISO-8859-1</encoding>
                </configuration>
            </plugin>

Collections.sort returns NullPointerException

NullPointerException (NPE) is a common but annoying runtime exception in production. 

Here is JDK documentation about null object or array
Thrown when an application attempts to use null in a case where an object is required. These include:

• Calling the instance method of a null object.
• Accessing or modifying the field of a null object.
• Taking the length of null as if it were an array.
• Accessing or modifying the slots of null as if it were an array.
• Throwing null as if it were a Throwable value.

Collections.sort() method returns NPE with below stack trace:
Exception in thread "main" java.lang.NullPointerException
    at java.util.Arrays.mergeSort(Arrays.java:1144)
    at java.util.Arrays.sort(Arrays.java:1079)
    at java.util.Collections.sort(Collections.java:117)

The reason is List, Set, HashMap etc allows null value which is ok to build the list/map/set, but will throw exception if we do sort using Collections.sort() method.

Here is one example about HashMap.

        HashMap<String, String> testMap = new HashMap<String, String>();
        testMap.put(null, "apple");
        testMap.put("key2", "orange");
        testMap.put("key3", "pear");
       
        List<String> keylist = new ArrayList<String>(testMap.keySet());
        List<String> valuelist = new ArrayList<String>(testMap.values());   
       
        System.out.println(keylist);
        System.out.println(valuelist);
       
        Collections.sort(keylist); // this will throw NPE
        Collections.sort(valuelist); // this will not throw NPE

However, for Hashtable, putting null to the map will throw NPE, for instance
        Hashtable<String, String> testTable = new Hashtable<String, String>();
        testTable.put(null, "1"); //this will throw NPE
        testTable.put(null, "2");
        testTable.put("key3", "3");        


Exception in thread "main" java.lang.NullPointerException
    at java.util.Hashtable.put(Hashtable.java:399)

Here is the difference between Hashtable and HashMap

	HashMap	Hashtable
Thread-safe	N	Y
Key nullable	Y	N
Better performance	Y	N

BufferedReader.readLine() hangs

I debugged a simple program to learn JavaCompiler in new JDK, but found BufferedReader.readLine() hangs for some reason. BufferedReader.readLine() returns null toindicate that you have reached the end of the data stream (i.e. the remote has closed). At all other times, readLine() will block until there is a complete
line (i.e. a message terminated by a newline). If you need to know that there is data before attempting to read, try to use BufferedReader.isReady() method.

Here is the sample code:
        Runtime run = Runtime.getRuntime();
        Process p = run.exec("java");

        BufferedInputStream in = new BufferedInputStream(p.getInputStream());
        BufferedReader br = new BufferedReader(new InputStreamReader(in));
       
        if (br.ready()) {
            String line;
            while ((line = br.readLine()) != null) {
                System.out.println(line);
            }
        }

Corrupted jar file complains invalid CEN header (bad signature)

When do maven build, we see below error info:
[INFO] ------------------------------------------------------------------------
[INFO] Compilation failure
error: error reading /home/root/.m2/repository/log4j/log4j/1.2.16/log4j-1.2.16.jar; invalid CEN header (bad signature)

error: error reading /home/root/.m2/repository/log4j/log4j/1.2.16/log4j-1.2.16.jar; invalid CEN header (bad signature)

[INFO] ------------------------------------------------------------------------
[INFO] For more information, run Maven with the -e switch
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1 minute 13 seconds
[INFO] Finished at: Thu Feb 10 09:54:17 PST 2011
[INFO] Final Memory: 52M/138M
[INFO] ------------------------------------------------------------------------

The root cause is jar file (in this case log4j-1.2.16.jar) corrupted. Jar file is de facto a Zip file which should be in binary format. The binaries can get corrupted if checked in from a windows system as text files, due to line ending conversion, and to keyword substitution. Therefore when we check in the jar file to CVS, need mark them as binary.

On Windows systems check-out of a non-binary file can lead to conversion of one-character line-endings to two-character - which corrupts a true binary like a jar. You might not notice this when checking out the same file on a unix/linux system as there is no special treatment for "text" files when checking out there.

We can use WinZip, 7-Zip to verify if the jar file is corrupted or not.

Java serialization

Definition:
In computer science, in the context of data storage and transmission, serialization is the process of converting a data structure or object into a format that can be stored (for example, in a file or memory buffer, or transmitted across a network connection link) and "resurrected" later in the same or another computer environment.[1] When the resulting series of bits is reread according to the serialization format, it can be used to create a semantically identical clone of the original object. For many complex objects, such as those that make extensive use of references, this process is not straightforward.

This process of serializing an object is also called deflating or marshalling an object.[2] The opposite operation, extracting a data structure from a series of bytes, is deserialization (which is also called inflating or unmarshalling).
http://en.wikipedia.org/wiki/Serialization

Java Serialization:
Java object serialization is used to persist Java objects to a file, database, network, process or any other systems. Serialization flattens objects into an ordered, or serialized stream of bytes. The ordered stream of bytes can then be read at a later time, or in another environment, to recreate the original objects. For instance, you can create an object on Windows server, serialize it, and then send it to a Linux server to deserialize it without worrying about different byte order or OS details.

Serializing an object needs to store object itself, but also needs to store all references, so all referenced objects are serializable to avoid NotSerializableException.

Java serialization is mainly for two major usage: RMI (remote method invocation) and JavaBean persistence.

How to serialize Java object?
ObjectOutputStream is the primary output stream class that implements the ObjectOutput interface for serializing objects (using writeObject method). ObjectInputStream is the primary input stream class that implements the ObjectInput interface for deserializing objects (using readObject method).

Keywords:

1. Serializable - tagging interface without method or attributes, to indicate the Java object is serializable. (
   writeReplace, readResolve, writeObject, readObject)
2. Externalizable - sub-interface of Serializable (writeExternal, readExternal)
3. transient - indicate the instance field not persistable (for security etc)
4. serialVersionUID - for version control

Sample Codes:
        // serializing
        List<String> list = new ArrayList<String>();
        list.add("xxx");
        list.add("yyy");
        FileOutputStream fos = null;
        ObjectOutputStream out = null;
        try {
            fos = new FileOutputStream("c:\\filename.txt");
            out = new ObjectOutputStream(fos);
            out.writeObject(list);
            out.close();
        } catch (IOException ex) {
            ex.printStackTrace();
        }

        // deserializing it
        FileInputStream fis = null;
        ObjectInputStream in = null;
        try {
            fis = new FileInputStream("c:\\filename.txt");           
            in = new ObjectInputStream(fis);
            list = (ArrayList<String>) in.readObject();
            in.close();
            for (String s : list) {
                System.out.println(s);
            }
        } catch (Exception ex) {
            ex.printStackTrace();
        }

Timestamp from Oracle DB

Background:
In recent cache node design, there is a timestamp field in its all interface for cache data push sequence. Keeping sequence (without expensive locking) is important to guarantee cache data and db data are consistent, esp for concurrency cache write case. It is not reliable to use cache node timestamp because from client to cache node there is network latency. In other words, first request might arrive at cache node later than second request. So the thought is to let caller (application server) to pass the timestamp when invoke cache node. In production, all application servers should be running NTP, but it is not 100% guaranteed due to human mistake (say forgetting turn on this service). In such a clustering environment, we propose to get the timestamp from backend Oracle 10g database because cluster application servers access the same database.

Side note: our current cache node is not using write-through or write-back policy, it is kind of client control for data store and cache node.

Oracle Date & Timestamp:
Starting from Oracle 9i, there is a new date type TIMESTAMP besides well known DATE. TIMESTAMP contains second fractions (millisecond), so it satisfies cache node requirement. They are data types about representing date and time values. They have the ability to store the month, day, year, century, hours, minutes, and seconds. The problem with the DATE datatype is its granularity when trying to determine a time interval between two events when the events happen within a second of each other. This issue is solved after Oracle introduced TIMESTAMP datatype. In order to represent the date stored in a more readable format, the TO_CHAR function has traditionally been wrapped around DATE or TIMESTAMP .

Here are some sample queries from dual:
select to_char(sysdate,'yyyy-mm-dd hh24:mi:ss')  from dual;
2011-02-09 01:37:29
select to_char(systimestamp,'yyyy-mm-dd hh24:mi:ss.ff')  from dual;
2011-02-09 01:38:39.809065
select sysdate from dual;
09-FEB-11
select systimestamp from dual;
09-FEB-11 01.39.33.058547000 AM +00:00

-- to get milliseconds part only
select 1000*(seconds-trunc(seconds)) millisec from
(select extract(second from systimestamp) seconds from dual)
986.874

SELECT to_char(sysdate, 'HH24:MI:SS'), to_char(systimestamp, 'HH24:MI:SS.FF6') FROM dual;
01:45:16
01:45:16.222255

-- there are 2 more timestamps we can use
select localTIMESTAMP from dual
08-FEB-11 05.46.05.630208000 PM
SELECT CURRENT_TIMESTAMP FROM dual;
08-FEB-11 05.46.14.940902000 PM AMERICA/LOS_ANGELES

JDBC to get Timestamp from Oracle:

// Get a statement from the connection
Statement stmt = conn.createStatement();

// Execute the query
ResultSet rs = stmt
.executeQuery("SELECT sysdate, systimestamp FROM dual");

// Loop through the result set
while (rs.next()) {
System.out.println(rs.getDate(1).toString());
System.out.println(rs.getTimestamp(1).toString());
System.out.println(rs.getTimestamp(2).toString());
}

Console Output:

2011-02-09
2011-02-09 01:12:25.0
2011-02-09 01:12:25.47468

iBatis SQLMap:
<select id="getDbSysDate" resultClass="java.util.Date">
   select sysdate from dual
</select>

<select id="getDbSysTimestamp" resultClass="java.sql.Timestamp">
   select systimestamp from dual
</select>

java.io.IOException: Read channel closed

I am involved in an integration project for billing system, and my side provides APIs for authentication and provisioning, they work pretty well in my JMeter test script. However, caller side complain their code gets "java.io.IOException: Read channel closed" error.

Here is the stack trace:
com.sun.jersey.api.client.ClientHandlerException: java.io.IOException: Read channel closed
 at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:128)
 at com.sun.jersey.api.client.Client.handle(Client.java:457)
 at com.sun.jersey.api.client.WebResource.handle(WebResource.java:557)
 at com.sun.jersey.api.client.WebResource.access$300(WebResource.java:69)
 at com.sun.jersey.api.client.WebResource$Builder.post(WebResource.java:491)

 Basically, it is a connection related issue. After they change to http from https for REST/SOAP call endpoint, the issue was gone.

For the error, it is possible the SSL is not enabled, or the hostname of the cert does not match the
hostname of the server. (Weblogic has an option to ignore hostname verification, not sure what is the option in tomcat) -Dweblogic.security.SSL.ignoreHostnameVerification=true

Why query by DB PK (index) is very slow?

The root cause is DB chained rows.


http://www.dba-oracle.com/t_preventing_chained_migrated_rows.htm

• Preventing chained rows - Chained rows can occur when a row is to large for a data block.  In these cases, moving large objects into a tablespace with a larger blocksize can often relieve chained rows.
  
• Preventing migrated rows - Migrated rows occur when a row expands (usually with a varchar2 data type), and there is not enough reserve defined by PCTFREE for the row to expand. In this case, you prevent future relocated (migrated) rows by adjusting the PCTFREE to ensure that future rows will have enough room to expand and then reorganize the existing table (using data pump, CTAS or dbms_redefinition) to remove the fragments. 

Here is the analysis from our DBA

How to find and eliminate Migrated and Chained rows

---------------------------------------------------

CONCEPTS:

* A row Migrates when a block was found to have the space available for

  a row which underwent an update that increased its size over and beyond

  its block's available space.

* A Chained row occurs when there is no block which can hold the row after

  it underwent an update which increased its size beyond the available free

  space in its hosting block. The solution is to split the row over several

  blocks.         

CAUSES and EFFECTS:

* Causes for migrated and chained rows:  Inserts, updates and deletes over

  periods of time

* Results from migrated and chained rows:  Degraded response for queries.

SOLUTION:

1)  Analyze the table:

    To prevent an ORA-1495 (specified chained row table not found), run the

    $ORACLE_HOME/rdbms/admin/utlchain.sql script.

  

    TRUNCATE TABLE CHAINED_ROWS:

    ANALYZE TABLE <table name> LIST CHAINED ROWS;

2)  List the Migrated or Chained rows.

     

    From SQL*Plus:

    col owner_name format a10

    col table_name format a20

    col head_rowid format a20

    select owner_name, table_name, head_rowid from chained_rows;

3)  You can now eliminate the Migrated or Chained rows by Create Table

    as Select (CTAS), exporting and then importing the table or by following

    the next steps:

    A) Create an empty copy of the table that has the Migrated or Chained rows.

       CREATE TABLE <temporary table name> AS

        SELECT * FROM <table name> WHERE ROWID IN

         (SELECT HEAD_ROWID FROM CHAINED_ROWS WHERE TABLE_NAME='<table name'>');

    B) Now delete the Migrated and Chained rows from the table. 

       DELETE FROM <table name> WHERE ROWID IN

        (SELECT HEAD_ROWID FROM CHAINED_ROWS

         WHERE TABLE_NAME='<table name>');

    C) Insert the rows back to the table.

       INSERT INTO <table name> SELECT * FROM <temporary table name>;

      Truncate the chained_rows table and drop the temporary table.

      Alternatively, you can move the table to a tablespace if the row cannot fit in the block and you need a tablespace with a   larger block size:

      alter table <table_name> move <tablespace>;