Archive for the ‘java’ Category

Parsing Appleworks and Clarisworks file formats

Thursday, January 17th, 2013

Over the past few years, when I have downtime, I sometimes like to reverse engineer abandoned file formats. It is kind of like working on a crossword puzzle with the bonus that any progress you make helps people out there who are trying to archive, index, or convert their old files.

output of hex fiend comparing two files

I’ve spent a lot of time trying to figure out file format for Appleworks and Clarisworks. My latest efforts have been to take a file, make a small change, then use Hex Fiend to compare what has changed in the binary format.

After years of off and on tinkering and documenting I finally wrote a basic parser for Appleworks and Clarisworks word processor files. I ‘believe’ this is the first free and open parser for this file format, even if it is ten years too late. I figured out a lot about the format, but it still has a long way to go. You can view my current documenting status here and download source for the parser on GitHub.

The parser so far can read:

  • document version
  • page size
  • margins
  • document content

output of parse

From what I have seen, most people trying to read Appleworks documents only really care about the document content, but I am very close to figuring out how to parse:

  • styles – (bold, italic, underline)
  • footnotes

I may not touch it again for another year, but who knows.

 Link to file format research

 Download source code at GitHub

Java keystore cert import on OSX Leopard

Wednesday, March 17th, 2010

This morning I needed to connect IDE (IntelliJ Idea) to a FishEye/Jira server that had a self signed security certificate. Since IntelliJ (or at least the Atlassian plugin) uses Java to connect to https, it fails because of the JVM’s strict security checking.

Normally when this happens, it is just a matter of installing the certificate into the JVM keystore. There is an article and code that does this for you here. This blog post even has a nice bash wrapper that will download and compile this code for you on OSX.

When I tried to do this today, I got this error every time I tried to run this tool:

java.security.UnrecoverableKeyException: Password verification failed

After a lot of digging on google. I finally found the problem.

On Java for OSX 10.6 u1 and 10.5 u6, Apple changed the default keystore password from ‘changeit’ to ‘changeme’.

Such a trivial change, but annoying because changeit had been the Sun default forever. There is a funny post on the Apple Java mailing list where an engineer at Apple apologized and just sort of said they didn’t think it would be a big deal for anyone.

This post from Matt Fleming, has some more info as well as how to change the keystore password if you decide you don’t like this change:

sudo keytool -storepasswd -new changeit -keystore /Library/Java/Home/lib/security/cacerts -storepass changeme

Monty Widenius is trying to regain control of MySQL and why this is bad for OSS

Saturday, January 2nd, 2010

One of the most widely discussed topics to go around the tech industry last year was the Oracle acquisition of Sun and what this meant for the MySQL database. This topic held up the merger with the US DOJ and currently has it stalled in the EU commission.

One of the primary forces behind these hold ups is a series of FUD articles written by Monty Widenius, the most recent just a few days ago. Monty has a huge following so whenever he writes up one of these articles it gets huge circulation and riles up the Slashdot and LAMP crowds.

I think that the open source community should be very skeptical about anything written by Monty on this topic, and should start looking at the big picture of what this merger means for themselves and the various players involved.

I don’t know why I haven’t been seeing many serious rebuttals to Michael’s posts. I can only guess that is because everyone working at Sun and Oracle are prohibited from speaking up on the matter.

Here is what I think everyone should consider:

1. Sun is the largest contributor to Open Source in the world

2. Java, which sun is responsible for, one of the largest ecosystem of open source software in the world.

IBM, RedHat, the Apache Foundation, Oracle, Google, and hundreds of other companies have based themselves on Java. Java is by far the most used platform out there today. Out of this wide adoption has sprung a massive open source ecosystem that can only be rivaled by Linux. I don’t have any studies but I wouldn’t be surprised if there was much more open source Java code out there than C.

The Java community in my experience, by and large, is very reluctant to touch anything that is not open source. In the past 10 years the community has moved from expensive application servers and IDEs to free alternatives. Projects like JBoss, Glassfish, Tomcat, Eclipse, and Netbeans are the dominant players in the space and have been driving the mindset that to be a player in this market you have to be free.

3. Sun is in trouble and risks going out of business if no-one buys it. Talks with IBM broke down, and there aren’t many other companies that can make a purchase.

Not much else to say here. Oracle has been having some rough years. Hardware sales are down and they’ve been spending too much on R&D. Sun needs someone to buy them and buy them quickly. They have been actively been doing layoffs that affect all of their open source efforts (including MySQL) while this drags out. Further delays, or the blocking of this merger will only further harm the OSS Community.

4. Oracle already owns Berkely DB and InnoDB, which current versions of MySQL rely on.

This same sort of noise and FUD was made years ago when Oracle bought these two products. Oracle has continued to maintain these and has been a more stable steward of the projects than when they were independent.

5. It doesn’t make business sense for Oracle to try to kill MySQL.

First off – MySQL does not compete with Oracle Database. Anyone who thinks it does does not understand what Oracle Database is. People that use Oracle tend to buy the whole oracle package (DB, App Server, IDE, Middleware, etc..). There are free alternatives to everything in this stack, including many products owned by Oracle but companies that want Oracle are companies that want the piece of mind that support for the stack brings.

There are no CTOs out there hemming and hawing about wether to use MySQL or Oracle. It would be like sitting around and trying to decide if you were going to buy a Ford Focus or an M1 Abrams Tank. I’m not using this analogy to point out that Oracle has many more features (it does) or that it is better than MySQL, only that it is different. You would never buy the tank to commute to work or for most of your driving needs. The same is true with MySQL, it is perfect for most projects and Oracle tends to be a little too heavyweight.

Furthermore, for companies that do insist on purchasing the Oracle stack but want to use MySQL would now be able to buy the support stack with MySQL in it. Oracle can now sell the complete support package and their customers can feel good about getting everything from one vendor. Companies that buy Oracle are most likely the companies that would be paying for MySQL support as well. If a customer comes to oracle, what do they care which database the customer wants to use when they own both.

The last thing Oracle would want to do is alienate a large developer community. Changing anything about MySQL would hugely upset not just the LAMP and Java communities but just about everyone on the planet. This is just bad business.

6. All Oracle will own is a trademark and some engineers.

The source code for MySQL is already free. Anyone can fork it off and start another project and attempt to gain community support around their new project. The only thing they can’t do is call it MySQL. Monty has already started one such fork called MariaDB.

Open source projects are about the community rallying around ideas, not around companies. Monty argues that a forked product could never compete with MySQL without the name recognition. This isn’t true. If the community feels that Oracle is doing a bad job as a maintainer, but someone else is releasing new features on some other project, people will switch very quickly. We see this all the time in the Linux world with the community switching from one fork to another of a large project.

Monty says that forks can never happen because they would need funding and resources. What this argument ignores is that large companies with a lot invested in MySQL could step up if the project is faltering. If Oracle were to stop releasing updates, do you think Google is going to sit around and do nothing? The community would jump ship to GoogSQL or whatever if it came to that and was seen as a better product.

6. Monty Widenius has the most to gain from Oracle divesting in MySQL.

Like everything else in the world, when there is an argument, you need to step back and ask yourself who has the most to gain. Widenius sold MySQL for a hefty personal gain and is now trying to wrestle back control by spreading fear throughout the MySQL community.

Monty has been making noise since September 2008 (before the announced Oracle-Sun merger) and complaining about the direction of the project. He didn’t feel Sun was doing a good job and started immediately calling for forks and and a change of direction. The community heard Widenius out but didn’t build up a ton of support for his ideas, because by and large, most people are satisfied with the job Sun is doing.

Right before the Sun ownership, MySQL was in the process of rolling out a non-free enterprise edition and telling people that they would have to pay for new features. The company I was working for at the time had MySQL sales reps and consultants flat out tell us that we would need to purchase a support agreement if we wanted to use the Falcon Engine or clustering past v5.5. Sun put a stop to this.

In Summary

I am not under any illusions that an Oracle-Sun merger would be all sunshine and roses. I think that Sun has developed a culture and business model around everything being open and free and Oracle has not. Oracle will need to make some big changes about how it does business in order for the merger to work.

I would prefer if Sun could remain an independent company but we have to face facts, Sun is in trouble and there aren’t many other companies that can bail them out. If Sun is allowed to continue it’s downward spiral then we are facing a great loss to the open source community.

I am not worried about the Oracle stewardship of any of Sun’s open source products precisely because of the community support. Oracle can’t afford to make huge changes and alienate hundreds of thousands of developers who have some say in how much money their companies give to Oracle. Doing something like killing or even changing GlassFish, VirtualBox, ZFS, or any other OSS project could lead to less sales in it’s existing stack of software.

Moving from Tomcat 6 to Jboss AS5 – notes

Monday, May 18th, 2009

I’m moving a bunch of single WAR sites from Tomcat 6 to Jboss AS5. Here are the configuration steps that need to be taken.

Assuming this is your tomcat config file:
%TOMCAT_HOME%/conf/server.xml:

<Host appBase="/home/wirelust/wirelust.com" autoDeploy="false"
     debug="0" deployXML="true" liveDeploy="true"
     name="domainname.com"
     unpackWARs="true">
 
    <Alias>www.wirelust.com</Alias>
 
     <Context cachingAllowed="true" cookies="true" crossContext="true" debug="0"
             displayName="wirelust" docBase="." path="" privileged="false"
             reloadable="true" swallowOutput="false" useNaming="true">
             <Resource auth="SERVLET" name="wirelustDatasource" scope="Shareable" type="javax.sql.DataSource"
                     username="username"
                     password="password"
                     driverClassName="net.sourceforge.jtds.jdbc.Driver"
                     url="jdbc:jtds:sqlserver://sql.wirelust.com/database"
                     removeAbandoned="true"
                     removeAbandonedTimeout="30"
                     logAbandoned="true"
             />
     </Context>
</Host>

Becomes the embedded tomcat config file:
%JBOSS_HOME%/server/default/deploy/jbossweb.sar/server.xml

<Host appBase="/home/wirelust/deploy/wirelust.com.war" autoDeploy="false"
     debug="0" deployXML="true" liveDeploy="true"
     name="domainname.com"
     unpackWARs="true">
 
    <Alias>www.wirelust.com</Alias>
</Host>

Notice that the appBase is now in a folder called “deploy”. You have to add this deploy folder to the config file:
%JBOSS_HOME%/server/default/conf/bootstrap/profile.xml

<bean name="BootstrapProfileFactory" class="org.jboss.system.server.profileservice.repository.StaticProfileFactory">
    <property name="bootstrapURI">${jboss.server.home.url}conf/jboss-service.xml</property>
    <property name="deployersURI">${jboss.server.home.url}deployers</property>
    <property name="applicationURIs">
        <list elementClass="java.net.URI">
            <value>${jboss.server.home.url}deploy</value>
 
            <value>file:/home/wirelust/deploy</value>
        </list>
    </property>
    <property name="attachmentStoreRoot">${jboss.server.data.dir}/attachments</property>
    <property name="profileFactory"><inject bean="ProfileFactory" /></property>
</bean>

Make sure your exploded directory ends in .war or jboss won’t see it.

Then in that new deploy directory, create a datasource file, in this case called “wirelust-ds.xml” with these contents:

<?xml version="1.0" encoding="UTF-8"?>
 
<!DOCTYPE datasources
    PUBLIC "-//JBoss//DTD JBOSS JCA Config 1.5//EN"
    "http://www.jboss.org/j2ee/dtd/jboss-ds_1_5.dtd">
<datasources>
    <local-tx-datasource>
        <jndi-name>wirelustDatasource</jndi-name>
        <connection-url>jdbc:jtds:sqlserver://sql.wirelust.com/wirelust</connection-url>
        <use-java-context>false</use-java-context>
        <driver-class>net.sourceforge.jtds.jdbc.Driver</driver-class>
        <user-name>username</user-name>
        <password>password</password>
 
        <Pool.LogAbandoned>false</Pool.LogAbandoned>
        <Pool.RemoveAbandoned>false</Pool.RemoveAbandoned>
        <Pool.RemoveAbandonedTimeout>50000</Pool.RemoveAbandonedTimeout>
   </local-tx-datasource>
</datasources>

In order to bind to the correct virtual host, you have to create a new file in your exploded WEB-INF directory called jboss-web.xml with these contents:

<!DOCTYPE jboss-web PUBLIC
    "-//JBoss//DTD Web Application 4.2//EN"
    "http://www.jboss.org/j2ee/dtd/jboss-web_4_2.dtd">
 
<jboss-web>  
    <context-root>/</context-root>  
    <virtual-host>wirelust.com</virtual-host>
</jboss-web>

Then the last change you have to make is how you are looking up your datasource in the application.
If you have code like this in your tomcat application it will fail because your jndi is not bound to java:comp/env:

Context ctx = new InitialContext();
Context envCtx = (Context) ctx.lookup("java:comp/env");
DataSource ds = (DataSource)envCtx.lookup("wirelustDatasource");
con = ds.getConnection();

This of course can be fixed with some further configuration and resource-ref settings but I couldn’t get that to work so I just changed the code to work either way:

Context ctx = null;
Context envCtx = null;
DataSource ds = null;
Connection con = null;
 
try {
	ctx = new InitialContext();
} catch (NamingException e) {
	e.printStackTrace();
}
 
if (ctx != null) {
	try {
		// look for the datasource in the tomcat location
		envCtx = (Context) ctx.lookup("java:comp/env");
		ds = (DataSource)envCtx.lookup("wirelustDatasource");
	} catch (javax.naming.NameNotFoundException nnfe) {
		try {
			// look for it in the jboss location
			ds = (DataSource)ctx.lookup(wirelustDatasource);
		} catch (Exception e) {
			e.printStackTrace();
		}
	} catch (Exception e) {
		e.printStackTrace();
	}
}
 
 
// open the connection
if (ds != null) {
	try {
		con = ds.getConnection();
	} catch (Exception e) {
		e.printStackTrace();
	}
}

Converting binary to signed decimal in Actionscript

Wednesday, April 8th, 2009

If you’re like me, you may have been living these past few years taking advantage of the luxury of new sleek programming languages that rarely require you to do low level operations. One of the old skills I seem to always let get rusty enough is bitwise operation. Most code written with the help of frameworks doesn’t require this type of code and most programmers probably don’t even bother to learn it anymore.

I came into an instance yesterday where I needed to do some bitwise work in actionscript. The code was reading a bytestream from a binary file and converting to an array of numbers. Since this project was ported from Java (by someone else), a simple operation was overlooked. It was taking the binary number and converting it to a signed short. Since actionscript doesn’t have the same casting mechanisms conversion isn’t as straight forward in java.

In java the operation looked like this:

//  binary: 1111 1111 1111 0001
int hexValue = 0xFFF1;
 
short value = (short)(hexValue);
// value = -15 

Since short in java is always stored with a signed bit, the cast appropriately set the value at -15.

The way this was translated into Actionscript looked liked this:

//  binary: 1111 1111 1111 0001
var hexValue:int = 0xFFF1;
 
var value:int = int(hexValue);
// value = 65521  // Wrong! 

Actionscript has no concept of casting for primitives so the int(hexValue) doesn’t do anything at all.

Here is the workaround I came up with, I suspect there is a shorter way to get the same result but I can’t think of one right now.

//  binary: 1111 1111 1111 0001
var hexValue:int = 0xFFF1;
 
// get everything except the signed bit "111 1111 1111 0001"
var unsignedValue:Number = (hexValue & 0x7FFF);  
// unsignedValue now equals 32,753
 
var signedValue:Number = unsignedValue;
 
// if the signed flag is set, flip the value
if ((hexValue >> 15) == 1) {
    signedValue = unsignedValue - 0x8000;   // 0x800 =  32,768 (maximum 15 bit number)
}
 
// signedValue = -15 

Picture Pump

Monday, April 6th, 2009

About five years ago Ben Sisto had an idea for an application that I thought would be useful. The idea being that if you want to download images from Google image search — it takes forever to go through the results, wait for them to load, and save the images that aren’t 404. The application Ben envisioned, would simply download every result and ignore the ones that were 404 or took too long to load. Once you have all the results on your hard drive it is a lot easier to sift through them.

I quickly banged out a release that was workable in a few days and never really looked at it again.

This week I had an exact need for this application again. When I fired it up, it seems Google changed their site so it wasn’t working anymore. I spent some time tonight to fix it up and got it working much better than the original.

So here it is. v1.0 (i guess) release of Picture Pump (named after the site Ben ran at the time – HoneyPump):

Click launch to start/install the application. OSX users should be all set, windows users you might have to get java first.

also, hey look. source code

Email me with any suggestions or patches.

UPDATE: 10/24/2009
– fixed my mime settings so that launch button works again (sorry moved servers and forgot to set it)
– fixed a few bugs
– added support for safe search
– added filtering by image type and license
– source code is now under the Apache Public 2.0

Howto set the cookie policy with Apache http client 4.0

Sunday, March 29th, 2009

The Apache Http Components client changed quite a bit from 3.0 to 4.0 beta. It wasn’t entirely clear to me from the documentation how to set request headers and cookie policies with the new client. After a lot of fishing around I was able to figure it out, here is the code for anyone trying to do the same thing.

Downloading a page with the 3.0 client:

HttpClient httpClient = new HttpClient();
 
GetMethod get = new GetMethod("http://www.twitter.com/teacurran");
 
// set a faux User-agent
get.setRequestHeader("User-Agent", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; .NET CLR 1.1.4322)");
 
// set the cookie policy
get.getParams().setCookiePolicy(CookiePolicy.RFC_2109);
 
int responseCode = httpClient.executeMethod(get);
System.out.println(get.getResponseBody());
 
get.releaseConnection();

here is the same thing with the 4.0 client:

DefaultHttpClient httpClient = new DefaultHttpClient();
 
HttpGet get = new HttpGet("http://www.twitter.com/teacurran");
 
// set a faux User-agent
get.addHeader("User-Agent", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; .NET CLR 1.1.4322)");
 
// set the cookie policy
get.getParams().setParameter(ClientPNames.COOKIE_POLICY, CookiePolicy.RFC_2965);
 
HttpResponse response = httpClient.execute(get);
 
int responseCode = int code = response.getStatusLine().getStatusCode();
HttpEntity entity = response.getEntity();
 
System.out.println(EntityUtils.toString(entity));

Creating an iText pdf with embedded JFreeChart

Monday, March 17th, 2008

Charts are simple to make with JFreeChart and there are tons of examples on the web about how to generate them and save them as .jpg or .gif files. I ran into an issue however, where I needed to embed a chart into a pdf file. My first thoughts were to generate the charts as a .jpg file and embed them, but the pdfs looked a little choppy and didn’t print well.

To solve the problem you need to have the chart rendered as vector graphics and inserted into the pdf. The code required to do this ended up being simple, but I wasn’t able to find any good complete examples on the web. This is a simple working example I made for anyone else who finds themselves in this position.

Downloads:

To do an ant build, all you need to do is set j2ee.lib.dir in the build.properties file to a directory that contains the j2ee.jar (in your jdk or app server) and you are good to go. The war file should easily deploy on any app server.

Here is the meat of the code required:

docWriter = PdfWriter.getInstance(doc, baosPDF);
 
doc.addProducer();
doc.addCreator(this.getClass().getName());
doc.addTitle("jfreechart pdf");
doc.setPageSize(PageSize.LETTER);
 
doc.open();
 
// add some text to the document
doc.add(new Phrase("Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum."));
 
// build up the dataset for the char
XYSeriesCollection dataset = new XYSeriesCollection();
 
XYSeries series = new XYSeries("XYGraph");
series.add(1, 1);
series.add(2, 3);
series.add(3, 9);
series.add(4, 11);
 
dataset.addSeries(series);
 
// set up the chart
JFreeChart chart = ChartFactory.createXYLineChart("XY Chart Sample, non default font", // chart title
        "x-axis",   // domain axis label
        "y-axis",   // range axis label
        dataset,    // data
        PlotOrientation.VERTICAL,   // orientation
        true,   // include legend
        true,   // tooltips
        false   // urls
    );
 
// trick to change the default font of the chart
chart.setTitle(new TextTitle("XY Chart Sample, non default font", new java.awt.Font("Serif", Font.BOLD, 12)));
chart.setBackgroundPaint(Color.white);
chart.setBorderPaint(Color.black);
chart.setBorderStroke(new BasicStroke(1));
chart.setBorderVisible(true);
 
int width = 260;
int height = 250;
 
// get the direct pdf content
PdfContentByte dc = docWriter.getDirectContent();
 
// get a pdf template from the direct content
PdfTemplate tp = dc.createTemplate(width, height);
 
// create an AWT renderer from the pdf template
Graphics2D g2 = tp.createGraphics(width, height, new DefaultFontMapper() );
Rectangle2D r2D = new Rectangle2D.Double(0,0, width,height);
chart.draw(g2,r2D,null);
g2.dispose();
 
// add the rendered pdf template to the direct content
// you will have to play around with this because the chart is absolutely positioned.
// 38 is just a typical left margin
// docWriter.getVerticalPosition(true) will approximate the position that the content above the chart ended
dc.addTemplate(tp, 38, docWriter.getVerticalPosition(true)-height);

Sun Spots are now Open Source

Wednesday, January 30th, 2008

sunspot.jpgSun has announced that the Sun Spots are now fully open source. This means that the API as well as the Squawk VM are now fully open. The Squak VM itself is an interesting project. I will be curious to see how it compares against google’s Dalvik VM on small devices (that is whenever google releases or tells us more about Dalvik).

The press release seems to imply that the Sun Spot hardware will also be open source:

The open source release of the versatile Java technology-based Sun SPOT platform will include hardware, software and the Squawk Virtual Machine.

but this is the only mention I see of hardware in the press release and on the several sites dedicated to Sun Spot. I really hope they expand on this and show the world everything that is included in the OSS releases.

If the hardware is Open Source that could make way for some third parties to make their own implementations, similar to the different implementaitons of Arduino. This would be especially welcome given the crazy high cost of a Sun Spot device ($250 or so per device compared with $20-150 depending on configuration for Arduino). Yea, Yea, I know Sun Spot is way awesomer than Arduino, but if I am going to make a little device to attach to the top of a rocket or the under carriage of my skateboard, you better believe I am going to be looking for the cheapest hardware I can find.

Building Chandler Server

Thursday, December 13th, 2007

I’ve been using Chandler for a few weeks now to try to organize my life.

I tried building Chandler Server from source today. Following the instructions on their site give’s me a build error because it can’t find a copy of Apache Abdera

Some of the error:

/mnt/drive2/chandler/cosmo/cosmo/src/main/java/org/osaf/cosmo/atom/provider/UserTarget.java:[20,46] cannot find symbol
symbol  : class AbstractTarget
location: package org.apache.abdera.protocol.server.impl
 
/mnt/drive2/chandler/cosmo/cosmo/src/main/java/org/osaf/cosmo/atom/provider/UserTarget.java:[29,32] cannot find symbol
symbol: class AbstractTarget
public class UserTarget extends AbstractTarget {
 
/mnt/drive2/chandler/cosmo/cosmo/src/main/java/org/osaf/cosmo/atom/provider/BaseItemTarget.java:[22,46] cannot find symbol
symbol  : class AbstractTarget
location: package org.apache.abdera.protocol.server.impl
 
/mnt/drive2/chandler/cosmo/cosmo/src/main/java/org/osaf/cosmo/atom/provider/BaseItemTarget.java:[25,45] cannot find symbol
symbol: class AbstractTarget
public abstract class BaseItemTarget extends AbstractTarget

After much fiddeling around, and even trying to install Abdera from source, I was able to get it to build by checking out everything related to cosmo and building that.

  • svn co http://svn.osafoundation.org/server/cosmo/trunk
  • cd trunk/cosmo
  • mvn package

ta-da.

I still get some errors with the unit tests, but I don’t know if those are serious or not yet.

Now let’s see if it run’s with Tomcat 6.0