Spatialite DLL hell

Spatialite is a geospatial-enabled extension to sqlite. I must have it. I intend to use it inside our commercial java product, which has windows and linux users. The development environment is Eclipse. The OS is Windows 10.

The author of spatialite mentions that the windows environment suffers from DLL-Hell. Well yes it does.

Installing sqlite and the xerial-jdbc software is easy and not covered.

You download the spatialite files here. It isn’t well explained, but you need the “mod_spatialite-.7z file.

My preferred eclipse setup is to create a /lib folder in my eclipse project and put the jars and dlls there. Below is a simple test project showing the locations of the files. Note that the sqlite-jdbc.jar is included with a “Native library location” that points to the folder containing the dll files. This is the standard way to do things. It works most of the time.

The author of spatialize has some good installation instructions and sample code. I added the sample file and tried to run it.

The result was the following error:
Failed to load extension mod_spatialite. Trying to find file manually
Exception in thread "main" java.lang.UnsatisfiedLinkError: Can't load library: C:\apps\eclipse-64-workspace\spatialite-test2\lib\mod_spatialite.dll
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1827)
at java.lang.Runtime.load0(Runtime.java:809)
at java.lang.System.load(System.java:1086)
at SpatialiteSample.main(SpatialiteSample.java:40)

This is the well-known dll hell. The eclipse launcher uses the “root” folder of the project as the root java location. Here are the details. The command line is:
"C:\Program Files\Java\jdk1.8.0_121\bin\javaw.exe" -agentlib:jdwp=transport=dt_socket,suspend=y,address=localhost:57143 -Djava.library.path=C:\apps\eclipse-64-workspace\spatialite-test2\lib -Dfile.encoding=Cp1252 -classpath C:\apps\eclipse-64-workspace\spatialite-test2\bin;C:\apps\eclipse-64-workspace\spatialite-test2\lib\sqlite-jdbc-3.18.0.jar SpatialiteSample

At this point, someone gets desperate and recommends adding the spatialite path to the windows PATH variable, or drop the dlls in the /system folder. But this is intended as a professional installation, not something for the enthusiatic developer. These are not acceptable recommendations.

To simplify the problem, a good next step is to move the dlls to the root folder. Can’t miss them there, can we?

And that works. The “mod_spatialite.dll” is found in the root folder. The command line is:
"C:\Program Files\Java\jdk1.8.0_121\bin\javaw.exe" -agentlib:jdwp=transport=dt_socket,suspend=y,address=localhost:57854 -Dfile.encoding=Cp1252 -classpath C:\apps\eclipse-64-workspace\spatialite-test3\bin;C:\apps\eclipse-64-workspace\spatialite-test3\lib\sqlite-jdbc-3.18.0.jar SpatialiteSample.

So we can run spatialite in eclipse if we are willing to place the dlls in the root directory. We are not.

At this point, I am extremely puzzled. To be continued, hopefully…

Java Null Pointer Static Analysis in Eclipse – JDT vs CheckerFramework – False-Positive comparison

We’re talking about this and this. For those of us who program in Eclipse using Java, what is the best tool(s) for checking null pointer exceptions.

There have been several attempts before, but nothing has made its way into the official language. Eclipse does provide good null pointer checking out of the box, but there is only so much it can do. To improve the power of the checkers, we need to add annotations.

Note: In the following, I am using JDK 1.8.

1) We have Eclipse itself, using the JDT annotations Eclipse help.
2) We have the standard checker framework.
3) We have FindBugs, which I believe is used by sonarcube.

There is great power to have something built into the environment, so the standard eclipse system has a strong advantage. Unless it has serious problems, it should be part of the standard development environment.

Unfortunately, it does have serious problems. Take a look at the following program:
[sourcecode lang=”java” highlight=”12″]
public class TestNullCode {

@org.eclipse.jdt.annotation.Nullable
private String testJdt;

public void testJdt() {
if (testJdt == null) {
testJdt = "not null";
}
// this line causes a false-positive warning
// Potential null pointer access: The field testJdt is specified as @Nullable
if (testJdt.equals("not null")) {
System.out.println("will work because testJdt is not null");
}
}
}
[/sourcecode]

This example shows a false-positive that is provably incorrect. The compiler issues a warning (or error, depending on your settings) on line 12, that is untrue.

Frankly, this is pretty sad. The reasons are justifiable, but the result is sad, nonetheless.

For non-final @Nullable fields the simplest strategy for avoiding any nullness related risks is to pessimistically assume potential null at every read. This means for strict checking no flow analysis should be applied to @Nullable fields..

How does the checkerframework do? Answer: No problems at all:
[sourcecode lang=”java” highlight=”11″]
public class TestNullCode {

@org.checkerframework.checker.nullness.qual.Nullable
private String testCheckerFramework;

public void testCheckerFramework() {
if (testCheckerFramework == null) {
testCheckerFramework = "not null";
}
// this line is ok
if (testCheckerFramework.equals("not null")) {
System.out.println("will work because testCheckerFramework is not null");
}

testCheckerFramework = null;
// following line is flagged by checker framework
// dereference of possibly-null reference testCheckerFramework
System.out.println("will fail because testCheckerFramework is not null" + testCheckerFramework.toString());
}
}
[/sourcecode]

How does FindBugs do? Answer: No problems at all. Same as checkerframework , so I won’t post the identical code.

Eclipse has a workaround that costs a line of extra code:

While this appears to be a very drastic restriction, the remedy is quite easy: before dereferencing a @Nullable field it has to be assigned to a local variable. Flow analysis is then safely applied to the local variable with no risk of side effects, aliasing nor concurrency, since local variables are not shared with any code locations that would be outside the scope of the analysis. I.e., the flow analysis can see everything it needs to consider regarding local variables.

[sourcecode lang=”java” highlight=”12″]
public class TestNullCode {

@org.eclipse.jdt.annotation.Nullable
private String testJdt;

public void testJdtCopy() {
String copy = testJdt;
if (copy == null) {
copy = "not null";
}
//This is fine
if (copy.equals("not null")) {
System.out.println("will work because copyis not null");
}
}
}
[/sourcecode]
I consider this “solution” ugly. I get it, but I don’t buy it.
At some point, I would expect the Eclipse engine to get better.

So at this point, we still need the other tools.

You can download my example eclipse project here.

Running Java apps as Windows Services

A tricky little problem, as it turns out. There is little out in google world to help you, and only a few tools. License issues with almost all I found, especially running on Windows boxes in 64 bit mode.

 I finally found Yet Another Java Service Wrapper at yajsw. License is LGPL and it supports 64 bit windows. I was able to get my application running very quickly. Worked perfectly the first time. Getting going was interesting:

 1) You download yajsw and extract the zip file.
 2) You run your app and lookup its PID.
3) You run the yajsw script generator, and it analyzes your running application to create a script file.
4) You then run the yajsw bat files to install-service, uninstall-service, start-service, stop-service, etc.

The only trick is to run the bat files with administrator permissions. (Right-click on the bat file and select “Run as administrator”.)

Right now the yajsw folder needs to exist on the server as well. Doubtless there are more convenient ways to package the install, but this was a good first step.

Debug xquery in eclipse. Configuration

So you want to run xquery in eclipse. You eventually find your way to XQDT. Some concern that we’re working with beta software (version 0.8.0 as of this post), but you bravely forge ahead. You survive the installation and run into your first roadblock at the getting started. Here you discover that they are working with Zorba 0.9.4 (later you suspect this is a misprint, but never mind) and do NOT SUPPORT DEBUGGING! If you can’t debug, this is DOA.

You follow the note to install Zorba from the subversion trunk, but this is a C project and no compiled distributions. Dead end 1.

You download and install Zorba itself (version 1.4.0), and while it runs, you are unable to debug. (Perhaps you can, but you never figured it out and got impatient. Plus, you refuse to depend on other programs installed in Windows anyway, because this is a distribution/installation hell.) Probably dead end 2.

You hear about the 28msec Sausalito project. They apparently have a version of Zorba that supports debugging. Looking better, but you don’t want a separate Eclipse installation. After a bit of exploration, you figure out to how attach the Sausalito project code to your existing Eclipse installation:

1) Install XQDT into Eclipse as normal. Steps described here

2) Download the Sausalito project from here, and extract the zipped contents.

3) Navigate to the plugins/com.28.msec.sausalito.win32/coresdk/bin directory.

Note: In my download, the path is com.28msec.sausalito.win32_1.2.10.201011291638

4) Copy the contents of this directory into your existing Eclipse project. (Put it somewhere convenient, of course, such as /lib/sausalito,)

Note: Obviously, you do not need all the contents of this directory. I do not yet know which files are required, and which are not. I know the zorba-sausastore.exe file is required, along with some others.

5) Now to add the new intepreter to Eclipse. Open Preferences -> XQuery -> Interpreters. Click the “Add…” button, then the “Browse” button. Navigate to the zorba-sausastore.exe file you just added, and select it. Eclipse should be able to inspect the file and set the default Interpreter name, e.g.,

Click ok and make the new Interpreter the default (or change it only for the one project).

And that should do it. Look see 🙂

I find it odd that Zorba is up to version 1.5.0, but XQDT is back on 0.9.4.

I also have NOT done anything with a Linux install. Obviously, you won’t have a win32 directory, but you’ll know what you’re doing anyway.