Playwright or Selenium?

Playwright and Selenium are the two big choices when choosing a browser user-interface test programming tool. Selenium has been around a long time. Playwright is the new kid in town.

I’ve used Selenium before. It is a pita but it mostly works. Unfortunately, even 99% “mostly works” is a problem when you are running hundreds of tests. That means something is always failing. So you wind up writing all your code with “wait-until-something-is true” and “try-this-multiple-times-until-it-succeeds” features. And then it still fails once in a while, so you just have to repeat the whole test and then it works. The bigger the project, the worse this problem becomes.

In short, we use Selenium because we have to, not because we like it.

I had the opportunity to start a new project so I tried Playwright. Still a learning curve. Still requires a lot of work. But they took all the “wait-until” stuff and moved it “behind the scenes”. It still has to be done, but you, the programmer, don’t have to handle it yourself. unless you want to. Much better.

After 2 weeks working with Playwright, I was still impressed. Progress was slow but steady.  The hardest part was that I am using Vaadin as the front-end development tool and they make serious use of the shadow DOM, so each new element type took trial and error to get working. This would have been the same amount of work in Playwright and Selenium.

I was also fighting against the “best-practices” of Playwright. I like to use “id” attributes whenever I am selecting something. And I really like to use XPath. Yes, XPath can be brittle, but don’t kid yourself: UI testing is always going to be brittle. Now, Playwright doesn’t support XPath inside the shadow-dom, and I was constantly running into that problem. Eventually, everything I was doing with XPath was easily handled using CSS. For example, select the element with type “vaadin-form-layout” and attributes class=”user=prefs” and id=”userPrefsId”. So I was writing “xpath-ish” and it was easily translated into CSS “vaadin-form-layout[user=”prefs”][id=userPrefsId”]” and so forth. Anyway, personal preferences and nothing to do with the subject of Playwright vs Selenium.

And then I read this guy and he scared me:

https://javascript.plainenglish.io/playwrights-auto-waiting-is-wrong-2f9e24beb3b8

https://zhiminzhan.medium.com/waiting-strategies-for-test-steps-in-web-test-automation-aaae828eb3b3

https://zhiminzhan.medium.com/why-raw-selenium-syntax-is-better-than-cypress-and-playwright-a0f796aafc43

https://zhiminzhan.medium.com/correct-wrong-playwrights-advantage-over-selenium-part-1-playwright-is-modern-and-faster-than-0a652c7e9ee7

https://zhiminzhan.medium.com/why-raw-selenium-syntax-is-better-than-cypress-and-playwright-part-2-the-audience-matters-a8375e6918e4

https://zhiminzhan.medium.com/playwright-vs-selenium-webdriver-syntax-comparison-by-example-4ad74ca59dcc

https://medium.com/geekculture/optimize-selenium-webdriver-automated-test-scripts-speed-12d23f623a6

His arguments were reasonable but not sufficiently detailed to be definitively persuasive. To summarize at a very high level, the best arguments were:

  1. Playwright did waiting wrong.
  2. Selenium is the web-standard and google/facebook will make sure it is up to date. Playwright could get left behind.

Ok, both of these are serious accusations. And I don’t feel qualified to comment on their validity.

Emotionally, the author, Zhimin Zhan seemed a bit cranky. Certainly seems like an expert, but sometimes experts get cranky when their favorite technology gets left behind. Either possibility seemed plausible.

So I decided I would redo the last 2 weeks work in Playwright in Selenium. It only took a few hours.

As soon as I ran the same tests in Selenium, I remembered why it was always so frustrating:

The first problem was with the Save-Menu. On startup it is inactive (disabled=”true”) and my tests assert that. However, the Selenium method isEnabled() returned true. Google “selenium isenabled not working”. My God! How many years now and that is STILL broken?

We shouldn’t all have to write this crappy code:

public static boolean isEnabled(WebElement element) {
   boolean enabled1 = element.isEnabled(); //this can be wrong
   String disabled = element.getAttribute("disabled"); //this is reliable
   boolean disabled2 = disabled != null && Boolean.parseBoolean(disabled);
   if (enabled1 == disabled2) {
      System.err.println("discrepancy in isEnabled");
      enabled1 = !disabled2;
   }
   return enabled1;
}

The next problem was when I clicked on a VaadinSideNavItem. I got this error:

org.openqa.selenium.ElementClickInterceptedException: element click intercepted: Element <vaadin-side-nav-item path="domain/person" 
id="CometPersonInit-peopleNav" role="listitem" has-children="">...</vaadin-side-nav-item> is not clickable at point (127, 92).
Other element would receive the click: <html lang="en" theme="dark">...</html>

The ‘theme=”dark”‘ element is adjacent to the side-nav button. So something was wrong with the point location calculation.

Setting an implicit wait period did not work. An explicit wait period did not work either. One thing that really sucks about wait code is that it swallows the exception so you don’t see the actual problem. (You are ignoring the exception and not logging it.) So when it fails, all you know is that it did not work for X seconds; not why.

The third issue is that the Selenium isVisible() method checks the element values but does not actually check to see if the element has been scrolled into view. Playwright does it correctly (my interpretation of “isVisible” is literal). Playwright also automatically scrolls elements into view when you try and act upon them. Very nice.

So I had 3 immediate frustrations with Selenium that Playwright took care of.

I sat back and googled some more. I found a video I liked where the speaker asserted that there was no comparison between the two.

At this point I was inclined to agree with him. And I’d invested a day to verify I was making the best decision. Out with Selenium. In with Playwright.

Now I am not saying Playwright is without difficulties. The codegen tool is a miss more than a hit when I use it to try and auto-code the Locators. And I believe I have found a bug where it just fails to work correctly with chromium. That was a huge frustration that cost a week of stoppage hell until I ran out of ideas and tried firefox instead of chromium. Firefox worked, and I was able to start moving again.

Spatialite DLL hell

Spatialite is a geospatial-enabled extension to sqlite. I must have it. I intend to use it inside our commercial java product, which has windows and linux users. The development environment is Eclipse. The OS is Windows 10.

The author of spatialite mentions that the windows environment suffers from DLL-Hell. Well yes it does.

Installing sqlite and the xerial-jdbc software is easy and not covered.

You download the spatialite files here. It isn’t well explained, but you need the “mod_spatialite-.7z file.

My preferred eclipse setup is to create a /lib folder in my eclipse project and put the jars and dlls there. Below is a simple test project showing the locations of the files. Note that the sqlite-jdbc.jar is included with a “Native library location” that points to the folder containing the dll files. This is the standard way to do things. It works most of the time.

The author of spatialize has some good installation instructions and sample code. I added the sample file and tried to run it.

The result was the following error:
Failed to load extension mod_spatialite. Trying to find file manually
Exception in thread "main" java.lang.UnsatisfiedLinkError: Can't load library: C:\apps\eclipse-64-workspace\spatialite-test2\lib\mod_spatialite.dll
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1827)
at java.lang.Runtime.load0(Runtime.java:809)
at java.lang.System.load(System.java:1086)
at SpatialiteSample.main(SpatialiteSample.java:40)

This is the well-known dll hell. The eclipse launcher uses the “root” folder of the project as the root java location. Here are the details. The command line is:
"C:\Program Files\Java\jdk1.8.0_121\bin\javaw.exe" -agentlib:jdwp=transport=dt_socket,suspend=y,address=localhost:57143 -Djava.library.path=C:\apps\eclipse-64-workspace\spatialite-test2\lib -Dfile.encoding=Cp1252 -classpath C:\apps\eclipse-64-workspace\spatialite-test2\bin;C:\apps\eclipse-64-workspace\spatialite-test2\lib\sqlite-jdbc-3.18.0.jar SpatialiteSample

At this point, someone gets desperate and recommends adding the spatialite path to the windows PATH variable, or drop the dlls in the /system folder. But this is intended as a professional installation, not something for the enthusiatic developer. These are not acceptable recommendations.

To simplify the problem, a good next step is to move the dlls to the root folder. Can’t miss them there, can we?

And that works. The “mod_spatialite.dll” is found in the root folder. The command line is:
"C:\Program Files\Java\jdk1.8.0_121\bin\javaw.exe" -agentlib:jdwp=transport=dt_socket,suspend=y,address=localhost:57854 -Dfile.encoding=Cp1252 -classpath C:\apps\eclipse-64-workspace\spatialite-test3\bin;C:\apps\eclipse-64-workspace\spatialite-test3\lib\sqlite-jdbc-3.18.0.jar SpatialiteSample.

So we can run spatialite in eclipse if we are willing to place the dlls in the root directory. We are not.

At this point, I am extremely puzzled. To be continued, hopefully…

Java Null Pointer Static Analysis in Eclipse – JDT vs CheckerFramework – False-Positive comparison

We’re talking about this and this. For those of us who program in Eclipse using Java, what is the best tool(s) for checking null pointer exceptions.

There have been several attempts before, but nothing has made its way into the official language. Eclipse does provide good null pointer checking out of the box, but there is only so much it can do. To improve the power of the checkers, we need to add annotations.

Note: In the following, I am using JDK 1.8.

1) We have Eclipse itself, using the JDT annotations Eclipse help.
2) We have the standard checker framework.
3) We have FindBugs, which I believe is used by sonarcube.

There is great power to have something built into the environment, so the standard eclipse system has a strong advantage. Unless it has serious problems, it should be part of the standard development environment.

Unfortunately, it does have serious problems. Take a look at the following program:
[sourcecode lang=”java” highlight=”12″]
public class TestNullCode {

@org.eclipse.jdt.annotation.Nullable
private String testJdt;

public void testJdt() {
if (testJdt == null) {
testJdt = "not null";
}
// this line causes a false-positive warning
// Potential null pointer access: The field testJdt is specified as @Nullable
if (testJdt.equals("not null")) {
System.out.println("will work because testJdt is not null");
}
}
}
[/sourcecode]

This example shows a false-positive that is provably incorrect. The compiler issues a warning (or error, depending on your settings) on line 12, that is untrue.

Frankly, this is pretty sad. The reasons are justifiable, but the result is sad, nonetheless.

For non-final @Nullable fields the simplest strategy for avoiding any nullness related risks is to pessimistically assume potential null at every read. This means for strict checking no flow analysis should be applied to @Nullable fields..

How does the checkerframework do? Answer: No problems at all:
[sourcecode lang=”java” highlight=”11″]
public class TestNullCode {

@org.checkerframework.checker.nullness.qual.Nullable
private String testCheckerFramework;

public void testCheckerFramework() {
if (testCheckerFramework == null) {
testCheckerFramework = "not null";
}
// this line is ok
if (testCheckerFramework.equals("not null")) {
System.out.println("will work because testCheckerFramework is not null");
}

testCheckerFramework = null;
// following line is flagged by checker framework
// dereference of possibly-null reference testCheckerFramework
System.out.println("will fail because testCheckerFramework is not null" + testCheckerFramework.toString());
}
}
[/sourcecode]

How does FindBugs do? Answer: No problems at all. Same as checkerframework , so I won’t post the identical code.

Eclipse has a workaround that costs a line of extra code:

While this appears to be a very drastic restriction, the remedy is quite easy: before dereferencing a @Nullable field it has to be assigned to a local variable. Flow analysis is then safely applied to the local variable with no risk of side effects, aliasing nor concurrency, since local variables are not shared with any code locations that would be outside the scope of the analysis. I.e., the flow analysis can see everything it needs to consider regarding local variables.

[sourcecode lang=”java” highlight=”12″]
public class TestNullCode {

@org.eclipse.jdt.annotation.Nullable
private String testJdt;

public void testJdtCopy() {
String copy = testJdt;
if (copy == null) {
copy = "not null";
}
//This is fine
if (copy.equals("not null")) {
System.out.println("will work because copyis not null");
}
}
}
[/sourcecode]
I consider this “solution” ugly. I get it, but I don’t buy it.
At some point, I would expect the Eclipse engine to get better.

So at this point, we still need the other tools.

You can download my example eclipse project here.

Running Java apps as Windows Services

A tricky little problem, as it turns out. There is little out in google world to help you, and only a few tools. License issues with almost all I found, especially running on Windows boxes in 64 bit mode.

 I finally found Yet Another Java Service Wrapper at yajsw. License is LGPL and it supports 64 bit windows. I was able to get my application running very quickly. Worked perfectly the first time. Getting going was interesting:

 1) You download yajsw and extract the zip file.
 2) You run your app and lookup its PID.
3) You run the yajsw script generator, and it analyzes your running application to create a script file.
4) You then run the yajsw bat files to install-service, uninstall-service, start-service, stop-service, etc.

The only trick is to run the bat files with administrator permissions. (Right-click on the bat file and select “Run as administrator”.)

Right now the yajsw folder needs to exist on the server as well. Doubtless there are more convenient ways to package the install, but this was a good first step.

Debug xquery in eclipse. Configuration

So you want to run xquery in eclipse. You eventually find your way to XQDT. Some concern that we’re working with beta software (version 0.8.0 as of this post), but you bravely forge ahead. You survive the installation and run into your first roadblock at the getting started. Here you discover that they are working with Zorba 0.9.4 (later you suspect this is a misprint, but never mind) and do NOT SUPPORT DEBUGGING! If you can’t debug, this is DOA.

You follow the note to install Zorba from the subversion trunk, but this is a C project and no compiled distributions. Dead end 1.

You download and install Zorba itself (version 1.4.0), and while it runs, you are unable to debug. (Perhaps you can, but you never figured it out and got impatient. Plus, you refuse to depend on other programs installed in Windows anyway, because this is a distribution/installation hell.) Probably dead end 2.

You hear about the 28msec Sausalito project. They apparently have a version of Zorba that supports debugging. Looking better, but you don’t want a separate Eclipse installation. After a bit of exploration, you figure out to how attach the Sausalito project code to your existing Eclipse installation:

1) Install XQDT into Eclipse as normal. Steps described here

2) Download the Sausalito project from here, and extract the zipped contents.

3) Navigate to the plugins/com.28.msec.sausalito.win32/coresdk/bin directory.

Note: In my download, the path is com.28msec.sausalito.win32_1.2.10.201011291638

4) Copy the contents of this directory into your existing Eclipse project. (Put it somewhere convenient, of course, such as /lib/sausalito,)

Note: Obviously, you do not need all the contents of this directory. I do not yet know which files are required, and which are not. I know the zorba-sausastore.exe file is required, along with some others.

5) Now to add the new intepreter to Eclipse. Open Preferences -> XQuery -> Interpreters. Click the “Add…” button, then the “Browse” button. Navigate to the zorba-sausastore.exe file you just added, and select it. Eclipse should be able to inspect the file and set the default Interpreter name, e.g.,

Click ok and make the new Interpreter the default (or change it only for the one project).

And that should do it. Look see 🙂

I find it odd that Zorba is up to version 1.5.0, but XQDT is back on 0.9.4.

I also have NOT done anything with a Linux install. Obviously, you won’t have a win32 directory, but you’ll know what you’re doing anyway.