Gimp batch mode with gmic

For photo work, I use Aurora HDR 2019 and Gimp. Within Gimp, I use the GMIC plugin a great deal, as it has the best noise reduction and hot pixel reduction.

The one big weakness of Gimp is lack of batch mode. You cannot record some action on an individual image, then save that action and apply it to a group of images.

And I do a lot of work on groups of images.

There is a heavy duty script processing engine in gimp, but I find it inaccessible. And I’m a programmer! The basic problem is that I am a lazy programmer and I really don’t need another language unless I really need another language.

Well there is BIMP for batch Gimp operation.

I’ve done practically no BIMP. The basic operations that BIMP provides I can can do in Irfanview.

Well I found myself needing to remove the hot pixels of a few hundred photos. The gmix function is remove_hotpixels. But this is the first time I have seen this documentation and the example is confusing. After reading this I thought I needed to write:

+remove_hotpixels _mask_size=3, _threshold=10,

And I tried many many options and nothing worked.

However, some of the simpler commands worked, so I knew it was just a matter of finding the right syntax.

This post is to record what worked. The image below shows the successful Bimp settings. (Except the input field would not expand.)

The function name is plug-in-gmic-qt

The input layer is something besides 0 (holy crap, that just generated out with no changes whatsoever and was a real pita to figure out)

The output mode is 0. Maybe some other options work but 0 works.

The command line string is “remove_hotpixels 3 10“, where 3 is the mask size and 10 is the threshold.

Don’t use a “plus” or “minus”. Don’t name your arguments

Well that is it so far. Another inch of knowledge.

 


Update from 2020:

 

Well, I have given up trying to run batch gmic from within gimp. Too much of a hassle, too iffy, and far too slow.

Instead, I installed gmic, the command line tool. You can find it at https://gmic.eu/download.html.

Better to have a horrendously hard to figure out uphill battle from the command line than the same thing from within a clunky gimp dialog.

The positive from using a command line is that we have well known tools for iterating over multiple files. This is important because the “iterating over multiple files” part of the gmic command line doesn’t seem to exist. I can only figure out how to write a gmic script to read one file and write one file.

I can’t yet even figure out how to tell gmic to read a jpg and write a tiff.

But the “remove hit pixels” command is (on a windows platform, using “cmd”):

for /r %f in (.\*) do gmic %f ^
-remove_hotpixels 3,10,,,Merged ^
-o %f

From the command line, cd to the folder containing the files you want to convert and paste the above text. It will recursively find all files in this folder and all sub-folders. It will open each file, run the “remove hotpixels 3,10” command, then save over the same file. And it will do it orders of magnitude faster than the same thing in gimp with bimp.

I also find anisoptropic smoothing to be very useful. It is a nightmare trying to find the right combination of arguments to a gmic command. The best way  is to setup the command in gimp then alter the settings to display the arguments. Still a pita, as you have to eyeball the text and re-type it into the command line; copy-paste does not work.

Here is a starting point for anisotropic smoothing:

for /r %f in (.\*) do gmic %f ^
-fx_smooth_anisotropic 80,0.7,0.3,0.6,1.1,0.8,30,2,2,0,1,0,0,50,50 ^
-o %f

The actual settings are listed here, but good luck converting the argument types to the integer command line values.

 

Not sure where the best documentation and tutorials for gmic are. Everything I have found requires you to know what everything means before being able to do anything. More notes added here as I learn.

 

https://manpages.ubuntu.com/manpages/trusty/fr/man1/gmic.1.html

http://gimpchat.com/viewtopic.php?f=10&t=19008

https://gmic.eu/tutorial/

G’mic Command Line: First steps

RIP Chester Williams

The death of Chester Williams hit me very, very hard today. I’ve written before how much the World Cup Final between South Africa and New Zealand mattered … the subject of the movie Invictus. And how important Chester was to that victory and the future of South Africa. In how he delivered the first big tackle to Jonah Lomu that set the tone for the entire game. It was at that moment I began to believe we could somehow win. Chester carried a heavy weight on his shoulders as the sole black on the team. People worried he was a “token player” and it was a fair concern because he was the first black to make it. I’d been studying him intensely all tournament (who hadn’t?) and knew he deserved to be there. But sports can be cruel and heroics seemed too much to hope for. Then, as I saw Lomu shudder and collapse, I began to hope and believe … and hope and believe … and then simply hoped and prayed and hung on until the end … like all the players on both sides. The greatest game of all time in which no one could score.

I cried like a baby after that game — the only game that ever mattered so much — and the deepest tears were because of Chester. If we’d won but he’d been a liability on the field, it would have been a setback to a country’s future. Instead, he had the game of a lifetime. And whites experienced pride and love for the gift of a black man’s pure courage in an arena that they understood viscerally. Many for the first time. The celebration of Chester was perhaps the first, honest, positive feeling that all colours could experience together.

I am sorry he died so young. But he is a legend and had 24 years of that knowledge. I will never forget my admiration of and debt to him.

Aurora HDR 2019 – Questionable RAW support of Sony RX10M4

I’ve had support communication with Aurora about problems reading Sony RX10M4 RAW files, which they claim to handle. At the time, I was complaining about invalid cropping: The corners of a RAW image containing vignetting (the dark edges), and Aurora did not remove it. Aurora support said that this is normal operation.

The following image shows this artifact. Same image in RAW and JPG processed by Auror without any subsequent processing. On the left is the RAW image and you can see the vignetting. On the right is the JPG image and you can see the vignetting was cropped and the image was enlarged to the same pixel dimensions.

Note that the JPG image was generated by the camera, i.e., I am saving in RAW+JPG format.

I also use the Sony Imaging Edge app to read RAW images on the computer and save them to JPG. When I do this with the RAW image, I get the identical JPG that is stored on the camera. So this tell me that Imaging Edge DOES choose to apply the vignetting when converting RAW.

Well I can live with this decision by Aurora. But that isn’t the full story. As you can tell, the above images are not the same otherwise. For example, the colors are different. Perhaps this is due to Aurora having better information from the RAW images and making more informed decisions. I can live with this too, if it is true. Color can be altered.

It is when you get to the details that the serious problems appear: 1. Aurora has made different decisions about the cropping/expansion. 2. The RAW image has bad noise artifacts. 3. The RAW image has chromatic aberrations.

Remember that the RAW image is stored with all the imperfections: The sensor itself has noise artifacts, chromatic and lens aberrations. But the RAW image also contains the information that lets the processor compensate for them. Aurora is not doing so.

Look at a close-up image. Below is a portion of the tent on the left-hand-side. On the left the RAW image processed by Aurora. On the right is the exact same region from the JPG image.

 

First, we see that we have different shapes. This is what leads me to suspect that Aurora is not using the lens information to correct for it. (I am assuming that Imaging Edge IS correcting for it, as well as the in-camera software.)

Second, look at the noise. The RAW image contains a lot of noise. In comparison, the JPG has eliminated that noise, but now has JPG noise artifacts.

Another close-up image below, showing the noise problem again AND the chromatic problem. On the left, the RAW images show serious color error along the borders. The JPG image does not.

If Aurora is doing “correct” RAW image processing, I don’t want it. (This is why I am currently saving in RAW+JPG and only working with the RAW when I absolutely have to.) But really, I am seeing so many issues that I am not convinced that Aurora is processing the RAW correctly.

Back to tech support …

…Back from tech support. Confirmation that this is a bug.

Aurora HDR 2019 – Blowouts

Much to learn about Aurora HDR 2019 still. This particular problem is that it is importing a set of bracketed images into an HDR and the starting point has blowouts, i.e., the whites are crushed to 100% and information is lost forever.

Below is a set of 9 images. These are bracketed exposures, with 1 EV separation.

After import into Aurora, this is what I see:

Notes: Actually importing the RAW images. auto-alignment, ghost reduction, color denoise, chromatic aberration reduction all turned off.

Ignore the vignettes. This is a separate issue I am dealing with (Aurora processes the RAW information but does not do lens correction).

I have “white highlighting” turned on. All the red areas in the image are 100% white, i.e., crushed/blown-out. If you look closely at the histogram, you will see a vertical line at the 100% mark. That is a problem.

That vertical line and the red splotches means that information is lost and cannot be recovered. No amount of adjustments in Aurora will let me fix this.

An HDR should not do this. This is a bug in the tool. We know for a fact that at least 1 (actually more than 1) image in the set is not blown out at these location. That information must be retained for further editing.

(Note: I proved that fact by importing only the darkest image into Aurora and noting that there were no blowouts.)

I could probably fix this by removing one or more of the over-exposed images and re-importing. But that isn’t the solution.

We’ll see what Aurora support has to say about it.

Sebago Resort dewinterizing – Memorial 2019

This was the first time that Robyn could not make it. We had some rain the first day, but otherwise great weather. Cold enough at night to need fires in the cabins. The lake water was really cold. My first time with waders, thanks honey. Replaced 2 dead trees and everything looking pretty good. And, of course, my new Sony RX10M4 camera.

Aurora HDR 2019 software. Sony RX10M4 5 image bracketing with 2 stop gaps. One sequence every 2 seconds. Pete Townshend “Dirty Water”. Many struggles with Aurora’s inability to accept the number “5” as the correct answer to the grouping problem.

Whassup with the Aurora HDR app sharpen halo?

Still a week 2 noob here, but going through some old HDR work to learn Aurora HDR 2019. I had an old series of HDR images from Sebago that was interesting: Boats on a dock. Boats that move constantly even on a calm day, and thus are very difficult to deal with.

The original base images are here:

base-1
base-2
base-3
base-4

I used hugin to align the images and produce the following HDR image shown below:

Hugin produced HDR image

I thought this was amazingly good. You can see the blur artifacts from the movement of the boats (the number 15; the motors, etc), but if you don’t look too closely this is a nice image.

Below is the attempt with Aurora HDR 2019, Version 1.0.0.2549. I imported the base images with Auto Alignment, Ghost Reduction, and Chromatic Aberration Reduction all turned on. Then I turned off all the Filters. (Meaning this is not going to be the final result.)

Aurora version

Obviously a fantastic job with the blurring. Amazing. I can work with this.

Except I can’t. Something is causing sharpening halos. I’ve tried turning all the filters on and off, but even with everything off, nothing eliminates those halos, afaict. So where did they come from?

Here is closeup of the problem: Look at the sides of the posts. Hugin on the left shows that it is possible to combine the images without excessive halos. Aurora on the right show unacceptable halos.

Not all my images show excessive halo. I did a series of work with a much better camera at http://www.clevercaboose.com/2019/05/07/tablerock-trip-20190414/ But now I needed to stop and figure out what is causing this.

I created a support ticket with Aurora. Quick response:

..it’s quite normal behavior of the software…the RAW photos you’ve sent to us are quite low-resolution ones, and they come out aligned pretty well considering their resolution and format. You might have achieved a better result with RAWs though. The halos you may see are not halos, but light on the photo increased by our tone-mapping powered by AI which increases the contrast.

Contrast that looks like a noob that just discovered the unsharp mask filter! I’d rather a more adult AI.

A note on the word “halo”. Specifically I am talking about overshoot and undershoot.

But perhaps the problem IS low resolution. These ARE old photos.

Try again with 2019 quality images:

hidef-base-1
hidef-base-2
hidef-base-3

Here is an HDR composite with ALL Aurora filters turned off.

ALL aurora filters turned off

The light on the horizon IS “haloed”, excuse me, AI-powered contrasting. But there is also contrast from the original images. Has Aurora added to this even with all filters turned off? Below is an area on the horizon zoomed in:

Zoomed in horizon

If there is any halo,. it is present in the originals as well. I have no cause for complaint. Aurora is excused.

Caveat: Play around with the filters (in particular the “HDR Smart Structure”) and you will find many ways to create crappy sharpening. So don’t turn the knob to 11.

So the issue is resolved. But be careful working with low resolution images. The AI isn’t very smart all the time.

Update 6/1/2019: Here is an extreme example with hi-rez images. Base images first. This is 9 images with 1EV bracketing

And here is an HDR of all 9 images. I aligned the images (this was hand-held), did no ghosting, and selected the “essential/vivid” template.

9-image HDR. Align images. No ghost reduction. Essential/vivid template.

Below is the same 9-image HDR with all the same settings EXCEPT medium ghost reduction is turned on. And here we see the problem: That is the worst halo I have ever seen.

Previously I was frustrated because I could not find a setting to turn off whatever was causing the halo. Now it is clear why: That halo is baked in when the image is created.

Robyn’s last game with Mike Cleary, Feb 18, 2018

Today was the last game of 10 years of my daughter playing for this man. When she started she could outrun anyone on the field, but a gust of wind might blow her over. A gazelle on a top team in the city. She got to travel far away to play in miserable weather. She got to learn that she is deeply competitive, even ruthless, but never dirty or unfair. She got to learn team success. She got to make mistakes that cost the team the tournament. She got to sit on a bench and wait for another chance. She got another chance and failed at that too. She got to endure two frustrating years playing only 1/3 of a game on a struggling no longer top team in the city. She got to learn that if failure is one’s fate, then not complaining and continuing to work hard every day is how you beat it. That character and perseverance is necessary to win in the long run.

She got to learn that she is a Marine by nature: That she’ll take that hill or die trying. Anything less is unthinkable. And that most people are unlike her, so she’ll probably spend much of her time climbing that hill alone. But that the view is worth it.

She got to learn that life is unfair. But that even if one is too small to hold onto the ball, one can always develop a great first touch so as never to be caught with the ball in the first place. Which now makes one a great passer. So weakness overcome at one stage of development becomes an advantage at the next. And that when height and strength finally do arrive, payback is beautiful.

The gazelle is now a panther.

I didn’t always have faith my daughter was going to be a good player. Those failures were painful, and it was my first rodeo. She had athleticism, but lacked skill and received no reward. I didn’t think anyone else had faith in her either, perhaps even her coach. 5 years in, after another disappointing year, I asked if we needed to move on. He was shocked at the question, answering: “Absolutely not. She is about to become very good and I have big plans for her.”

(What he only told us later was that he was alone in that opinion. All the other coaches had recommended he drop her.)

That meeting ended our concern. I trusted the man. That summer she grew, and that fall everything changed for the better.

3 girls who played all 10 years together. 8 who played the last 6. That just doesn’t happen. You have to be incredibly lucky.

Lucky to have a coach powerful enough to win the internal club battles and keep the team together. Who has always done the right thing for the player, as opposed to himself. Who cares about character and chemistry and playing soccer the right way. Who praises a loss played well, and criticizes a win played badly. Who criticizes without belittling. Who believes that fundamentals cannot be taught via shortcuts. Who demands you play it out of the backfield. That you pass to build up an attack instead of just kicking it downfield. Even if it means you lose for a few years. That you need to take that hill the way he tells you to take that hill, and if you die today while trying, today isn’t the most important matter.

Where you are in 10 years is the most important matter.

Thank you, Michael Cleary. Your character matters, and that is why this team always stayed together. My daughter has been incredibly lucky to have you as her coach. May you have many more teams of quality, and may you be respected and rewarded for being the great coach you are.

Kitchen remodeling before and after

So we’re remodeling the kitchen. The impetus was getting fed up with the electrical stove and lack of vent. That plus watching the excellent contractor working next door and his crew showing up 7AM to 7PM including weekends. Hard to find good people so we pulled the trigger.

Expected to take 1 month. Removing the wall to the dining room. Hard wood floors. New appliances except for the fridge. Custom cabinets.

Following picture was taken the night before the work started, Jan 21.

Kitchen the night before

Day 1 was a full day of demolition. Dramatic and happens fast. They even managed to add the frame for the pantry. Jan 22

24 hours later after demolition

Electrical dry walling was next. After that was the hard wood floors. Thus endeth week 1, day 4, Jan 25.

Day 5. Electrical, dry wall, and hardwood floors.

Next was the custom cabinets. 6 full days of 2-3 people. We had a tricky situation with the bay windows and future sink. No problem. These guys figured it out as they went and did it perfectly. We added a peg board to an open space, and neat triangular diagonal section facing the dining room. Do not buy standard cabinets and expect to like the results. I repeat, do not buy standard cabinets. The customization is critical and well worth it.

Day 14. Custom cabinets completed

Onto painting. And this gets stressful. First, you’re going to come home to a house where you can’t breathe. Meaning the entire area is either walled off or completely open with fans blasting the air out windows into 10 degF outside weather. You bundle up and watch the house temperature drop to 53 degF and do your best.

Worse is that you’ve picked the wrong color and it is overwhelming. The ripe apricot looks bright orange in warm light (see peg board) and coral pink in cool light (see fridge cabinet)! Lesson learned: Do not use water-based paint to test when the final paint will be oil-based.

Day 18: Ripe Apricot looks like orange and coral pink!

So we cost ourselves a day. We didn’t want to go into the brown-ier shades, even though that would have been safer, given the different lighting conditions. We tried the next darker shade of ripe apricot and that did what we wanted. Even that was scary, as practically the entire kitchen was one dominant color, and we had to take to visualize and gamble that the walls and countertop would balance things out.

Day 21: Corrected color for cabinets. French Vanilla for the walls completed.
Day 22: Cabinet doors replaced. Neutral/warm lighting, early morning sunlight. Still a balance of warm (left) and cool (right) lighting

And next is the backsplash. We started off wanting something plain and simple that would match the wall color. But plain tile disappears. So we thought we need smaller tiles. That was better, but still not quite there, see below.

This wasn’t bad backsplash.

Also, prices range from $3-80 per square foot. We also had to consider that black pearl countertops would change the balance as well. It came down to the two options below:

Final two choices in backsplash options. We decided to go bold with the tile on the right, as it should go better with the Black Pearl countertop.
Mixing up the lighting

Construction got behind 1 day and then a 2nd day because of bad weather and someone getting sick. On the day they were supposed to put in the countertops, no one showed up. I went shopping for lights at 7PM and when I got home three men were fitting the countertops, grinding granite in the driveway and fitting the new sink. One enjoyed his 0% alcohol heineken. All done by 10PM.

Countertop and sink installation, day 24

Shortly after we got the big stuff in. And then I had the epiphany of getting some art work from our good friend Dierk Van Keppel at
http://www.rockcottageglassworks.com/ . And then we had a functional kitchen and things slowed down a lot. I got obsessed with wifi lighting and invested heavily in the Lutron Caseta system. A premium price, but only by 10% and they are the best. Electronic stuff that just works is worth a premium. Took some time to get drawer handles in. Bought back the cabinet people to add some internal shelves on rollers. And all the little loose ends.

Fast forward to March 23, or 2 months since we started. 98% done and all is fine. The color grabs your attention and changes throughout the day. I’ve been trying to create a good HDR image and failing so far. So here is the way it looks when fully lit:

Near completetion panorama. Bright

And a little darker

Near completetion panorama. Darker