Advance and be Recognized

Another important component of my weather server (MetCommand-Home) is the sky-recognizer program.

Every five minutes, a webcam aimed out of an upstairs window snaps a shot of the sky. Usually, that's as far as it would get. But I wanted to have real observations, and not something I'd be having to update manually every few hours.

I finally settled on programming something, wxSky, that would do the job.

wxSky examines the pixels of an image and comes up with an observation. Currently, the list of observations is:

- Sunny (daytime only)
- Cloudy
- Twilight
- Clear (nighttime only)

The sky condition is gauged by examining the colour channels of the image, both separately and together. The rules are relatively simple and are accurate at least 90% of the time, although it can be difficult to detect clouds on a summer's night--contrast that with winter, when the ground is covered by snow and glowing in all the light pollution, and the clouds light up just fine. Partly cloudy would be more difficult to detect, and I find that, 90% of the time, sunny or cloudy does just fine.

It is, of course, a GUI program, with an editable parameters file, Linux-style (in fact, the production version runs on Linux).

wxSky v 1.27

It's difficult to see in this photo, but in the upper-left-hand corner of the window is mostly administrative stuff--the time and date; the latest observation, the middle is a view of the picture as nine averaged zones, with the Sweet Spot (more on this shortly) highlighted. Below that is a box simply recording the last five observations.

The upper right-hand corner is a view of the average values for the nine zones of the picture. Three buttons--R, G and B--are for selecting Red and/or Green and/or Blue channels.

Finally, there is one composite (master) value--displayed for R, G and B channels (and combined). These are averaged values for the entire picture.

Below that there is the same display, but for the Sweet Spot, which is a user-selectable zone (1/9 of the pic).

Why all the RGB breakouts? Because different sky conditions are based on colour-channel values. Twilight is a deep blue, a bit of green, and almost no red. Sunny is a deep blue with a good helping of green and a little red. Cloudy is almost equal values of red, green and blue; or, at night, with significant red, some green and almost no blue. Easy access to those values is important when calibrating the program to a new camera.

Why a Sweet Spot? Because your camera could be facing in any direction or (like mine) be partially obscured in a particular zone. I personally find that zone 7 (lower-left corner) has the best values.

The output of the program is a single text file simply containing one line with the date and time of the observation, and the observation itself. This is handed off to wxOCR, the LCD recognizer, for incorporation into an observations packet.

The weather server (Centretown Observatory) is available at:
http://mizar64.dyndns-home.com:64180/weather.html.

-Bill

Avast, Damned Thing!

[Edit 2015-08-22: I see there is an existing product for OCR'ing LCDs called, coincidentally, WxOCR.  I will have to change the name of the program.]

I really should have blogged, before now, on the wonderful little OCR (WxOCR) program that reads my home weather station's statistics from snapped JPEGs. It was a couple of years in the making, and put my coding skills to the test; but it's been running for at least three years, and running very well.

The Hardware
I have a home weather station, which consists of several outdoor sensors, wirelessly linked to an LCD display pad. I also have a spare ancient laptop and a 1024x768 webcam.

The Mission
To deduce the values of the readings on the display, accurately, cheaply.

It's Dependable
(In essence) what happens is that at five-minute intervals, cron fires off a message telling a shell script to snap a picture of the weather station's LCD display. Having done so, the photo is handed off to WxOCR, which analyzes the green portion of the image (for maximum contrast, and it's configurable--you can use red, green, blue, or all channels). Data are defined as Groups, Characters and Segments; for each character, the expected location of each segment area is scanned and compared to a control value to determine whether it is lit, or extinguished. The LCD character has seven segments (we ignore any decimal point) which can correspond to the bits in a binary series; we add up the seven resulting bits to get a value from 0 to 127.  This is compared to a list of permissible values; if it's a match, then you've done one character, and move on to the next. If no match, a null is generated, which immediately halts processing for that number-group.

Later in the processing, the output values are herded into CSV format, along with the Sky Recognizer program's output, and handed off to the weather server.

So far, in about three years' operation, it has yielded probably hundreds of thousands of observations.  Most weeks, it is about 99.98% accurate.

It's Highly Configurable
All the configuration is via two files, one of which can be altered at any time and one of which is reserved by the system.  Most configuration directives are concerned with the placement of characters and segments.  Also, for each character, a control spot (a nearby area guaranteed always to be clear) can be defined by both position and dimension.  Characters can be configured horizontally or vertically, and optionally their individual segments can be defined.  Character groups can be interpreted is several modes: direct-read (for most things), darkest-segment-wins (useful for displays with a windrose), and (for precipitation) a running-totals mode.  A decimal point may be placed at any character position. Math (+-*/) can be performed on the character group at several points in its processing.

It's Error-Correcting
Because all of the values permissible must be defined, error-correction can be handled.  Partial characters, or characters with an extra segment in them, can be converted to their proper values.  In general, this works exceedingly well.

It's GUI
The program was designed from the outset to be a GUI program (screenshot below).  It's got everything I need to be able to position data groups quickly--or even the whole screen.  It even has 'second sky condition' buttons, plus a timer value; these are for situations where it's difficult for the system to detect; fog, snow, etc.

It's Linux
I need a rock-solid operating platform; but even the most-advanced Windows version needs to be rebooted regularly.  Therefore, WxOCR is built on Debian Linux and has essentially no downtime (it reboots weekly, three minutes early each Sunday morning).

But it ain't Perfect
It's a minor annoyance, but for whatever reason the character '2' is often missing the top bar, even though it's clearly in the image.  Technically, this shouldn't be happening.  It does anyway.  And, during the day, differences in lighting can throw off the threshold values established for each character. [Edit 2015-10-25: This is now essentially fixed.]

Nor Ever Will Be
It's almost impossible to get a perfect week's readings.  In fact, in all those three years, I've only seen it happen once.  This is for various reasons: the digits were themselves in transition as the pic was snapped; the camera shifted (damned cats); the contrast was poor in one particular image; and so on.  Plus, the camera can drift in aim and focus over time. I'd frankly like to see a commercial system (if one exists; I can't find one) with a better accuracy than WxOCR.

It's Still Evolving
I can still see more work I'd like to do on the program.  I'd like to add a four-way doodad for the individual characters in a group (refer to photo below).  I'd like to change slightly how the precipitation sensor works, to cut down on n/a readings from it.

It Needs to Go Open-Source
So far, I've not found another program that does what WxOCR does.  I really think this program needs to be open-sourced, and I'll do that in the near future.  I think it could really fill a (highly vertical, admittedly) need.

Here it is, in all its glory:

WxOCR v. 2.82

A Proposal for Superluminal Communications Using Entangled Photon Pairs

For the life of me, I cannot understand why the following proposal should not work:

Exactly between two points (points A, B) in space time separated by an arbitrary distance (point C), position a photon generator and a beam splitter.  Then generate photons, split them into entangled pairs, and send them towards (A) and (B).

An entangled pair of photons is linked; the destiny of the one is the destiny of the other.  The effect is instantaneous, no matter how far the particles end up separated in space time.

At each endpoint (A, B), there are just two pieces of equipment:  a photon detector, and a shutter positioned slightly ahead of the detector.  Each endpoint takes turns sending and receiving.

When it's A's turn to send, sliding the shutter closed will intercept the incoming photon, causing its entangled partner also to extinguish; thus the detector at (B) will detect nothing; call that a 1.  If A does not slide the shutter, the photon can continue, then the detector at (B) will register a photon; call that a zero.

When it's B's turn to send, the same situation obtains at (A).

Granted, this method requires a source of photons to be positioned in advance.  Depending upon the separation of (A) and (B), it could be years before the stream of entangled photons is available to be manipulated.  Also, it requires extreme coordination for the entangled pairs to arrive at both stations simultaneously.

This communication method could be tested using present-day equipment.

-Bill

Upgrade Hell

I was 'forced' into upgrading Ubuntu on the server last night; there was a brief outage between 0800 and 0900.

I grow weary of the Ubuntu frequent-upgrade game.  Every six months, upgrade or lose out on support.  In view of the Shellshock vulnerability currently out there, that carries new weight.

Generally, when I install Linux, I install Debian.  It's got an upgrade cycle of once every two or three years.  I made a mistake opting for Ubuntu for the server, one which I'm determined to correct before the next Ubuntu upgrade cycle.  Which is due in about a month.

Because things go wrong when you upgrade.  Case in point:  through two upgrades now, Ubuntu has insisted on installing lxdm window manager, even though I run with kdm.  Now, it's true that a simple apt-get install kdm made short work of it; but still.

Stop the Ubuntu, I wanna get off.

Data Glitch

The past few days, readings have been sparse. The OCR program was crashing out, and I didn't have time to see why.

I finally got a look a few moments ago and immediately noticed why: one of the elements of the OCR program is the ability to define 'validation mappings' for each character. In other words, this combination of elements lit up equals character X. I had originally allocated for 12 validation mappings per character; but recently I realized that the OCR program could actually do error-corrections: if a certain pixel is added to or removed from certain characters, a distinctive pattern appears. So why not harness that and make the OCR progra much more fault-tolerant. Well, it turned out that in adding the additional mappings, I overshot the limit of 12. Easy fix; five minutes.

The lesson to be learned here? Always allocate amply for your data needs.

A New Project

I've embarked on a new project, one that actually ties in very nicely with the weather system and keeps me marching in the right direction development-wise.

I'm working on a way to synchronize the hosts file amongst various machines on a network, as well as Google-Chrome bookmarks (and, in future, files to be backed up).  There are various kludges to make this happen; I just said screw it and have begun writing myself a couple of programs.

The first is the server.  It will respond to XML requests on the server's CGI-bin interface.  The server is (at least for now) designed only to run when queried.

The second is the client.  It will send XML queries to the CGI-bin interface, and listen to the results.

I've decided upon XML as the way to do this, simply because it's good at encapsulating string information, and just a bit more precise than the typical query of key1=value1&key2=value2&etc.

So, for now I'm working on the skeleton of the server and in particular the XML module.

In the meantime, I'm still also working on my weather system, plus on my Comedy Radio Show.

-Bill

Getting Ready to Open Source
2014-02-10

My weather system is starting to mature - it's at Version 3.10 now, and I frankly can't think of much to add to it, apart from an expanded environmental dataset - Snow-on-Ground, that sort of thing.  I recently added queries and a data-extant flag, which tidies things up a bit.

So, I'm getting ready to open-source the whole works.  It turns out that's taking some time.

First, I've got to document what I have.  It's no easy proposition, but it has to be done.  I've been at it for two days, now, and have some 22 pages complete.  It's gonna be one of those projects.

Next, I'll have to tidy things up a bit and include a sample dataset.  That'll take only a little work.

Thirdly, I've got to document the install procedure.  That's a little bit involved but, again, a necessary evil.

All that, not just for the weather system, but for all of the support programs as well.

Once I've got all that together, then I've got to upload the works to sourceforge.

Bleah.

-Bill