Avast, Damned Thing!
[Edit 2015-08-22: I see there is an existing product for OCR'ing LCDs called, coincidentally, WxOCR. I will have to change the name of the program.]
I really should have blogged, before now, on the wonderful little OCR (WxOCR) program that reads my home weather station's statistics from snapped JPEGs. It was a couple of years in the making, and put my coding skills to the test; but it's been running for at least three years, and running very well.
The Hardware
I have a home weather station, which consists of several outdoor sensors, wirelessly linked to an LCD display pad. I also have a spare ancient laptop and a 1024x768 webcam.
The Mission
To deduce the values of the readings on the display, accurately, cheaply.
It's Dependable
(In essence) what happens is that at five-minute intervals, cron fires off a message telling a shell script to snap a picture of the weather station's LCD display. Having done so, the photo is handed off to WxOCR, which analyzes the green portion of the image (for maximum contrast, and it's configurable--you can use red, green, blue, or all channels). Data are defined as Groups, Characters and Segments; for each character, the expected location of each segment area is scanned and compared to a control value to determine whether it is lit, or extinguished. The LCD character has seven segments (we ignore any decimal point) which can correspond to the bits in a binary series; we add up the seven resulting bits to get a value from 0 to 127. This is compared to a list of permissible values; if it's a match, then you've done one character, and move on to the next. If no match, a null is generated, which immediately halts processing for that number-group.
Later in the processing, the output values are herded into CSV format, along with the Sky Recognizer program's output, and handed off to the weather server.
So far, in about three years' operation, it has yielded probably hundreds of thousands of observations. Most weeks, it is about 99.98% accurate.
It's Highly Configurable
All the configuration is via two files, one of which can be altered at any time and one of which is reserved by the system. Most configuration directives are concerned with the placement of characters and segments. Also, for each character, a control spot (a nearby area guaranteed always to be clear) can be defined by both position and dimension. Characters can be configured horizontally or vertically, and optionally their individual segments can be defined. Character groups can be interpreted is several modes: direct-read (for most things), darkest-segment-wins (useful for displays with a windrose), and (for precipitation) a running-totals mode. A decimal point may be placed at any character position. Math (+-*/) can be performed on the character group at several points in its processing.
It's Error-Correcting
Because all of the values permissible must be defined, error-correction can be handled. Partial characters, or characters with an extra segment in them, can be converted to their proper values. In general, this works exceedingly well.
It's GUI
The program was designed from the outset to be a GUI program (screenshot below). It's got everything I need to be able to position data groups quickly--or even the whole screen. It even has 'second sky condition' buttons, plus a timer value; these are for situations where it's difficult for the system to detect; fog, snow, etc.
It's Linux
I need a rock-solid operating platform; but even the most-advanced Windows version needs to be rebooted regularly. Therefore, WxOCR is built on Debian Linux and has essentially no downtime (it reboots weekly, three minutes early each Sunday morning).
But it ain't Perfect
It's
a minor annoyance, but for whatever reason the character '2' is often
missing the top bar, even though it's clearly in the image.
Technically, this shouldn't be happening. It does anyway. And, during
the day, differences in lighting can throw off the threshold values
established for each character. [Edit 2015-10-25: This is now essentially fixed.]
Nor Ever Will Be
It's almost impossible to get a perfect week's readings. In fact, in all those three years, I've only seen it happen once. This is for various reasons: the digits were themselves in transition as the pic was snapped; the camera shifted (damned cats); the contrast was poor in one particular image; and so on. Plus, the camera can drift in aim and focus over time. I'd frankly like to see a commercial system (if one exists; I can't find one) with a better accuracy than WxOCR.
It's Still Evolving
I can still see more work I'd like to do on the program. I'd like to add a four-way doodad for the individual characters in a group (refer to photo below). I'd like to change slightly how the precipitation sensor works, to cut down on n/a readings from it.
It Needs to Go Open-Source
So far, I've not found another program that does what WxOCR does. I really think this program needs to be open-sourced, and I'll do that in the near future. I think it could really fill a (highly vertical, admittedly) need.
Here it is, in all its glory:
WxOCR v. 2.82
For the life of me, I cannot understand why the following proposal should not work:
Exactly between two points (points A, B) in space time separated by an arbitrary distance (point C), position a photon generator and a beam splitter. Then generate photons, split them into entangled pairs, and send them towards (A) and (B).
An entangled pair of photons is linked; the destiny of the one is the destiny of the other. The effect is instantaneous, no matter how far the particles end up separated in space time.
At each endpoint (A, B), there are just two pieces of equipment: a photon detector, and a shutter positioned slightly ahead of the detector. Each endpoint takes turns sending and receiving.
When it's A's turn to send, sliding the shutter closed will intercept the incoming photon, causing its entangled partner also to extinguish; thus the detector at (B) will detect nothing; call that a 1. If A does not slide the shutter, the photon can continue, then the detector at (B) will register a photon; call that a zero.
When it's B's turn to send, the same situation obtains at (A).
Granted, this method requires a source of photons to be positioned in advance. Depending upon the separation of (A) and (B), it could be years before the stream of entangled photons is available to be manipulated. Also, it requires extreme coordination for the entangled pairs to arrive at both stations simultaneously.
This communication method could be tested using present-day equipment.
-Bill
Upgrade Hell
I was 'forced' into upgrading Ubuntu on the server last night; there was a brief outage between 0800 and 0900.
I grow weary of the Ubuntu frequent-upgrade game. Every six months, upgrade or lose out on support. In view of the Shellshock vulnerability currently out there, that carries new weight.
Generally, when I install Linux, I install Debian. It's got an upgrade cycle of once every two or three years. I made a mistake opting for Ubuntu for the server, one which I'm determined to correct before the next Ubuntu upgrade cycle. Which is due in about a month.
Because things go wrong when you upgrade. Case in point: through two upgrades now, Ubuntu has insisted on installing lxdm window manager, even though I run with kdm. Now, it's true that a simple apt-get install kdm made short work of it; but still.
Stop the Ubuntu, I wanna get off.
Data Glitch
The past few days, readings have been sparse. The OCR program was crashing out, and I didn't have time to see why.
I finally got a look a few moments ago and immediately noticed why: one of the elements of the OCR program is the ability to define 'validation mappings' for each character. In other words, this combination of elements lit up equals character X. I had originally allocated for 12 validation mappings per character; but recently I realized that the OCR program could actually do error-corrections: if a certain pixel is added to or removed from certain characters, a distinctive pattern appears. So why not harness that and make the OCR progra much more fault-tolerant. Well, it turned out that in adding the additional mappings, I overshot the limit of 12. Easy fix; five minutes.
The lesson to be learned here? Always allocate amply for your data needs.
A New Project
I've embarked on a new project, one that actually ties in very nicely with the weather system and keeps me marching in the right direction development-wise.
I'm working on a way to synchronize the hosts file amongst various machines on a network, as well as Google-Chrome bookmarks (and, in future, files to be backed up). There are various kludges to make this happen; I just said screw it and have begun writing myself a couple of programs.
The first is the server. It will respond to XML requests on the server's CGI-bin interface. The server is (at least for now) designed only to run when queried.
The second is the client. It will send XML queries to the CGI-bin interface, and listen to the results.
I've decided upon XML as the way to do this, simply because it's good at encapsulating string information, and just a bit more precise than the typical query of key1=value1&key2=value2&etc.
So, for now I'm working on the skeleton of the server and in particular the XML module.
In the meantime, I'm still also working on my weather system, plus on my Comedy Radio Show.
-Bill
Getting Ready to Open Source
2014-02-10
My weather system is starting to mature - it's at Version 3.10 now, and I frankly can't think of much to add to it, apart from an expanded environmental dataset - Snow-on-Ground, that sort of thing. I recently added queries and a data-extant flag, which tidies things up a bit.
So, I'm getting ready to open-source the whole works. It turns out that's taking some time.
First, I've got to document what I have. It's no easy proposition, but it has to be done. I've been at it for two days, now, and have some 22 pages complete. It's gonna be one of those projects.
Next, I'll have to tidy things up a bit and include a sample dataset. That'll take only a little work.
Thirdly, I've got to document the install procedure. That's a little bit involved but, again, a necessary evil.
All that, not just for the weather system, but for all of the support programs as well.
Once I've got all that together, then I've got to upload the works to sourceforge.
Bleah.
-Bill
A Raspberry Pi Security Camera
2013-11-04
I'd been looking for uses for my raspberry pi, and it finally hit me: a security camera!
Given I already had the pi, and a charger, all else I needed was a webcam. For this, I chose the Nexxtech HD Webcam; it's a decent little unit, with a 1280x720 CMOS sensor and plenty of low-level light pickup, and a relative bargain for $29 at The Source.
I rigged up a case for the pi from an empty cigarette pack; the new 25s are perfect for this. With that, I just had to rig a little shelf, in the window beside the door, on top of which went the webcam, and away I went. Perhaps the most technically challenging moment came in figuring out a way to run power to the downstairs entrance!
Now, you know me; I don't do things the easy way; I do them the efficient way. I didn't want some GUI setup to worry about; everything would have to be done from the command line. With that in mind...
The preliminaries: I had to install a few things that weren't in the default Raspbian setup:
- Apache2
- fswebcam
- ImageMagick (just in case)
- Possibly others
The capture portion was easy enough to set up, with the proviso that the brightness be adjusted for day and night. The image should be in two sizes, and should be saved to /var/www. I put it into a shell script:
nano /home/pi/webcam.sh
#!/bin/bash
time=$(date "+%H%M")
if ((630 < 10#$time && 10#$time < 1700)); then
fswebcam -D 1 -f 2 --set brightness=20% -r 1280x720 --jpeg 95 /var/www/door.jpg --scale 320x180 --jpeg 95 /var/www/door-sm.jpg
else
fswebcam -D 1 -f 2 --set brightness=100% -r 1280x720 --jpeg 95 /var/www/door.jpg --scale 320x180 --jpeg 95 /var/www/door-sm.jpg
fi
chmod +x /home/pi/webcam.sh
I wanted to capture four frames per minute from the camera (the relevant modules were already installed in Raspbian).
The other issue was the with the WiFi dongle setup, the pi was unstable; so I'd throw in a reboot every hour; it takes just seconds. The easiest way to do all this was through cron jobs, as follows:
crontab -e
# m h dom mon dow command
0 * * * * /sbin/reboot
* * * * * /home/pi/webcam.sh
* * * * * sleep 15; /home/pi/webcam.sh
* * * * * sleep 30; /home/pi/webcam.sh
* * * * * sleep 45; /home/pi/webcam.sh
And that, literally, is all I needed to do.
The rig handles day- and nighttime illumination levels very well; attached is a jpeg. I access it locally through the network at http://pi/door.jpg. Works like a charm!
WxOCR v. 2.82
