Fresh Install of Ubuntu Server 12.10 on a Working System from an ISO Image and Non-Bootable USB Drive
2012-12-23

See the last section of this post for a concise summary

New (less old) Server
Yesterday, I replaced my home/weather server, a venerable Toshiba Tecra 8000, circa 1998, with something a little more recent--my old Acer laptop, circa 2005.

I've been using Ubuntu Server for some years, now, and like it a lot more than their desktop flavours.  The server was running version 11.10; quite frankly, I was averse to updating the poor old thing with such cramped disk space (10 Gigs). Switching to the newer machine gives me lots of free space for future upgrades, and it should remain capable of running mainstream Linux distros for quite a few years to come.

Installation was simple.  I'd planned carefully and kept a list of files that needed to be modified on both old and new machines to make the changeover.  The weather system I transferred in stages, first the support scripts and programs, and then the main daemon itself.  A few hours of work, and the new server was up and running, and the old one was back up as just-a-normal-client.  The new server boots to the XFCE desktop and now hosts the webcam analyzer, a GUI program.  It's working surprisingly well and laughs at the puny load the weather system puts on it; the old server wheezed and chugged to keep up.

But It Ain't Dead Yet
Now, time to repurpose the old server; I'm going to replace the thin client, which currently captures the local readings from an LCD display.  It runs Arch Linux (by necessity, due to a CPU peculiarity) off a thumb drive, takes about ten minutes to boot, and has a big old, space-eating CRT monitor attached.  The old server--even though its hardware is several years older than that thin client's--can handle the task of capturing the readings and transmitting them to the server--and nothing else, while, being a laptop, saving a load of space.

The plan was to install Ubuntu Server in a very minimal configuration, then XFCE and the readings-capture software.  I wanted to do a fresh install, rather than an upgrade, to get rid of all of the server stuff it wouldn't actually need anymore.  And, I just like to start fresh when repurposing a system.  Additionally, U'Server uses a console-based install.  I wasn't up to trusting a 14-year-old laptop with a GUI-based install; let it sort out the basics, first.

So, now it was merely a matter of installation method.  I evaluated my limited resources.

And here the issues began.

Can't Boot from CD or USB
I knew already that I couldn't boot from a thumb drive; the damned BIOS is just too old, though it does have a USB port.  No, I wasn't willing to risk updating the BIOS; I bought that laptop second-hand a good ten years ago.

Okay, so I'll burn a CD and install from that; that's how I'd done the last fresh install on that machine.  I slapped one in, booted, and nothing.  I could hear the disc scraping inside the drive. Apparently Teeka the Human-Like Sphynx kitty had perched on the keyboard too many times and... harmed the CD-ROM drive.  I didn't feel like futzing with it.  (I did, incidentally, install an anti-cat-perching device on the laptop after I catching her doing it.  She wasn't trying to be naughty, either; sphynxes are hairless and love warm places.  QED.)

So, no CD, and no thumb drive.

...or Network... or not...
Next alternative:  network install.  Nope; Ethernet access is via a PCMCIA card, which requires drivers which are not, of course, available at boot time.  So that's out.

There are ways of installing Linux from files already present on the hard drive.  I decided to try that route.

Issue:  the previous installation had carved off a separate /home partition; but in order to run a GUI, I needed all the space consolidated into one partition.

Issue:  Can't repartition the drive while running Linux which resides on that drive--and, again, no way to boot, say, System Rescue CD or Partition Magic from USB or the network.

I partly addressed the partitioning issue by moving /home off its separate partition and onto the root partition.  That freed up its partition, which I unmounted, deleted and replaced with a smaller (700 MB) partition (sda3 - remember that for later).  Onto that I copied the Ubuntu Server ISO.

Leverage the Existing Boot Manager
The old server  boots with Grub2, and the boot manager is the key to booting unconventionally.  Following some examples on the Internet (none of which worked straight-up for me), I added (as Superuser) the following to /etc/grub.d/40_custom:

(This didn't actually work for me)

menuentry "ubuntu-12.10-server-i386" {
    set iso="/ubuntu-12.10-server-i386.iso"
    loopback loop (hd0,3)$iso
    linux (loop)/install/vmlinuz file=$iso/preseed/ubuntu-server.seed ro quiet --
    initrd (loop)/install/initrd.gz
}

What we're doing is setting a memory variable (iso) to point to the ISO file on the partition.  Next, we're mounting that ISO - (hd0,3) above is Grub's way of referring to /dev/sda3; the partition where my ISO file resided for the installation.  Then we're booting the Linux kernel and related files from inside the ISO.

Backtracking slightly:  I verified that the files referenced in "linux (loop)..." and "initrd (loop)..." above are correct.  To do this, I mounted the ISO's partition, then mounted the ISO file onto the loopback device, and had a look around:

mkdir /media/hst
mount /dev/sda3 /media/hst
mkdir /media/iso
mount -o loop,ro -t iso9660 /media/hst/ubuntu-12.10-server-i386.iso /media/iso
cd /media/iso
ls (etc., etc...)

That said, I saved the Grub file previously mentioned, then ran update-grub to get the entry added into Grub's boot-time menu.

Next, I rebooted.

False Start:  What CD-ROM?
Up flashed the Grub boot-time menu.  I arrowed down to the new entry, then hit 'e' to view/edit the commands.

Issue:  seems grub-update had removed the (hd0,3) bit from the code.  I manually added it back in, took a deep breath and hit Ctrl+X to go ahead and boot with those commands.

It trundled away for a long moment, then up came the Ubuntu installer, to ask me some questions.  Hooray!

Issue:  after a number of questions, it reported that it couldn't mount the installation 'CD-ROM'.  Okay, wait--I'd seen something online about that.  I hit Alt+F2 for a command shell (console 2) and entered the following commands to mount the ISO file (which the installer, running from memory, had unmounted) to the /cdrom mountpoint:

mkdir /media/iso
mount /dev/sda3 /media/iso
mount -o loop,ro -t iso9660 /media/iso/ubuntu-12.10-server-i386.iso /cdrom

(This is very similar to how I'd temporarily mounted it earlier, to look at its contents.)  Ultimately, I didn't need to do this part (see below); but take note of it in case the entire installation must come only from the hard drive.)

Alt+F1 took me back to the installer (console 1).  Ha, I thought, yes, go ahead and retry mounting the CD-ROM.

What CD-ROM? 
Issue:  same error.

Okay.  I flipped back to the command shell and remounted the ISO (the third command line, above), then flipped back to the installer.

This time, I said no, don't try to mount the CD-ROM.  That took me out to the main installation menu (detect and mount CD-ROM, something like that).  Just for fun, I chose the next entry (something about debconf).  Lo and behold, the installation process carried on!

Issue:  A few screens in, I'm informed that I cannot re-partition the drive when a partition is already mounted.  Yes, /dev/sda3 was mounted, holding the ISO file.

Dammit!

Boot from ISO, Install from USB
I consulted with my coffee for a long moment, trying to think this through.  Possibly I could instead boot from the System Rescue CD ISO, and try re-partitioning from that.  But, even then, I'd have to leave that 700 MB partition holding the ISO file.

Huh.  If only I could use that damned thumb drive.

Something was trying to catch my attention.  Quite literally--the blinking lights of that plug-in Ethernet adapter.

The PCMCIA bus was initialized and working.  So would the USB port!

I aborted the installation, shut down, wrote the ISO to the thumb drive and plugged it in, and rebooted, following the steps above.

Problem:  installer starts up but then can't find the preseed file.  Clue:  it's looking for it on /cdrom.  As it turns out, that's where it's mounted the thumb drive.  so it's looking for it after the installer unmounts the original ISO image, and after it's mounted the thumb drive.  Okay.

Reboot back to the existing OS and tweak /etc/grub.d/40_custom, replacing $iso/preseed... with /cdrom/preseed... .  Then one last round of update-grub.

I rebooted and again selected the installer entry in Grub.  The installer fired up, asked the usual questions and then carried right on. Now, the installer was running in memory, and the installation files were coming from the thumb drive.  The hard drive wasn't being used at all.

Moments later, the partitioner popped up with nary a complaint.  I set up a small swap partition, then created one single partition with the remaining free space.  Nine-point-five Gigabytes--that'll be enough for the GUI and what few bits of software I'll have to add.

It chugged along merrily, finished the installation, and rebooted.  I'm now configuring and installing packages.  Hot damn!

Knowing what I do now, if I need to do a fresh install in future, I can simply download the ISO to that main partition (remembering that the installer unmounts it after starting up, leaving the drive free for reformatting) and 'burn' it onto a thumb drive, and repeat the above steps.

Concise Procedure:

• This works if you have at least one available USB port.
• These instructions may be specific to replacing Ubuntu Server with Ubuntu Server 12,10
• Grub configuration is specific to Grub2
• Follow the instructions as Superuser

• Download the installation ISO  to a partition on the hard drive.
• Use one of the USB installer-type programs (UNetBootIn, etc.) to 'burn' the ISO to a thumb drive.
• Add the following to /etc/grub.d/40_custom (edit (hd0,3) and anything else as needed):

menuentry "ubuntu-12.10-server-i386" {
    set iso="/ubuntu-12.10-server-i386.iso"
    loopback loop (hd0,3)$iso
    linux (loop)/install/vmlinuz file=/cdrom/preseed/ubuntu-server.seed ro quiet --
    initrd (loop)/install/initrd.gz
}

• run the command:

update-grub

• Plug in the thumb drive so it's available later.
• Reboot.
• At the Grub menu (you may have to be quick about it and/or additionally pre-configure Grub2 even to show a menu at boot time), highlight the new entry for the ISO and press 'e'.
• Verify that the commands are as you need them to be (watch the device references), and/or edit as needed (per "menuentry" in box above)..
• Press Ctrl+X to boot with that configuration.
• Installation should proceed uneventfully from there.You may partition the hard drive, as it's not being used.
• Reboot and enjoy that new-install smell and rich, Corinthian Leather crontab.

If you don't have a USB port or otherwise can't use a USB drive at any point in the process... meh; it gets a whole lot messier.

Some thoughts:

• During installation, do not reformat the partition containing the ISO file!
• The ISO can't be on the same partition as an existing installation, because the existing must be reformatted during the install process.
• If you have a small partition available, place the ISO file there and install per the above.  When the 'CD-ROM' mounting problem occurs, you'll need to mount the ISO as described earlier, and continue without re-trying.
• After installation, you can delete the ISO from that small partition and use it for something else.
• You could also connect the drive to another system and consolidate the partitions from there.
• You might also do this by ISO-booting to a specialized distribution that runs from memory (which lets you unmount the hard drive and adjust its partitions).
• Or...
• With the hard drive connected to another computer (or, again, by ISO-booting to a run-in-memory distro), you could repartition/reformat it, install Grub/2 and edit its config as needed, toss the ISO into the consolidated partition, then boot in the intended computer and install from the ISO
• Or...
• Boot from ISO file, continue installation from network (presuming network drivers had been loaded by that point).  I don't even know if you can mount an ISO over the network from a limited installation environment.

There are other options which I haven't thought of, or run into in my Extended Knowledge Base (i.e. the Internet).  I leave their discovery to you.

Sometimes, the Bug's in the Data
2012-12-19

I have a strict ritual, first thing every morning.  After making a coffee, I plunk down at the computer, visit Environment Harper Canada, and note the previous day's weather stats into a spreadsheet, for safekeeping.  Then I check the news and get on with my day.  I've been doing that for years, originally for comparative figures while hand-tracking local readings.  Since automating the setup, I've found the data handy in checking on my weather system's accuracy.  And, in the past year or so, it's been invaluable, as EnviroCan has actually lost data from their newfangled setup and may never recover all of it.  If they need help, I have a copy of their data...

Fast-forward to this morning.  I fire up my web browser.  It defaults to a portal page of my own design, complete with summary weather information.  The page is updated and timestamped by the weather system, every five minutes.

This morning, the time read 23:59.  No updates since midnight.

Uh oh.  I log into the server and check on things.  Yes, the weather program's running.  I send it a stop command.  Nothing happens; it's hung.  I kill the process.  A quick check shows nothing overtly wrong with the data folders.  Okay, start it up again, and turn on some debug logging.  It starts up, but disk activity stops after initialization.  I check the input folders for each of the locations tracked by the system; they're brimming with data files from overnight.  So... the system's starting up okay, but it must be choking on input data.  The debug log quickly confirms that it's choking on a particular input file; the second one it tries to process.

I open the file in my text editor.  Nothing out of place here; it even includes the local-sky condition inserted by my recently-mentioned webcam analyzer.  I look at the first file, which had been successfully processed.  The only difference here is that the sky condition hasn't been inserted.  This is expected behaviour when the weather system stops taking inputs.

Okay; could there be something wrong with the sky-condition text?  I take another look at the problem file.

The sky condition includes a timestamp.  I had added that a couple of days ago, as a check to make sure the analyzer was continuing to function.

A little bell went off in the back of my head.  The weather system keeps an indexed list of unique sky conditions it's encountered; these are pointed to in the data records, saving significant storage space.

There's a maximum of 255 entries.  Each timestamped sky condition will be treated as unique.

I fire up the text editor again and take a look at the list for the 'home' location (where the problem is occurring).  Sure enough, it's packed with 254 entries, the last 249 of which are timestamped.

I do a little cleanup on the file and quickly scrub the timestamps from the remaining input files.  It's only the home location that receives input from the webcam analyzer.

Next, I figure up Lazarus and pull up the webcam-analyzer project.  A tiny tweak to the source code has it once again outputting just 'Cloudy' instead of '2012-12-19 @ 09:37: Cloudy'.

I fire up the weather system, and it chugs away for a surprisingly few seconds.  Moments later, everything's up-to-date, and the system's ticking along.

Gremlin smacked; job done.  Back to earning some money.

You Don't Have to Be a Scientist...

2012-12-14

A recurring hobby in my life has been amateur meteorology.  I was recording daily temperatures and precipitation back in elementary school, preparing colour-coded charts and trying (largely in vain) to issue forecasts of my own.  I picked up on the hobby again shortly after my bout with cancer, eleven years ago.  First it was a simple recording thermometer, from which I dutifully took daily readings and reset the little magnetic floats; then a wireless digital model which I could read from indoors (a boon during the winter); more recently a full-blown, computerized amateur station (that died); and, finally, with the current wacky setup, which has been running for about eight months.

Those who know me well know that I have a passion for improvising and squeezing performance out of obsolescence.  I hate throwing out a computer unless it's simply too old to do anything useful.  My home weather station involves an ancient thin client with just enough horsepower to snap images of my weather station's LCD display and OCR it for the current readings; and a decade-old laptop running Linux in text mode, as the server.  And it works!

I wish I could have a better weather setup.  The reality is, my location is hemmed in with trees and other buildings, and wind and precipitation data wouldn't be accurate enough to be useful.  Be that as it may, I'm always looking to add new functionality.

Recently, I've been teaching my webcam to interpret the sky and report on it.  It captures a sky image every five minutes and archives one image per hour, for possible future use.  This week, I found that use.

My initial inspiration was to generate a poster, showing a running representation of the weather through the year.  I took each image, averaged all of the pixels together for one representative value, then generated a block pixel in the poster.  The days run in vertical stripes, top-to-bottom, left-to-right.  The result was interesting--and you'll be surprised how much information can be gleaned from it.

Here's how it looks (January at left; Midnight at top)

One of the first things you'll notice is the varying length of the day, giving the lens-shaped daylight map.  Neat, eh?  It's reminiscent of one of those composite images of the galactic centre.

You can also see the effects of the spring and fall time changes, as daylight abruptly shifts one row up or down.  (The graph was prepared using civil time; i.e. EST in winter, DST in summer.)

Notice, too, how much darker the images are from mid-year through early November.  This one puzzled me for a couple of minutes, until I realized it was foliage.  Leaves.  Lots of trees at my location, which makes the Centretown Observatory an ironic joke.

There's more.  Note how, especially in the summer months, it appears to be clear much more consistently in the morning, and much less often in the afternoon and early evening.  This could be meteorological, or it could be environmental, as in backlit trees further darkening the afternoon images.

On cloudy nights during the winter months, especially with bright snow on the ground, the night sky lights up much more.

Looking at that, I thought, I wonder if I could just use the parts of the image which are least obscured by the trees, to get a truer representation of the sky itself.  It didn't take much to produce that.  Here's a comparison of that sweet spot versus the whole frame:

Okay--that's a definite improvement; it's much easier to discern a clear sky during the summer months.

I further took the sweet spot and broke it into its red-green-blue components:

These plots just serve to confirm that the best way to read the state of the sky is to examine the ratio of Red+Green over Blue.  As you can see, there's about as much blue light on a cloudy day as on a clear one.  This rule-of-thumb also works at night:  if there's light in the sky, and it's reddish, then it's cloudy; else it's likely twilight.

Taking all this together, I put together a simple set of rules and wrote the code.  To test and calibrate it, I had it colour-code its interpretation of each image into a test poster.

Not bad at all.  It's about 90% right in recognizing Cloudy (white = day, grey = night), Clear (dark blue(), and Twilight (pale blue).  Compare with the first image.

With a bit more calibration, and factoring in sunrise/sunset times ("sunny" vs "clear") and recent history (last several images), I should be able to boost the daytime accuracy to about 95%.  At some point, I'll also add zone comparisons (i.e. some blue and some white = "scattered clouds").  For nighttime, I don't know.  I'll have to analyze the numbers in greater detail, to see if I can pull sufficient intel from a very dark camera frame.  I notice, too, that the images are generally dark in the afternoons, when the trees are backlit; I could colour-correct for that.

And so, as of today, my weather system is reporting local sky conditions.  And, for now, it's good enough.