Another fun project
last update: 03/21 @ 17:59

I really enjoyed using my hands and some tools to help Taylor build a bed platform in her van. She and Nanette are touring the southwest now; expected back around here in about 10 days. It was great to see something physical take shape in the real world. And Taylor sounds like she’s happy with it — it’s always nice to have positive reinforcement.

My latest project highlights how well my pre-retirement jobs fit my interests. I’m very happy to be retired, and recommend it to almost everyone (I do think there is a lot of learning and growing to be gained by the variety of jobs, bosses and organizations you can experience over a few working decades).

There is a subtle update to this blog as a result of that project. I’ve given thought to adding topic tags here for a while. And there is a Blosxom plugin to do that, but I’ve also been interested in the buzz around Rust lately, and thought it would be fun to learn a new language.

So at the bottom of blog posts now is a little “Topics” section. As part of that, I realized the monthly list had grown pretty long (over 1500 posts over 22 years is a fair amount). So on the left are two HTML details sections: one with the monthly list and one with the new topic list. Finally, I was able to replace a JavaScript library that I was only using for help handling the comments. It was a large file with lots of cool functionality, that I was barely using. JavaScript has progressed to more standardized support; so I think my “fetch” replacement for “AJAX” will serve me well.

That JavaScript file was from the early 2000s, as is Blosxom, the blog engine I use here. As a happily retired person, I appreciate the wisdom and experience which comes with age and don’t want to discount age just because there are newer, younger things out there. But another reason I wanted to use Rust for my tagging project is that Perl is slowly being diminished. When I started working on the integration between Rust and Perl, I realized my workstation didn’t have some of the needed Perl modules and getting them was not going to be super quick and easy. I just pushed along with “development in production” — something I would have had to frown on in my prior, tech managerial jobs.

I am giving a little bit of thought to rewriting Blosxom in Rust so I don’t run into an update to our servers that makes it hard to get the required Perl modules installed. I do have to say that Blosxom is very flexible and does all I need it to do (if I do rewrite it, it won’t be that cool and flexible). And I was really pleased to be able to graft in my updates and still benefit from the core reverse chronological ordering of posts. Also, with a little manipulation of the URL, topic tags restricted to dates work as well (just as they do by default): Alexandria vs. Alexandria in March, 2026. It was cool to me that both those just worked as expected.

I won’t build a full Rust-based web server if I go down that path. I’m happy with the wisdom and experience that comes with GGI, which has been a part of the web since almost it’s very first days. There are more modern ways to build out interactivity, but I know I don’t need that level of sophistication or performance for my little world. I happened upon this post recently about performance of Rust and CGI.

If you’re still reading this, I will also say that I was in no rush and wanted to learn a bit about this new language. So, I chose not to use AI for this at all. I expect AI would have been happy to help me out, would have been much quicker and likely would have produced more solid code. But I would not have learned as much about what makes Rust work and how it differs from other languages. This NYT article about AI assisted programming was an interesting read as I worked on this.

And this fun project has been a throwback, in some ways, to the very first work I did on the web for AARP’s website. In those early days, the AARP IT leadership was all about supported software. Our DEC servers came with a C compiler from DEC so I was told I had to use C for our web interactivity even though Perl was around then (because open source Perl was not commercially supported). Perl, being a higher level language was much safer for public interaction than homegrown C code, and DEC was certainly not going to support my crazy programs. (Eventually, we convinced them that Perl was a better fit.)

As a reward for anyone reading this far: my tagging code.

I expect I’ll slowly go back over some of those 1500+ entries and add some appropriate tags. But for now, back to regular blogging about life rather than reflecting on blogging itself :)

Merry Christmas and Happy New Year!
last update: 01/01 @ 11:47

The Christmas present that cried wolf:

Panzer did his job keeping all mice still. And we had a lovely Christmas at home.

However, the flu hit the next day and we’ve been slowly recovering for the last week. Fortunately, with all of us vaccinated, the flu has been only bad, not the super terrible that is making the news.

But back to crying wolf: I saw a news story which convinced me it really would be good to have a NOAA weather radio and so both Sarah and Robert got these Midland radios under the tree.

We might have taken heed when the alarms went off and the lights started to strobe the morning after Christmas. It seemed to be a test of the emergency broadcast system.

But we did not take heed, we took that as a sign that the emergency function works well.

However, we did not need to be alerted to the snow squalls in the foothills to the west of us at 3:00 this morning (as we were tucked into our cozy beds and getting the sleep needed to recover from the flu).

Nor did we need to know about the special marine warning at 4:15am.

So now, the radio is quiet, with the battery (which seems pretty beefy) removed. And we’ll have to think about when we want to be more aware of weather emergencies and when we think sleep is preferable to knowing what might be happening outside and unlikely to impact us.

It is reassuring to know the emergency function works, and it’s plenty loud and flashy to wake us up.

Personal Tech History
last update: 07/31 @ 13:33

25-ish years ago, I read someone I believed to not be crazy (Tim Bray, or Cory Doctorow, I think) extol the virtues of “owning” a piece of the Internet. Having returned from Costa Rica not too many years before then, I liked “kayakero” (some of my raft company coworkers had taken to calling me kayakero mystico due to my squirt boat safety boating — thank you Ken for hauling the boats, er surfboards, down). And I hoped to homestead on the Internet as kayakero.org. But that was not available at the time.

As Register.com merged with (was subsumed by) Network Solutions I went to make sure I still had access to the configuration items I need. And NetSol happily pointed out that kayakero.org was now available. So I picked that up.

I don’t expect I’ll shift things over and I’m not sure what I’ll do with that domain. But a small part of me feels like my patience paid off. Of course when the renewal comes due, at higher than the initial offer price, I may let that domain go back into the pool for someone else to pick up.

And those 25-ish years have blunted my prior interactions with NetSol — I vaguely remember not wanting to use them as a registrar to start with. But I don’t recall specifically why anymore. Optimistically speaking, I won’t be reminded now that I’m their customer.

Epic Mother’s Day Gift
last update: 05/11 @ 10:56

For Mother’s Day this year, Robert mapped all the previous Mother’s Days with photos of the two of them.

So, of course, I need to add a photo of the two of them looking at that gift for this year’s Mother’s Day.

Sarah was very moved — it really was a great and unexpected gift (and proof he’s in the right program at UW — and maybe something I should look at for use here on the blog…).

We’ve been Tinged!
last update: 04/02 @ 09:50

And I think it’s a good thing.

(Sorry, another geeky post.)

Fiber Internet finally became an option for us this year; Verizon was never interested in Alexandria for some reason (I prefer to think the city was unwilling to give them too sweet of a deal).

My hope is: because they are not competing with themselves for cable TV viewers (as Comcast is), they will be more reliable and more transparent. I was surprised when we moved from Verizon (DSL Internet) to Comcast (cable Internet) several years ago that I had to change the way we get DNS resolution.

About 18 months later, during the Covid pandemic work from-home-days, I got a bit more insight into what seemed to be going on. And I was very unhappy about it. That experience broke any trust I may have had in Comcast and led me to start looking at options. One recommendation from the upshot of Comcast blocking access to VPNs (including those used by federal employees working from home) was to turn off the “Security Edge” service. In my case, that meant a new service bundle, higher priced, and (theoretically) less featureful. But DNS resolution still required forwarding our queries upstream, which only made me more unhappy. I am very happy to welcome Ting to the neighborhood and trust they will not interfere with our use of the Internet as Comcast did.

I’ve removed the forwarding clauses from our DNS configuration (mentioned in that earlier post) and not seeing any problems. Ting is not consistently 1 GB symmetrical — it is, however, consistently faster than any prior connection we’ve had and way more bandwidth than we could ever need (and less expensive than Comcast).

Skipping down memory lane
last update: 03/10 @ 14:07

Sarah and I were looking over some old photos and she pointed out that she had this copy of Linux Journal featuring a photo of Robert (with less hair) and me (with more hair).

That giant data set from the cover article: 200GB — still respectable but not what we call big data these days.

I miss the magazine and the metro ride to read it on.

RTFM
last update: 02/28 @ 18:29

Well, that’s likely something I should have done but didn’t in the last few weeks.

As noted earlier, I had a power supply fail and the server was old enough it was not clear I could replace it. Also, I’d had a hard drive fail on the same server a few months back. Both have been through a remodel at the house and so it was time to replace them.

For the first server, I was really under the gun: I wanted to get our internal network back up and running quickly. Clearly no time for manual reading.

For the second server, just recently “Completed,” I was not under quite as much personal pressure. But I still just installed the OS (yes, two major versions more recent than what I had), installed the software and laid down my configurations. It likely would have been a good idea to read the release notes to understand what all had changed, but I was too lazy or thought I could figure it out as I went.

Which, I’m happy to say, I could (thank you Internet).

If you happen to be jumping from RHEL 7 (or, more likely CentOS 7) to RHEL 9 (or, more likely Rocky 9), here are the final couple gotchas I ran into:

External FirewallD forwarding/routing change

With the changes to FirewallD and the underlying firewall layer, my old forwarding didn’t work. I saw a lot of this in the logs, “filter_FWD_internal_REJECT: IN=eno1 OUT=eno2” The fix, thanks to Internet searches:
firewall-cmd —permanent —new-policy allowForward
firewall-cmd —permanent —policy allowForward —set-target ACCEPT
firewall-cmd —permanent —policy allowForward —add-ingress-zone internal
firewall-cmd —permanent —policy allowForward —add-egress-zone public
firewall-cmd —reload

Some updates needed for ruby scripts

The new, modern Ruby doesn’t like instance variables (for good reasons). The quick (but perhaps less than ideal) fix was to make them global variables. My scripts run one at a time, no worry about race conditions so likely that’s OK.

Testing red network with only default route

I thought I was being smart, I added my public IP addresses to the new server while I was building it and then went to test by plugging it into a switch that only had my laptop connected. I gave my laptop an IP in the same class C network and started testing. And nothing worked. I had the server’s default route set up to go out the internal side so I could use the current Internet connection to get updated software. Once I added a route to that class C on the NIC, I could test and see traffic flowing.

My new servers have redundent power supplies now — I learned that lesson. But maybe I didn’t learn the hard disk lesson: I’m not doing any sort of RAID, just frequent backups. And, thanks to the new servers and new disks, I have my old disks as spares should I need them.

Now I can say I’m finishing up some post migration work; prepping for Ting as our new ISP, and getting on with life.

Recovering
last update: 01/28 @ 23:19

Win some and lose some…

Refurbished power supply was a bust; wrong item (it was hot swapable, I didn’t have redundent, hot swapable — but I do now).

So I’ve moved on to server rebuild. And with Centos pretty much gone, seemingly a causality of IBM’s purchase of Red Hat, I’m moving on to Rocky Linux. Going from 7.X to 9.X was bound to have some growing pains (having them in a more controlled build and cutover rather than replacing a dead server would have been nicer).

Lessons from my experience:

ssh

First, Rocky uses AuthorizedKeys vs. Centos 7 using AuthorizedKeys2; not a big problem but my first, bad, assumption was the upgrade meant I needed newer keys. And then (did I mention I was trying to go fast), I just used default file permissions on the new file. Final stumbling block: the new key pushed me over the limis so I started to get “too many failed attempts” errors. Lesson: copy AuthorizedKeys2 to AuthorizedKeys and check the file permissions.

samba

It’s been a long time since I set it up. First, bad assumption: samba users are pulled from OS users. Eventually the errors made sense and I remembered I had to create the samba users/passwords. Second issue: selinux boolean to allow samba to get to home directories — that generates a really weird error on Linux workstation mapping in those drives (not sure what Windows would have looked like). Lessons: create the samba users/passwords and ensure selinux is permitting home directories (or relabel the file system).

httpd

Really went pretty well: don’t forget to install CRS with mod_security. And the new rulesets added a few more false positives for subversion access.

subversion

Speaking of subversion; it just works. I was pleasantly surprised to see no issues with access the repositories via httpd (after the mod_sec updates). (Do be sure selinux context is correct.)

postfix and spamassassin

It took a lot longer than it should have to understand that the error message about pipe failed due to unknown user really meant just that. The Spamassassin package for CentOS 7 used the “spamfilter” user (I think); it seems like the Rocky 9 package piggybacks off of the “mail” user.

dovecot

I had to refresh my memory on SSL keys for dovecot; but mistakenly figured I could just remove the Thunderbird saved key and add back the new key (see below). For me, dovecot uses standard key files in standard tls directory. I was surprised that dovecot seems to log Thunderbird’s ssl error which Thunderbird seemed to just mask.

Thunderbird

When I hit the certificate issue, I thought (next bad assumption) I could just remove the key I loaded from the old server and add a new exception for the new server’s key. I spent a long time trying but could never coach Thunderbird to ask for that offending key and give me a chance to accept it.

Thunderbird

So I remembered having to create a new profile not that long ago when the hard drive failed on the old server. (The power supply was the camel back breaking straw.) So I created a new profile and that let me import the certificate. But…

Thunderbird

The Thunderbird new profile/account set up presumed username would include domain (davewill@kayakero.net, for example). Of course, dovecot on Linux, using Linux accounts just needed the name. The unknown user / invalid password threw me for my final loop of the rebuild session. Lessons: Delete imported SSL cert only as a last resort and look closely at the assumptions made in setting up a new profile/account.

tar

Final note this this entry: when I started my backup scripts, I was getting astronomically large backup files. tar, in Rocky 9 does not like “—exlude” clause after the source of the files to tar up. CentOS 7 was OK with that. Lesson: tar .

The server has been working well for a few days now and I’m feeling pretty good. I have a bunch more disk space to work with (but I’ll likely need new, bigger USB drives for offsite storage). And I’ve seen a whole new slew of error messages and eventually worked out what they mean and what I need to do.

Of course, the public server (serving this blog and website) is the same era as the internal server and lived through the same very dusty remodel. So that’s next on my plate…

Oh, and Ting is setting up fiber to the house too…

Ack!!!
last update: 01/13 @ 13:12

I may be a bit distracted for a while.

I can now say I’ve experienced a power supply failure for a server — new to me before today. I knew these needed to be replaced, I was just hoping for 6 more months…

New (refurbished) power supply on order — if that’s really the cause (could be motherboard as well), I may be up and limping in a week or so.

New servers ordered as well — clearly it was time, but I was hoping not to be forced into a rush job :( And, learning my lesson, redundant power this time; that seemed excessive for home use before today).

Oh, and Ting says they may be able to hook us up next month.

Gonna be a techy few weeks here…

Flumoxed … Resolved
last update: 01/10 @ 20:42

Feel free to skip this, it’s going to be geeky…

I’ve been running Bind as a DNS server for years now; both for our internal network and as authoritative for this website. It’s been stable and working fine for many iterations of DSL and now cable ISP service. It’s possible, though I don’t honestly remember, that I was running (an older version of Bind) way back in the Ricochet wireless modem days.

And I’ve always been a control freak so I’ve had it configured as the resolver for our internal network. If the query is not for something on our network, it goes to the root servers and tracks down the correct IP address.

So I was flumoxed when my Comcast service was upgraded and suddenly my internal DNS servers could no longer get resolution for external systems. The upgrade was meant to be simple: new modem, faster service. “This year’s deal.” And it worked fine for everything except DNS resolution — which means it effectively worked fine for public access of this site, but was significantly broken for our use here going out to the Internet.

My flumoxation was of the compound variety: a simple upgrade, which clearly didn’t break connectivity, did break that one key function. And rather than querying our internal Bind servers, if we queried a public DNS server, it worked fine.

But, being a control freak, our internal network is a different domain from the public network (but registered and paid for). And that domain is not published. So relying on public DNS would leave us blind to our intranet.

I had iftop running and was tailing logs and doing tcpdumps for wireshark and nothing was standing out to me as the problem. I just got status: SERVFAIL for external addresses and everything internal was fine.

As I was scratching my head and looking for better Bind debug options, I ran across the documentation for forwarding. Since I could query external DNS on the command line fine, I thought it was worth a shot (though not my preferred option).

That worked. I’m sending more traffic to (and I’m more reliant on) Comcast and Google’s DNS servers. But it’s working.

I would welcome explanations or theories, but now I’m just going to bed. I expect it’s something in the format of an “internal” query from Bind (following the path from the root name servers to the authoritative name servers) versus just asking someone else to do that leg work.

But first a shout out to Bill at Comcast. He was not able to point me to a fix, but he tried very hard and was as helpful as he could be. (And I know my situation is not very common.)

Agile Software Development
last update: 08/09 @ 09:58

Sorry a bit off the normal family topic…

My summation of Agile Software development. I don’t see any results when I search for that on the web so I’m going to claim ownership of it.

And I’ll quote myself again from a short presentation I’m putting together for work:

“It’s always hard to find the sweet spot between over planning yourself into a box and flying by the seat of your pants into a cliff. We try to make sure we know where we’re generally going and what next one or two steps are most likely to get us closer.”

I think it applies more generally than just software development: when remodeling, you don’t need to choose the colors before the drywall is up.

Back to normal family travel and milestones…

New Geek Fun…
last update: 01/26 @ 15:20

I decided I needed to play with AJAX a bit after seeing all the fun other folks are having with that on their web sites. Generally speaking, I’m not a big fan of eye-candy for the sake of eye-candy.

But this seems like a good use of AJAX to me: having the comments and comment form for each entry in this page seems redundant and wasteful of page real estate (and perhaps of bandwidth, though that’s less of an issue as all the regular visitors are now on broadband connections). However, loading a page just to see the comments and form seems less than ideal.

So, now the comments will (should) reveal in-line on this page. I’m not completely sold on this, so let me know if you find it difficult or confusing; it’s easy to change back.

And for anyone interested, it was also pretty easy to set up: I’m using Scriptaculous (and Prototype) as the AJAX libraries. Actually, Scriptaculous is providing the reveal effect, Prototype is the real AJAX library doing the work. So I put those files in place and referenced them from the main blog page template. Then I Wrote a couple javascript functions to use the libraries to get and display my contents. Then I just needed to create a new “flavour” (that’s what Blosxom calls styles of content or sets of templates) to display the comments and form as a snippet of HTML.

The couple of “tricks” I tracked down to make it work reasonably well: set display to none for the container (not the CSS for the container) that holds the AJAX content (the comments in my case) and let Scriptaculous’s effect make it visible. And tell the web server (Apache, of course in my case) to send some no cache headers on the pages and page snippets involved in the AJAX content — otherwise they will be cached and not reflect updates as they should.

Of course, when I hear from folks that this really isn’t working, I’ll try to get it working better and update the tricks here. Perhaps for extra credit I’ll replace the comment submission button with an AJAX post to keep everything in this page…

The only downside I see to this approach is that I don’t expect it plays well for screen readers or for search engines. We’ll see if I have any time to make the comments more accessible for those two specialized types of visitors.

Add new entry (owner only)