The Tinker Trials

The Computer: Tinker

Tinker, the 486 Toshiba Satellite T2130CS.
(Click for larger view.)

Specs

This is a T2130CS Toshiba Satellite with a 486DX 75Mhz CPU, 8MB of RAM, and a 500MB hard drive.

Full details (which I found here): T2130.txt

History

This was my girlfriend's old computer, once named "the Toshibasaurus". It was new in 1995 and still had Windows95 on it when I received it. It also had an external CD drive (including sound card) that plugged in using a PC card. This CD drive has since died.

I've always wanted to find a use for this machine. Linux always advertises that it will let you resurrect old machines. In 2004, I managed to get Slackware 10.1 installed on it. This was no mean feat, as it had no ethernet connection at the time and I couldn't boot from a PCMCIA CD drive, so I was transferring data from another machine using floppies. As I recall, even the lowmem boot disk wouldn't work with only 8MB of RAM, and for some reason I couldn't turn on swap before the install crashed. (Something like that anyway; this was 4 years ago, so the details are fuzzy.) I ended up copying the boot disk to its own partition on the hard drive and booting it that way.

I eventually got Slack installed and running (command line only; no X windows), proving it could be done. But I never got the network connection configured correctly, and I had no real need for the thing... so it ended up back in storage again.

The Goal

However, lately I have need for a home web server to host an automated grader (not yet written) for my TA work. Tinker seems a good candidate for the job.

Requirements

OS: Debian Linux
I want some form of Linux, for the practice of it and to support the other current software I'll need to run on this low tech machine. I didn't really "get" Slackware last time. (I don't really get the Church of the Sub-Genius either--which is where the notion of Slack comes from. I'd like to like it, but... it just doesn't do it for me.) There are some small distros around--Puppy and Damn Small Linux--but I don't really need X or a working desktop--just a server. So I've decided to go with Debian.
Java: 6.0
For the grader to work, I need the latest version of Java. This will be the toughest requirement to satisfy, and it doesn't seem likely that 8MB will be enough to run it. (But here goes nothing!)
Python: Semi-recent
I'm just learning Python, but I plan to write the grader AI in it. So I'll need some version of that, though it probably doesn't need to be the most recent.
Web server: Small, stable, with CGI
I need a decent web server that can handle CGI requests. For the practice, I'll probably just go with Apache.

First Attempt

Debian

Debian's site can be a bit tricky to navigate at times, but I found all the old stable releases. I read through the various install manuals and decided to go with 2.2 (potato), last updated in 2002. The install guide says 12MB of RAM are needed.

Once I tracked down a copy of potato (see http://archive.debian.org/), I did a network install. This required 3 floppies and a network connection. I have another old laptop (a Pentium III) with a 3.5" drive, and a few months ago I found an abandoned 10/100 ethernet PC card.

The only real problem with the install was configuring apt using the installer. It was assuming certain things about the directory structure of the package server which weren't true of archive.debian.org. I had to boot the new system and edit the apt config file manually, but things went smoothly from there.

Java

I had a couple old partitions I'd left on the drive in case I wanted to dual-boot to either DOS or Win95 at some later point. So my partitions looked like this:

/dev/hda1 - 75MB - fat16 (DOS)
/dev/hda2 - 50MB - ext2 (Linux: temp)
/dev/hda3 - 50MB - linux swap
/dev/hda4 - 350MB - ext2 (Linux: Debian)

Hard drive space: The bare-bones Debian install took up about 75MB. I downloaded the Java installer (70MB) and tried to install it. It needed 212MB to install, meaning I needed to fit 75 + 70 + 212 = 367MB on my 350MB partition. (MB values are approximate.) I tried trimming as much as I could from Debian, but couldn't quite get the space. I kicked myself for not originally devoting the whole drive to Debian. I thought about trying to combine the first three partitions into one to hold the Java installation, but that still wasn't enough. And I couldn't move the hda4 partition.

After a couple hours of this, realization struck and I kicked myself twice as hard for overlooking this: I only had a couple MB of data on hda1 (DOS is small). So I mounted it and moved the installer program over to there. I started installing from hda1 to hda4 with 50MB to spare on hda4.

C libraries: Then the installer bombed about 3/4 the way through: my C libraries where too old for the installer's unpacker to use. I was worried that this would be a major problem. But I changed my apt config file again, this time to point to the woody distro. Then I just used console-apt (works way better in low mem than aptitude!) to upgrade libc6. It went fine!

Low memory: Okay, trying to install Java again... It got to unpacking rt.jar and started swapping big time. Now I know why people say you never want to swap running programs. Swap should instead be used when multitasking to swap out idle processes. I watched the unpacking process for a couple hours... and about 43/50MB of my swap partition was in use... but it finally started writing rt.jar to disk, so I figured it'd be down hill from there and went to bed. I don't know how many hours it actually took to unzip this single file.

In the morning, I tried javac -version. It took about 5 minutes because javac looks to be about 7.5MB or so--and about 3.5MB of that ended up in swap. But it finally spit out the version line. Yay, it worked! [Update: Or not.] But I definitely needed more than 8MB of RAM to continue the project.

Disk space was also tight, so I went through the java directory and removed all the db, demo, sample, and man directories. That freed up about 50MB, leaving 152MB of java files.

Memory

The largest memory module that will fit into Tinker is a very rare 24MB module that I could find only at expensive memory stores. But I found a 16MB module for $4.50 (including shipping) at eBay... that then got snatched up (Buy It Now!) while I was trying to determine if it'd work in my machine. (There's a big sticker in the memory access area: 5V ONLY.) There were no more compatible 16MB modules left for sale, but I did find a few 16MB PC cards for sale.

PC Card RAM: There are a number of these "memory credit card" things floating around--mostly 8 or 16MB. But there's precious little information about them out there. The manufacturers don't support (or even provide info about) them any more. And PC card RAM seems to have been a short fad that went out of style about the time the WWW started, so there's not much general info online either.

As near as I can figure--from Wikipedia entries, memory sites, online Solaris pcram documentation, and ancient discussion threads--is that there were indeed PCMCIA RAM cards at one point. However, these did not directly add to your computer's available memory. Instead, you could use them as temporary ramdrives--reading and writing to a virtual disk at almost the speed of RAM, rather than at hard disk speeds. Perfect, I thought, for a swap partition!

However, there's some confusion as some PC cards are Flash storage devices--which have limited read/writes and so would make poor swap devices. Also, a number of old laptops--particularly the IBM Thinkpad 700 series--have similar cards that seem (as near as I can tell) to actually add memory directly to the laptop's usable total.

I wrote one eBay seller to find out if the cards were really PCMCIA. He said yes, so I bought a Kingston KTM-TP750/16 "16MB credit card" for $13.50 (including shipping). It arrived with an extra 8MB Toshiba memory card. However, neither of them would fit the last 1/4 inch into my PCMCIA slot!

Memory PC cards.

Here are the two cards compared to my PCMCIA Linksys network card. As you can see, they are the same shape and size as a PCMCIA card, but the holes are smaller, there are more of them, and the two rows are slightly offset. Searching on the number of pins (holes) revealed that these are 88-pin JEIDA DRAM cards. JEIDA was the original Japanese PC memory card standard. However, America responded with the incompatible 68-pin PCMCIA standard. In 1991, the two standards were unified into a single 68-pin form named JEIDA 4.1/PCMCIA 2.0. (Learn more about this in this 1993 article.)

So I struck out on that one. (I contacted the seller, and returned the cards for a refund... which, after a few weeks and a couple more emails, actually happened.) The DRAM would have been perfect, but I needed it in proper PCMCIA form. This was an interesting learning experience, and, since these cards are so poorly documented online, it prompted me to create this whole page. (Giving back to the web and all that.)

Trimming processes: The other thing I tried was to cut the number of processes running. Things were pretty bare-bones already, but I turned off a few of the gettys (which let you log in on multiple consoles, accessible by hitting ALT+F#, where # is 1 through the number of gettys you have running). I also switched off portmap and inetd. Normal use is pretty responsive, with just a few hundred KB of swap in use. But I only have 6.5MB of memory available (the kernel, etc. uses the rest), and most of the time about 5.5MB or so of that is in use.

Memory modules: As mentioned above, Java wasn't going to run without more memory. So I continued the search for memory modules. Searching for the 16MB part number from the specs (given above) wasn't much help, as few of the memory sites give the actual memory specs: voltage, access time, even whether it's SIMM or SODIMM. Searching for "T2130CS memory" was more helpful, but note how some memory sites say you need 72-pin SODIMM (with picture!) (and also SODIMM here, with a very different picture), while other sites say you need 72-pin SIMM (if they say at all, which most don't).

I should have paid closer attention to the specs, which do say SIMM (though it's easy to overlook). However, all images and reading I did (such as this nice visual comparison) showed that 72-pin SIMM modules looked way too long, whereas 72-pin SO-DIMMs were the right size. So I went for a 16MB SO-DIMM module from eBay ($11) that seemed to meet the specs otherwise.

When it arrived, it... almost fit. The notch on the side was only half as high as it should have been; all the other dimensions were correct. Since they had sent me exactly what I ordered, I didn't feel I had reason to request a refund. I figured, "Ah, what the hell. Let's force it." I filed off the notch and tried it anyway. No luck: instead of 8000KB, I now had only 3904KB.

Back to the search, I then stumbled across the 24MB module with the correct part number--NW2029U--at eBay for $21.90. (The listing had no other specs though.) When it arrived, it fit a little tightly, but correctly; and the notch was the right size. So now I have 32MB of RAM! I memtested it to make sure all was well.

Here's an image of the 24MB SIMM (above), the 16MB SODIMM (after notch-filing; below), and the empty bay they go in (to right):

Memory modules: NW2029U 24MB SIMM and 16MB SODIMM.
Click to enlarge

Conclusion: This laptop takes a rare and bizarre SIMM module that looks like a SODIMM module--hence the widespread confusion. However, look closely for the silver (not gold) pin connectors and the left-side notch, which is twice as deep as on SODIMMs.

Apache

I installed potato's Apache using console-apt. I edited the config files manually to move the logs, htdocs, etc onto /www, which is my mounted 50MB /dev/hda2 partition. It served up the static welcome page with no trouble.

Umasking a directory: My one complaint is that you can't umask a directory in Linux. That is, you can set the global umask (usually 022), which is subtracted from the normal file permissions whenever a new file or directory is created. However, I'd really like it to be set to 002 for my /www directory, so I can easily edit files created by various users--whether me, root, or the web server itself. It's possible to set the SGID on a directory (chmod g+s /www), which means all files and directories created in /www will now have the same group owner as /www itself. However, the permissions of the new files are still the default. That is, www group members have group ownership of the file, but they still don't have group write permission to the file. Lame.

The one (cleanish) work-around I've found is that it is possible to specify a umask when mounting a vfat (but not an ext2) partition. So, I might reformat the /www to vfat if this issue keeps irking me. [Update: Actually, I ended up doing this.]

SSH

I got this from the woody distro when I updated the C libraries. (I think I had to include the non-US packages to find it, but I can't remember for sure.) It installed itself fine and worked right out of the box.

Memory (update)

Processes at this point: The processes I had running at this point were init, the various kernel processes (kflushd, kupdate, kswapd, and keventd), klogd and syslogd, 4 gettys (2 currently logged in), cardmgr and pump (necessary for my PC card ethernet connection), sshd, and 6 apache processes. I still had about 16 MB free, with no swap in use.

Java (again)

Crashes: I didn't actually bother to try compiling a file with only 8MB of memory. I assumed if javac -version worked, compiling a file should work. After I upgraded to 32MB, I tried compiling a Hello World program, but, after about a minute of processing, javac crashed with the following error:

# An unexpected error has been detected by Java Runtime Environment:
#
#  Internal Error (nmethod.cpp:1707), pid=198, tid=1026
#  Error: guarantee(cont_offset != 0,"unhandled implicit exception in compiled code")
#
# Java VM: Java HotSpot(TM) Client VM (10.0-b22 mixed mode, sharing linux-x86)

For the interested, here's the extended file it dumped: hs_err_pid198.log

My concern here is that, near the end of the log, Java thinks it's out of memory and can't see any swap memory (although I have about 50MB of it). Running top while trying this shows that free memory does get down to about 500K or so, but never 0k.

My first thought was to try it again a few times. My second thought was to try to reinstall, since I did previously delete a number of (non-essential?) files. It took only 37 minutes this time (thanks to 32MB of RAM). It did crash after successfully installing but while trying to register; this is probably because I have no X-windows or browser. The reinstall did not fix the problem: still the same crash error, even running as root.

This was all with Java 1.6.0_06. I downgraded to Java 1.5.0_16 and it works.

Python

Python 2.1 proved easy to install with console-apt. However, since I was writing my grading system on a modern machine with 2.5 installed, I ended up including a number of more recent Python features. So I uninstalled 2.1 and tried to install 2.5.2 from source. This necessitated installing gcc, make, and the whole build environment, which took a lot of space. Python 2.5.2 built okay, but then I ran out of disk space installing it.

Outcome

At this point, I was fed up and ready to wipe the machine and start over.

Second Attempt

I reformatted (now that I have everything off of my old dos partition):

/dev/hda1 - 450MB - ext2 (Linux: /)
/dev/hda2 - 38MB - linux swap
/dev/hda3 - 55MB - ext2 (Linux: /www)

Debian (again)

Although I had more memory now, I repeated the potato installation.

Apache (again)

From potato. I still pointed it to use a different partition (/www) though as a security measure: an uploading DoS or something shouldn't bring the whole system down, just httpd.

SSH (again)

From woody. At this point, my system was still only about 75MB or so.

Python (again)

I grabbed the build environment from woody, which included an update of libc6, etc. This ran about 24.3 MB.

Then I compiled Python 2.5.2 from source. This took a long time, but went pretty smoothly. With the install files still in place, this ran over 140MB. Now that those are gone, Python is largely confined to /usr/local/lib/python2.5, which is 52MB. There's a 17MB test/ directory in there I could maybe delete if pressed for space.

A little tweaking (such as adjusting my scripts' #!she-bang lines) and I was serving up my ported, half-written grading system with no problems! Success.

Java (again)

I went with 1.5.0_16 again: a 50MB download, installing to 135 MB. After removing the demo, man, and sample directories, this was only 118MB.

It takes 25 seconds to spit out the javac -version and 150 seconds to compile a Hello World program. It takes 3 seconds to spit out the java -version and 15 seconds to run Hello World (or 3 seconds to run it again). This is too slow for instant web-based grading feedback, so I had to modify my grading program to use a grading pipeline.

Outcome

I completed this second attempt in early August 2008. Tinker ran fine as a web server for that fall semester. The first version of my grading system (named Tamarin) didn't compile submissions online after all, so I never used Java. But Tinker served up a combination of static HTML pages and Python-based CGI pages for the whole semester, allowing 64 students to upload their homework twice a week and to check on their grades.

Tinker went down twice during the semester. Both times I was away from home, but I think it may have been due to power outage. Each time I had to come home to press the power button (and then run fsck and such). Still, I was overall very happy with his performance. I shut him down properly 24 Dec 08 for the Christmas break and packed him away in his case.

Upon returning home, I fired him up on 12 Jan 09, ready to do some more tinkering and upgrading. However, he wouldn't boot: the hard drive would repeatedly whir quietly up to speed, tick/click 4 or 5 times, and then whir down again. I inserted a linux rescue disk floppy and, after waiting a couple minutes, it finally booted to the floppy... but the kernel panicked when the hard drive proved to be inaccessible.

I hate sudden, inexplicable hardware failures! It looks like we're back to the beginning again...

P.S.: Toshiba BIOS Access

Since they all tend to different and this one doesn't bother to display how to do it: to access the old "TOSHIBA Video BIOS V1.40" (©1995), press Esc during boot. You can do this before or after memory initialization. (If, like me, your hard drive is missing/failing, you may have to wait a couple minutes while nothing happens.) The computer will eventually beep and then wait, displaying: Check system. Then press [F1] key. Press F1 to enter the BIOS.

Third Attempt

Hard Drive: Removal

This is easy... once you know how. Here's the easy way: Flip the computer over and remove the group of 4 screws from the bottom next to the memory bay; these hold the hard drive in place. Flip the laptop right-side up and lift the screen. Open the palm rest (flip the two front latches up, then slide the palm rest towards you). On the left is the hard drive heat shield; on the right is the battery bay. No more unscrewing it required. Pry the hard drive cover off the left-side of the computer by pressing on the front of the left corner of the computer. (This might make more sense when you see the piece you're trying to remove in the picture below.) Now, reach between the heat shield and the battery bay and push the hard drive to the left so that it unplugs from the IDE connectors. It should eventually come popping out the left side of the computer:

Hard drive removed.
Click to enlarge

If you want to go further with disassembly, the trick is to next remove the thin line of plastic that runs along the front of the keyboard, holding the base of the keyboard down. It runs under the connector to the mouse buttons on the palm rest. Wiggle it until you can get all the little tabs free. Then the keyboard just flips up/back, and you're on your way. Three screws hold the hard drive heat sink in place. You can remove the palm rest by popping the two hinges out of their sockets. Careful doing this though--I managed to break the left hinge off putting it back on. (Grr.)

Hard Drive: Replacement

The drive, though old and small (labeled at 527MB), appeared to be a 44-pin IDE drive. There are 44 pins in a group on the back (with a central one missing), then a separate group of 4 pins, which are for jumper settings. These are pretty common and standard, so I picked up a 10GB IBM Travelstar laptop drive (Model: DJSA-210) up off eBay for $11 (+$4 shipping).

When the drive arrived and I flipped it over, I discovered a problem. (Again, this is why this page exists: to document all these little gotchas you don't otherwise even know to look for.) The next picture shows the problem:

Hard drive removed.
Click to enlarge

The original Toshiba drive is on the left. Note the 4 screw holes, located in 2 pairs each about 1/3 the way in from the ends of the drive. These screws anchor the drive in place.

The silver thing in the middle was the case originally on the underside of the new replacement drive. Note the lack of actual screw holes in the same places as on the Toshiba drive. Instead, the open screw holes are near the very ends of the drive. However, there are paper-covered recesses in the case where the screws should go. This gave me hope.

However, removing the case revealed the drive on the right in the picture: the only screw holes are indeed at the ends of the drive.

In short, I cannot screw this drive into place. This doesn't worry me too much since, as a server, Tinker doesn't get moved much. However, I know at some point I'm going to forget that the HD is not screwed down, and I'll flip the laptop over. I can just imagine the free, unplugged end of the drive falling upwards (now downwards), bending the connector pins and ruining the drive. I'm pretty sure that's overly dramatic though, since, with the drive heat shield in place, how far can the thing really go? The greater danger is just that the drive will wiggle and eventually wiggle loose of the IDE connectors.

This discovery does annoy me though, just because you'd think that, if the hardware itself is standardized and compatible, that the ability to screw the thing into place would also be pretty standardized. Guess not.

The new drive's dimensions are not quite the same as the old one: it's only about 2/3 as thick. This means that it didn't slide into place very easily. I had to remove the heat shield so I could guide the pins into place. When it's plugged in, it wiggles downwards a bit, suggesting that it might actually be suspended by the pins about a millimeter or so over the floor of the bay. I considered putting in some sort of shim, but eventually decided it was probably a worse idea to have loose pieces of junk floating around in there that could either melt, catch fire, or cause a short. Overall, the connection is snug and I'm no longer worried about it the drive wiggling enough to cause problems. (To be on the safe side, I'll still try not to move Tinker around too much.)

I booted the computer before closing everything up, and it looks like we're good to go--a bad partition table, but it can read the drive again.

Reinstallation: Base System

Okay, I guess doing this a third time might be a chance to better document the process. I grabbed the 3 idepci disk images: the rescue, root, and driver disk. (See the potato installation guide for more info.)

Partitions: I have way more space now. I guess Tinker is a web server, so I should leave room mostly for that. And I don't really want to be swapping much.

/dev/hda1 - ~2000MB - ext2 (Linux: /)
/dev/hda2 - ~128MB - linux swap
/dev/hda3 - ~7920MB - ext2 (Linux: /www)

System install: Things went relatively smoothly. I chose to edit the source list by hand this time, specifying this single uncommented line:

deb http://archive.debian.org/debian potato main contrib non-free

This configured apt fine, but I chose the advanced option for choosing packages and got dumped into dselect. I decided I didn't want to bother with advanced selection after all, but couldn't back up. I got dumped onto the command line with only the default packages installed (which was no big deal). So I used apt-get to install console-apt and then used capt to install everything else from there. I added a few extra tools, like the man pages, nano, etc. I also installed Apache.

Woody upgrades: From woody, I grabbed only libc6, locales, ssh, build-essential and all the package upgrades they depended on. (I figure I'm running potato here, so I should use those packages unless it's really necessary to upgrade for some reason.)

Reinstallation: Extra Software

Python: I downloaded and compiled Python 2.5.4 from source.

Java: I tried version 1.6.0_11, but still got the crash described above. I tried running it restricted to a max of 8MB of RAM, as follows:

java -J-Xms8m -J-Xmx8m Hello.java

But it still crashed. [Update: I later realized this only restricts the size of the heap.] The error log is still saying there's 0k of physical memory free, and it can see no swap. <sigh>. Back to 1.5.0_17.

I'm running the compiler under nice to try to keep the web server responsive. However, the difference wasn't particularly noticeable during the one test I ran. I think the lag comes more from having to swap apache back in than it not having immediate access to the CPU. Still, I'll leave javac niced--every little bit helps!

Configuration

I trimmed processes even more ruthlessly than last time. So I have: init, kernel processes (kflushd, kupdate, kswapd, and keventd), klogd and syslogd, 1 mingetty, cardmgr and pump (necessary for my PC card ethernet connection), sshd, and 4 apache processes.

Umask update: I mentioned earlier the annoying inability to umask a directory. This means that whenever I create directories the web server needs write access too, I often need to do a bunch of chgrp and chmod-ing. Also, if I happen to create something there as root, no other users can change or fix it.

In the end, I did the following instead: For my user account ztomasze, I left my default group as ztomasze. This means I can safely set my umask to 002, since there's no one else in that group. (Actually, there are no other real users at all, but, still... trying to do it up right here!) I also added myself to groups such as www-data, users (which all future real users should also be in), and staff (which I share with root).

Then, in the web server directories, I set the group ownership of various directories to either staff, users, or www-data, depending on who needs what sort of write access. By setting the SGID for those directories (chmod g+s dirname), any future files created by anyone in that directory will recursively inherit the same group ownership as the directory itself. And if all normal users are using a 002 umask, they'll also have proper write/access permissions too. I thought this a pretty clean fix.

Outcome

This concludes the Tinker Trials (at least for now). I've achieved the initial goals--albeit running Java 5 rather than 6. It serves up static HTML pages in a fraction of a second--seemingly as fast as any normal web server. Python scripts take a little longer--usually about 4 or 5 seconds, which is slightly slow. Compiling a Java program on Tinker still takes a little over 2 minutes, which is pretty sad. It also drags everything else down a little while it's doing it.

I love my new hard drive, though--it's completely silent, even while being accessed. I imagine that this machine is doesn't draw much energy; it's small and silent, and I don't mind having it run constantly in my living space.

Postscript: 2.5 years later

Tamarin was eventually retired in favor of Tinker2--a Pentium 200Mhz with 96 MB of RAM. I simply made an image of Tinker and copied it over with minor configuration changes.

Some months later, I got Tinker out of storage again, but he now looks like this:

Screen so discolored as to be unreadable.
Click to enlarge

I'm not sure what happened here: perhaps heat damage from living in a warm Hawaiian bedroom? His 3.5" floppy drive has also failed. This makes it impossible to do any significant maintance on him, since he lacks a USB drive or bootable CD drive.

It is thus time to finally retire Tinker come the next eWaste recycling fair. R.I.P! He is remembered in life by Tinker2 and a possible Tink3r project.