I ended up abandoning the effort because I simply could not squeeze in X windows in 4MB of RAM no matter how many things I threw out or not loading. Correction: I could not squeeze X windows and a window manager in 4MB of RAM. X windows came up ok but not having a window manager is such a pain when you have to resize windows. I went with cross-platforming the project by encapsulating the graphics primitives into simple procedures. That way I could program on Turbo-C in ANSI-C mode and test out the graphic subroutines at home. Later I would swap out the code in the simple procedures with their X windows equivalents.
But that taste of power with Linux was addictive. I had the same capabilities as the big Sun and HP boxes on my small (even by those day's standards) PC. I have been using them in one way or another ever since. And it has rewarded me by saving my skin on more than a few occasions.
Happy (belated) 21st birthday Linux!
Thanks to fitofinsanity for posting on reddit.com the image of the message by methodshop.com at Flickr.