Disabling swap file

I have an imac from 2009. At the time, I maxed out the memory to a whopping 4GB. This may not seem much these days but it is still plenty for most things.

Over time, my disk has gotten a bit fragmented, applications have gotten a bit heavier (e.g. Firefox commonly uses well over a GB), and things have gotten a little slower.

I had already half decided to buy a new mac to ‘solve’ this problem by simply getting the latest model and maxing out the RAM. The only problem is that Apple seems to be dragging their heels refreshing their iMac line, which is now more than a year old (ancient basically). So, I’ll have to survive for a bit and figure out how to do that until they get around to fixing that.

Basically the primary reason for slowness on macs is swapping. You may think you have enough RAM, and you may in fact have plenty of available RAM. It doesn’t matter, your mac will swap applications to disk even when you don’t need the memory. That takes time. Especially on fragmented disks. Basically on my machine it got to a point where the disk would be churning for minutes every time I opened applications and there was considerable lag when switching applications that I had already opened. All this despite still having a full GB of memory available. So even though I am nowhere close to running out of memory, the OS decides to spend time swapping stuff to and from disk almost constantly. Reason: it was designed well over a decade ago during a time when 16MB was massive amount of memory. I have about 250x that amount.

I found this nice post explaining how to turn swapping off (you have to reboot for this to work). Now there are all sorts of reasons why swapping and virtual memory are good things in theory and all sorts of advice warning that all sorts of evil things can and will happen if you mess with it.

Consider yourself warned.

There are also people claiming that swapping “improves performance”. The latter point is of course massive bull shit: how does using a massively slower disk help make things faster compared to only using RAM, which has massively better bandwidth and latency?. There are also loads of people confusing virtual memory and swap space whining that applications won’t run without virtual memory. Virtual memory is the notion that applications have access to a memory space that is much larger than the available RAM. On a 64 bit machine, this typically is way more than the capacity of your disk or indeed most storage solutions (think a couple of hundred TB).

Swap space on the other hand is used for temporarily swapping memory to disk that has actually been allocated and is in use by some application to free it up for some other application. I.e. you don’t swap virtual memory but only memory that is actually used. So, swapping is only useful when you need more RAM than you have and when you somehow prefer to not close applications and instead have their memory written to disk and read from disk when you switch between them.

Provided you have enough RAM to run all the applications that you need, this should be exactly never. On servers this is the reason you generally turn swapping off because by the time you need it, your server is dead anyway. On desktops, the same argument can be used, provided you can fit your applications into memory comfortably.

So, I turned off swapping on my mac a few days ago and was rewarded with a massive reduction in disk activity and nearly full restoration of the performance I enjoyed when this machine was still new and 4GB was four times what you actually needed. I call that a huge win.

Now, there is one caveat: modern operating systems run hundreds of processes. Those processes use memory. Nasty things happen if it cannot be allocated. The price of running out of memory is that random processes may crash and fail. If that happens to a critical process, your entire OS becomes unstable and you may have to reboot. Solution: make sure you don’t run out of memory. The good news is that normally you shouldn’t run out of memory except when running buggy software with memory leaks or when running a lot of software or software with extreme memory needs (e.g. Adobe Aperture or some games). A little awareness about what is running goes a long way to avoiding any trouble.

I routinely have Firefox open with a few dozen tabs, spotify, skype, a terminal, eclipse, and maybe preview and a few other small things. With all that open, I’m still below 3GB. That leaves about 1 GB for file caches, misc other OS needs, and a little bandwidth to open extra applications. If I run low on memory, or suspect that I will, I simply close a few applications.

I used to do this with windows XP as well and only recall a hand full of cases where applications crashed because they ran out of memory. Well worth the price for a vast increase in performance. Turning off swapping on old PCs is pretty much a no brainer and especially on new ones with way more memory than you need (and considering memory prices, why would you run with less …), I cannot possibly imagine any scenario where swapping adds any value whatsoever but I guess it would be a nice fallback solution as a last resort when the only alternative would be for applications to crash. Unfortunately, neither macs nor windows pcs have a setting that would enable that. So, turning off swapping entirely is the next best thing.

Quora and a bad user experience

I compulsively sign up for all sorts of stuff I don’t really need. Call it professional curiosity. Yesterday I was talking with a friend and he mentioned quora, a popular Q&A site that I have never really bothered with. So, on a whim I did my thing and signed up. This blog post is about the bad stuff that happened after that. In short the experience was appalling and mildly offensive and illustrative of how some companies just don’t get it.

So, here’s what happened.

I went to their home page. Basically there’s nothing there except a login box and a message stating “Quora connects you to everything you want to know about.”. I thought I had Google for that. Whatever. Despite this shocking lack of information and spectacularly bad marketing, I went ahead and attempted to sign in using the Facebook button. So, I clicked that and did the oauth exchange with Facebook. So far so good. Many sites offer this style of signups with Google, Facebook, Twitter, and other sites and it is sort of convenient.

After that it showed me a page to create a password. Eh… didn’t I just sign in?! Surely the intent of that was avoiding this. Annoyed, I just went ahead anyway and typed one of my  passwords that I reserve for sites I absolutely don’t care about one bit.

Then I was presented with a page that required me to select five topics out of some huge list. That just sucks. I’m a new user. Don’t force me to make choices like that right away. Since I wasn’t particularly interested in following anything at this point, I just clicked next and got another page with yet more crap to choose from. It just kept insisting I had to follow topics in order to use the site. I declined and closed the tab, which sort of my way of saying fuck you very much and have a nice life. Basically, I had lost all interest in quora at this point. Bad UX and I’m not really convinced I need it to begin with.

It could be I’m missing out on something really great. Whatever. Bad UX causes you to lose users like that. I wasted five minutes of my time and Quora never even got around to explaining what they do. Shit happens.

Then I started getting emails like X is now following you on Quora. Apparently, quora just went ahead and spammed my Facebook friends on my behalf. That’s a definite red flag in my book. There is no way in hell I would give permission to any company to do that.

I just deactivated my account to prevent more damage and because frankly I felt offended. If you got spammed, sorry.

There are a couple of patterns here that I observed with other sites:

  • Signup is needlessly complex. Every extra mouse click loses you customers. Every usability expert knows this. Don’t do this. At this point even having an email signup + password is fairly pointless for most users. Make it optional, if you bother implementing this at all (why would you?).
  • The site is useless without signup. So how am I to decide I want to sign up at all? Give me samples or partial access.
  • The site assumes you want to receive loads of email notifications. Off is a good default for any such notifications except the truly urgent ones. If I didn’t turn shit like this off, my inbox would have hundreds of new messages every morning from all sorts of sites I really don’t care about. Sorry, you are not the most important thing in my life.
  • The site hijacks your Facebook feed to promote itself. No I don’t endorse your shitty site and I certainly don’t want you anywhere near my friends until I’ve had a chance to evaluate what you are about. I clicked the sign in button to sign in, not to give you permission to abuse privileges you sneaked in with that request. If you are going to post to my Facebook feed, make sure I know it is going to happen and have an opportunity to prevent it.

Quora failed on all four points and displayed a level of arrogance and aggressiveness that may or may not have been intended that caused me to de-activate my account right away.

GeoTools

In the past few weeks I’ve been working on a little project on git hub called GeoTools that you might be interested in if you are into geo spatial stuff.

GeoTools is a set of tools for manipulating geo hashes and geometric shapes in the wgs 84 coordinate system. A geo hash is a representation of a coordinate that interleaves the bit representations of the latitude and longitude and base32 encodes the result. This string representation has a very useful property: nearby coordinates will have the same prefix. As is observed in this blog post: http://blog.notdot.net/2009/11/Damn-Cool-Algorithms-Spatial-indexing-with-Quadtrees-and-Hilbert-Curves, geo hashes effectively encode the path to a leaf in a quad tree.

The algorithms used for implementing the functionality in GeoTools are mostly well known and commonly used in 2D graphics. However, I have not found any comprehensive open sourced implementation for the wgs 84 coordinate system.

The key functionality that motivated the creation of this library was the ability to cover shapes on a map with geo hashes for indexing purposes. To support that, I needed some elementary geometric operations implemented.

Features

  • GeoGeometry class with methods that allow you to:
    • Calculate distance between two coordinates using the haversine algorithm.
    • check bounding box containment for a point
    • check polygon containment for a point
    • get the center for a polygon
    • get bounding box for a polygon
    • convert circle to a polygon
    • create a polygon from a point cloud
    • translate a wgs84 coordinate by x & y meters along the latitude and longitude
  • GeoHashUtils class with methods that allow you to:
    • check containment of a point in a geohash
    • find out the boundingbox of a geohash
    • find out neighboring geohashes east, west, south, or north of a geohash
    • get the 32 sub geo hashes for a geohash, or the north/south halves, or the NE, NW, SE, SW quarters.
    • convert geo hash to a BitSet and a BitSet to a geo hash
    • cover lines, paths, polygons, or circles with geo hashes

I’ve deliberately kept the design simple and non object oriented. These classes have no external dependencies and only use the java.util package and java.lang.Math class. Consequently it should be easy to port this functionality to whatever other language. Also, I’ve not attempted to implement Point, Polygon, Path, Circle, or other classes to support this library. The reason for this is very simple: these things are commonly implemented in other frameworks and any attempt from my side to impose my own implementation would conflict with the need of others to reuse their own classes. I think it is somewhat of a design smell that world + dog feels compelled to implement their own Point class.

So instead, a point is represented as a latitude, longitude pair of doubles or as an array of two doubles (like in geo json). Likewise, paths and polygons are represented as arrays of points. A circle is a point and a radius. This should enable anyone to integrate this functionality easily.