One day, when we’re sitting in our rocking chairs recounting our past IT glories (“Why, when I was a young man, computers had ‘wires’”), we’ll invariably start talking about our storage war stories. There will be so many. We’ll talk of frisbee tossing stuck disks or putting bad drives in the freezer. We’ll recount how we saved a company’s entire financial history by recovering an alternate superblock or fixing a byte swapping error on a tape with the “dd” command. I’m sure our children will be transfixed.
No…no, they won’t be transfixed, any more than we would be listening to someone telling us about how their grandpa’s secret pot roast recipe starts with “Get a woodchuck…skin it.” You simply have to be in an anthropological state of mind to listen to something like that. More likely, they walked into the room to ask you your wifi password (Of course, only us old folk will have wifi. Your kids are just visiting. At home they use something far more modern and futuristic. It’ll probably be called iXifi or something).
Unfortunately for us, many of these war story issues remain serious problems today. Disks “do” get stuck and they “do” often get better and work for a while if you freeze them. It’s a great way to get your data back when you’ve been a little lazy with backups.
Another problem is fragmentation. This is what I wanted to focus on today.
Disks today are still spinning platters with rings of “blocks” on them, where each block is typically 512 bytes. Ideally, as you write files to your disk, those bytes are written around the rings so you can read and write the blocks in sequence. The head doesn’t have to move. Each new block spins underneath it.
Fragmentation occurs because we don’t just leave files sitting on our disk forever. We delete them. We delete emails, log files, temp files, render files, and old projects we don’t care about anymore. When we do this, those files leave “holes” in our filesystems. The OS wants to use these holes. (Indeed, SGI used to have a real-time filesystem that never left holes. All data was written at the end. I had to handle a few cases where people called asking why they never got their free space back when they deleted files. The answer was “we don’t ever use old holes in the filesystem. That would slow us down!”)
To use these holes, most operating systems use a “best fit” algorithm. They look at what you are trying to write, and try to find a hole where that write will fit. In this way, they can use old space. When you’re writing something extremely large, the OS just sticks it into the free space at the end.
The problem occurs when you let things start to fill up. Now the OS can’t always find a place to put your large writes. If it can’t, it may have to break that large block of data into several smaller ones. A file that may have been written in one contiguous chunk may get broken into 11 or 12 pieces. This not only slows down your write performance, it will also slow down your reads when you go to read the file back.
To make matters worse, this file will remain fragmented even if you free more space up later. The OS does not go back and clean it up. So it’s a good idea not to let your filesystems drop below 20% free space. If this happens and performance suffers, you’re going to need to look into a defragmentation tool.
Soon, this issue won’t matter to many of us. SSDs (Solid State Disks) fragment just like spinning disks, but it doesn’t matter near as much. SSDs are more like Random Access Memory in that data blocks can be read in any order, equally as fast. So even though your OS might have to issue a few more reads to pull in a file (and there will be a slight performance hit), it won’t be near as bad as what a spinning disk would experience. Hence, we’ll tell our fragmentation war stories one day and get blank looks from our grandkids (What do you mean “spinning disk?” The disk was “moving??”).
Personally, I long for the days when disk drives were so large, they would vibrate the floor. I liked discovering that the night time tape drive operator was getting hand lotion on the reel to reel tape heads when she put the next backup tape on for the overnight runs. It was like CSI. I’m going to miss those days. Soon, everything will be like an iPhone and we’ll just throw it away, get a new one, and sync it with the cloud. Man that sucks.