RoHS CompliantAll Small Tree cards, switches and storage products are RoHS compliant!

  1. Snapshots…your trashcan, on steroids

    May 19, 2014 by Steve Modica

    I have to admit, as an old time UNIX guy that’s been around inodes, fsck and corrupted filesystems all my life, snapshots sounded a little too good to be true.

    The word was long known to me.  Customers would say, “I took a snapshot of that disk so I could upgrade it and revert if I screwed something up.”  It’s just that imaging a disk would take hours.  You’d start the copy and go home for the night.

    These new snapshots (like those supported by ZFS) were instantaneous.  One click and you would “instantly” have a new copy of your data.  How?  That’s not even possible.  To make it even weirder, the new copy takes up no space!?  Now it’s starting to sound like perpetual motion.

    The actual explanation is a lot simpler. Every filesystem is composed of data (your file data) and metadata (the name of the file, permissions, location of blocks, inode number, etc.).  All this metadata is what organizes your data.  You have what’s called an “inode table” where all that stuff lives, and it “points to” the actual data you wrote.  It might be video data, or your mom’s apple pie recipe.

    When you create a snapshot, you are instantly making a copy of that inode table.  You now have two. All these inodes point to the same data.  So the data was not copied.

    Now the magic happens. When a user deletes a file from the original data, the inode for that file is removed, but the snapshot inode remains.  ZFS will keep the data around as long as there’s an inode in some snapshot somewhere pointing to it.  The same is true if you edit a file.  The old data is saved, but the new data gets written.

    All this old stuff (old data) essentially becomes part of the snapshot.  As more things change, the snapshot grows larger. If you were to delete “all” the data on the original filesystem, the snapshot would essentially grow to the size of the original filesystem. (The original filesystem would drop to 0.)

    In some ways, it’s a little like a trashcan. When you delete something, it doesn’t really go away. It goes into the trash. If you wanted to, you could drag it out of the trash.

    There’s a similar way of recovering snapshots.  You simply “clone” (or mount) them.  When you do this, the snapshot inode table is mounted and it still points to all the old data.  That file you deleted yesterday?  If you mount yesterday’s snapshot, it’s right back where it was.  Simply drag it back out.

    Obviously, while snapshots make for a great method of saving previous images of a set of data, they are not a backup solution.  If your RAID dies and can’t be recovered, your snapshots die too!  So for true backup protection, consider rsync or some other method of moving your data to another system.

    Small Tree’s TitaniumZ servers support snapshots and rsync and we have a very nice graphical interface so you can manage it all yourself. If you have any questions about snapshots or a backup solution that’s right for your editing team, don’t hesitate to contact me at


  2. Graveyard Carz Post Production Workflow Case Study

    April 16, 2014 by Joe DiBenedetto



    With over 26 episodes to be completed in the coming months, The Division, the team behind Velocity by Discovery’s Graveyard Carz, needed to accelerate their post production workflow.

    The growing company was three seasons into the production of their hit show, Graveyard Carz.  They had scraped by on consumer-grade storage for the past five years, and The Division was long overdue for a storage solution that made sense.

    The post production workstations consists of four Windows PCs and an iMac running Premiere Pro, a Windows PC running DaVinci Resolve, and a Mac Pro using Adobe Audition.

    Online Editor and Executive Producer Aaron Smith was eager to find a way to help post production at the Division run as a well-oiled machine.  “We needed a way to share all of our media between several editors, our audio mixer, and the colorist.  Every time we had to copy or transcode files it added a couple minutes to a task, and those minutes quickly add up to hours and days.  We aren’t a big shop, so we needed something easy to set up, and simple to use.”


    The Division has a mixture of Macs and Windows operating systems.  All of the workstations can now share the same media at the same time, eliminating the need for file transfers between computers.  The speed of the TitaniumZ made it possible to move terabytes of assets onto the storage system in the morning and share them in an afternoon.  “Within a day, we were completely set up and accessing media on the TitaniumZ.  Editors were picking up media and editing segments in no time.”

    When a segment is shot, the media from the camera cards are copied to the appropriate folder on the TitaniumZ.  Each segment can be any mixture of media shot on Canon 60D, 7D, 5D MkII, GoPro, and even Blackmagic Pocket Cinema Cameras.  “At the moment we’re just working in HD, but 4K is definitely a possibility for the future, and it’s nice to know the Titanium Z can handle those kinds of data rates through 10 gigabit Ethernet.” The media is then organized, renamed, and assigned a specific serial number which is used to identify media and log metadata later using Trello, a web-based project management application. The serial number is prepended to all of the media for that segment.

    Once the media is on the TitaniumZ and renamed with serial numbers, the media is ready for any of the segment editors to throw into Adobe Premiere.  All of the segments are edited in the native camera formats at full resolution directly off the TitaniumZ. The main content, QuickTime H.264 footage shot on Canon DSLR cameras, is shot in the Technicolor Cinestyle color setting at 45mbps, while the GoPro footage is shot using the ProTune color settings at 35mbps. Post production supervisor Casey Faris explains, “We don’t make proxies anymore because there’s no need to do so. The TitaniumZ is beyond capable of handling the data rates we throw at it.  Even with several editors working, editing multiple streams of HD video is a breeze.  We have yet to see any slowdown or hiccup in performance.  Our editors were actually surprised at how fast the TitaniumZ is.  It’s like working from a local drive, but faster.”

    “With the Small Tree solution we have a mix of gigabit and 10 gigabit Ethernet connections hooked straight into the TitaniumZ. No need for external switches or servers running complex software. We have the color grading and online seats hooked up via 10 gigabit Ethernet, which provides plenty of room when handling finishing codecs like ProRes 4444 and Cineform, while the other editors have more than enough bandwidth cutting native camera codecs.” Says Smith.

    Once segments are edited, the program editor assembles them into the program sequences.  The Premiere projects are all imported into one project to be laid out into a watchable show.  Because all of the workstations access the same media, the Premiere project will open on any computer connected to the TitaniumZ. “It’s a huge time saver,” notes Faris. “Shared storage allows us to open a Premiere project on any computer without transferring media.  Those transfer times can really nickel and dime your day away. With the TitaniumZ, we can work faster, and more efficiently, cutting our time in half per episode compared to before.”

    After the program is locked, it’s sent to Audition for audio mix.  On the audio station, the project can be opened referencing the original sound files.  After the mix, the audio is exported as several different “stems” or parts of the mix, which, include dialogue, sound effects, and music tracks.  The .WAV files can be bounced right to the TitaniumZ, where they can be picked up by the online editor to add to the final Premiere project.

    Color correction is done on a high-end Windows PC running Blackmagic Design’s DaVinci Resolve.  The program is rendered in sections as ProRes 422 or Cineform QuickTime files with EDLs out of Premiere.  The colorist can then open the renders in Resolve right from the TitaniumZ.  After color is finished, the sections are rendered back onto the TitaniumZ as Cineform QuickTime movies, which can be added into Premiere.

    Once all the final media has been added to the program, graphics and supers are added, and the episode is rendered to 1920×1080 190mbps Cineform QuickTime movies for FTP upload.  All of the original footage is stored on the TitaniumZ for the rest of the season and is then copied to external hard drives for archival.

    “Working with Small Tree has been wonderful.  The biggest thing we appreciate about a company is always their customer support.  With Small Tree, we can call them up and talk to a real person who can walk us through anything we need.   They’ll even remotely troubleshoot any of our systems using screen sharing.  That kind of support is really rare these days,” notes Smith.

    “After just a few days, we were all working off the TitaniumZ with our existing systems and had forgotten the files weren’t on the stations we were using.  It’s a solution that just feels natural.  You forget it’s there.  When you don’t have to worry about the technical side of things, you can focus on the creativity and the storytelling.”

    “After experiencing shared storage with Small Tree, we’re never looking back.”

  3. Small Tree Unveils its Own Storage Operating System and Project Sharing Interface at NAB 2014 to Strengthen Company’s Shared Storage Solutions

    March 28, 2014 by Joe DiBenedetto

    ZenOS and Project Wrangler Deliver Super Fast Access to Video Editors

    3U_TitaniumZ with ZenOS

    Oakdale, Minn., March 27, 2014 — At this year’s NAB, Small Tree (Booth SL11105) will unveil  its own storage operating system and new Interface to strengthen their already robust TitaniumZ all-in-one, real-time, Ethernet-based shared storage systems.

    Based on FreeBSD 10, Small Tree’s ZenOS operating system provides video editing teams using TitaniumZ with even better performance by capitalizing on Samba 4 and Native Avid Media sharing high speed multipath iSCSI support for super fast access from individual clients. Small Tree’s newly developed Graphical User Interface, Project Wrangler, is an iSCSI sharing tool that allows users to create, mount and share iSCSI projects quickly and easily, while also enabling Active Directory and Open Directory support for seamless integration, as well as NFS support for Avid and Final Cut X project and library sharing.

    “Talk to video editors and you learn quickly that what they want most of all are intuitive tools that eliminate waiting around for access to files and minimizing rendering times,” said Steve Modica, Small Tree’s Chief Technology Officer. “We developed ZenOS and Project Wrangler to ensure our TitaniumZ shared storage systems provide optimal performance and efficiency unmatched throughout the industry.”

    Small Tree’s TitaniumZ is capable of supporting up to 24 video editing workstations without needing an Ethernet switch, and was designed and tested to provide maximum performance for real-time video editing workflows.

    Offered in three scalable models – TitaniumZ-5, TitaniumZ-8 and TitaniumZ-16 – Small Tree’s popular shared storage product can be configured with up to 20 10GbE ports or 24 GbE ports with storage capacities (raw) from 10TB to 1PB (Petabyte). TitaniumZ can be upgraded on-site through advanced ZFS technology, which enables servers to seamlessly integrate newly added storage while keeping existing files intact.

    Providing high performance sharing, quickly and directly from the RAID, TitaniumZ systems include simplified setup and management, RAID protection for multimedia editing and Small Tree’s industry-leading tech support. Creative teams will feel the difference in increased performance and productivity immediately with any of Small Tree’s all-in-one systems featuring ZenOS and Project Wrangler.

    TitaniumZ was designed to be simple to setup and manage. The storage can be made available to users within minutes of initial startup, while the powerful and convenient browser-based Project Wrangler interface makes management of TitaniumZ straightforward. Using Project Wrangler, TitaniumZ can be managed from anywhere with an Internet connection.

    All three TitaniumZ systems offer optimal flexibility with the capability to work across multiple protocols (AFP, SMB, NFS and iSCSI) and platforms, including Adobe Creative Suite, Avid Media Composer, Avid Pro Tools, Apple Final Cut Pro 7, Apple Final Cut Pro X, and Blackmagic Design Davinci Resolve. Available in a variety of drive configurations, from 5 to 144 disks, with multiple network scalability options, Small Tree’s newest line of shared storage solutions can meet the demands of today’s 4K workflows whether they’re using Red, Arri or Phantom.

    For more information on TitaniumZ featuring ZenOS and Project Wrangler, visit Small Tree at NAB (SL11105) or Follow Small Tree LinkedIn, or @smalltreecomm.

  4. Thunderbolt Updates

    March 20, 2014 by Steve Modica

    We’ve been working pretty hard on Thunderbolt products over the last few weeks and I thought I’d write up some of the interesting things we’ve implemented.

    I’m sure most of you are aware that Thunderbolt is an external, hotplug/unplug version of PCIE.  Thunderbolt 1 provided a 4X PCIE bus along with an equivalent bus for graphics only. Thunderbolt 2 allows you to trunk those two busses for 8X PCIE performance.

    PCIE Pause

    This is a new feature of Thunderbolt designed to deal with the uncertainty of what a user may plug in.

    Normally, when a system boots up, all of the PCIE cards are in place. The system sorts out address space for each card and each driver is then able to map its hardware and initialize everything.

    In the Thunderbolt world, we can never be sure what’s going to be there.  At any time, a user could plug in not just one device, but maybe five!  They could all be sitting on their desk, daisy-chained, simply waiting for a single cable to install.

    When this happens, the operating system needs the capability to reassign some of the address space and lanes so other devices can initialize and begin working.

    This is where PCIE Pause comes into play.  PCIE Pause allows the system to put Thunderbolt devices into a pseudo sleep mode (no driver activity) while bus address space is reassigned. Then devices are re-awakened and can restart operations.  What’s important to note is that the hardware is “not” reset.  So barring the odd timing issue causing a dropped frame, a PCIE Pause shouldn’t even reset a network mount on a Small Tree device.

    Wake On Lan

    We’ve been working hard on a Wake On Lan feature.  This allows us to wake a machine from a sleep state in order to continue offering a service (like File sharing, ssh remote login or Screen sharing).  This may be important for customers wanting to use a Mac Pro as a server via Thunderbolt RAID and Network devices.

    The way it works is that you send a “magic” packet via a tool like “WakeonMac” from another system.  This tells the port to power up the system far enough to start responding to services like AFP.

    What’s interesting about the chip Small Tree uses (Intel x540) is that it requires power in order to watch for the “magic” wake up packet. Thunderbolt wants all power cut to the bus when the machine goes to sleep.  So there’s a bit of a conflict here.  Does a manufacturer violate the spec by continuing to power the device, or do they not support WOL?

    This is most definitely true for the early Thunderbolt/PCIE card cage devices.  They were all very careful to follow the Thunderbolt specification (required for certification and branding) and this leaves them missing this “powered while sleeping” capability.

    Interested in learning more about how you could be using Thunderbolt? Contact me at


  5. Small Tree Introduces Specialty Thunderbolt 2 Gigabit Connectivity Solutions

    by Steve Modica

    New Line Up Works Seamlessly for Mac and Windows Systems

    Oakdale, Minn., March 20, 2014 — With its high performance I/O capabilities, Thunderbolt™ 2 is an extremely attractive tool for post-production professionals seeking to optimize productivity and efficiency. To further strengthen the workflow environment for those relying on Thunderbolt™ 2, Small Tree is introducing a new line up of specialty Gigabit connectivity solutions that work seamlessly with Mac and Windows systems.

    “Small Tree has updated its current Gigabit chip to the new Intel i350, which gives our customers a new range of features for supporting Thunderbolt™ 2 performance and power specifications,” said Steve Modica, Small Tree’s Chief Technology Officer.  “These new cards allow us to offer a wider range of products for connecting Mac or Windows systems with Thunderbolt™ 2 ports into optical and multiport environments for things like Link Aggregation, traffic monitoring and high security environments.”

    Highlighted by its recently introduced ThunderNET2 system, Small Tree’s Thunderbolt™ 2 inspired technologies add network bandwidth to any computer with a Thunderbolt port.  Features include the ability to daisy chain up to six devices to a single port on the host computer, whisper quiet normal operation and portable protection for any environment.

    ThunderNET2 provides creative media professionals a cost-effective option to integrate Thunderbolt equipped platforms into high performance storage and data networks. Small Tree designed the system, as well as it entire line of Gigabit connectivity solutions, to support post-production pros harnessing the increased processing power available in today’s all-in-one desktop and laptop computers.

    For more information on Small Tree’s line up of products dedicated to harnessing the full power of Thunderbolt™ 2, visit, or call 1-866-782-4622 for a price quote.

    Follow Small Tree on LinkedIn, or @smalltreecomm.

  6. Brand New Media Selects Shared Storage Solution from Small Tree

    March 17, 2014 by Steve Modica

    Multi-Office International Digital Media Company
    Accelerates Workflow with Plug and Play Implementation


    Singapore, March 17, 2014 Ideal Systems, an industry leading, multi-national systems integrator providing innovative solutions to broadcasting, telecoms and enterprise companies, recently integrated technology from Small Tree at Brand New Media’s broadcast facility in Singapore. Brand New Media (BNM) is a global digital media company that required a robust shared storage solution to accelerate its workflow without disruption to their infrastructure and manpower.

    “We came across Small Tree at the NAB conference (last year) and were really impressed by the company and their high performance networking and shared storage products,” said Fintan Mc Kiernan, CEO at Ideal Systems Singapore. “They have a great blend of hardware and software skills that allow them to create unprecedented bandwidth to their storage devices. This is going to be vital for our customers in Asia, such as Brand New Media, as they start to build out their capacity for 4K Ultra HD production work flows.”

    Working with many of the world’s biggest brands and broadcasters including Seven Media Group, Prime Media Group, TVNZ, StarHub, the Health Promotion Board of Singapore and NTUC Fairprice, BNM provides turnkey solutions via the creation and distribution of new content via their transmedia channels  television and all connected devices. Before installing a Small Tree TitaniumZ-16 with 128TB to accommodate the company’s continued growth, BNM relied on a series of portable hard drives and a 32TB server to meet its storage needs.

    “Due to the nature of our business, BNM is constantly looking for  solutions that are cutting-edge yet cost-effective to expand our server space and Small Tree’s TitaniumZ storage systems offer just the performance we need,” stated Colin Ng, Broadcast Manager at Brand New Media. “We look forward to the upkeep of our infrastructure and automating our data and server management to an unparalleled degree.”

    Capable of supporting numerous multimedia content creation workstations with a highly flexible mix of gigabit Ethernet and 10 gigabit Ethernet configuration options featuring storage capacities (raw) from 10TB to 576TB (Terabyte), TitaniumZ is designed and tested to provide maximum performance for real-time video-editing workflows. BNM has its eight workstations – Mac and Windows – at its Singapore location, seamlessly running a variety of multimedia software applications, including Apple Final Cut

    Pro 7, Adobe After Effects and Adobe Photoshop. The shared storage system works flawlessly with the entire Adobe Creative Suite in addition to Apple Final Cut Pro X, Avid Media Composer and Autodesk Smoke, offering creative facilities optimal flexibility.

    Available in a variety of drive configurations, from 5 to 144 supporting 16 to 144 concurrent ProRes 422 streams, with multiple network scalability options, and extremely easy to install, Small Tree’s TitaniumZ line of shared storage solutions can meet the demands of any facility, large or small.

    “The Small Tree system was simply ‘plug and play’,” said Colin Ng. “It happens to be one of the easiest pieces of hardware we have in our facility.”

    For more information about Small Tree and its growing line of shared storage and networking products, visit Follow Small Tree on Facebook at or on Twitter @SmallTreeComm.

    About Brand New Media

    Brand New Media (BNM) is a global digital media company with offices and dedicated studios operating across Australia, Singapore, US and Europe. Brand New Media owns, creates and operates digital channels allowing leading brands to become broadcasters. Brand New Media’s ChannelPLAY platform delivers content and channels to all devices, anywhere and anytime, delivering an innovative and integrated marketing solution to brands. For more information, visit

  7. Testing with Adobe Anywhere

    March 7, 2014 by Steve Modica

    Small Tree has been working closely with Adobe to make sure our shared editing storage and networking products work reliably and smoothly with Adobe’s suite of content creation software.

    Since NAB 2013, we’ve worked closely with Adobe to improve interoperability and performance, and test new features to give our customers a better experience.

    Most recently, I had the chance to test out Adobe Anywhere in our shop in Minnesota.

    Adobe Anywhere is designed to let users edit content that might be stored in a high bandwidth codec, over a much slower connection link.  Imagine having HD or 4K footage back at the ranch, while you’re in the field accessing the media via your LTE phone and a VPN connection.

    The way it works is that there’s an Adobe Anywhere server sitting on your network that you connect to with Adobe Premiere and this server compresses and shrinks the data “on the fly” so it can be fed to your machine much like a YouTube video.  Except you are scrubbing, editing, cutting, dubbing and all of the other things you might need to do during an edit session.

    This real-time compression/transcoding happens because the Adobe Anywhere system is taking advantage of the amazing power of GPUs.  Except rather than displaying the video to a screen, the video is being pushed into a network stream that’s fed to your client.

    I tested my system out with some Pro Res promotional videos we’ve used at trade shows in the past, and did my editing over Wi-Fi.

    What I found was that the system worked very well.  I could see that the Adobe Anywhere system was reading the video from Small Tree’s shared storage at full rate, then pushing it to my system at a greatly reduced rate.  I had no trouble playing, editing and managing the video over my Wi-Fi connection (although Adobe recommends 1Gb Ethernet as the minimum connectivity for clients today).

    This type of architecture is very new and there are caveats.  For example, if you are very far from the server system or running over a very slow link (like a vpn connection), latency can make certain actions take a very long time (like loading an entire project, or using Adobe’s Titler app which requires interactivity).  Adobe cautions that latencies of 200msecs or more will lead to a very poor customer experience.

    Additionally, just because the feed to the clients is much lower bandwidth (to accommodate slower links), the original video data still needs to be read in real-time at full speed. So there are no shortcuts there.  You still need high quality, low latency storage to allow people to edit video from it. You just have a new tool to push that data via real-time proxies over longer and slower links.

    All in all, I found the technology to be very smooth and it worked well with Small Tree’s shared network storage.  I’m excited to see the reach of Small Tree shared storage extended out to a much larger group of potential users.

    For a demonstration of Adobe Anywhere over Small Tree shared storage, visit us at the NAB Show in Las Vegas this April (Booth SL11105).

  8. Small Tree to Showcase ThunderNET2 at NAB 2014

    March 4, 2014 by Steve Modica

    Capitalize on Thunderbolt 2™ Technology and Support
    4K Video File Transfer and Display SimultaneouslySmall Tree

    Oakdale, Minn., March 4, 2014 Small Tree, the Ethernet-based shared storage and networking specialist, will showcase its new ThunderNET2 solutions for post-production professionals at NAB 2014 (Booth SL11105). With ThunderNET2, users will be able to support 4K video file transfer and display simultaneously.

    Created to support post-production pros harnessing the increased processing power available in today’s all-in-one desktop and laptop computers, ThunderNET2 opens up a whole world of possibilities. Combining high performance I/O capabilities of Thunderbolt™ 2 with the flexibility of PCIe, Small Tree’s ThunderNET2 provides creative media professionals a cost effective solution to integrate Thunderbolt equipped platforms into high performance storage and data networks.

    ThunderNET2 features improvements to Small Tree’s ThunderNET solution introduced last year and which already has provided major performance enhancements for post-production companies. One company whose efficiency has been optimized with ThunderNET is 3-Legged Dog (3LD), a community-oriented and artist-run production and development studio for emerging and established artists and organizations that create large-scale experimental artworks of all kinds.

    “Using Small Tree’s ThunderNET, we’re pushing an insane amount of data – like composites up to 20K,” said Cameron Vokey, line producer at 3LD. “If we were using straight up Gigabit, this would not be happening. It needs to be as fast as Thunderbolt. We wouldn’t be getting the read/write speeds we’re achieving without ThunderNET.”

    ThunderNET2 boasts the widest range of 1 or 10 Gigabit Ethernet connectivity to Mac Pro, iMac, Mac Mini, MacBook Pro or MacBook Air systems, providing higher levels of network performance and connecting seamlessly to Small Tree’s Ethernet shared storage appliance, TitaniumZ, or 10 Gigabit Ethernet switches. ThunderNET2 can also provide additional Gigabit Ethernet connectivity to the user’s current network.

    “With its more compact design combined with the 20% increase in performance, ThunderNET2 provides a very cost effective solution for Thunderbolt equipped systems,” said Corky Seeber, president of Small Tree. “Typically, these same systems don’t provide enough network connection flexibility for professionals in the creative fields. ThunderNET2 capitalizes on the new Thunderbolt™ 2 technology to create an affordable and extremely powerful solution for the post-production industry.”

    Adding network bandwidth to any computer with a Thunderbolt™ 2 port, Small Tree’s ThunderNET2 can be configured to support specific requirements, allowing customers to select the proper amount of bandwidth to meet both workflow demands and budgets.

    For more information on ThunderNET2, or any of Small Tree’s cost-effective shared storage and networking solutions, visit Follow Small Tree on LinkedIn, or @smalltreecomm.

  9. Another Couple of Reasons to Love SSDs

    February 26, 2014 by Steve Modica

    One day, when we’re sitting in our rocking chairs recounting our past IT glories (“Why, when I was a young man, computers had ‘wires’”), we’ll invariably start talking about our storage war stories.  There will be so many.  We’ll talk of frisbee tossing stuck disks or putting bad drives in the freezer. We’ll recount how we saved a company’s entire financial history by recovering an alternate superblock or fixing a byte swapping error on a tape with the “dd” command. I’m sure our children will be transfixed.

    No…no, they won’t be transfixed, any more than we would be listening to someone telling us about how their grandpa’s secret pot roast recipe starts with “Get a woodchuck…skin it.”  You simply have to be in an anthropological state of mind to listen to something like that. More likely, they walked into the room to ask you your wifi password (Of course, only us old folk will have wifi. Your kids are just visiting. At home they use something far more modern and futuristic. It’ll probably be called iXifi or something).

    Unfortunately for us, many of these war story issues remain serious problems today.  Disks “do” get stuck and they “do” often get better and work for a while if you freeze them. It’s a great way to get your data back when you’ve been a little lazy with backups.

    Another problem is fragmentation. This is what I wanted to focus on today.

    Disks today are still spinning platters with rings of “blocks” on them, where each block is typically 512 bytes. Ideally, as you write files to your disk, those bytes are written around the rings so you can read and write the blocks in sequence. The head doesn’t have to move.  Each new block spins underneath it.

    Fragmentation occurs because we don’t just leave files sitting on our disk forever. We delete them.  We delete emails, log files, temp files, render files, and old projects we don’t care about anymore. When we do this, those files leave “holes” in our filesystems. The OS wants to use these holes.  (Indeed, SGI used to have a real-time filesystem that never left holes. All data was written at the end.  I had to handle a few cases where people called asking why they never got their free space back when they deleted files.  The answer was “we don’t ever use old holes in the filesystem. That would slow us down!”)

    To use these holes, most operating systems use a “best fit” algorithm.  They look at what you are trying to write, and try to find a hole where that write will fit. In this way, they can use old space. When you’re writing something extremely large, the OS just sticks it into the free space at the end.

    The problem occurs when you let things start to fill up.  Now the OS can’t always find a place to put your large writes. If it can’t, it may have to break that large block of data into several smaller ones. A file that may have been written in one contiguous chunk may get broken into 11 or 12 pieces.  This not only slows down your write performance, it will also slow down your reads when you go to read the file back.

    To make matters worse, this file will remain fragmented even if you free more space up later. The OS does not go back and clean it up.  So it’s a good idea not to let your filesystems drop below 20% free space. If this happens and performance suffers, you’re going to need to look into a defragmentation tool.

    Soon, this issue won’t matter to many of us.  SSDs (Solid State Disks) fragment just like spinning disks, but it doesn’t matter near as much.  SSDs are more like Random Access Memory in that data blocks can be read in any order, equally as fast. So even though your OS might have to issue a few more reads to pull in a file (and there will be a slight performance hit), it won’t be near as bad as what a spinning disk would experience.  Hence, we’ll tell our fragmentation war stories one day and get blank looks from our grandkids  (What do you mean “spinning disk?”  The disk was “moving??”).

    Personally, I long for the days when disk drives were so large, they would vibrate the floor. I liked discovering that the night time tape drive operator was getting hand lotion on the reel to reel tape heads when she put the next backup tape on for the overnight runs. It was like CSI. I’m going to miss those days. Soon, everything will be like an iPhone and we’ll just throw it away, get a new one, and sync it with the cloud.  Man that sucks.

    Follow Steve Modica and Small Tree on Twitter @smalltreecomm.  Have a question? Contact Small Tree at 1-866-782-4622.


  10. Buying Storage

    February 13, 2014 by Steve Modica

    I’ve been in the computer industry for quite some time.

    Back in the early days, we worried a lot about running out of space on a computer or a server. If you filled up your Novell Netware system, what could you do?  Adding drives was an option, but it was expensive and “scary” and you’d still end up with another volume you had to train your users to use (we didn’t have the ability to stripe all that stuff together). Further, it was likely your disk controller only supported two drives and your motherboard only supported a couple controllers.  If you ran out of space in that scenario, it meant buying an entirely new platform (software included) that would be extremely expensive. There was also no guarantee all of your stuff would migrate cleanly.

    This led many of our early computer system design people down the path of expandability and modularity.  We wanted SCSI and later, Fibre Channel, so we could add device after device to a system and never run out of space. We wanted expandable filesystems so these new devices could be merged in without moving data around.  We wanted clusters so as we ran out of CPU power and IO slots, we could just add more. Never again would we find ourselves sitting on the floor at 10 p.m. trying to figure out why our second IDE drive wasn’t being seen by the new controller we installed last week.  (You forgot to change its address knucklehead. It’s conflicting with the first disk you put in there).

    So today, we have lots of options.  There are blade servers, clusters, and all manner of scalable this and that. You simply buy the first bit and start using it, and if you ever need more, you just buy some more bits and plug them in and it all gets bigger.

    The problem I have with this sort of model is the price for those first bits. You aren’t simply paying for the disks.  You’re also paying for the ability to expand. This expansion capability is extremely important if your business has the chance of wild and uncontrolled growth (and wouldn’t we all like that), but most of us are running smaller businesses. We’re like pizza places, but instead of selling pizza, we’re selling services. We’d be happy to see our businesses growing at 20% year over year.

    When I think about servers and storage, I like to focus on what I expect to need this year, and what will likely get me through next year.  Beyond that, I should expect to refresh the entire system.  Even if I “could” double the storage capability, will I really want to? Will 6Gb SATA drives be fast enough for the new 4K codecs coming along in two years?  Will I want to spend “expansion capable” dollars on storage technology that’s two years old?

    My personal opinion is that things are changing far too quickly to buy for a horizon past two years, and if you really think you might need to expand that quickly, you should probably be buying that storage now rather than hoping to add on in six months or a year.

    Follow Steve Modica and Small Tree on Twitter @smalltreecomm.  Have a question? Contact Small Tree at 1-866-782-4622.