RoHS CompliantAll Small Tree cards, switches and storage products are RoHS compliant!

  1. Storage (and Productivity) Innovations for 2015

    December 10, 2014 by Tom Jennings

    In today’s servers, the primary storage system is spinning disk connected to hardware RAIDs to achieve speed and redundancy. The limitation on performance deals with how many spindles you can get operating at the same time. With 10Gb Ethernet, there is usually a way to have the network output keep up with the storage. Processors and backplanes are not well utilized, as they continue to be much faster than the rotating storage.

    Over history, the question has been “what can I change to make the bottleneck be something else?” The problem is that a number of these pieces are balanced to match the rotating disk speed. RAID sets of 8 drives are balanced with the RAID being about as fast as the 8 spinning disks.

    To add speed, we add more RAIDs and more drive sets, but eventually you run out of slots without fully utilizing the processor or the backplane speed. The LSI RAID cards we uses are limited to two ports, with each port handling four drives. Even maxing out the two SAS ports with faster disks would limit us to 1.5 gigabytes per second per slot, so just replacing the drives with SSDs would gain us less than a 2X improvement.

    So where do we go now? If we replace the RAID with a 12Gb SATA or SAS host adaptor and change out the drives to SSDs, we start to see some improvement. Now we use ZFS to gain redundancy and provisioning. Gone are the 2-port limits, so we can connect enough SSDs to more efficiently use the PCI-Express bus, and we see the total throughput rise to 8GB per second.

    We now turn our attention to the output side. Where we had been using 10 gigabit Ethernet connections, we see 40Gb Ethernet connections making more sense. Now a twin 40Gb port card gets us 10GB per slot on the output side.

    With the increased bandwidth on both storage and networking, we see the backplane better utilized, and we start to see some stress on processor speed. Life is great, once again, except how do we get all that data out to users? All this does nothing for us if the user is still operating on a 1Gb Ethernet port over SMB2 or AFP.

    Let us look at the connection first. We see 40Gb Ethernet cards being available in add-on configuration, but how about laptops or Mac Pros? Enter Thunderbolt 3, which is due out next year, with enough bandwidth to handle a 40Gb connection. Hold that thought for a moment as we see opportunities for SMB3 with multi-path. With Thunderbolt 3 and SMB3, you could run multiple 10Gb connections to the switch or server, or run a pair of full 40Gb connections.

    In 2015, it looks like we’ll finally see server systems less defined by the number of spindles than by the number of cores or 40Gb connections.

    Tom Jennings is with Small Tree,

  2. My Hopes for IBC this Year

    August 28, 2014 by Steve Modica

    I’m heading out to IBC and there are a number of things I hope to see there. Of course, I’ve got customers asking me about SSDs, and engineers working on 40Gb Ethernet and people want to bring it all together. Really, what’s the hold up here?

    My short wish list:

    • 3.5” Server Chassis (8, 16, 24) with 12Gb SAS expanders onboard
    • 2.5” Server Chassis (12 and 24) with 12Gb SAS expanders onboard
    • Balanced 40Gb switches that can legitimately aggregate 16 or 24 10Gb ports into 4 or 6 40Gb ports
    • 4TB or larger SSDs that can handle enterprise workloads but cost less than $1000 per Terabyte
    • Thunderbolt 3 previews
    • 40Gb Ethernet Adapters
    • 8 or 10Terabyte 7200RPM SAS drives
    • New Wi-Fi technology that can run full duplex and offer backpressure and bandwidth reservation (can you imagine editing wirelessly?)

    Obviously, I have a few of these technologies in hand already, but there are some major roadblocks to building a balanced server with them. SSDs are very expensive and still too small. We’ll need those 400MB/sec devices to justify putting 40Gb ports in a server.

    Shifting gears, in September, we are going to be running a special at Small Tree. If you purchase a TitaniumZ (8 or 16), we’re giving away two SANLink2 10Gb Ethernet Adapters. Using the two onboard 10Gb ports on the Titanium, you can immediately connect two clients and be editing over 10Gb.

    Contact Small Tree today to purchase your TitaniumZ system with two Promise SANLink2 10Gb Ethernet Adapters included – or 866-782-4622. Purchase must be completed by 9/30/14.

  3. Data choke points and a cautionary tale

    August 14, 2014 by Steve Modica

    During a normal week, I help a lot of customers with performance issues. Some of the most common complaints I hear include:

    “I bought a new 10Gb card so I could connect my Macs together, but when I drag files over, it doesn’t go any faster.”

    “I upgraded the memory in my system because Final Cut was running slow, but it didn’t seem to help very much.”

    “I bought a faster Mac so it would run my NLE more smoothly, but it actually seems worse than before.”

    All of these things have something in common.  Money was spent on performance, the users didn’t have a satisfying experience, and they would be much happier had the money been spent in the right place.

    Of course, the first one is easy.  Putting a 10Gb connection between two Macs and dragging files between them isn’t going to go any faster than the slowest disk involved. If one of those Macs is using an old SATA spinning disk, 40-60MB/sec would be a pretty normal transfer rate.  A far cry from the 1000MB/sec you might expect from 10Gb Ethernet!  Who wouldn’t be disappointed?

    Similarly, the second case where a user upgrades memory based on an anecdotal suggestion of a friend is all too common.  On the one hand, memory upgrades are typically a great way to go, especially when you run a lot of things simultaneously. More memory almost always means better performance.  However, this is assuming that you didn’t have some other serious problem that was overwhelming your lack of memory.

    In the case of Final Cut 7, which is a 32 bit application, more memory isn’t going to help Final Cut directly.  In fact, it’s much more likely that Final Cut would run better with a faster disk and perhaps a faster CPU.  Since FCP 7 didn’t use GPU offload, even moving to a better graphics card might not have delivered a huge gain.

    The last one, where buying a faster Mac actually made things worse, is a classic case of mismatched performance tuning.  For this customer, the faster Mac also had a lot more memory.  It turns out that Mac OS X will dynamically increase the amount of data it will move across the network in a burst (the TCP Receive Window).  This resulted in the network overrunning Final Cut, causing it to stutter.  The solution?  Dial back the receive window to make sure FCP 7 can keep up.  This will be corrected by some other changes in the stack that are coming soon.  One day, slower applications will be able to push back on the sender a little more directly and a little more effectively than today.

    These cases bring to mind a discussion I had with a 40Gb Ethernet vendor back at NAB in April. They wanted me to use their cards and perhaps their switches. The obvious question:  Don’t your users want the speed of 40Gb Ethernet? Wouldn’t they want to run this right to their desktops?!

    Of course they would.  Everyone wants to go fast.  The problem is that those 40Gb ports are being fed by storage. If you look closely at what raid controllers and spinning disks can do, the best you can hope for from 16 drives and a raid card is around 1GB/sec.  A 40Gb cards moves about 4GB/sec. So if I sold my customers 40Gb straight to their desktops, I would need somewhere around 64 spinning disks just to max out ONE 40Gb port.  It could be done, but not economically. It would be more like a science project.

    Even worse, on Macs today, those 40Gb ports would have to connect with Thunderbolt 2, which tops out around 2.5GB/sec and is yet another choke point that would lead to disappointed customers and wasted money.

    I think 40Gb Ethernet has a place. In fact, we’re working on drivers today. However, that place will depend on much larger SSDs that can provide 1GB/sec per device.  Once we’re moving 8 and 16GB/sec either via a RAID card or ZFS logical volumes, then it will make sense to put 40Gb everywhere.  The added advantage is that waiting to deploy 40Gb will only lead to better and more stable 40Gb equipment. Anyone remember the old days of 10Gb back in 2003 when cards were expensive, super hot, and required single mode fiber?

  4. Snapshots…your trashcan, on steroids

    May 19, 2014 by Steve Modica

    I have to admit, as an old time UNIX guy that’s been around inodes, fsck and corrupted filesystems all my life, snapshots sounded a little too good to be true.

    The word was long known to me.  Customers would say, “I took a snapshot of that disk so I could upgrade it and revert if I screwed something up.”  It’s just that imaging a disk would take hours.  You’d start the copy and go home for the night.

    These new snapshots (like those supported by ZFS) were instantaneous.  One click and you would “instantly” have a new copy of your data.  How?  That’s not even possible.  To make it even weirder, the new copy takes up no space!?  Now it’s starting to sound like perpetual motion.

    The actual explanation is a lot simpler. Every filesystem is composed of data (your file data) and metadata (the name of the file, permissions, location of blocks, inode number, etc.).  All this metadata is what organizes your data.  You have what’s called an “inode table” where all that stuff lives, and it “points to” the actual data you wrote.  It might be video data, or your mom’s apple pie recipe.

    When you create a snapshot, you are instantly making a copy of that inode table.  You now have two. All these inodes point to the same data.  So the data was not copied.

    Now the magic happens. When a user deletes a file from the original data, the inode for that file is removed, but the snapshot inode remains.  ZFS will keep the data around as long as there’s an inode in some snapshot somewhere pointing to it.  The same is true if you edit a file.  The old data is saved, but the new data gets written.

    All this old stuff (old data) essentially becomes part of the snapshot.  As more things change, the snapshot grows larger. If you were to delete “all” the data on the original filesystem, the snapshot would essentially grow to the size of the original filesystem. (The original filesystem would drop to 0.)

    In some ways, it’s a little like a trashcan. When you delete something, it doesn’t really go away. It goes into the trash. If you wanted to, you could drag it out of the trash.

    There’s a similar way of recovering snapshots.  You simply “clone” (or mount) them.  When you do this, the snapshot inode table is mounted and it still points to all the old data.  That file you deleted yesterday?  If you mount yesterday’s snapshot, it’s right back where it was.  Simply drag it back out.

    Obviously, while snapshots make for a great method of saving previous images of a set of data, they are not a backup solution.  If your RAID dies and can’t be recovered, your snapshots die too!  So for true backup protection, consider rsync or some other method of moving your data to another system.

    Small Tree’s TitaniumZ servers support snapshots and rsync and we have a very nice graphical interface so you can manage it all yourself. If you have any questions about snapshots or a backup solution that’s right for your editing team, don’t hesitate to contact me at


  5. Thunderbolt Updates

    March 20, 2014 by Steve Modica

    We’ve been working pretty hard on Thunderbolt products over the last few weeks and I thought I’d write up some of the interesting things we’ve implemented.

    I’m sure most of you are aware that Thunderbolt is an external, hotplug/unplug version of PCIE.  Thunderbolt 1 provided a 4X PCIE bus along with an equivalent bus for graphics only. Thunderbolt 2 allows you to trunk those two busses for 8X PCIE performance.

    PCIE Pause

    This is a new feature of Thunderbolt designed to deal with the uncertainty of what a user may plug in.

    Normally, when a system boots up, all of the PCIE cards are in place. The system sorts out address space for each card and each driver is then able to map its hardware and initialize everything.

    In the Thunderbolt world, we can never be sure what’s going to be there.  At any time, a user could plug in not just one device, but maybe five!  They could all be sitting on their desk, daisy-chained, simply waiting for a single cable to install.

    When this happens, the operating system needs the capability to reassign some of the address space and lanes so other devices can initialize and begin working.

    This is where PCIE Pause comes into play.  PCIE Pause allows the system to put Thunderbolt devices into a pseudo sleep mode (no driver activity) while bus address space is reassigned. Then devices are re-awakened and can restart operations.  What’s important to note is that the hardware is “not” reset.  So barring the odd timing issue causing a dropped frame, a PCIE Pause shouldn’t even reset a network mount on a Small Tree device.

    Wake On Lan

    We’ve been working hard on a Wake On Lan feature.  This allows us to wake a machine from a sleep state in order to continue offering a service (like File sharing, ssh remote login or Screen sharing).  This may be important for customers wanting to use a Mac Pro as a server via Thunderbolt RAID and Network devices.

    The way it works is that you send a “magic” packet via a tool like “WakeonMac” from another system.  This tells the port to power up the system far enough to start responding to services like AFP.

    What’s interesting about the chip Small Tree uses (Intel x540) is that it requires power in order to watch for the “magic” wake up packet. Thunderbolt wants all power cut to the bus when the machine goes to sleep.  So there’s a bit of a conflict here.  Does a manufacturer violate the spec by continuing to power the device, or do they not support WOL?

    This is most definitely true for the early Thunderbolt/PCIE card cage devices.  They were all very careful to follow the Thunderbolt specification (required for certification and branding) and this leaves them missing this “powered while sleeping” capability.

    Interested in learning more about how you could be using Thunderbolt? Contact me at


  6. Testing with Adobe Anywhere

    March 7, 2014 by Steve Modica

    Small Tree has been working closely with Adobe to make sure our shared editing storage and networking products work reliably and smoothly with Adobe’s suite of content creation software.

    Since NAB 2013, we’ve worked closely with Adobe to improve interoperability and performance, and test new features to give our customers a better experience.

    Most recently, I had the chance to test out Adobe Anywhere in our shop in Minnesota.

    Adobe Anywhere is designed to let users edit content that might be stored in a high bandwidth codec, over a much slower connection link.  Imagine having HD or 4K footage back at the ranch, while you’re in the field accessing the media via your LTE phone and a VPN connection.

    The way it works is that there’s an Adobe Anywhere server sitting on your network that you connect to with Adobe Premiere and this server compresses and shrinks the data “on the fly” so it can be fed to your machine much like a YouTube video.  Except you are scrubbing, editing, cutting, dubbing and all of the other things you might need to do during an edit session.

    This real-time compression/transcoding happens because the Adobe Anywhere system is taking advantage of the amazing power of GPUs.  Except rather than displaying the video to a screen, the video is being pushed into a network stream that’s fed to your client.

    I tested my system out with some Pro Res promotional videos we’ve used at trade shows in the past, and did my editing over Wi-Fi.

    What I found was that the system worked very well.  I could see that the Adobe Anywhere system was reading the video from Small Tree’s shared storage at full rate, then pushing it to my system at a greatly reduced rate.  I had no trouble playing, editing and managing the video over my Wi-Fi connection (although Adobe recommends 1Gb Ethernet as the minimum connectivity for clients today).

    This type of architecture is very new and there are caveats.  For example, if you are very far from the server system or running over a very slow link (like a vpn connection), latency can make certain actions take a very long time (like loading an entire project, or using Adobe’s Titler app which requires interactivity).  Adobe cautions that latencies of 200msecs or more will lead to a very poor customer experience.

    Additionally, just because the feed to the clients is much lower bandwidth (to accommodate slower links), the original video data still needs to be read in real-time at full speed. So there are no shortcuts there.  You still need high quality, low latency storage to allow people to edit video from it. You just have a new tool to push that data via real-time proxies over longer and slower links.

    All in all, I found the technology to be very smooth and it worked well with Small Tree’s shared network storage.  I’m excited to see the reach of Small Tree shared storage extended out to a much larger group of potential users.

    For a demonstration of Adobe Anywhere over Small Tree shared storage, visit us at the NAB Show in Las Vegas this April (Booth SL11105).

  7. Another Couple of Reasons to Love SSDs

    February 26, 2014 by Steve Modica

    One day, when we’re sitting in our rocking chairs recounting our past IT glories (“Why, when I was a young man, computers had ‘wires’”), we’ll invariably start talking about our storage war stories.  There will be so many.  We’ll talk of frisbee tossing stuck disks or putting bad drives in the freezer. We’ll recount how we saved a company’s entire financial history by recovering an alternate superblock or fixing a byte swapping error on a tape with the “dd” command. I’m sure our children will be transfixed.

    No…no, they won’t be transfixed, any more than we would be listening to someone telling us about how their grandpa’s secret pot roast recipe starts with “Get a woodchuck…skin it.”  You simply have to be in an anthropological state of mind to listen to something like that. More likely, they walked into the room to ask you your wifi password (Of course, only us old folk will have wifi. Your kids are just visiting. At home they use something far more modern and futuristic. It’ll probably be called iXifi or something).

    Unfortunately for us, many of these war story issues remain serious problems today.  Disks “do” get stuck and they “do” often get better and work for a while if you freeze them. It’s a great way to get your data back when you’ve been a little lazy with backups.

    Another problem is fragmentation. This is what I wanted to focus on today.

    Disks today are still spinning platters with rings of “blocks” on them, where each block is typically 512 bytes. Ideally, as you write files to your disk, those bytes are written around the rings so you can read and write the blocks in sequence. The head doesn’t have to move.  Each new block spins underneath it.

    Fragmentation occurs because we don’t just leave files sitting on our disk forever. We delete them.  We delete emails, log files, temp files, render files, and old projects we don’t care about anymore. When we do this, those files leave “holes” in our filesystems. The OS wants to use these holes.  (Indeed, SGI used to have a real-time filesystem that never left holes. All data was written at the end.  I had to handle a few cases where people called asking why they never got their free space back when they deleted files.  The answer was “we don’t ever use old holes in the filesystem. That would slow us down!”)

    To use these holes, most operating systems use a “best fit” algorithm.  They look at what you are trying to write, and try to find a hole where that write will fit. In this way, they can use old space. When you’re writing something extremely large, the OS just sticks it into the free space at the end.

    The problem occurs when you let things start to fill up.  Now the OS can’t always find a place to put your large writes. If it can’t, it may have to break that large block of data into several smaller ones. A file that may have been written in one contiguous chunk may get broken into 11 or 12 pieces.  This not only slows down your write performance, it will also slow down your reads when you go to read the file back.

    To make matters worse, this file will remain fragmented even if you free more space up later. The OS does not go back and clean it up.  So it’s a good idea not to let your filesystems drop below 20% free space. If this happens and performance suffers, you’re going to need to look into a defragmentation tool.

    Soon, this issue won’t matter to many of us.  SSDs (Solid State Disks) fragment just like spinning disks, but it doesn’t matter near as much.  SSDs are more like Random Access Memory in that data blocks can be read in any order, equally as fast. So even though your OS might have to issue a few more reads to pull in a file (and there will be a slight performance hit), it won’t be near as bad as what a spinning disk would experience.  Hence, we’ll tell our fragmentation war stories one day and get blank looks from our grandkids  (What do you mean “spinning disk?”  The disk was “moving??”).

    Personally, I long for the days when disk drives were so large, they would vibrate the floor. I liked discovering that the night time tape drive operator was getting hand lotion on the reel to reel tape heads when she put the next backup tape on for the overnight runs. It was like CSI. I’m going to miss those days. Soon, everything will be like an iPhone and we’ll just throw it away, get a new one, and sync it with the cloud.  Man that sucks.

    Follow Steve Modica and Small Tree on Twitter @smalltreecomm.  Have a question? Contact Small Tree at 1-866-782-4622.