Pure CTO drills down on key-value, no DFMs on FA//ST, and quick object
On this podcast, recorded ultimately week’s Pure//Speed up 2025 occasion, we discuss to Pure Storage’s chief expertise officer (CTO), Rob Lee, to get a drill-down on use of key-value shops within the firm’s Purity flash storage working system, why it doesn’t use its much-trumpeted DirectFlash Modules within the newly introduced FlashArray//ST, and what makes its quick object storage so quick.
Are you able to clarify how a key-value retailer is utilized in Pure’s storage merchandise?
The rationale we use a key-value retailer, and I’ll get into the advantages, is loads of the identical causes that if you’re organising giant quantities of knowledge, you utilize a relational database.
You organise your info in a really orderly manner into tables. You’ll be able to construct indexes. You’ll be able to look issues up very effectively. You have a tendency to do this as an alternative of simply storing piles and piles of knowledge unorganised and making it very arduous to lookup.
Now, the genesis of utilizing key-value shops in our merchandise and software program goes again to day one of many firm.
One of many issues we did very in another way than all people else out there was we designed and rethought storage software program, inclusive of file methods and the way we map logical blocks to bodily places. We rethought the way you construct these for the way flash works on the most native stage. One of many key concerns with flash is, not like magnetic arduous disks, you don’t overwrite the contents in place.
To alter the contents, you must write a brand new copy and you must rubbish accumulate the outdated factor. And if you’re doing this, you need to keep away from rewriting the identical piece of flash time and again to burn out the media.
Effectively, it seems that by organising our metadata – which is successfully that mapping, if you’ll, of file names, file system to bodily location, in a key-value retailer, from the analysis neighborhood, there are many nice methods to minimise that write amplification – the variety of instances that now we have to rewrite that metadata construction and keep it over time.
In order that was the important thing – no pun supposed – perception driving us to organise our metadata in key-value shops.
The second piece although – and once more, borrowing from the database world and why you see so many key-value shops used at cloud scale – is that they make it actually, very easy to partition and distribute and create concurrency and parallelism.
And so if you have a look at FlashBlade and why FlashBlade is uniquely good at metadata efficiency – I talked a bit about this on stage [at Pure//Accelerate 2025] with FlashBlade EXA – is as a result of we retailer all that metadata in a key-value retailer that permits us to very, very linearly and with very excessive concurrency, scale out efficiency in a manner you merely can’t do with different information buildings.
To attract a quite simple comparability, traditionally, most storage methods have organised their information in a tree-like construction. Effectively, if you consider the way you look one thing up in a tree, you begin on the high, you go left, you go proper, you go left, you must comply with it step-by-step by step.
[It’s] very arduous to parallelise that, proper? With a key-value retailer, you possibly can reap the benefits of the media, the flash, parallel entry, you possibly can reap the benefits of our distributed expertise, and you’ll look stuff up with very excessive pace, with very excessive concurrency.
Does the key-value retailer come into operation solely on the metadata stage, and is the file system intact elsewhere, working in parallel with that?
What’s very nice about how we’ve constructed our software program is we use one method to managing the file system metadata, the person metadata, in addition to our extra bodily metadata, if you’ll. All the mappings between … as you understand, we do information discount, proper? Effectively, if you do information discount and you discover deduplication, you must hold a mapping that claims, “Oh, I don’t have this block bodily saved right here, there’s a separate copy over there.”
Effectively, that’s a mapping; we put that within the key-value retailer. So, by utilizing the identical method to managing all our metadata, effectively, A, it’s much less software program to put in writing, B, we will make that actually, actually sturdy and actually, actually performant, however then C, all the components of our system, whether or not it’s the file system, whether or not it’s our bodily media administration, get the advantages of the properties I simply mentioned.
Pure makes an enormous deal about using its DirectFlash Modules (DFMs) and the capability they will obtain, and but the newly introduced FlashArray//ST doesn’t use them. What does it use and why?
So it’s an incredible query, and look, I’ll begin with the philosophy now we have behind our {hardware}. I believe it was Steve Jobs who mentioned, if you wish to construct actually good software program, you’re going to construct {hardware} to assist that, and that’s actually form of our philosophy. In case you have a look at DirectFlash, what makes DFMs work is the software program that permits them.
We attempt to put little or no into the DFM {hardware}. It’s to allow the software program.
Now, the explanation why we’re not utilizing DFMs in FAST at the moment or the information path of FlashBlade EXA is, with any design, you design for a spread of the design area, efficiency, effectivity, price, and so forth, and our DFMs are very a lot designed for a really big selection of the enterprise wants when it comes to effectivity, capability, and so forth.
With FAST and with FlashBlade EXA, we’re aiming on the extremely, extremely excessive finish, high finish of efficiency, and we merely haven’t designed our drives for that high tier of extremely efficiency. There’s a broader ecosystem of {hardware} distributors that do have extra specialised merchandise for these components of the design area, and so the place it is sensible to faucet into that element market, we’re going to go do this.
What particularly do DFMs not have that you’ve got within the {hardware} that you simply’re utilizing there?
Effectively, once more, there’s no particular element that you simply’re going to go level to. It’s how have I optimised the design of the DFM and the way it’s used, versus buying and selling off latency, energy, area, capability, and so forth, and we simply haven’t optimised these for microsecond latencies as a result of, once more, they’re simply not that a part of the design area.
One other of the issues that Pure executives typically seek advice from is the power to offer very high-performance object storage. I’ve by no means actually heard an reason why that’s attainable. Is it only a case of chucking sources at it, or is there one thing else there?
Completely, and we really hit it within the first a part of the podcast. Plenty of it has to do with how we organise our metadata in a key-value retailer, and that drives a ton of efficiency for us.
If we stroll this again and we have a look at the legacy approaches to object storage, object storage within the enterprise has grown up within the period of low-cost and deep. Of us have tried to implement the item protocols usually on high of a file system, on high of an underlying block system.
You’ve got layers and layers and layers of inefficiency. So, primary is we implement an object natively. There usually are not layers and layers of performance-sucking inefficiency. After which quantity two is, if you happen to have a look at efficiency in two items, one is the metadata, the executive work, after which the information piece. Which means, I need to look one thing up, I’ve acquired a reputation, I’ve acquired to determine the place it’s, and as soon as I determine the place it’s, then I’ve the information piece of truly loading and transferring the information.
It seems with trendy object workloads, the executive piece, the metadata, finally ends up being a really giant portion of the general efficiency calls for. And once more, that’s the place our native method, not layering it on high of the file system and having a extremely distributed, extremely parallel key-value retailer, permits us to ship that efficiency. After which actually now we have a very quick information path. That’s since day one.
