-->

We’re Building the Future of Data Infrastructure

Products
Company
Support

Archive for the ‘Fibre Channel’ Category

August 27th, 2020

How to Reap the Benefits of NVMe over Fabric in 2020

By Todd Owens, Technical Marketing Manager, Marvell

As native Non-volatile Memory Express (NVMe®) share-storage arrays continue enhancing our ability to store and access more information faster across a much bigger network, customers of all sizes – enterprise, mid-market and SMBs – confront a common question: what is required to take advantage of this quantum leap forward in speed and capacity?

Of course, NVMe technology itself is not new, and is commonly found in laptops, servers and enterprise storage arrays. NVMe provides an efficient command set that is specific to memory-based storage, provides increased performance that is designed to run over PCIe 3.0 or PCIe 4.0 bus architectures, and — offering 64,000 command queues with 64,000 commands per queue — can provide much more scalability than other storage protocols.

A screenshot of a cell phone

Description automatically generated

Unfortunately, most of the NVMe in use today is held captive in the system in which it is installed. While there are a few storage vendors offering NVMe arrays on the market today, the vast majority of enterprise datacenter and mid-market customers are still using traditional storage area networks, running SCSI protocol over either Fibre Channel or Ethernet Storage Area Networks (SAN).

The newest storage networks, however, will be enabled by what we call NVMe over Fabric (NVMe-oF) networks. As with SCSI today, NVMe-oF will offer users a choice of transport protocols. Today, there are three standard protocols that will likely make significant headway into the marketplace. These include:

  • NVMe over Fibre Channel (FC-NVMe)
  • NVMe over RoCE RDMA (NVMe/RoCE)
  • NVMe over TCP (NVMe/TCP)

If NVMe over Fabrics are to achieve their true potential, however, there are three major elements that need to align. First, users will need an NVMe-capable storage network infrastructure in place. Second, all of the major operating system (O/S) vendors will need to provide support for NVMe-oF. Third, customers will need disk array systems that feature native NVMe. Let’s look at each of these in order.

  1. NVMe Storage Network Infrastructure

In addition to Marvell, several leading network and SAN connectivity vendors support one or more varieties of NVMe-oF infrastructure today. This storage network infrastructure (also called the storage fabric), is made up of two main components: the host adapter that provides server connectivity to the storage fabric; and the switch infrastructure that provides all the traffic routing, monitoring and congestion management.

For FC-NVMe, today’s enhanced 16Gb Fibre Channel (FC) host bus adapters (HBA) and 32Gb FC HBAs already support FC-NVMe. This includes the Marvell® QLogic® 2690 series Enhanced 16GFC, 2740 series 32GFC and 2770 Series Enhanced 32GFC HBAs.

On the Fibre Channel switch side, no significant changes are needed to transition from SCSI-based connectivity to NVMe technology, as the FC switch is agnostic about the payload data. The job of the FC switch is to just route FC frames from point to point and deliver them in order, with the lowest latency required. That means any 16GFC or greater FC switch is fully FC-NVMe compatible.

A key decision regarding FC-NVMe infrastructure, however, is whether or not to support both legacy SCSI and next-generation NVMe protocols simultaneously. When customers eventually deploy new NVMe-based storage arrays (and many will over the next three years), they are not going to simply discard their existing SCSI-based systems. In most cases, customers will want individual ports on individual server HBAs that can communicate using both SCSI and NVMe, concurrently. Fortunately, Marvell’s QLogic 16GFC/32GFC portfolio does support concurrent SCSI and NVMe, all with the same firmware and a single driver. This use of a single driver greatly reduces complexity compared to alternative solutions, which typically require two (one for FC running SCSI and another for FC-NVMe).

If we look at Ethernet, which is the other popular transport protocol for storage networks, there is one option for NVMe-oF connectivity today and a second option on the horizon. Currently, customers can already deploy NVMe/RoCE infrastructure to support NVMe connectivity to shared storage. This requires RoCE RDMA-enabled Ethernet adapters in the host, and Ethernet switching that is configured to support a lossless Ethernet environment. There are a variety of 10/25/50/100GbE network adapters on the market today that support RoCE RDMA, including the Marvell FastLinQ® 41000 Series and the 45000 Series adapters. 

On the switching side, most 10/25/100GbE switches that have shipped in the past 2-3 years support data center bridging (DCB) and priority flow control (PFC), and can support the lossless Ethernet environment needed to support a low-latency, high-performance NVMe/RoCE fabric.

While customers may have to reconfigure their networks to enable these features and set up the lossless fabric, these features will likely be supported in any newer Ethernet switch or director. One point of caution: with lossless Ethernet networks, scalability is typically limited to only 1 or 2 hops. For high scalability environments, consider alternative approaches to the NVMe storage fabric.

One such alternative is NVMe/TCP. This is a relatively new protocol (NVM Express Group ratification in late 2018), and as such is not widely available today. However, the advantage of NVMe/TCP is that it runs on today’s TCP stack, leveraging TCP’s congestion control mechanisms. That means there’s no need for a tuned environment (like that required with NVMe/RoCE), and NVMe/TCP can scale right along with your network. Think of NVMe/TCP in the same way as you do iSCSI today. Like iSCSI, NVMe/TCP will provide good performance, work with existing infrastructure, and be highly scalable. For those customers seeking the best mix of performance and ease of implementation, NVMe/TCP will be the best bet.

Because there is limited operating system (O/S) support for NVMe/TCP (more on this below), I/O vendors are not currently shipping firmware and drivers that support NVMe/TCP. But a few, like Marvell, have adapters that, from a hardware standpoint, are NVMe/TCP-ready; all that will be required is a firmware update in the future to enable the functionality. Notably, Marvell will support NVMe over TCP with full hardware offload on its FastLinQ adapters in the future. This will enable our NVMe/TCP adapters to deliver high performance and low latency that rivals NVMe/RoCE implementations.

A screenshot of a cell phone

Description automatically generated
  • Operating System Support

While it’s great that there is already infrastructure to support NVMe-oF implementations, that’s only the first part of the equation. Next comes O/S support. When it comes to support for NVMe-oF, the major O/S vendors are all in different places – see the table below for a current (August 2020) summary. The major Linux distributions from RHEL and SUSE support both FC-NVMe and NVMe/RoCE and have limited support for NVMe/TCP. VMware, beginning with ESXi 7.0, supports both FC-NVMe and NVMe/RoCE but does not yet support NVMe/TCP. Microsoft Windows Server currently uses an SMB-direct network protocol and offers no support for any NVMe-oF technology today.

With VMware ESXi 7.0, be aware of a couple of caveats: VMware does not currently support FC-NVMe or NVMe/RoCE in vSAN or with vVols implementations. However, support for these configurations, along with support for NVMe/TCP, is expected in future releases.

  • Storage Array Support

A few storage array vendors have released mid-range and enterprise class storage arrays that are NVMe-native. NetApp sells arrays that support both NVMe/RoCE and FC-NVMe, and are available today. Pure Storage offers NVMe arrays that support NVMe/RoCE, with plans to support FC-NVMe and NVMe/TCP in the future. In late 2019, Dell EMC introduced its PowerMax line of flash storage that supports FC-NVMe. This year and next, other storage vendors will be bringing arrays to market that will support both NVMe/RoCE and FC-NMVe. We expect storage arrays that support NVMe/TCP will become available in the same time frame.

Future-proof your investments by anticipating NVMe-oF tomorrow

Altogether, we are not too far away from having all the elements in place to make NVMe-oF a reality in the data center. If you expect the servers you are deploying today to operate for the next five years, there is no doubt they will need to connect to NVMe-native storage during that time. So plan ahead.

The key from an I/O and infrastructure perspective is to make sure you are laying the groundwork today to be able to implement NVMe-oF tomorrow. Whether that’s Fibre Channel or Ethernet, customers should be deploying I/O technology that supports NVMe-oF today. Specifically, that means deploying 16GFC enhanced or 32GFC HBAs and switching infrastructure for Fibre Channel SAN connectivity. This includes the Marvell QLogic 2690, 2740 or 2770-series Fibre Channel HBAs. For Ethernet, this includes Marvell’s FastLinQ 41000/45000 series Ethernet adapter technology.

These advances represent a big leap forward and will deliver great benefits to customers. The sooner we build industry consensus around the leading protocols, the faster these benefits can be realized.

For more information on Marvell Fibre Channel and Ethernet technology, go to www.marvell.com. For technology specific to our OEM customer servers and storage, go to www.marvell.com/hpe or www.marvell.com/dell.

August 18th, 2020

From Strong Awareness to Decisive Action: Meet Mr. QLogic

By Nishant Lodha, Director of Product Marketing – Emerging Technologies, Marvell

Marvell® Fibre Channel HBAs are getting a promotion and here is the announcement email –

I am pleased to announce the promotion of “Mr. QLogic® Fibre Channel” to Senior Transport Officer, Storage Connectivity at Enterprise Datacenters Inc. Mr. QLogic has been an excellent partner and instrumental in optimizing mission critical enterprise application access to external storage over the past 20 years. When Mr. QLogic first arrived at Enterprise Datacenters, block storage was in a disarray and efficiently scaling out performance seemed like an unsurmountable challenge. Mr. QLogic quickly established himself as a go-to leader and trusted partner for enabling low latency access to external storage across disk and flash. Mr. QLogic successfully collaborated with other industry leaders like Brocade and Mr. Cisco MDS to lay the groundwork for a broad set of innovative technologies under the StorFusion™ umbrella. In his new role, Mr. QLogic will further extend the value of StorFusion by bringing awareness of Storage Area Network (SAN) congestion into the server, while taking decisive action to prevent bottlenecks that may degrade mission critical enterprise application performance.

Please join me in congratulating QLogic on this well-deserved promotion.

Soon after any big promotion, reality sets in and everyone asks what you are going to do for them and how are you going to add value in your new role. Will you live up to the expectations?

Let’s take a journey together (virtually) in this three-part blog and find out how Mr. QLogic delivers!

Part 1: Heterogeneous SANs and Flash bring in new challenges

In the era of rapid digitalization, work from home, mobility and data explosion, increasing numbers of organizations rely on a robust digital infrastructure to sustain and grow their businesses. Hosted business-critical applications must perform at high capacity and in a predictable manner. Fibre Channel remains the connection of choice between server applications and storage array data. FC SANs must remain free of congestion so workloads can perform at their peak or risk business interruption due to stalled or poorly performing applications.

Fibre Channel is known for its ultra-reliability since it is implemented on a dedicated network with buffer-to-buffer credits. For a real-life parallel, think of a guaranteed parking spot at your destination, and knowing it’s there before you leave your driveway. That worked well between two directly connected devices and until the recent past, congestion in FC SANs was generally an isolated problem. However, SAN Congestion has the potential to become a more severe problem occurring at more places in the datacenter for the following reasons:

  • Heterogenous SAN Speeds: While datacenters have seen widespread deployment of 16GFC and 32GFC, 4/8GFC still remain part of the fabric. Servers and Storage FC ports operating at mismatched speeds tend to congest the links between them.
  • CapEx and OpEx Pressure: Amidst the global pandemic, businesses are under unprecedented pressure to optimize OpEx and increase their bottom lines. More applications/VMs are being hosted on the same servers increasing stress onto existing SANs causing previously balanced SANs to be prone to congestion.
  • Flash Storage: FC SANs are no longer limited in performance due to spinning media. With the advent of NVMe and FC-NVMe, SANs are being pushed to their limits and existing links (4/8GFC) may not be able to sustain AFA bandwidth, creating oversubscription scenarios that lead to congestion and congestion spreading.
  • Data Explosion: Datasets and databases are growing in scale and so are the Fibre Channel SANs that support these applications. Scale out SAN architectures, more ports and end points in a domain, mean that a singular congestion event can spread to and impact a wide set of applications and services.

FC SANs are lossless networks and all frames sent must be acknowledged by the receiver. The sender (e.g. a Storage Array) will stop sending frames if these acknowledgments are not received. Simply put, the inability of a receiver (e.g. FC HBA) to accept frames at the expected rate results in congestion. Congestion can occur due to misbehaving end devices, physical errors or oversubscription. Oversubscription due to the reasons outline above is typically the main culprit.

A screenshot of a cell phone

Description automatically generated

Introducing the “Aware and Decisive” FC HBAs

Marvell’s QLogic Fibre Channel HBAs (aka. Mr. QLogic) and its StorFusion set of end-to-end orchestration and management capabilities are being extended to build a solution that is “aware” of the performance conditions of the SAN fabric and “decisive” in terms of the actions that it can take to prevent and mitigate the conditions like congestion, that can degrade the fabric and thus application performance.

Marvell QLogic Universal SAN Congestion Mitigation (USCM) technology works independently and in coordination with Brocade and Cisco FC fabrics to mitigate SAN congestion by enabling congestion detection, notification, and avoidance.  QLogic congestion “Awareness” capability means that it can either be informed of or automatically detect signs of congestion. QLogic “Decisive” action capability intends to prevent or resolve congestion by either switching traffic to a more optimized path or quarantining slower devices to lower priority virtual channels.

Available immediately, QLogic Enhanced 16GFC (2690 Series) and Enhanced 32GFC (2770 Series) Adapters have the ability to deliver a wide range of these SAN Congestion Management capabilities.

Preview of Part 2 and Part 3

If you think Mr. QLogic is up to something here and has the right vision, motivation and expertise to rescue FC SANs from Congestion, then come back again for the sequel(s).

In Part 2, I will talk about the underlying industry standards-based technology called “Fabric Performance Notification” (FPINs) that form the heart of the QLogic USCM solution.

In Part 3, you will see the technology in action and the uniqueness of the Marvell QLogic solution – for it is the one that can deliver on congestion management for both Brocade and Cisco SAN Fabrics.

August 12th, 2020

Put a Cherry on Top! Introducing FC-NVMe v2

By Nishant Lodha, Director of Product Marketing – Emerging Technologies, Marvell

Once upon a time, data centers confronted a big problem – how to enable business-critical applications on servers to access distant storage with exceptional reliability. In response, the brightest storage minds invented Fibre Channel. Its ultra-reliability came from being implemented on a dedicated network and buffer-to-buffer credits. For a real-life parallel, think of a guaranteed parking spot at your destination, and knowing it’s there before you leave your driveway. That worked fairly well. But as technology evolved and storage changed from spinning media to flash memory with NVMe interfaces, the same bright minds developed FC-NVMe. This solution delivered a native NVMe storage transport without necessitating rip-and-replace by enabling existing 16GFC and 32GFC HBAs and switches to do FC-NVMe. Then came a better understanding of how cosmic rays affect high-speed networks, occasionally flipping a subset of bits, introducing errors.

Challenges in High-Speed Networks

But cosmic rays aren’t the only cause of bit errors. Soldering issues, vibrations due to heavy industrial equipment, as well as hardware and software bugs are all problematic, too. While engineers have responded with various protection mechanisms – ECC memories, shielding and FEC algorithms, among others – bit errors and lost frames still afflict all modern high-speed networks. In fact, the higher the speed of the network (e.g. 32GFC), the higher the probability of errors. Obviously, errors are never good; detecting them and re-transmitting lost frames slows down the network. But things get worse when the detection is left to the upper layers (SCSI or NVMe protocols) that must overcome their own quirks and sluggishness. Considering the low latency and high resiliency data center operators expect from FC-NVMe, these errors must be handled more efficiently, and much earlier in the process. In response, industry leaders like Marvell QLogic, along with other Fibre Channel leaders, came together to define FC-NVMe v2.

Introducing FC-NVMe v2

Intending to detect and recover from errors in a manner befitting a low-latency NVMe transport, FC-NVMe v2 (standardized by the T11 committee in August 2020) does not rely on the SCSI or NVMe layer error recovery. Rather, it automatically implements low-level error recovery mechanisms in the Fibre Channel’s link layer – solutions that work up to 30x faster than previous methods. These new and enhanced mechanisms include:

  • FLUSH: A new FC-NVMe link service that can quickly determine if a sent frame does not quickly reach its destination. It works this way: if two seconds pass without the QLogic FC HBA getting a response back regarding a transmitted frame, it sends a FLUSH to the same destination (like sending a second car to the same parking spot, to see if is occupied). If the FLUSH gets to the destination, we know that the original frame went missing en route, and the stack does not need to wait the typical 60 seconds to detect a missing frame (hence the 30x faster).
  • RED: Another new FC-NVMe link service, called Responder Error Detected (RED), essentially does the same lost frame detection but in the other direction. If a receiver knows it was supposed to get something but did not, it quickly sends out a RED rather than waiting on the slower, upper-layer protocols to detect the loss.
  • NVMe_SR: Once either FLUSH or RED detects a lost frame, NVMe_SR (NVMe Retransmit) kicks in, and enables the re-transmission of whatever got lost the first time.

Complicated? Not at all — these work in the background, automatically. FLUSH, RED, and NVMe_SR are the cherries on top of great underlying technology to deliver FC-NVMe v2!

Marvell QLogic FC HBAs with FC-NVMe v2

Marvel leads the T11 standards committee that defined the FC-NVMe v2 standard, and later this year will enable support for FC-NVMe v2 in QLogic 2690 Enhanced 16GFC and 2770 Enhanced 32GFC HBAs. Customers should expect a seamless transition from FC-NVMe v2 via a simple software upgrade, fulfilling our promise to avoid a disruptive rip-and-replace modernization of existing Fibre Channel infrastructure.

So, for your business-critical applications that rely on Fibre Channel infrastructure, go for Marvell QLogic FC-NVMe v2, and shake off the sluggish error recovery, and do more with Fibre Channel! Learn more at Marvell.com.