Archive | as a Services RSS feed for this section

PaaS Element Types

8 Dec PaaS Element Types

Please Note : This post builds directly on the previous post “A viable PaaS Model“

What are PaaS Element Types?

PaaS Element Types are the constructs required to build a PaaS.  Each PaaS Element Type builds upon the previous, I’m not the first to come up with the overall concept of Types building upon one another, this was inspired by Data Types from Software Development.  It is important to understand that all PaaS Element Types end up being abstractions built upon one another.

Why are they important?

As we move to the new paradigm of application development, architecture, and management, applications will be the comprised of these new element types vs. the traditional patterns and designs we have all become used to.  By fully understanding PaaS Element Types, you will gain not only an understanding of how different PaaS solutions work, but what their capabilities, strengths, and weaknesses are and why they have these characteristics.

Compute, Networking, Storage, what about Memory?

You may have noticed that Memory is not listed as a column, why is that?  Storage is meant to encompass all forms of storage, which includes Memory (Memory is just a very fast form of volatile storage, just as an L1 Cache is even faster than standard memory it is all for storing bits and bytes – albeit briefly).  All of the PaaS Elements have the ability to leverage memory for whatever purpose they may need, but I see no reason to separate memory in the Element Types.

Primitives as defined in the previous blog post

Primitives are the Core Building Blocks of Resources.  What does this mean?  Primitives cannot be reasonably reduced to a more basic/granular bit of functionality.  You may be thinking that an Operating System could most certainly be reduced to a more simplistic bit of functionality and you would be correct if we were talking about us in a traditional Infrastructure, but not PaaS.  Part of the magic of PaaS is the prescriptive nature that PaaS brings, along with the obfuscation in most cases of components such as the Operating System.  In the case of the Operating System, it is also important to recognize that as a Primitive, the OS has not actually been instantiated (it isn’t a running OS).  One final note is that PaaS eliminates nearly all direct ties between code and the OS (There are still limits imposed by the runtimes, etc. which are difficult to avoid – e.g. Mono for .Net support on Linux)

Sophisticates : Composites / Combinations / Extensions of Building Blocks (Primitives)

Sophisticates are built from Primitives, meaning that a Sophisticate cannot exist unless it is backed by a Primitive with some addition, change, or a second PaaS Element Type such as a Primitive or Sophisticate.  A Sophisticate could be built from a combination of another Sophisticate and a Primitive!  Let’s take RDBMS as an example (note that this would just as easily apply to NoSQL solutions, etc).  RDBMS will likely leverage a RuntimeVM, an Operating System, Processes, an Interface, Block Store, Cache, and a File System.  While this may be complex it is hidden by the RDBMS Interfaces, which is part of the beauty of PaaS is exposing this combination as a Service or Extension that can be consumed by an API call or a DB connection.

Definitives : Instantiations (Running) of Primitives and Sophisticates either directly in use or wrapped in Services/APIs creating easily leverage abstractions.

These abstractions allow complex configurations of Primitives and Sophisticates coupled with Application Logic, Dynamic Configuration capabilities, and more.  Definitives are the live and fully implemented abstractions as they consume resources.  This could be a Schema in a RDBMS or a Collection with Documents in it in a Document Store.  Definitives are where all of the specificity occurs in PaaS and ultimately what differentiates applications from each other once they are running.

What can you do with all of this stuff?

If you use the prescriptive side (what a developer writing to a running PaaS offering would do) of PaaS and simply consume it, you can quickly and without a deep understand deploy applications within the limits of the PaaS App Space.  Alternatively you can design your own PaaS with whatever capabilities you require to support the behaviors and design patterns that you need for your App Space.  This is done by using the PaaS Element Types to make choices on the Control Space, which in turn creates the boundaries of the PaaS App Space.

Control Spaces share some characteristics of the App Spaces that they provide, this is because the Control Space operates on the same infrastructure that the App Space operates on.  This is true in the majority of PaaS cases that I have seen to date (it could at least in theory, change).  Control Space components are Definitives built from Primitives and Sophisticates to provide the prescriptive approach to the App Space that makes PaaS an attractive alternative to traditional software builds, configuration, deployments, etc.

The work in mapping Primitives, Sophisticate, and Definitives is not yet finished!

What I have provided is the beginning, not the end.  More work needs to be done in adding additional examples of PaaS Element Types, along with mapping all of the current PaaS offerings to both the PaaS Model and what Element Types are used/comprise each PaaS offering.  In future blog posts with help from different collaborators I hope to accurately map the major PaaSes to the PaaS Model and their PaaS Element Types.

 Comment here on the blog or ping me on Twitter (@mccrory) with your thoughts and ideas.

A viable PaaS Model

7 Dec

What makes a PaaS a PaaS?

I’ve seen many discussions on blogs and twitter around this topic, so much so that many people are tired of talking about it because it always leads to cyclical discussions.  I for one haven’t been satisfied with any of the answers that I have seen.  Some people try to define PaaS with requirements such as it must be on demand, while others say that an API and services need to be exposed.  I disagree with these requirements/constraints on describing PaaS.  I think there is a model should be applied to define what is PaaS, to make this model credible there is a necessity that it can be mapped to most (if not all) current and future offerings defined as PaaS (Which will be a follow-on series to this post).

The PaaS Model

The PaaS Model is made up of two different constructs, which are called Spaces.  These two Spaces serve different purposes, but are composed of the same PaaS Element Types (elements will be explained in detail, in an App Space deep-dive follow-on post).  The Control Space and App Space are show in the diagram below, notice that the App Space is fully wrapped/contained within the Control Space.

Control Space:

The Control Space performs all of the automation, management, provisioning that is required by the PaaS.  Interaction with other lower level components such as an Orchestrator is achieved through API abstractions if/when necessary. The Control Space and its implementation also determine what elements are allowed/exposed to the App Space. Further, the Control Space is responsible for maintaining App Space coherency and dependencies.  While the Control Space is comprised of several separate functions, they may be combined in different manners depending on the specific PaaS implementation.  All of this will also commonly be exposed through one or more API interfaces (this however, isn’t necessarily a requirement in the model presently).

App Space:

The App Space is where end-user/customer applications are deployed, updated, and run.  The App Space is controlled (and commonly coordinated) by the Control Space.  The exposure of PaaS Element types by the Control Space to the App Space is one of the key differentiating factors between different PaaS implementations.  App Space characteristics are controlled by how the Control Space is built/designed along with what PaaS Elements were used to build the Control Space.

  • App Network: The path which applications communicate with app each other and services/resources exposed to apps.  The App Network exposes Network connectivity to the App Space.
  • Executor: The application bootstrapping mechanism for apps being or currently deployed.  The executor provides compute/memory resources to the App Space.
  • Code Processor: Examines code, libraries, and dependencies before sending to the Engine and/or Executor.  This can also be thought of as a code inspector or post/pre-processor.
  • Coordination Network: Where the Control Space components communicate/coordinate with each other.  Think of this as a management network that is in most cases out-of-band from the App Network.  The Coordination and App Networks could be combined, however in a production system this would introduce too large of a security risk.
  • Engine: Coordinates the distribution and provisioning of code, services, and their dependencies (most frequently in the Control Space).  The Engine decides the where and how of what happens in the App Space.  Also, the Engine may be capable of coordinating with an Orchestration Layer or other automation tools outside of the Control Space to provide new/additional resources to either/both the Control Space and the App Space.
  • Monitor: Looks at the state of the App Space and the Control Space, signaling other Control Space components to resolve conflicts.  The component most likely to resolve conflicts would be the Engine or a specialized component designed purely for conflict resolution.
  •  Notes: These components/functions may be grouped differently based on the specific PaaS implementation.  Also, some of the functionality or components could be put into an API, split into sub-components, or even externalized through/in a Client or Client side API.

PaaS Elements:

PaaS Elements are abstractions on top of different layers of resources.  Most PaaS Element abstractions are done through Service based or Service centric abstractions.  This is not the only way of creating/doing an abstraction, but it is one of the most flexible ways.  PaaS Elements are broken up into three primary types which are defined below:

Please note there will be an in-depth post on PaaS Elements following this blog post.

  • Primitives : The Core Building Blocks of Resources
  • Sophisticates : Composites / Combinations / Extensions of Building Blocks (Primitives)
  • Definitives : Instantiations of Primitives and Sophisticates either directly in use or wrapped in Services/APIs

All PaaS Elements have the ability to provide or interact through a pass-through interface or abstraction to any other PaaS Element type all provided by the Control Space.  Elements are combined to create an Application (App) in the App Space.  The most interesting twist to the model is that the Control Space is made up entirely of PaaS Elements as well.  There is a great deal more to be covered on this topic both here on this blog and out in the community.  Please provide feedback and questions in the comments section of this post or live on Twitter (@mccrory)

I would like to give a special thanks to Shlomo Swidler (@shlomoswidler) and Derek Collison (@derekcollison) for providing valuable feedback in several draft iterations of this post.

Defying Data Gravity

2 Apr

How to Defy Data Gravity

Since I have changed companies, I have been incredibly busy as of late and my blog has had the appearance of neglect.  At a minimum I was trying to do a post or two per week.  The tempo will be changing soon to move closer to this….

As a first taste of what will be coming in a couple of weeks I thought I would talk a bit about something I have been thinking a great deal about.

Is it possible to defy Data Gravity?

First a quick review of Data Gravity:

Data Gravity is a theory around which data has mass.  As data (mass) accumulates, it begins to have gravity.  This Data Gravity pulls services and applications closer to the data.  This attraction (gravitational force) is caused by the need for services and applications to have higher bandwidth and/or lower latency access to the data.

Defying Data Gravity, how?

After considering how this might be possible, I believe that the following strategies/approaches could make it feasible to come close to Defying Data Gravity.

All of the bullets below could be leveraged to assist in defying Data Gravity, however they all have both pros and cons.  The strengths of some of the patterns and technologies can be weaknesses of others, which is why they are often combined in highly available and scalable solutions.

All of the patterns below provide an abstraction or transformation of some type to either the data or the network:

  • Load Balancing : Abstracts Clients from Services, Systems, and Networks from each other
  • CDNs : Abstract Data from it’s root source to Network Edges
  • Queueing (Messaging or otherwise) : Abstracts System and Network Latency
  • Proxying : Abstracts Systems from Services (and vice versa)
  • Caching : Abstracts Data Latency
  • Replication : Abstracts Single Source of Data (Multiplies the  Data i.e. Geo-Rep or Clustering)
  • Statelessness : Abstracts Logic from Data Dependencies
  • Sessionless : Abstracts the Client
  • Compression (Data/Indexing/MapReduce) : Abstracts (Reduces) the Data Size
  • Eventual Consistency : Abstracts Transactional Consistency (Reduces chances of running into Speed of Light problems i.e. Locking)

So to make this work, we have to fake the location and presence of the data to make our services and applications appear to have all of the data beneath them locally.  While this isn’t a perfect answer, it does give the ability to move less of the data around and still give reasonable performance.  Using the above patterns allows for the movement of an Application and potentially the services and data it relies on from one place to another – potentially having the effect of Defying Data Gravity.  It is important to realize that the stronger the gravitational pull and the Service Energy around the data, the less effective any of these methods will be.

Why is Defying Data Gravity so hard?

The speed of light is the answer.  You can only shuffle data around so quickly, even using the fastest networks, you are still bound by distance, bandwidth, and latency.  All of these are bound by time, which brings us back to the speed of light.  You can only transfer so much data across the distance of your network, so quickly (in a perfect world, the speed of light becomes the limitation).

The many methods explained here are simply a pathway to portability, but without standard services, platforms, and the like even with the patterns etc. it becomes impossible to move an Application, Service, or Workload outside of the boundaries of its present location.

A Final Note…

There are two ways to truly Defy Data Gravity (neither of which is very practical):

Store all of your Data Locally with each user and make them responsible for their Data

If you want to move, be willing to accept downtime (this could be minutes to months) and simply store off all of your data and ship it somewhere else.  This method would work now matter how large the data set as long as you don’t care about being down.

IaaS and SaaS lead to PaaS

13 Feb

Most people currently portray the Public Clouds as a stack consisting of IaaS, PaaS, and SaaS.  Worse still, they try to define Private Clouds in the same way. In a discussion I had with Christofer Hoff – @Beaker he accurately pointed out that this isn’t entirely the case, that they are all really Integrations.  This blog post was about three sentences in when a discussion broke out on Twitter regarding IaaS and ultimately where IaaS and SaaS converge – hint: PaaS.

IaaS and SaaS lead to PaaS

IaaS is becoming established by many Public Cloud providers by providing Virtual Machines, Block Based and File Based Storage, Databases and Key Value Stores, and several other core capabilities. As these capabilities become more sophisticated and their complexity increases, the drive to abstract and re-simplify pushes IaaS toward PaaS. IaaS is beginning to also take root in Private Clouds (Enterprises) where there is a motivation for more automation and self-service in an attempt to drive more efficiency. What Private Cloud providers (I’m thinking of IT) will find is that they are unable to achieve the same levels of efficiency that Public Clouds can. The difference between Public Cloud IaaS and Private Cloud IaaS efficiency isn’t entirely based on scale as you might initially think. The disparity is also driven by the storage philosophies that they follow or rather the difference.

Public Cloud IaaS is driven by very cheap and very scalable storage using software based redundancy with many replicas of data (think multiple copies on JBOD). Private Cloud IaaS in contrast is driven by highly available, fault tolerant, hardware redundant disk arrays (think SANs). This difference in storage methodology translates into a difference in Platform Methodology, Applications Methodology, and Operations Methodologies. Data is king and the assumptions that are made around how data is persisted and retrieved drives decisions about platforms, applications, and operations.

When Private Clouds embrace some of the newer approaches in storage, they will also embrace change in their Platform, this is where PaaS enters the picture in a big way. PaaS will also be pushed by applications developers because of the increases in productivity and efficiency that it brings to Enterprise Development Organizations. The changes to the Enterprise Development Orgs mean changes to Application Design Methodologies. These changes will drive IT to make changes as well because previous Operations Methodologies will no longer work (I’m looking at you traditional ITILv3). There will also be deep security implications to all of these changes, however this will neither make security better or worse, just applied differently than it is presently.

We’ve covered the IaaS angle and how it pushes toward a PaaS centric world, but what about SaaS? I find it interesting that SaaS has been with us for 10 years now, yet many people are just realizing its value. The defacto example of SaaS is Salesforce.com. Virtually every article that talks broadly about SaaS mentions them, mainly because almost everyone has heard of Salesforce and has probably interacted with it at some point in their career. Public Cloud SaaS has several issues that it struggles with, two of which are the need to maintain a shared infrastructure in order to keep costs low but still keep customer data separate. The second issue being the need allow customers to customize the solution to meet their needs (this could be through integrations, add-ons, or other applications). Making this work if you are simply delivering pure software as a service is difficult if not impossible. This has led Salesforce and other providers to slowly transform into more of a Platform that offers Software and less of a traditional or pure Software as a Service solution.

Public SaaS Cloud moves toward PaaS because of customer demands for isolation and customization, but what about Private SaaS Clouds? Private SaaS Clouds are more about the delivery and management of the software by IT. This pattern of an Enterprise being its own provider or contracting to a Hosting Provider for these services will still drive them to want capabilities that are not out of the box. This drive will lead them down the PaaS path as well.

PaaS no matter what it ends up looking like is much closer to what the end state looks like than IaaS or SaaS.

The Future of PaaS in the Enterprise – The Service Oriented Platform

26 Jan

Anyone who followed Cloud Computing last year watched several changes occur throughout the space.  These changes were happening in the view of what Public, Private, and Hybrid Clouds were defined as, the interest in IaaS solutions in the Enterprise and within Service Providers, and finally a renewed interest in the potential of PaaS.

Currently Public PaaS solutions are focused on core compute functionality and persistent services all of which are being backed by some mix of IaaS (with or without Virtualization, more with than without though).  Why are so many Public PaaS focused on their IaaS underpinnings?  Two reasons come to mind, IaaS is easier for customers and IT to understand and yet generic enough to have the broadest appeal.  This is important in the continuing evolution of the Cloud market as a whole, we need the widest adoption possible early on for any Cloud Platforms to take off.

Now, focusing in on the set of events around PaaS in the past year brings us to the rumors that surrounded EngineYard’s acquisition by VMware (which never happened and now we see that Amazon is working with them on a Ruby on Rails Elastic Beanstalk), Heroku was acquired by Salesforce.com, VMware announced the Open PaaS solution, and Microsoft will fight with anyone who calls Azure anything but a PaaS.  All of this Public facing PaaS news sounds great, but I’m sure you are thinking how does this translate to Enterprise?

I wondered this same thing and began to try to figure out just where it all leads.

What is the Future of Enterprise PaaS?

Public Cloud Services are always different from Private Cloud Services, many people think that Hybrid is something easy but miss the nuances involved with implementing Hybrid Clouds.  I’m not saying that achieving a Hybrid model is impossible, just that there are many hurdles and few have working solutions that seem to address most or all of them.

PaaS in an Enterprise will operate as a Service Oriented Platform.  That sounds like a silky smooth sentence doesn’t it?  But there is substance behind the statement.  Today the most advanced Public PaaS platforms are focusing on generic, on-demand, multi-tenant, infrastructure components, and other core capabilities.  What Enterprises will need for PaaS will be this plus many other services including specialized components such as vertical specific ERP and CRM solutions exposed as services, legacy solutions with service bridges, Big Data service connectors, and many more.  As these capabilities get built and each former “Silo” gets exposed as a service a Platform will emerge.

About the Service Oriented Platform:

Something to understand is that each business is different, it was formed differently and while it operates most likely in a similar fashion to other businesses in that market, it is still different.  This difference becomes even more pronounced when the needs are pushed to IT (Operations and Development).  This would be the equivalent to the Butterfly Effect in IT.

Why does the Butterfly Effect matter for an Enterprise Service Oriented Platform?

Because it means that each Enterprise Service Oriented Platform will be DIFFERENT than all others (maybe only marginally, but still different).  Each service exposed to the ESOP will be a new capability that Devs can leverage as they put together Apps on top of the Enterprise PaaS.

Something that may be important to realize is that IT Operations may view the solution as an ESOP while Devs may view it as an Enterprise PaaS.  I think both of these are actually correct and will talk about the same thing, it is just how it is being viewed and by who, but it is still the same thing.

Hybrid Clouds will be exposed as another service capability:

Hybrid Clouds will be Public Cloud services that are exposed either directly or indirectly to the ESOP.  The ESOP will have the control, authentication, metering, and likely the QoS.  The EPaaS will have the allowed/exposed services and resources from the Public Cloud.  This will be at a different level than simply pushing and pulling VMs, Virtual Hardisks, or Creating VPNs.

I’m interested in people’s thoughts and comments:

Comment on this Post or Follow Me on Twitter – @mccrory

Current PaaS Patterns – Types of PaaS

23 Jan

There are many definititions of PaaS that I have run across, the most succinct of which is “A Service where Code is uploaded and executed”.  While succinct, this leaves a lot of “wiggle room” for what PaaS really is.  The secret is PaaS isn’t one thing, it is a broad array of things that can be used and mixed together.  What model ore TYPE of PaaS determines what capabilities and limitations the platform will have.

The Type 1 PaaS – Upload Your Environment

Type 1 PaaS Examples

This style of PaaS is an option from several different providers including Amazon EC2 and Microsoft Azure with the VM Role.  By imaging and then uploading an environment packaged as a VM, you are allowed to run legacy systems in Cloud environments

The Type 2 PaaS – Upload Your Compiled Application

Type 2 PaaS Examples

This PaaS is where an environment is already provided and you are uploading and executing an Application that was written/is supported by that PaaS environment. This could be a preconfigured Amazon EC2 VM, a RackSpace Cloud VM, Microsoft Azure Role, or any of the other dozens of Cloud providers.

The Type 3 PaaS – Upload Your Packaged Application

Type 3 PaaS Examples

PaaS here is presented as a container that is more restricted than a traditional environment and is more stringent on code.  This could be a Java WAR file for example that leverages a container such as Apache Tomcat, Amazon’s Elastic Beanstalk is a good example of this.  The solution could also just be a custom JVM such as the one that Google App Engine uses (where your App is a JAR).  Why is this different than a Type 2 PaaS?  It is a container in an environment, not just an OS environment like Linux or Windows.

The Type 4 PaaS – Upload Your Code

Type 4 PaaS Examples

This type of PaaS allows source code to be uploaded where the compilation happens in the PaaS itself.  By accepting the code as the uploaded payload a greater level of insight as to what the resource requirements are of the code can be gotten, which offers more versatility from the platform.  An example would be VMware’s Open PaaS (which is in Alpha currently).  Open PaaS accepts many different code types, looks at what the requirements are and adjusts the environment based on what the needs of the code are.  Other examples include Heroku and Engine Yard, both do Ruby on Rails. When the Ruby on Rails code is uploaded for example, then a MySQL server instance is created to support the application amongst other things that happen.  Nearly everything that is needed is allocated automatically on demand.

The Type 5 PaaS – Upload App/Platform Specific Code

Type 5 PaaS Example

PaaS here would only allow specific code which is designed to work as logic or an extension of the Platform is allowed.  While this is a highly restrictive form of PaaS, it can also be powerful and is a natural move for a Software as a Service provider looking to allow cutsomization.  Salesforce.com is a great example with their App Logic using Apex Code.  Apex Code allows a level of customization and extension to the Salesforce solution using their own blended Java/C# like syntax.

The future of PaaS is in leveraging many of these models to best meet design decisions and customer needs.  The biggest question will be what happens in the future when Enterprises want to adopt and adapt PaaS in their Data Centers.  While the economies of scale will always be with Public Cloud PaaS providers, there are benefits to having PaaS implementations inside the Enterprise.

In my next post I will talk about where PaaS could go.

CLASH – CLoud Admin SHell

11 Jan

It has been several weeks since I have posted to this blog.  I would blame this on the holidays, but that would be inaccurate as it has been something far more insidious!

What is CLASH?
CLASH is a universal shell.  What is a universal shell?  First, by universal I mean that it is intended to run on all major desktop operating system platforms including:
Windows XP, Windows Vista, Windows 7

Mac OSX Leopard, Mac OSX Snow Leopard
Ubuntu Linux, and most other flavors of Linux
——————————–
Now the shell portion is a bit different.  Since this is a CLoud Admin SHell, Cloud is an important part of idea.  In this initial prototype I went through many experiments in learning the nuances of JRuby <follow on post link here>, but have been able to get a reasonable working version of VIJava (the interface I’m using to get JRuby to work with VMware vCenter/vSphere).

Great, so what can I do with this prototype?

-You can try it out on any of the platforms listed above!
-If you are on a Windows System, you can either edit the clash.bat file and put in your server’s IP or name along with a username and password to connect
-If you are on Mac or Linux, you can simply type ./clash –Server 127.0.0.1 –Username administrator –Password password to connect 

-The following are the working commands:
> Get-VM SomeVMName
Above gets the VM object and prints out the Guest Operating System type

> $result
This command prints the name of the VM object
> $result $
This displays the methods available to the VM object
> $result $.get
This shows only the methods with “get” in them
> $result $.set
This shows only the methods with “set” in them
> $result $.find something
This shows only them methods that contain “something” in them
> $result method
This will execute the method against the object (i.e. $result getName will display the name property of the VM object from Get-VM)
> disconnect
This will cleanly disconnect from the vCenter / vSphere server
> Start-VM SomeVMName
This command is flakey at the moment, this will be fixed when the next prototype is released in a few weeks
> Stop-VM SomeVMName
This is in the same state as Start-VM
> Get-VM SomeVMName > $result getName
This allows a limited form of piping in clash by using the > as an operator (you must have whitespace on both sides of the > symbol.

Where is this headed?
After many experiments, it will be broken out into a flexible system that allows many different options and cool capabilities against not only vCenter and vSphere, but most Cloud platforms as well.  Currently planned platforms include:
-Amazon EC2 and S3
-Rackspace
-(Looking for the next provider for this list)

Other interfaces to the shell (for both input and output) will include a Web interface, I’m looking for thoughts on other types of interfaces desired.

How will this work?
Below is my latest planned diagram for how I hope/think things should work:

Where can I get the Prototype?
You can get the code from GitHub here
You can download the entire package here

Installation Directions (AGAIN, this is a PROTOTYPE, it does NOT follow best practices)
1.) Make sure that you have Java 1.5 or Above Installed
Download Java from Here

2.) Install JRuby 1.5.6
Download JRuby from Here
3.) Follow the JRuby Setup Instructions Here
4.) Download the CLASH prototype/alpha-1 from GitHub (See Above Links)
Unzip/Tar it to c:\clash or /clash directory in ROOT
5.) Start clash by going to c:\clash\bin\ or /clash/bin
On Windows edit clash.bat to contain the correct IP Address, Username, and Password
then run clash.bat
On Mac and Linux type ./clash –Server 127.0.0.1 –Username administrator –Password password
Make sure to select the IP or Name of a valid vCenter / vSphere Server
6.) Play with clash
A Request:
Please supply feedback to me through comments to this post or communicate directly with me through Twitter – my handle is @mccrory
I’m looking for ideas/things that you would like to see clash do, better commands, capabilities, features, etc.

Cloud Escape Velocity – Switching Cloud Providers

18 Dec

The term Escape Velocity is the speed needed to “break free” from a gravitational field without further propulsion according to Wikipedia.org.  Data Gravity as explained in THIS previous post is what attracts and builds more Data, Applications, and Services on Clouds.  Data Gravity also is what creates a high level of Escape Velocity to move to another Cloud.

Some background on why this post is timely:

A few days ago Amazon announced a new AWS service for importing VMware disk images (VMDKs) into EC2.  VMware already offered a method for converting EC2 instances through their Converter tool into Workstation VMs and with a 2nd pass conversion into ESX VMs.  While all of this sounds wonderful and it does have value, it brings to light an entirely different issue.  Only Stateful / Fully encapsulated applications can be moved around in this way.

Examples of sources of Cloud Gravity (App, Service, and Data Gravity Combined) on your specific Application.

If someone selects a Cloud provider and writes an application leveraging anything more than a handful of VMs, Data Gravity will make it virtually impossible to move to a new/different Cloud provider.  Don’t believe it?

See the Diagrams Below:

Here is a diagram of an app that has a Low Escape Velocity because of Lower Cloud Gravity:

Cloud Escape Velocity with Low Gravity

Below is a diagram of an app that has a High Escape Velocity because of High Cloud Gravity:

Cloud Escape Velocity with High Gravity

Some potential dependencies include:

Database with a specific API

Web Worker which serves as a web interface and uses internal Authentication (Your user logins are here!)

Application code that uses the Database and Web Workers specific APIs and/or depends on Low Latency and High Throughput access to them.

Here are a few additional things to think about:

- The longer (more time) an Application stays in a specific Cloud the more difficult it is to move.  Why?  Data Gravity increases due to more Mass (data being stored).  Imagine accumulating 100′s of GBs of Data, how easy will it be to shuffle/transfer that much data around?

- The more provider APIs and Services that you depend on the harder it is to move.  Why?  Because there are only two paths that can be taken in a move.  The first is to find another provider that has the exact same set of APIs and Services (this will limit your choices).  The second is to change or rewrite your application to take advantage of the new Cloud provider’s APIs and/or Services.

- Different providers have different charges for the consumption of the same resources.  Your current provider gives free usage of queues for applications. The provider you are looking to go to charges after the first X number of messages on the queue.  Now what do you do?  You will either pay more when you move, rewrite your application to fit the new provider’s model, or pick another provider that has free queue usage.

- Different QoS guarantees from Cloud provider to Cloud provider.  Some Cloud providers offer SLAs with reimbursements for outages, others only offer best effort.  Some providers offer tiered Services, others only offer a single tier.  What happens if you want to move and you can’t get the minimum level of QoS that you need?

This is NOT an attempt to dissuade anyone from using Public Clouds (they are incredibly valuable and powerful), but I would like more people to go in eyes wide open.

Data Gravity – in the Clouds

7 Dec

Today Salesforce.com announced Database.com at Dreamforce.  I realized that many could be wondering why they decided to do this and more so, why now?

The answer is Data Gravity.

Consider Data as if it were a Planet or other object with sufficient mass.  As Data accumulates (builds mass) there is a greater likelihood that additional Services and Applications will be attracted to this data. This is the same effect Gravity has on objects around a planet.  As the mass or density increases, so does the strength of gravitational pull.  As things get closer to the mass, they accelerate toward the mass at an increasingly faster velocity.  Relating this analogy to Data is what is pictured below.

Data Gravity

Services and Applications can have their own Gravity, but Data is the most massive and dense, therefore it has the most gravity.  Data if large enough can be virtually impossible to move.
What accelerates Services and Applications to each other and to Data (the Gravity)?
Latency and Throughput, which act as the accelerators in continuing a stronger and stronger reliance or pull on each other.  This is the very reason that VMforce is so important to Salesforce’s long term strategy.  The diagram below shows the accelerant effect of Latency and Throughput, the assumption is that the closer you are (i.e. in the same facility) the higher the Throughput and lower the Latency to the Data and the more reliant those Applications and Services will become on Low Latency and High Throughput.
Note:  Latency and Throughput apply equally to both Applications and Services
How does this all relate back to Database.com?  If Salesforce.com can build a new Data Mass that is general purpose, but still close in locality to its other Data Masses and App/Service Properties, it will be able to grow its business and customer base that much more quickly.  It also enables VMforce to store data outside of the construct of ForceDB (Salesforce’s core database) enabling knew Adjacent Services with persistence.
The analogy holds with the comparison of your weight being different on one planet vs. another planet to that of services and applications (compute) having different weights depending on Data Gravity and what Data Mass(es) they are associated with.
Here is a 3D video depicting what I diagrammed at the beginning of the post in 2D.

 

More on Data Gravity soon (There is a formula in this somewhere)

Public Cloud Comparison and Calculator v2

5 Dec

After some time away from the Public Cloud Compute Comparison that I did a couple of months ago (which got X hits), I decided to update it based on feedback and new ideas.  What follows is a brief walkthrough with instructions on how to use the Calculator.

Before I go any further, a brief disclaimer:  I do not warrant the accuracy of this Comparison and Calculator, it may have errors and omissions (all unintentional if they exist).  I also take no responsibility if your bill turns out to be something very different than what the Calculator shows.  And finally, I’m employed by Dell, which has relationships with Microsoft, Joyent, and Amazon, and possibly the others and I’m just not aware.

First, let’s cover the updated Compute Comparison:

I’ve updated the Compute with Terremark as an additional provider, I would be interested in adding others if people are interested simply add a request by commenting at the bottom of this post.  Also added, is the ability to Calculate/Estimate your costs by using the add Quantities fields.  To use the Quantities, add the number of each type of Compute instance as you like and get a rough idea of what the cost will be.  Please note that there are assumptions, however for the most part I have annotated in the spreadsheet what those assumptions are.  Once you are happy with your compute instances, you can go down to the bottom and move to the new Cloud Storage Comparison and Calculator.

Cloud Compute Comparison and Calculator

Next, The Cloud Storage Comparison and Calculator:

It was quite an ordeal getting the tiered storage pricing models to work correctly as formulas, but it is all in there.  The Cloud Storage Comparison and Calculator attempts to cover the other side of the Cloud equation, by taking into account the following:

Monthly Persisten Storage Requirements

Data Transfer (In and Out of the Storage Cloud)

API Calls (In and Out bound requests)

Redundancy Costs

By entering some estimates across the top and choosing a quantity (this would usually be 1, which acts as a trigger to calculate Monthly Cost) you can easily get an idea of what your storage cost would be.

Cloud Storage Comparison and Calculator

And finally we have the Cost Summary Page.  This page combines the Total Monthly Cost from the Compute and Storage sheets into one place.

Cloud Comparison Summary Monthly Total Costs

To switch from the Cloud page to the Storage page and from the Storage page to the Summary page (or worksheet if you want to be precise), go to the bottom of the page an select (as shown below)

Tabs and Sheets

In the next post I will cover different results that came out as I used the Calculator.

An Administrator’s view of Open PaaS and VMforce

23 Nov

After posting about How Development works on Open PaaS and VMforce, I felt it was time to provide an equivalent view from an Administrator’s perspective. Before going deep, I thought I would provide a comparison of what things look like between the Developer’s view of things vs. the Administrator’s.

Comparison of Developer vs. Admin View


Please note that this is derived information and in some cases speculative (but I bet I’m close)
Starting at the top:

  • The URL and Mapping matches a DNS entry with an External IP (Host), a Path, and a Port to Access the Application
  • The Application contains an App Instance matching the Virtual Machine with a Workload (Potentially multiple Workloads)
  • The Internal IP operates off of the assumption that the VM is either multi-homed or has a NAT based interface with an Inside and Outside Address
  • The Service Instance matches a VM with a specific Running Service inside. This could be a shared Service instance or a Multi-User/Tenant Service Instance (There isn’t enough info. from what I have found to know which)
  • The Service Catalog is the equivalent of a Template/Gold Image based VM (in the describe model)

    • There are several different ways VMware could choose to implement isolation and multi-tenancy.

The diagram below gives an Administrator’s view of Open PaaS and its implementation inside of VMforce. The current implemented resource model shows a quota system as the chosen way of limiting/controlling consumption of resources in the Open PaaS Cloud.

Administrator's View of Open PaaS / VMforce

The Account is the line between where the Administrator turns over the resources to the Developer. This seems like it would create an environment like the wild west, but this is a deceptively simple view. The Architects and Administrators both have the ability to constrain the system before any code is pushed into it. This is achieved by decisions on what types of code can supported in the system, potentially constraining allowed frameworks, available services, the ability to create services, and allocated resources. Quota based allocated resources include number of CPU cores, Memory, and Disk space.

From what I have been able to find so far, there is a focus on isolation by Account using a quota system.
The strongest isolation model would be to assign each workload its own VM, this however would consume far more resources than isolation at a process level (a typical trade-off). Implementing isolation at a process level would work well but you wouldn’t want all Workloads (App Instances) for a single Application running in a single VM, because if the VM fails so does your Application. As more is revealed, I will provide more indepth information on how isolation and distribution is done.

There is also an unknown as to if and how a load-balancing mechanism is implemented. I haven’t come across how/if this is implemented, perhaps this is done in the Mapping (via. DNS/round robin?). This is purely speculative.

AppCloud appears to be VMware Open PaaS Cloud backend name

20 Nov

As I continue to go through VMC related code, I have come across a few code entries talking about AppCloud.  At first I thought this might be a reference to EngineYard’s AppCloud solution. This brought to mind the rumors I mentioned in previous posts, but after further digging and reading the following code and code comments:

AppCloud Gist

Code references to AppCloud


I am fairly convinced that AppCloud is referring to the Open PaaS Cloud Controller (and possibly some of the other components collectively), not EngineYard’s AppCloud. The big question in my mind is when will the first version of “AppCloud” be launched/shipped/released.

How Development works on Open PaaS & VMforce

14 Nov

After having gone through the materials available (both the easy to find and the difficult to find) I have created what should be an accurate view of what the environments inside a VMware Open PaaS and VMforce world should look like.  In this post is a series of diagrams that I have created based on what I have concluded is the way the current system works.

Developer's overview of Open PaaS & VMforce

In the diagram above, starting at the top:

  • Organization Context – This is likely a sub-Cloud of the overall Cloud, but I haven’t been able to clarify this in the code yet.  It provides authentication and determines which services are available to be used and shared amongst User Accounts (aka Service Domains)
  • User Account – This is done by registering an e-mail address and a password, each account is allocated a quota which is controlled by an administrator.
  • URL – This is how external Applications, Users, APIs, etc. access the Application
  • Mapping – Connects the URL to the Application (Allowing Applications to be Switched beneath the URL)
  • Application – Also know as a “Droplet”, is made up of App Instances and connect to Service Instances
  • Service Instances – Are useable/consumable/invoked instances of services from the Service Catalog
  • Service Catalog – This is the listing of all available services that can currently be invoked for use by an Application

Logical Diagram of Open PaaS - VMforce

The above diagram shows a slightly more filled out Service Catalog. These are the services that were provided as examples by the VMware presentations and documentation that I have seen so far.  The diagram also shows an even larger number of applications running, although each only has a single App Instances associated with it.

Detailed Logical Diagram of a more realistic example

In this diagram (above), there are two URLs each providing access to an Application.  The first application on the left has a single Application Instance and that App Instance is bound (see Binding Labels) to a MySQL Instance (Service Instance) and a RabbitMQ Instance (Service Instance).  The two Service Instances are created from the Service Catalog’s MySQL and RabbitMQ entries.

The second Application has three App Instances inside of it, all of which are bound to the SAME RabbitMQ Instance that the first Application is (this means that the two Applications can share information through the RabbitMQ Instance).  The MySQL Instance is a separate MySQL Instance from the first Application MySQL Instance, although both are based/invoked from the MySQL Service in the Service Catalog.  The Redis, Memcache, and MongoDB instances are all bound to each of the App Instances in the second application and are used by all three instances.

Diagram of Open PaaS - VMforce Service Tiers

The final Diagram is from information I came across while digging through the VMC Ruby code.  The code has fields in it for “Service Tiers”, which based on some poking around on Salesforce’s website, I came up with the above possibilities.  I don’t know if this is 100% accurate, but based on the information I think it provides a pretty reasonable approach to exposing provider services to Developers looking to write code on a Cloud platform such as VMforce.

There are several more interesting things that I have come across since I began going through the code.  I will be blogging about them soon.

Walk-through of the VMforce / Cloud OS / OpenPaaS Demo

13 Nov

This post attempts to walk-through the demo that was shown at the Ruby Conference.  I was not actually at the conference, but I am reconstructing what happened based on materials and information that was tweeted and the presentation materials.

Diagram of VMware Cloud OS - PaaS - VMforce Demo CLI Walk-through

The walk-through above shows a sophisticated PaaS layer (reminding me of the Google AppEngine PaaS) where code is uploaded/pushed and then inspected and compiled.  The resulting “App Instance” (also referred to as a “Slug” in Heroku terms) is ready to respond to requests.  Can’t service enough requests with just one App Instance?  Type “vmc instances fu 5″ and instead of a single App Instance, the App Instance clones/copies/ Scales Out to 5 instances.

Need to Scale Down from 5 App Instances to 3 because demand has fallen?  ”vmc instances fu 3″ Scales Down the App Instances from 5 to 3 killing the last 2 App Instances – 3 and 4 (note that instances are numbered 0-4).

 

Diagram of VMware Cloud OS - PaaS - VMforce Demo CLI Walk-through - 2

Now the requests have fallen to the point where there is only 1 App Instance needed, so the command “vmc instances fu 1″ shrinks (Scales Down) the App Instances from 3 to 1.  Now we look to see where the fu Application actually is resource utilization wise (This is the point where we see the overlap with the underlying VM) by typing “vmc stats fu”

This allows a Developer on a Public Cloud – in the above case vCloudLabs.com is hosted by Terremark (using vCloud if I had to guess) to deploy their code with little to NO knowledge of the underlying Infrastructure beneath (IaaS).  This ultimately mirrors the functionality of what VMware’s Cloud OS will provide to  Private Clouds and the Enterprise Development groups inside them.  What is demoed is supposed to be the same as the system was designed for VMforce in Salesforce.com ‘s Data Center as well.

VMware quietly shows Cloud OS, OpenPaaS, and VMforce at Ruby Conference

13 Nov

Yesterday, VMware previewed the first concrete evidence that they are moving forward on the OpenPaaS initiative, the VMware Cloud OS, and VMforce at the 2010 Ruby Conference in New Orleans. At the conference Derek Collison demonstrated an early preview of the VMware Cloud OS via. a command line interface that he and Ezra Zygmuntowicz created.

The demonstration included showing Ruby code being auto deployed (pushed) into an a VM (where it becomes an App), coming online and then scaling both up and down in real-time. Also shown in the presentation itself was a screenshot of the control panel providing a view into what is being referred to as an application centric view of PaaS.

VMware Cloud OS Dashboard

In addition, the architecture was described, covering how the system is coordinated for auto-scaling and how resources are controlled.  Below is the high level architecture that was presented, a more detailed and in-depth walkthrough of how the system works will be published on this blog tomorrow.

VMware Cloud OS / OpenPaaS Architecture

VMware is moving quickly to fully support all modern/popular languages in the Cloud OS, including:

  • Ruby
  • Java
  • Node.js
  • Python
  • .NET
  • and more!

This strategy is critical to VMware being able to uplift itself from being considered a purely infrastructure company, to that of a Platform company (beyond owning SpringSource).  It is also important to VMware in attempting to grab developer mindshare by trying to better meet the capabilities that developers have traditionally been going to the Public Cloud for, by enabling Enterprises to provide those same capabilities internally (real Private Clouds).

Several more posts to follow very soon…

Where most Enterprise IT Architectures are today

4 Nov

Most Enterprises are architecturally in a rigid and fragile state.  This has been caused by years of legacy practices in support of poor code, design patterns, underpowered hardware (which focused on increasing MHz not parallelism/multi-cores).  What follows is a brief review of what has led us here and is needed background for the follow-on post which exercises a theory that I’m testing.


Architecture Phase 1 – How SQL and ACID took us down a path
Early on in the Client/Server days even low power x86 servers were expensive. These servers would have an entire stack of software put on them (i.e. DB and Application functions with Clients connecting to the App and accessing the DB). This architecture made the DB the most critical component of the system. The DB needed to ALWAYS be online and needed to have the most rigid transactional consistency possible. This architecture forced a series of processes to be put in place and underlying hardware designs to evolve in support of this architecture.
This legacy brought us the following hardware solutions:

RAID 1 (Disk Mirroring) -> Multi-pathed HBAs connecting SANs with even more Redundancy

Two NIC Cards -> NICs teamed to separate physical Network Switches

Memory Parity -> Mirrored Memory

Multi-Sockets -> FT Based in Lock Step CPUs

All of this was designed to GUARANTEE both Availability and Consistency.
Having Consistency and Availability is expensive and complicated.  This also does not take into account ANY Partition tolerance.  (See my Cap Theorem post)

 

 

 

 

 

 

 

 

 

 

 

 

Architecture Phase 2 – The Web
Web based architectures in the enterprise contributed to progress with a 3-Tier model where we separated the Web, Application, and Database functionality into separate physical systems. We did this because it made sense. How can you scale a system that has a Web, Application, and Database residing on it? You can’t, so first you break it out and run many web servers with a load balancer in front. Next you get a big powerful server for the Application tier and another (possibly even more highly redundant than the Application tier server) for the Database. All, set right? This is the most common architecture in the enterprise today. It is expensive to implement, expensive to manage, and expensive to maintain, but it is the legacy that developers have given IT to support.  The benefit being that there is better scalability and flexibility with this model and with adding virtualization (which helps further the life of this architecture).

Where is Virtualization in all of this?
Virtualization is the closest Phase 2 could ever really get to the future (aka Phase 3, which is covered in my next post). Virtualization breaks the bond with the physical machines, but not the applications (and their architectures) that are running on top. This is why IT administrators have had such a need for capabilities in products like VMware ESX in conjunction with VMware vSphere like HA (High Availability), DRS (Distributed Resource Scheduling), and FT (Fault Tolerance). These things are required when you are attempting to keep a system up as close to 100% as possible.

Today

The trend toward Cloud architectures is forcing changes in development practices and coding/application design philosophies.  Cloud architectures are also demanding changes in IT operations and the resulting business needs are creating pressures for capabilities that current/modern IT Architectures can’t provide.

This leads us to what is coming….

CAP Theorem and Clouds

3 Nov

A background on CAP Theorem:

CAP Theorem is firmly anchored in the SOA (Service Oriented Architecture) movement and is showing promise as a way of classifying different types of Cloud Solution Architectures.  What follows is an explanation about CAP Theorem, how it works, and why it is so relevant to anyone looking at Clouds (Public, Private, Hybrid, or otherwise).

Distributed Systems Theory – The CAP Theorem:
CAP Theorem was first mentioned by Eric Brewer in 2000 (CTO of Inktomi at the time) and was proven 2 years later.  CAP stands for Consistency, Availability, and Partitioning tolerance.  CAP Theory states that you can only have TWO of the three capabilities in a system.  So you can have Consistency and Availability, but then you don’t have Partitioning tolerance.  You could have Availability and Partitioning tolerance without rigid Consistency.  Finally you could have Consistency and Partitioning tolerance without Availability.

The KEY assumption is that the system needs to persist data and/or has state of some type, if you don’t need either Data persistence or State ANYWHERE, you can get very close to having Consistence, Availability, and Partitioning simultaneously.

Understanding Consistency, Availability, and Partitioning:

Consistency is a system’s ability to maintain ACID properties of transactions (a common characteristic of modern RDBMS). Another way to think about this is how strict or rigid the system is about maintaining the integrity of reads/writes and ensuring there are no conflicts.  In an RDBMS this is done through some type of locking.
Availability is system’s ability to sucessfully respond to ALL requests made.  Think of data or state information split between two machines, a request is made and machine 1 has some of the data and machine 2 has the rest of the data, if either machine goes down not ALL requests can be fulfilled, because not all of the data or state information is available entirely on either machine.
Partitioning is the ability of a system to gracefully handle Network Partitioning events.  A Network Partitioning event occurs when a system is no longer accessible (Think of a network connection failing). A different way of considering Partitioning tolerance is to think of it as message passing.  If an individual system can no longer send/receive messages to/from other systems, it has been effectively “partitioned” out of the network.
A great deal of discussion has occurred over Partitioning and some have argued that it should be instead referred to as Latency.  The idea being that if Latency is high enough, then even if an individual system is able to respond, the individual system will be treated by other systems as if it has been partitioned.
In Practice:

CAp – Think of a traditional Relational Database (i.e. MS SQL, DB2, Oracle 11g, Postgres), if any of these systems lose their connection or experience high latency they can not service all requests and therfore are NOT Partitioning tolerant (There are ways to solve this problem, but none are perfect)
cAP -  A NOSQL Store (i.e. Cassandra, MongoDB, Voldemort), these systems are highly resilient to Network Partitioning (assuming that you have several servers supporting any of these systems) and they offer Availbility.  This is achieved by giving up a certain amount of Consistency, these solutions follow an Eventual Consistency model.
CaP – This isn’t an attractive option, as your system will not always be available and wouldn’t be incredibly useful in an Cloud environment at least.  An example would be a system that if one of the nodes fails, other nodes can’t respond to requests.  Think of a solution that has a head-end where if the head-end fails, it takes down all of the nodes with it.
A Balancing Act
When CAP Theorem is put into practice, it is more of a balancing act where you will not truly leave out C,A, or P.  It is a matter of which two of the three the system is closest to (as seen below).
NOTE:  Post to follow tying this more closely to the Cloud coming tomorrow.
In my research I came across a number of great references on CAP Theorem (In order most favored order):

The Real Path to Clouds

2 Nov

I’ve been spending a great deal of time as of late researching the background and roots of Cloud Computing in an effort to fully understand it. The goal behind this was to understand what Cloud computing is at all levels, and is quite a tall order. I think I have it figured out and am now looking for the community’s feedback to vet and fully mature my theory.

First a brief review of CAP Theorem, which states that all implemented systems can only sucessfully focus on two of the three capabilities (Consistency, Availability, and Partitioning tolerance).  If you aren’t familiar with CAP Theory, please check out yesterday’s post and the resources at the bottom of that page.

First some background – Below are the two phases deployed today in >90% of Enterprises.  See “Where most Enterprise IT Architectures are today” for in-depth discussion on Phase 1 and Phase 2.

 

What follows is the theory:

Architecture Phase 3 – Cloud & The Real Time Web Explosion

The modern web has taken hold and Hyperscale applications have pushed a change in Architecture away from the monolithic-esque 3-Tier that traditional Enterprises still employ to that of a loosely coupled Services Oriented queued/grid like asynchronous design. This change became possible because developers and architects decided that a Rigidly Consistent and Highly Available system wasn’t necessarily required to hold all data used by their applications.

This was brought to the forefront by Amazon when it introduced its Dynamo paper in 2007 where Werner Vogels presented the fact that all of Amazon is not Rigidly Consistent but follows a mix of an Eventual Consistency model on some systems and a Rigidly Consistent model on others. Finally, the need for a system that operates at as close to 100% uptime and consistency, depending on a single system, was broken. Since then, we have found that all Hyperscale services follow this model: examples include Facebook, Twitter, Zynga, Amazon AWS, and Google.

Deeper into Phase 3
Why is Phase 3 so different than Phase 1 and 2?
Phase 3 Architecture not only stays up when systems fail, it assumes that systems can and will fail! In Hyperscale Architectures, only specific cases require RDBMS with Rigid Consistency, the rest of the time an Eventually Consistent model is fine. This move away from requiring systems to be up 100% of the time and maintaining Rigid Consistency (ACID compliance for all you DBAs) lowers not only the complexity of the applications and their architectures, but the cost of the hardware they are being implemented on.

Moore’s Law is pushing Phase 3
Until around 7 years ago, CPUs were constantly increasing in clock speed to keep up with Moore’s Law. Chip makers changed their strategy to keep up with Moore’s Law by changing from an increase in clock speed to increasing the numbers of cores operating at the same clock speed. There are only two ways to take advantage of this increase in cores approach, the first is to use Virtualization to slice up CPU cores and allocate either a small number of cores or even partial cores to an application. The second is to write software that is able to asynchronously use and take advantage of all of the cores available. Currently most enterprise software (Phase 2) is not capable of leveraging CPU resources in the second way described, however many systems in Phase 3 can.

How do IT processes fit into this?
Today most Enterprise IT shops have designed all of their processes, tools, scripts, etc. around supporting the applications and architectures mentioned in Phase 1 and Phase 2, NOT Phase 3. Phase 3 requires an entirely different set of processes, tools, etc. because it operates on an entirely different set of assumptions! How many Enterprise IT shops operate with assumptions such as:
Replicating Data 3 or more times in Real Time
Expecting that 2 or more servers (nodes) in a system can be down (this is acceptable)
Expecting that an entire Rack or Site can be down
(These are just a few examples)

Why will Enterprises be compelled to move to Phase 3 (eventually)?
The short answer is cost savings and operational efficiency, the longer answer is around the reduction in systems complexity. The more complex the hardware and software stack is (think of this as the number of moving parts) the higher the likelihood of a failure. Also, the more complicated the stack of hardware and software become, the more difficult it is to manage and maintain. This leads to lower efficiencies and higher costs, at the same time making the environment more brittle to changes (which is why we need ITILv3 in Enterprises today). To gain flexibility and elasticity you have to shed complex systems with hundreds of interdependencies and an operational assumption of all components being up as close to 100% as possible.

Conclusion:

Different systems have different requirements, most systems do not in fact need an ACID compliant consistency model (even thought today most have been developed around one.  Some specific cases need ACID properties maintained and a level of 100% Consistency, but these are in the minority (whether in the Cloud or in the Enterprise).  This is causing a split in data models between the current CA (Consistency and Availability – “Traditional Enterprise Applications) and AP (Availability and Partitioning tolerance – many Cloud Applications).  A combination of these is the long term answer for the Enterprise and will be what IT must learn to support (both philosophically and operationally).

Follow me on Twitter if you want to discuss @mccrory

Is it me or is it EngineYard ? Updated – (The Answer is BOTH)

30 Oct

UPDATED:

So after doing some additional reading at the suggestion of @drnic, I read what @tmornini said about the changes occurring.  What is the stated view of events?  EngineYard is growing up.  I can believe this as a plausible explanation, I state this because I have personally gone through this type of change at several of the startups that I have founded (specifically @Surgient & @Hyper9).  Companies change and mature (at least hopefully they do).  As these changes happen, the companies aren’t always the right fit for the same group of employees that were there in the very beginning.

This is actually a very frequent occurrence at successful companies, one of which that comes to mind is VMware.  I watched from the outside as a customer and partner, VMware went from a <100 person company when I first worked with them to an independent subsidiary of EMC with >100K employees.  Now VMware has multiple companies that are subsidiaries of it!

I still am left wondering what EngineYard’s real plan is, although I did see a mention that the customers leaving might have been on an old hardware platform as well.  I will post more if/when I find out more.  EngineYard looks like it is in an exciting phase and a great place to work (great momentum).

ORIGINAL POST:

Am I the only one that finds odd things afoot at EngineYard? Since August @ezmobius, @wycats and @carllerche have all left EngineYard (under friendly terms from what I can tell). Now High Profile EY customers are deciding to leave on friendly terms? In the past week I have seen two examples of this;  the first was with New Relic deciding to change to their own model + a different provider (yet said they had no issues with EngineYard)

and then Pivotal Labs app Pivotal Tracker is leaving under similar circumstances (IMHO).

In my mind, NONE of this would be odd, except that it has all happened in the window from Sept. 1st through Nov. 1st. and both companies have chosen early November for their move (one heck of a coincidence). What is going on over at EngineYard? BTW, in case any of you out there are wondering, I’m not an EngineYard customer, but I may be in the future.   I am actually impressed with what they have built and I’m a fan of both Ruby and JRuby.  :)

Follow

Get every new post delivered to your Inbox.