Cloud foresight

i hope that this weekend i will find time to further comment. I hope for people to realy give this a thought as what we are talking about here concerns almost everyone of you.

Rahmu, Once again you took the bait! you nailed most of the things that are going on right on the dot. Rahmus summary is well composed and worth for others to revise with care. there are things to add and note but more of that later. people ready Rahmus reply and try to grasp it. One thing i would note out at this point and that is Rahmus well composed summary is very technology centric. something you would expect from a technologist. Rahmus compositions keep getting better and better across all discussion threads.

It is rare that i would complement someone, this time i can not deny the respect that rahmu has earned in the various responses he has written. keep up the good work rahmu. I dont know you in person but i dont need to know you in person to appreciate your effort in this forum.

there is an angle i hope people would consider thinking about after giving rahmus summary a review and those are:
- why "join" the cloud
- where will the cloud be?
- what does the cloud impact?
- what is the momentum driving the cloud?

@Samer, regardless of opensource or closed source solution or services, there always will be services. the services are becoming standardized with the cloud. take a good look on how over the past years, commercial companies have recreated themselves via opensource. in addition, remember that tools are tools, they are tools, you will always get a newer and a better model, the important thing
is to remember that a tool is a tool and not the business operation itself. but more of this later. my neighbors are dragging me to the bar for some pool and beer.

Cerio!

When old idealistic beards talk, their activist tone is very biased by idealism. Something that Rahmu has already pointed out. In turn, Rahmu so far has given a granular description on various areas concerning the cloud. most of the product and tool refered to were opensource related. in the cloud there is room for commercial products and operations. opensource will not be killing the commercial industry. the commercial industry is recreating itself thru opensource.

In regards to hypervizors, the market has become saturated with options. you do not need to have many to state that there are many. in between what exists right now there is more than enough. esx,hyperv,xen etc. once again these are just a tool, the hypervizor tool with each having its pro and cons. the pro and cons are not wheight by the product itself alone. but as well by how well it gets integrated with other tools thru which you can orchestrate and automate. another aspect is the legal policy limitations or staff knoweldge and experience. one might be easier to administer than the other. another might offer more resiliency features than the other. it all depends on what you want to acheive.

there has been a huge misconception in regards to opensource and that is, as an open code, anyone can audit, but in reality how many audit? very few are interested, its more of the geeks or those who have too much free time at hand who would be sifting thru codelines without compensation. we all want our bread and butter. which only states that there is no such thing as a free meal. opensource is not a free meal. any thoughts or ideas that are built upon that is a misconception.

the industry is changing, and it will change dramatically. Rahmus first reply is as good as it gets for a blueprint of the cloud. lets roll back a bit and recall what linux was like back in 1995. linux was deemed to be a lost cause. it was scrutinized by microsoft and sun. and then they began to admit that they can no longer undermine what it is. this enforced and obliged such companies to make more effort on improving their products and sell it at a smaller rate. a big diferentiator between the two (commercial and opensource) uptill latelly has been standardization. lets be honnest, it is a nightmare on the number of linux distros that exist out there. this alone has caused lots of challenges for hosting legacy opensource based OS and apps on just about any hypervizor. one aspect of the cloud is standardization. but why do we want to standardize, simply because we expect to see and have larger operations. we all know that IT yet still is in its infancy. that is where not all other industries have yet fully exploited IT. this continous growth for IT need is of the most important driving forces to standardize in order to be able to control and operate with sanity and feasable cost.

grid computing is no longer an HPC world architecture. every day, more and more concepts from the HPC world are blooming under the "cloud" with scalout solutions. the age of having to scaleup or forklift upgrade an environment is coming to an end. the concept of a scaleout is simple, get more with less. have endless amount of given resource without having to make dramatic investments and changes in the architecture and infrastructure. scaleout as a feature is available in each of the primary categories; file, object and block. for example glusterfs scales nicely with files. it still requires features but is simpler than lustre to operate. a commercial competitor and my personal selection is Isilon. it has lots of features than neither of the other two have and realy serves a requirement. with block storage, you can use scaleio, or vsan with vmware, you have options. you do not need to have a "heavy duty" storage backend. hyperconverged products are grabing a slice of the market such as nutanix and simplivity or then again vblock or upcoming evo:rail. just being scaleout doesnt speak out for itself. each of these have their architectural limitations so do not blindly trust what the whitepapers claim. they do offer a solution. hyperconverged and scaleout are not killer products. they are not a one solution fits all requirement products. they are products that enable to construct infrastructures for given requirements.

Rahmu mentioned the containers, and presented a good comparative analogy of thinking of chroot on steroids. that is about what it is. it grants the possibility to run more processes over a common kernel thus giving you a better ROI. of the few key characteristics of virtualiztion was to standardize configuration and hardware compatibility, hence less admin work. and to be able to further leverage your current hardware. if in the past you had one app runing on top of one kernel runing on top of one physical cpu operating at an average utilization of 5%. that was waste of 95% of your investment. with virtualization, you were able to sitaute about 30 vm, hence 30 app on their own kernels on a single hardware hence multiplying your utilization and increasing your ROI. with containers, you can host even more with less number of kernels making it able to leverage even more from your hardware. but one of the actual reason and benefits of containers is to be able to scale deployement with less administrative work. why have an uber guru administer a single apache instance on a single virtual machine when you can have the same admin guru administrating 200 apache instances on the same virtual machine.

so as you see, with all the variants of virtualization and its morphing into a "cloud" is more of standardizing and optimizing. but for what? again, this same old question. well, IT is not a leading force by itself. it is an enabler. it grants technologies that other industries demmand and it is that demmand that creates the need for solutions. IT based solutions. industries are ever more reinventing themselves with new means of working. this alone has lead numerous researches to deduce that at an average of 50% of common jobs today will be voided within 20 years. the postman wont be delivering your mail. it could be a drone. your farmer could be replaced with a robot machine that multitasks. no more cashiers in the shops, one would monitor about 8 self service cashiers cutting staff by 7/8. yes the world is changing and it is this change that is the driving force to standardize the technology and the way of working with it. that is a lot of just about what the cloud is.

think of what rockefeller did about a hundred years ago. think of the same phenomena but instead of crude oil bonanza. think of it bonanza. even thou the cloud will cause change in employment roles, at the same time it is enabling new roles. with every change, there are casulties but overtime its worth going thru it as on the long run, it is enabling many new things that currently are impossible to achieve without extreme huge operations costs.

one of the key descriptive terms as for what is a fundemental philosophy of the cloud is: to abstract, to pool and to automate. by now each one should have started to think of why go thru the hastle of learning to do things in a different way when i can just do it the same as right now. well, the fact to the case is that there is going to be a division, one part is going to go fast forward with the cloud, and the other part remaining with an arkeic bazar like way of working. to put that into an example, western nations will be going into the cloud, other remaining nations with poor infrastructures, inconsistent level of education and unstable political policies will retain the old way of working.

it is always going to be a bipolar dilema, those who want full control, and those who want to concentrate on the core operations, hence consolidate and outsource. i believe the later is what will dominate. why go thru the haslte of maintaining, contracts, network, protocol, hardware, os, framework applications to run your application on top of that when you can cross out a few ending up with contract and application? that is a lot of saving!

just as what linux did to the IT industry so far, OpenStack will be doing just about the same. in otherwords OpenStack in good and bad is the next linux phenomena. there already are dilemas in regards to positioning and using it and just as with linux there always will be a need for it and maintaining it. just as linux has closed and open developers maintaining it, so does openstack. but openstack alone will not suffice. just as linux has been an underlying OS.

openstack is a tool that underlies something else. time will tell what things we are to see.

more later got to go to sleep now.

BL

@Arithma,

what you have listed are new products and services to enable business and end user accomplish new things. those are the type of things that are created to satisfy a demand, a business demand. these new services and products enable new business and ventures. these operations as well are dependent on an infrastructure. the cloud serves as an infrastructure for these services and operations. old business and new business both are dependent on IT infrastructure and in the future, the cloud. the cloud is the underlying layer that will host and cause a change in the industry from which there is no turning back.

do we really need 20ghz cpu when we can have the equivalent amount of computing power as a scaled solution without additional cost? the industry can put billions into developing such cpu, or then again invest less by exploiting grid and scale approach.

we currently are living in the beginning of the end of conventional resources. this alone is as well a driving force to optimize things. we currently are consuming almost twice the amount of resources in comparison to what our planet is able to yeld. the irony.

that is the challenge with openstack, it has just one too many moving parts
without any assurance that individual features would be developed and maintained
thruout openstacks lifetime.

one of the big things of this century is the philosophy of an ecosystem. i have
always looked down on numerous commercial solutions because of their closed
and limited features and lack of the ability to integrate or further use.
its such things that over employed people in the IT industry who are currently
becoming obselete with what ecosystems facilitate and enable.

big companies are biding on ecosystems. the control of the whole stack. that
is where we are going to see merges between big companies, dog eat dog competition
between ecosystems and with time only to tell who are to survive and coexist
down the road.

in regards to learning, i see that to be a financing issue of your employer and
not yourself. there are numerous official training courses that exist to enable
any person become familiar and compitent with propriatary products. all the major
vendors create curriculum to guarantee a clear comprehention of the products
and solutions which would insure a level of mastery of the product. this is not
a bad thing but a good thing. ive seen guys who have redhat certification
and yet do not fully grasp what LVM realy is and why it exists. that is like
the simplest of examples i can give at this point. today the internet is infested
with pirated materials, dont get me wrong, i am not encouraging, just stating.
but reading such material alone will not suffice to enlight on why certain
solutions are designed as such. i have been to numerous costly trainings
and always found value in them because the instructors have real knowledge
and references. it becomes more of a training and workshop experiencen.
something you can not compensate with just reading documents. but yes, there is
truth to the statement, with opensource there is a saturated amount of documentation
and howto websites thru which you could learn. well for isilon there are also
open communities at emc website and google groups where you can discuss and enquire.
some of those threads are even addressed by developers themselves. ive also
attended training courses concerning opensource and those were just as expensive
delivered e.g. by redhat, hp, etc. so when it comes to training for a solution
and certification, there is no workarround, there is always a cost if you
want to get certified. look at vmware, you cant do the exam if you havent
participated in their course. ive been thinking of doing VCAP since i already have
my VCP. but the amount of training i have to do is redicoulous. the bigger joke
here is to self finance such training to obtain a certification and still
lack on experience that would help to get a job position. voiding the investment
unless you have an actual need for it. one of the things that gets on my nerves
are people who brag about having this or that certification or amount of certification
yet are totally incompitent. as an example, i once had an instructure who bragged
of having over 30 certifications (even listed on linkedin :P ) all in various fields
with many within microsoft technologies yet the person did not know what diskpart
was or how to use it. he was recommending thirdparty tools to do what diskpart
does in windows. grr... i can go on ranting about that but i think i will stop
here. in short, for a company that plans to use a technology, regardless of
whether it is commercial or opensource based, the company always reserves a
budget for proper staff training. one of the reasons for that is that insurance
companies refuse to compensate if something goes wrong because of the lack of
official training which is considered part of a preventative good way of working habbit
and readiness.

in regards to containers, no they will not replace hypervizors. they will replace
certain implementations. a well scaled and orchestrated environment is where containers
and hypervizors complement each other. i recall there were container products in the
market about 15 years ago. one at least got bought out by vmware, i think its derivative
is thinapp. not sure got to verify that ;)

openstack has many oportunities, it is a phenomena just as linux even thou it has
a different driving force form commecrical companies. it will be interesting to see
how far it will get. there is lots of commitment to it. one fo the reason for that is
no single vendor has been able to provide a standardization based on collaboration with
other vendors. the openstack story differs here where all intersted commercial companies
have an equal opportunity to a stnadardized solution stack of which everyone gains.

more later, hopefully in reply as to xAAS that samer enquired about ;)

@Arithma,

20ghz, i dont mind as long as power consumption doesnt grow in multiples. cpu speed has grown for decades and the speed has always been an impact. computig power nowadays is not based on single thread operations but on multi-thread operations. if you use legacy 8 or 16bit single thread code, yeah a 20ghz can help you up to a point. such is refered to as a scaleup patchup. if you are multithreading, you do not need such big cpu speed and you can distribute across to many cpu. the bottleneck of cpus nowadays has more to do with incompatible kernel semaphor values or using the wrong schedulers. each application environment can be optimized. optimiziation would include noting kernel semaphores and schedulers.

the constraints of scaleout has less to do with the scaleout concept itself but with the attempt to use technologies at their limits. there are numerous technology components that we are dependent on today whose fundemental architecture was never designed with scaleout in mind. they have been patched and extended. and when you have a stack of protocols, processes, etc. you will be facing limitations at some point. those limitations are not scaleout concept specific. a performance bottleneck of a single device does not define nor states that a whole distributed scaleout solution suffers the same. this reminds me of a military idiom, a chain is as strong as its weakest link.
well with scaleout distributed solutions, they are designed to be resilient to spof and mpof. you can experience a peformance degradation during a spof or mpof. but you will not experience a service outage. resiliency is the new attribute when making use of erasure code based technologies.

@Rahmu

the issue of certifications really ticks me of as it realy doesnt prove much. a persons portfolio proves even more. it is true that this is a chicken and egg dilema. this is why i like the idea where educational institutions attempt to facilitate the obtaining of such certifications. but having a certification without actual experience is as worth as toilet paper. during the last 15 years i have seen numerous indians who have and are proud of their certifications. only a tenth realy lived up to it. the remainder made me question their compentence even more. especially when they openly brag of how they pay a friend like 100£ to do the certification exam on their behalf. its such kind of great achievements that has caused lots of excess unecessary headaches in the industry. a colleague of mine has just been lately traveling to various destinations across the world working in customer environments. he was personally picked to do this because he as one person was able to accomplish alone what 14 indians with equivelent "training and certification" were not able to do. he delivered quality and in time.

the challenge with selflearning is the learning curve pace. for others it can be short, for others very long. there is no consistency with that. thru profesional training and certification, you can expect a level mastery because you are being mentored and absorbing at a higher more intensive rate. IT has various fields and opensource is present in most of them. but opensource does not have ideally composed training material for each or across several fields. there always is a room for improvement whichever way you look at things. a chicken and egg dilema. i wish that commercial products were as open as to what you refered to with bugzilla and the works but believe me when i say of what i have seen, not even big companies are able to oblige developers to document 100% all that they coders to. for example, with a product X, they might release an upgrade patch and document about 20 things fixed in it while in truth, 20 is what they bothered to publically report and under the hood they have done so much more. this has really pissed me of. its because of such human behavior that even tools such as bugzilla can not defeat. my means to overcome these is selfconfidence on my troubleshooting and a network of people with and thru whom i can push things forward. with a company xyz, i identified a bug, the process took over 6month and nothing happened. then i remembered that i had a friend who works in the same department. i contact that person and in two weeks i had the patch that i needed. what was a surprise was that even thou the company has "internal open KB", they had a closed bugzilla accessible only by a few and another unofficial kb/bugzilla that the developers used in between of which no one knew but the coders themselves.. hip hip hurraa.....&%#€#"#"%#"#%" got a bit of topic here but i think you got the idea ;)

in regards to openstack, regardless of its challenge and flaws, i still have high expectations for it. i just hope that it wont be the next SNMP. SNMP was supposed to be a common thing among many vendors. the bickering lead to nothing other than the misuse of the protocol killing its actual potential. yes it is in use and yes lots has been done with it. it had a potential the potential to do more.

its usually the small teams that can do more because they have a clear objective, dedication and get version 1.0 completed and take it on form there. in an environment where you have more players and agendas, you are lucky to get version 1.0 released even with half of what was initially planned! such small units/companies are the ones who get bought out because they yeld result. they cash in and then they quit the jobs and everything stays to gather dust. this has happened over and over with opensource and commercial solutions. the big fuss in regards to openstack or other cloud related has more to do than with technology alone. so far the dialog in this thread has been very technology centric.

with the cloud you have several areas that are impacted, the software, the infrastructure and the IT operations. solutions such as OpenStack are infrastructure related, a 1/3 of the whole which alone does not tell the whole story. the big picture includes those three main areas and openstack covers only one of them. yes it has features and promises to do more. but as we know by now, not all features or promises are kept.

the biggest challenge of course is the human factor. it is not easy to accept a change, embrace a change and live with the cahnge. here, there is going to be a change, why so? simply because the maintaining of current contemporary solutions are already classifiable as legacy solutions on the long run. this kind of reminds me of the talks that were going on on why should i port my software form 16bit to 32bit and even to 64bit. wasnt that also a big change? in the more positive direction i beleive...

as i mentioned earlier, there are three distintive areas with the cloud and its transformation.
- the applications
- the infrastructure
- the it operations

in regards to the application, to cut the story short, its about replatforming and porting the solution and as may be required maybe change some of its architecture in order to comply or better fit with the new underlying platforms. think of java, what was java like when it came arround? a huge thing, code once, use on many platform (almost). well, with the transformation, take that a notch up, code once, port to many cloud platforms. look at pivotal or then again at hazelcast. these kind of give you the idea.

in regards to infrastructure, yeah there is lots here, maybe a bit saturated as well but still with lots ahead. the key play here is, to abstract the resources, put the resources into pools and automate and orchestrate. this is where you will see lots of devops activity. whether it is openstack, ceph or viper, the concept yet remains the same.

the dejavu that i get sometimes gives me headaches. i remember 15 years ago tweaking my own automation and orchestration to achieve what is a trend now. i obliged developers to use virtual machines for their java development. i reset it every night overwriting everything from a gold copy. they hated me for that but look at things today and what is happening. you no longer have the fixed devbox. you deploy one when you need it and let it expire after that. i made use of dfs and terminal server where i obliged people to use corporate productivity tools thru that. i had control on the data and how it was accessed. what is the first generation of VDI doing today? just about the same thing. second generation of VDI today sources applications from across platforms.

a bit headache with the infrastructure often has been compliance. there are over 4000 security compliance standards and requirements in europe alone. i do not know of any single product solution that can qualify for all of them. this alone is causing a headache with what the cloud could be and where the data resides or how it is accessed. or then again what about backup and restore solutions, do you want to only backup or do you want to restore or both? and where do you want to backup form and where do you want to restore to? things can get complicated. because of such things, things in the cloud infrastructure will be bundled. for example you order a silver level virtual machine. that would include average storage, sufficient computing, basic backup and restoration and minimum auditing and monitoring, you do not have to select these separatly. they are prebundled and end of story. or then again you have a security access control application that monitors where people walk arround in the bank. you need fault taulerance, high availability, security compliance, zero unplaned downtime, backup, high security auditing, etc in the old days, that would cost you lots of man hours to design, implement and maintain. now you just select a platinum or gold level virtual machine with a preinstalled "apache" for example. it has all the features required at the tick of the box.

in regards to it operations, that is where the big change is occuring as well. how to tackle the weakest link in a chain? the human factor is what i am refering to. well bundle stuff, lay out simplified and standardized processes and workflows. etc. i remember having worked for a customer who used lots of man hours to emulate what today can be done thru defined processes and automation. 10 years ago when i dared to suggest a simplification, i was decapitated simply because the ideas were job position threatening. as an animal survival instict, people always perceive the "danger" first fending of without giving things more thought. in this case people wanted to secure their jobs and way of life. they were not ready to embrace a change. today, if you are not ready to embrace a change, regardless of your competence or character, you will loose your job. here there is a big misunderstanding. what simplification and standardization of workflow and processes is not to cut down on head count but more about enabling the staff to do more. cover more and yeld more. i dont know if you have ever read the comic book "asterix in rome", there is this one scene i never forget. that is when asterix goes to get a document and then faces tremendous amount of beaurocracy. this is what current IT world is like. there is no ill intention, but the various branches of the beaurocracy are not collaborating or supporting each other hence making IT work slow an hard to work in large organizations. a little like once working for a customer, the customer requested access to a nas share. the ticket was dispatched, job done, closed and signed of. it didnt work. so they reopened another ticket for the same thing. this time i was able to map, but i wasnt able to browse. then another ticket was created. i was able to browse and was not able to write. then another ticket was opened becuase i could delete files. makes sense, eh? who is to blame for that.. reminds me of the youtube satire where akhmed the xxx xxxx says "i-kill-you". now that i once again went of track back to topic.

the cloud, the cloud in regards to it operations is to standardize, in my previous example, for example, grant user access request would include the other attributes not to have to recreate a dozen separate tickets and request to access a nas share. certain requirements will be bundled into ready processes and packages to cut down on these. "i need a mailing distribution list", instead of being your own architecture, project manager, delivery manager, implementor, etc. a predefined tick-in-the-box workflow would exist as a template eliminating tidious labour and wasted man hours.

lets be realistic here, private clouds will exist but in not the same abundance as to the resident IT hardware that exists today. there are various reasons for this of which the most important are:
- compliance
- legacy apps
- impact on business
- network infrastructure reliability
- political impact
- natural disaster impact
- etc

legacy app, it could be that a legacy app is cheaper to run in your office than ont he cloud, hence a mini cloud in your office.

in some cases, data "can not leave the building", hence you will need to have a private infrastructure, hence a private cloud something that an be runing for example on convereged solutions such as, nutanix, simplivity or evo:rail. they are small enough to maintain local operations and flexible enough to use locally and connect to a public or remote private cloud.

in some cases (which reminds me of a project 20 years ago where they wanted to remotely monitor and control beverage botling in africa in europe), disregarding network limitations, its one thing to monitor and control remotely, but what if something went wrong? the production line stall and you loose dollars by the hour. in such a case or another that i know of, the cost of such a stall was 30m$/hour. so no, you will not be handling such operations remotely for the next 10-15 years!

if you want to go all public cloud, which woul work for most, you are fully dependent on 99.999% uptime reliability from your infrastucture. can you guarantee that from your local telcom? i can not guarantee that anywhere in europe! just last summer i laughed at the irony where a local company was selling home security solutions based on Mobile tech, it would send you alerts vial sms, mms etc. then we have a long term elecrictal shortage, over 30min. even my work cell phone was down!! nothing was working! this might be ok for some businesses, but not for many. so this would oblige to have a small private cloud to maintain operations during such outages.

politics is a sensitive topic but a world we have to live with. look at russia and ukrain, imagine that you had a DC or used public services from a company residing in the krim. what kind of impact would that have? a nastyone i guess.

what if you have disaster in the area? believe it or not, even big hosting companies are failing with this topic. it was just this week where i heard of an incident where there was a water leakage in a building that the hosting company had leased facilities from. the building maintainer decided to shutdown all electricity in the building not knowing what the isp does there. there went the primary electricity source and shortly after, the backup power. but the backup power wasnt of much benefit because the building cooling system was initially shut which caused an overheat killing the datacenter long before the secondary power ran out. the irony.. all because of a pipe leak and the lack of organization and communication between the landlord and the tenant.

look at solutions such as what riverbed provides, this is part of what the future could be like. in short whehter it is such or a small "cloud" in your office is a case specific question. a huge percentage will be consolidated to the public cloud where you require only the internet access. but then you would plan your business redundancy in accordance as well. if not then the insurance company wont compensate for loss of revenue or business if you do not do your homework yourself. always be up to standards and compliant.

one thing to remember about the cloud is that one of its key characteristics is elasticity. you could have a single vsphere or xen instance in your office yet you might need every once in a while to host more vm than what fits into it. it might not be feasable to install another due to cost or other reasons, what do you do? well, connect it to the cloud. you could elastically push non-critical vm from your box to another hosting company cloud and host the additional vm that you require. and then retrieve them when done and pay as you go. practical, eh?


i once again ran out of time to rant on stuff and neglected to comment on samers enquiry regarding xAAS. sorry dude, maybe tomorrow ;)

now to sleep and dream of beer that i will consume this weekend.. maybe..

i wonder if anyone is reading or thinking of we have been talking about so far. so far it has been cloud related, maybe this could be retitled as cloud foresight. it is a ridicule if Rahmu and I are about the only ones who pitch in. Arithma has been brave and his pitch has been welcome. if anything his pitches have indirectly emphasized some of the things that are being stated.

is there a possibility to add poles to this thread or something?

i took the liberty of retitling this thread as cloud foresight since most of this thread if not all has been cloud centric.