I spent more than 21 (at the time of write this post) working on I.T. and I have the opportunity of being involved as early adopter and researcher of cutting edge technology that later becomes a standard and still can remember things like:
Can sounds crazy now, but remember things like Cloud providers, Docker, SDN, SDWAN.... just didn't exist or being research projects some years ago.
Time goes fast (in technology flies) and I was so luckily to work with those disruptive technologies from their born to what are now in many different roles and organizations.
One of the most valuable thing on my career it's the big number of IT people I had opportunity to engage with. My favorite conversation usually start when a "non believer" approach you saying:
Well, the reality is that something must being doing right the main CSPs (AWS, GCP, OCI, Azure) when Cloud market is growing year by year. But, what they offer to catch the attention of organizations?, why is so difficult to build something similar ?.
Lets start from the beginning
The business case:
Each cent you plan to spend in IT must have a business case, no matter if is your homelab (where you plan develop new skills than will let you become a better paid professional), an startup (straggling with the need of raise funds (each cent counts), a big company (running a private or hybrid Cloud) or a Cloud Service provider.
Your business case will define the objectives and IT will translate it to pieces of technology like services, features, etc to bring value to the business.
Build a clear business case with clear objectives is the foundational stone, before start installing the last Linux distro to run Open Stack, Cloud Stack. VMware Cloud... or whatever you plan to use. Maybe you don't need as many (and as complex) services like one of the big four. Think about what you really need and try to keep it simple. Simplicity is the way to other goals like reliability, security, etc.
The Services:
The number or services can be almost endless, but keeping in mind simplicity you can start thinking on provide at least :
You need compute to run your vm instances. In the beginning of cloud days most of the providers start using Xen, later other software become more mature and used and currently KVM is the leader Hypervisor.
You need Storage to make your data persistent and ideally two types of storage one block based and other object based. Later you can start thing about provide different storage layers based in performance (Cold, hot, ..)
You need network to make your services available and you need to thing about how close is your network to your hardware. Concepts like vLans, VxLans/Geneve, SDWan will raise probably.
You need IAM to manage users and roles to assign the right permissions.
You need monitoring capabilities.
Finally you will need a way to control billing (alerts, budgets, etc).
Of course there are other pieces than can be considered foundational (depend of your use case), but this is a minimum set of services.
The software
That's another question that people usually raise because they're curious about what software use AWS, GCP, OCI or Azure to run a cutting edge Cloud.... and there is no simple answer for that question :-), but I can tell you that most of the services can be implemented using software available for everyone. For example, the leading Hypervisor today is KVM that is OPenSource available for everyone. Can we use other Hypervisors? of course Xen, VMware can be used as Hypervisor in most of the Cloud Suites available.
Probably the main question is, do you want to go with a proprietary solution (because the support, software limitations, etc.)? or try to do it yourself using a Open Source project. Maybe an OpenSource and proprietary mix? . There are no perfect answer to that question, only options that can match with your current requirements (take an eye also in you expected evolution).
In the proprietary side you can find between others:
if you prefer be free as a bird them:
can be an option and yes.... probably there are others, but those cover almost 100% of deployments.
Network or how to connect to your cloud
Your new cloud services must me reachable by some way and that is the network. Typically in a CSP services will be accessible through public internet (if you publish them), via IPSec VPN or by dedicated connection.
Use the public internet or a VPN is not rocket science if we talk about dedicated connection you will need to find a provider that connect your remote locations to your cloud datacenters, depend of that location you will have more or less options and prizes. If we pretend also to has dedicated connectivity with one or more of the big 4 providers, then you will need in addition a dedicated link to one of the PoC of those providers.
Big CSPs not always run their own facilities (remember simplicity principle), in many occasions they're collocated in specialized providers liker Equinix, Digital Realty, etc.. If you collocate your cloud or provide easy access to one of those providers, then the provider backbone can be used to reach other regions or other CSPs collocated in the same region.
Multi-cloud is the future of cloud!!. You choose a provider based in their strengths and suffer it, because their weakness. Why get limited by one provider?, maybe you can choose more than one, interconnect it and deploy each service in the best cloud possibe.
Deal with complexity
Don't underestimate how complex can be run something like a CSP and most important don't pretend to be AWS, GCP , OCI or Azure. They are big corporations with huge budgets and an army of engineers, SREs, developers, support teams, field engineers, a big footprint of datacenters globally distributed... What I mean is, only because the big four has amazing services like serverless, Database PaaS, BigData, Managed Kubernetes, etc. you don't need to build it if there is no reason to do that (remember the business case).
If you need a service you can't provide (or even if you can), maybe the cost of consume it from a external provider will be smaller than the cost and effort needed to run it yourself.
Multicloud ready
In the beginning there was only AWS and all other vendors deal with this situation, fighting to get part of the AWS (current or potential) market share. Choose your cloud provider was like choose your football team and vendors invest a lot of in build communities of fan boys :-).
Today situation is becoming different and is no unusual find organizations running a few services on provider A and other in provider B, with both clouds connected. Find what you do great and integrate with others that can bring you the value you business need and you're not ready to deliver yourself.
A perfect example of that approach can be found in the multi cloud strategy from Oracle OCI. They provide some of their key services to other providers like AWS, Azure or GCP so users of those providers can consume OCI services like any other native service from their provider or through high performance and resilience connectivity.