You Should Be Using Email “Plus Addressing”

There’s a great trait of most major email providers sometimes called plus addressing.

It gives you access to unlimited email addresses based off your one single email account. If you add a “+” sign in your email name, followed by anything, it will still get delivered to your inbox. So:

  • will+extra@example.com will go to will@example.com
  • steve+august2014@example.com will go to steve@example.com
  • ops+salesforce+account@example.com will go to ops@example.com
  • and so on….

Lots of power-users use this feature to help automatically filter email into folders/labels. Some give out a unique email to every site to help figure out which services are selling their address to spammers.

This feature is probably supported by your mail provider. At least including GMail, Yahoo, Microsoft, Apple, and FastMail. (And if that doesn’t cover you, you probably have bigger problems, like you are running your own inbound mail server, yikes!)

Support Emails With +’s

Pretty please, if you have a signup form in your application that takes email addresses, test to be sure it accepts emails with plus-signs in the recipient part. If it doesn’t, give your development team a demerit for me.

Use Plus Addressing Liberally When Testing

You probably have to regularly test a signup process. In fact, I’d wager that signup flows get manually tested more than any other class of feature.

Rather than jump through hoops creating all sorts of dummy email addresses, or aliases to your Google Apps users, use plus addressing. If I’m testing a new welcome flow, I’ll use addresses like:

  • will+welcome1@example.com
  • will+welcome2@example.com
  • will+welcome3@example.com

Bonus Tip: example.com is a special reserved domain, which you can guarantee won’t actually send anyone an email. Of course, if you need to receive a welcome email or other part of your workflow, be sure to plus-address your own actual email address!

3 Rules for Choosing Startup Technologies

Black, White, and Oh-So-Gray

Despite our more impassioned wishes, technology choices (like all worthwhile choices in life) are rarely black or white.

Show me a piece of technology advice on the Internet, and I’ll show you three pieces of conflicting advice. Such is the nature of both debating on the Internet, and making decisions in a rapidly evolving and still relatively nascent industry.

I see founders all the time paralyzed by technology choices, deciding to wait to build until the right tech falls in their lap. Or deciding to keep up with horrible home-grown solutions just in case some software package might not meet exactly their needs.

How To Make A Technology Choice

When in doubt, follow these three rules:

  1. Choose tech that has lots of customers like you and a thriving ecosystem. Avoid solutions that are only used by the Fortune 500, or by the one pre-revenue startup that hired the guy who built the tech in the first place.
  2. Choose tech that’s specifically designed for your purpose. Avoid solutions that try to solve all your problems with one monolithic package.
  3. Choose tech that your developers are excited about. They will work harder, faster, and more creatively with tools and services that inspire them. Avoid forcing anything on your team that will be an excuse for building anything less-than-awesome.

Embrace the impossibility of a one best choice, and relax. Choose with informed data, and your gut. There are as many ways to solve a given technology problem as there are technologists, so chances are good that whatever informed choice you make will be capable of delivering for you.

Only You Can Choose

Choices aren’t made by committee (at least not smart choices…)

Make a confident choice, and go solve some problems with it!

12-Factor Apps in Plain English

Popular platform-as-a-service provider Heroku (now a subsidiary of Salesforce…fancy that…) maintains a manifesto of sorts called The Twelve-Factor App. It outlines a methodology for developers to follow when building modern web-based applications. Despite being partly self-serving (apps built like this will translate more naturally to running on Heroku), there’s a lot of meaty best-practices worth examining.

Strive for These Best Practices

I think these concepts are important for readers of ClearlyTech, even if you aren’t the developer! For those who desire to know why this stuff is important, or who want to have an intelligent conversation with their development team about these issues, I present to you 12-Factor Apps in Plain English:

I. CodebaseOne codebase tracked in revision control, many deploys

Put all your code in a source control system. Heck, just put it up on GitHub from the start.

All your application code lives in one repository. Once you get big, you may have a distributed system with multiple apps talking to each other (like a web application and a backend API), at which point you can treat them as separate apps with their own code repositories (still in source control, of course).

A codebase is run by developers on their local machines, and deployed to any number of other environments, like a set of testing machines, and the live production servers.

Importance: Non-negotiable Everyone does this, and developers will laugh at you if you aren’t.

II. DependenciesExplicitly declare and isolate dependencies

All the environments your code runs in need to have some dependencies, like a database, or an image processing library, or a command-line tool. Never let your application assume those things will be in place on a given machine. Ensure it by baking those dependencies into your software system.

Most languages and frameworks provide a natural way to do this. You list all the versions of all the libraries you expect to have in place, and when the code is deployed, a command is run to download all the right versions and put them in place. No guesswork, everything as it needs to be.

This philosophy extends to your devs or devops team managing entire machine configurations using management tools like Chef and Puppet

Importance: High Without this, your team will have a constant slow time-suck of confusion and frustration, multiplied by their size and number of applications. Spare yourself.

III. ConfigStore config in the environment

Configuration is anything that may vary between different environments. Code is all the stuff that doesn’t.

The code that talks to your database will always be the same. But the location of that database (which machine it’s running on) will be different for a local developer machine than it will for your production servers. Likewise, in your testing environment, you might want to log debugging information about each web request, but in production that would be overkill.

Usernames and passwords for various servers and services also count as configuration, and should never be stored in the code. This is especially true because your code is in source control (see I. above) which means that anyone with access to the source will know all your service passwords, which is a bad security hole as your team grows.

All configuration data should be stored in a separate place from the code, and read in by the code at runtime. Usually this means when you deploy code to an environment, you copy the correct configuration files into the codebase at that time.

Importance: Medium Lots of companies get away without this, but you’re sloppy if you do.

IV. Backing ServicesTreat backing services as attached resources

Your code will talk to many services, like a database, a cache, an email service, a queueing system, etc. These should all be referenced by a simple endpoint (URL) and maybe a username and password. They might be running on the same machine, or they might be on a different host, in a different datacenter, or managed by a cloud SaaS company. The point is, your code shouldn’t know the difference.

This allows great flexibility, so someone from your team could replace a local instance of Redis with one served by Amazon through Elasticache, and your code wouldn’t have to change.

This is another case where defining your dependencies cleanly keeps your system flexible and each part is abstracted from the complexities of the others…a core tenet of good architecture.

Importance: High Given the current bindings to services, there’s little reason not to adhere to this best-practice.

V. Build, release, runStrictly separate build and run stages

The process of turning the code into a bundle of scripts, assets and binaries that run the code is the build. The release sends that code to a server in a fresh package together with the nicely-separate config files for that environment (See III. above). Then the code is run so the application is available on those servers.

The idea here is that the build stage does a lot of heavy lifting, and developers manage it. The run stage should be simple and bullet-proof so that your team can sleep soundly through the night, knowing that the application is running well, and that if a machine gets restarted (say, a power failure happens) that the app will start up again on launch without the need for human intervention.

Importance: Conceptual From a practical perspective, the tools and framework you use will define best-practices for building, deploying, and running your app. Some do a better job than others of enforcing strict separation, but you should be okay if you follow your framework’s suggested mechanisms.

VI. ProcessesExecute the app as one or more stateless processes

It’s likely you will have your application running on many servers, because that makes it more fault tolerant, and because you can support more traffic. As a rule, you want each of those instances of running code to be stateless. In other words, the state of your system is completely defined by your databases and shared storage, and not by each individual running application instance.

Let’s say you have a signup workflow, where a user has to enter 3 screens of information to create their profile. One (wrong) model would be to store each intermediate state in the running code, and direct the user back to the same server until the signup process is complete. The right approach is to store intermediate data in a database or persistent key-value store, so even if the web server goes down in the middle of the user’s signup, another web server can handle the traffic, and the system is none-the-wiser.

Importance: High Not only is a stateless app more robust, but it’s easier to manage, generally incurs fewer bugs, and scales better.

VII. Port bindingExport services via port binding

We’re getting a bit technical now, but stick with me. This factor is an extension of factor IV. above. The idea is that, just like all the backing services you are consuming, your application also interfaces to the world using a simple URL.

Usually you get this for free because your application is already presenting itself through a web-server. But let’s say you have an API that’s used by both your customers in the outside world (untrusted) and your internal website (trusted). You might create a separate URL to your API that your website can use which doesn’t go through the same security (firewall and authentication), so it’s a bit faster for you than for untrusted clients.

Importance: Medium Most runtime frameworks will give you this for free. If not, don’t sweat it. It’s a clean way to work, but it’s generally not hard to change later.

VIII. ConcurrencyScale out via the process model

When running your code, the idea is that lots of little processes are handling specific needs. So you might have dozens of handlers at the ready to process web requests, and another dozen to handle API calls for your enterprise users. And still another half-dozen processing background welcome-emails going to new users, or sending tweets for your users sharing things on your social media service.

By keeping all these small parts working independently, and running them as separate processes (in a low-level technical sense), your application will scale better. In particular, you’ll be able to do more stuff concurrently, by smoothly adding additional servers, or additional CPU/RAM and taking full advantage of it through the use of more of these small, independent processes.

Importance: Low Don’t worry about this factor until you get pretty deep into scaling considerations. Trust your chief architect or CTO to raise the red flag if this is going to become an issue for you.

IX. DisposabilityMaximize robustness with fast startup and graceful shutdown

When you deploy new code, you want that new version to launch right away and start to handle traffic. If an application has to do 20 seconds of work (say, loading giant mapping files into RAM) before it’s ready to handle real traffic, you’ve made it harder to rapidly release code, and you’ve introduced more churn on the system to stop/start independent processes.

With the proliferation of so many 3rd party libraries in today’s software systems, sub–1-second startup times are less and less common. But beyond loading code, your application should have everything it needs waiting in high-speed databases or caches, so it can start up snappily and be ready to serve requests.

Further, your application should be robust against crashing. Meaning, if it does crash, it should always be able to start back up cleanly. You should never do any mandatory “cleanup” tasks when the app shuts down that might cause problems if they failed to run in a crash scenario.

Importance: Medium Depending on how often you are releasing new code (hopefully many times per day, if you can), and how much you have to scale your app traffic up and down on demand, you probably won’t have to worry about your startup/shutdown speed, but be sure to understand the implications for your app.

X. Dev/prod parityKeep development, staging, and production as similar as possible

It has become in vogue in recent years to have a much more rapid cycle between developing a change to your app and deploying that change into production. For many companies, this happens in a matter of hours. In order to facilitate that shorter cycle, and the risk that something breaks when entering production, it’s desirable to keep a developer’s local environment as similar as possible to production.

This means using the same backing services, the same configuration management techniques, the same versions of software libraries, and so on.

This is often accomplished by letting developers use a tool like Vagrant to manage their own personal virtual server that’s configured just like production servers.

Importance: Medium Developers will feel like taking shortcuts if their local environment is working “well enough”. Talk them out of it and take a hard-line stance instead, it’ll pay off long-term.

XI. LogsTreat logs as event streams

Log files keep track of a variety of things, from the mundane (your app has started successfully) to the critical (users are receiving thousands of errors).

In an ideal situation, those logs are viewed by developers in their local consoles, and in production they are automatically captured as a stream of events and pushed into a real-time consolidated system for long-term archival and data-mining like Hadoop.

At the very least, you should be capturing errors and sending them to an error reporting service like New Relic or AirBrake. You can take a more general approach and send your logs to a service like PaperTrail or Splunk Storm.

Importance: Low If you are relying on logs as a primary forensic tool, you are probably already missing out on better solutions. Be sure to consolidate your logs for convenience, but beyond that, don’t worry about being a purist here.

XII. Admin processesRun admin/management tasks as one-off processes

You’ll want to do lots of one-off administrative tasks once you have a live app. For example, doing data cleanup on bad data you discover; running analytics for a presentation you are putting together, or turning on and off features for A/B testing.

Usually a developer will run these tasks, and when they do, they should be doing it from a machine in the production environment that’s running the latest version of the production code. In other words, run one-off admin tasks from an identical environment as production. Don’t run updates directly against a database, don’t run them from a local terminal window.

Importance: High Having console access to a production system is a critical administrative and debugging tool, and every major language/framework provides it. No excuses for sloppiness here.

Summary

Some of these items may seem esoteric, as they are rooted in some fundamental systems design debates. But at the heart of a happily running system is an architecture that is robust, reliable, and surprises us as little as possible. These 12 factors are being adopted by most major software platforms and frameworks, and to cut corners against their grain is a bad idea. Discuss these issues with your development team, see if there are some quick wins to improve the quality of your application design.

Best Cloud Infrastructure Provider

New Cloud IaaS providers are coming out of the woodwork these days. Here are ClearlyTech’s current picks for the top-tier solutions you should consider before you deploy to the Cloud.

Large Scale Production Needs

Amazon AWS is the 400lb. gorilla in the space. Their pure-play IaaS product is their Elastic Compute Cloud (EC2). It’s the largest, oldest, and most feature-rich of its peers.

Amazon has had its issues over the years, sometimes bugs, sometimes bizarro pricing, and their share of highly publicized downtime. But they are a giant, and with all that experience comes a maturity that should count a lot when hunting for a reliable partner to run your high-traffic application. They are a tried-and-true solution at this point, warts and all.

Note that Amazon may not be (probably isn’t, in fact), the best technical solution. Other newer entrants to the game have had the opportunity to watch Amazon’s trials over the past 7 years, and improve in the few areas that Amazon has shown weakness.

I count two auxilliary areas I’ve seen AWS fall short.

  1. Amazon has terrible customer support for AWS. Even if you pay for the Gold support, I’ve found it’s impossible to get a truly knowledgable person on the phone, especially during a crisis situation. Amazon appears to be taking the Google approach to support. Make sure everything runs right, so you never have to interact with your customers. And when things don’t work, throw them just enough of a bone so they don’t leave, while you fix the problem.
  2. Amazon is starting to feel like an aging platform. Network and Disk I/O in particular have inconsistent performance. While they are upgrading all the time and rolling out new tiers of machine (like Provisioned IOPS, which is consistent, but still not particularly fast), and recently SSD drives, AWS is at risk of getting passed by some players investing heavily in newer infrastructure with a fresher architectural approach.

Despite these minor issues, ClearlyTech favors Amazon AWS for large scale production needs. Gartner Group agrees so strongly that AWS lives in a quadrant all its own among cloud hosting providers in their 2013 report.

Gartner 2013 Cloud Hosting Comparison
Gartner 2013 Cloud Hosting Comparison

Another Mature Player

ClearlyTech recommends Rackspace as another long-standing player in the hosting business worth a look. Rackspace was a pioneer in the managed-hosting space (they run your traditional infrastructure), and has recently put a lot of attention into a Virtualized Cloud hosting model to compete with AWS.

If the lack of good customer support from Amazon is a deal-breaker for you, Rackspace is a very appealing alternative. Every time I’ve called Rackspace, a real person answers the phone, is sufficiently technical, and handles my needs quickly and painlessly. Their commit to customer support shows.

Personal Projects, Prototypes, and Developer Playgrounds

For pure ease-of-use, low-prices, ClearlyTech supports Digital Ocean, the self-described “Simple Cloud Hosting”. They are built on a new clear technology stack, including all SSD drives, mitigating some of the risk of bad disk I/O slowing down an otherwise sufficient server.

This is a great option if your developers need extra horsepower without a lot of hassle, or if you need to get a prototype up for customer testing on a public IP address with easy setup/teardown process. They are a pleasure to use, and time will tell whether they keep expanding their offering into something mature enough to use for more serious production deployments.

Your Own Iron

If you want dedicated hardware, you could go to Rackspace. But for that, ClearlyTech supports Softlayer

Thousands of reputable companies rely on them for no-frills managed data-center hosting. You may still need some sysadmins to run the show, but SoftLayer consistently has good prices on excellent hardware and extremely reliable hosting.

Some to Watch

There are so many players out there, we won’t even try to mention more than a few alternatives. But we’re keeping an eye on a few

  • Google Compute and associated services is going to make a run at AWS in the next few years. They’ve learned a lot from watching Amazon, from their own experience with huge scale provisioning of cloud resources, and from App Engine, their initial platform as a service cloud play, which failed to gain traction among serious startups and open-source developers.
  • Microsoft Azure has a tough row to hoe, as fewer and fewer companies are deploying Microsoft technologies to the cloud vs the rapidly evolving open-source stacks. But Microsoft isn’t taking it lying down. We’ll be watching what innovations they can bring to the table, especially given that they control the hardware and the software stack. And if you need to deploy Windows to the cloud, we prefer them over Amazon EC2.

The Cloud – Infrastructure As A Service

cloud-ethernet

This article is part of a series on the defining value propositions of cloud computing platforms. The whole series includes:

Infrastructure As A Service

The main value proposition usually associated with the cloud is that it replaces the need for purchasing, racking, and managing your own computers, storage, and networking hardware in a datacenter. This is far from the only benefit of the cloud, but it’s certainly a major one. Amazon CTO Werner Vogels describes it nicely:

The cloud lets its users focus on delivering differentiating business
value instead of wasting valuable resources on the undifferentiated
heavy lifting that makes up most of IT infrastructure.

The “undifferentiated heavy lifting” that Vogels describes includes things like: procurement of hardware; building/renting space, network, and cooling in specialized datacenters; physical machine security; hardware monitoring; network administration; and all the associated capital costs and bookkeeping.

The general model for IaaS is that you can click a few buttons in a web interface (or make a few API calls, if that’s how you roll), and a new computer boots up somewhere, just for you, an instant toy/tool, that acts just like a machine that you unboxed and racked yourself, except it’s powered, networked, and cooled, with an OS ready to go.

Let’s savor the appeal of having powerful thousand-dollar gadgets just appearing with the click of a button. Seriously, just savor that. It’s like geek Christmas, every day….. Man, do I the cloud.

The Dirty Truth of The Cloud

IaaS providers do their best to hide from you the dirty truth of the cloud: cloud machines are almost always virtual machines, not real ones. When you launch a new machine, it’s the server-grade equivalent of running Windows on your Mac with VirtualBox or Parallels. Major cloud providers use what’s called a Hypervisor process, usually Xen or VMWare ESX to run many logical (guest) servers on one physical hardware (host) box.

Underlying this simple idea is a lot of complexity. Sometimes the abstraction of “just like a regular machine” leaks out in how you provision, manage, or monitor cloud machines. But in the past few years, as the startup world has moved rapidly to managed infrastructure, there are fewer and fewer issues with running your production system in a virtualized cloud environment.

If you want a managed infrastructure, but on dedicated hardware, you can either opt for something like EC2 Dedicated Instances, or you can move to a provider that allows you to select your hardware, but then manages it for you, like IBM’s SoftLayer

IaaS is Often Cheaper, Especially At First

Cloud Services providers are constantly promoting how much cheaper it is to run in the cloud. And it is. Sort of…. Like most cost calculations, the truth is it depends.

Capex becomes Opex

At first, there’s no doubt that the Cloud is cheaper. The capital expenditure for hardware and people to run it is a significant cost for any startup, especially in the modern era of lean startups (both financially and philosophically lean).

Swapping that capital expense for monthly operating costs allows a startup the low up-front cash, and the operating flexibility to modify infrastructure as necessary to meet the growing or (more often than we’d like to admit) shrinking needs of the startup.

The Total ROI Is Huge

The most common mistake I see founders making when evaluating the ROI of a cloud deployment, is that they stop at running the raw price calculations. And of course, they forget all the components involved with a robust infrastructure. They focus solely on the CPUs and datacenter cost, but skip all the networking gear, KVMs, spare parts, hard drive replacements, etc.

There’s little doubt that IaaS pricing is a win for the little guy (and the providers aren’t faring too badly either!). The providers get to buy in bulk, and rent you instance access at a reasonable markup. Given the rates a giant like Amazon gets on hardware and bandwidth, it’s quite likely that even with their markup, it still costs less than if you paid small startup prices yourself. Not to mention your likely preference for operating vs capital expense.

However, the real win comes in time-savings for your technical team. I can’t stress enough how much of a distraction it is to run your own infrastructure. Servers are not a set-it-and-forget-it kind of resource! Any savings you get by penny-pinching onto your own hardware (maybe not much savings at all) will quickly be made back when your developers aren’t woken up in the middle of the night to replace hardware or debug network issues.

Is Cloud IaaS Right For Me?

If you have to ask the question, then the answer is quite likely yes.

Most startups these days that decide to rack their own hardware know in advance that it’s a core competence (a pillar for your business). My friends at ObjectRocket launched to build the best hosted MongoDB solution available. They are systems and performance experts, and that’s what they were selling, so owning their entire hardware stack was crucial to maintain the control they needed. And they were willing to spend the capital up-front, knowing they could pass the long-term savings on to their customers. They did such a good job building their own infrastructure play that RackSpace acquired them very early for a tidy sum.

You are paying two kinds of overhead when you deploy to the cloud:

  1. A virtualized environment comes at a performance cost. It’s very efficient in terms of CPU and RAM performance (maybe a 2–5% hit vs. dedicated hardware), but shared network, shared disk I/O and other factors mean that you won’t have screaming fast hardware at your disposal. And that’s okay. Very few companies need that.
  2. There’s a man in the middle (your IaaS provider) marking up the hardware on you. If you need lots of hardware, and can get bulk rates when buying it, and have the capital to spend, simple economic theory suggests that cutting out the man in the middle will give you the most bang for your buck.

If you absolutely have to have the most hosting performance for your dollar, and you don’t require the flexibility to scale up and down very much, you might consider building it yourself. But if you are going to build your own infrastructure, remember that you need to budget for at least one (recommended three) full-time system and network admins. In my experience, you shouldn’t even start to think about building your own infrastructure until you know you’ll need at least 100 machines.

Working With Makers

As you grow a team of developers, designers, and other creatives (yes, programming is creative. Don’t make me explain it to you), it’s crucial that you pro-actively remember to treat your team as the makers they are.

I’m not going to recap the different between Makers and Managers, because the venerable Paul Graham, as usual, said it best.

Go read Paul’s short essay on the topic this very minute.

And for you visual learners, I offer this gem of a diagram to aid your learning:

A Day In The Life Of A Programmer

You might be forgiven for assuming I’m being funny, but I assure you, it’s very true. Underestimate the power of its message at your company’s peril!