After self-hosting my email for twenty-three years I have thrown in the towel. The oligopoly has won.
Many companies have been trying to disrupt email by making it proprietary. So far, they have failed. Email keeps being an open protocol. Hurray?
No hurray. Email is not distributed anymore. You just cannot create another first-class node of this network.
Email is now an oligopoly, a service gatekept by a few big companies which does not follow the principles of net neutrality.
I have been self-hosting my email since I got my first broadband connection at home in 1999. I absolutely loved having a personal web+email server at home, paid extra for a static IP and a real router so people could connect from the outside. I felt like a first-class citizen of the Internet and I learned so much.
Over time I realized that residential IP blocks were banned on most servers. I moved my email server to a VPS. No luck. I quickly understood that self-hosting email was a lost cause. Nevertheless, I have been fighting back out of pure spite, obstinacy, and activism. In other words, because it was the right thing to do.
But my emails are just not delivered anymore. I might as well not have an email server.
So, starting today, the MX records of my personal domain no longer point to the IP of my personal server. They now point to one of the Big Email Providers.
I lost. We lost. One cannot reliably deploy independent email servers.
This is unethical, discriminatory and uncompetitive.
*Record scratch*
*Freeze frame*
Wait, uncompetitive?
Please bear with me. We will be there in a minute.
First, some basics for people who may not be familiar with the issue.
This doesn't only affect contrarian nerds
No need to trust my word. Google has half a billion results for "my email goes directly to spam". Search any technical forum on the internet and you will find plenty of legitimate people complaining that their emails are not delivered.
What's the usual answer from experienced sysadmins? "Stop self-hosting your email and pay [provider]."
Having to pay Big Tech to ensure deliverability is unfair, especially since lots of sites self-host their emails for multiple reasons; one of which is cost.
Newsletters from my alumni organization go to spam. Medical appointments from my doctor who has a self-hosted server with a patient intranet go to spam. Important withdrawal alerts from my bank go to spam. Purchase receipts from e-commerces go to spam. Email notifications to users of my company's SaaS go to spam.
You can no longer set up postfix to manage transactional emails for your business. The emails just go to spam or disappear.
One strike and you're out. For the rest of your life.
Hey, I understand spam is a thing. I've managed an email server for twenty-three years. My spamassassin database contains almost one hundred thousand entries.
Everybody receives hundreds of spam emails per day. Fortunately, email servers run bayesian filtering algorithms which protect you and most spam doesn't reach your inbox.
Unfortunately, the computing power required to filter millions of emails per minute is huge. That's why the email industry has chosen a shortcut to reduce that cost.
The shortcut is to avoid processing some email altogether.
Selected email does not either get bounced nor go to spam. That would need processing, which costs money.
Selected email is deleted as it is received. This is called blackholing or hellbanning.
Which email is selected, though?
Who knows?
Big email servers permanently blacklist whole IP blocks and delete their emails without processing or without notice. Some of those blacklists are public, some are not.
When you investigate the issue they give you instructions with false hopes to fix deliverability. "Do as you're told and everything will be fine".
It will not.
I implemented all the acronyms1, secured antispam measures, verified my domain, made sure my server is neither breached nor used to relay actual spam, added new servers with supposedly clean IPs from reputable providers, tried all the silver bullets recommended by Hacker News, used kafkaesque request forms to prove legitimity, contacted the admins of some blacklists.
Please believe me. My current email server IP has been managed by me and used exclusively for my personal email with zero spam, zero, for the last ten years.
Nothing worked.
Maybe ten years of legitimate usage are not enough to establish a reputation?
My online community SDF was founded in 1987, four years before Tim Berners Lee invented the web. They are so old that their FAQ still refers to email as "Arpanet email". Guess what? Emails from SDF don't reach Big Tech servers. I'm positive that the beards of their admins are grayer than mine and they will have tried to tweak every nook and cranny available.
What are we left with?
You cannot set up a home email server.
You cannot set it up on a VPS.
You cannot set it up on your own datacenter.
At some point your IP range is bound to be banned, either by one asshole IP neighbor sending spam, one of your users being pwned, due to arbitrary reasons, by mistake, it doesn't matter. It's not if, it's when. Say goodbye to your email. Game over. No recourse.
The era of distributed, independent email servers is over.
Email deliverability is deliberately nerfed by Big Tech
Deliberately?
Yes. I think we (they) can do better, but we (they) have decided not to.
Hellbanning everybody except for other big email providers is lazy and conveniently dishonest. It uses spam as a scapegoat to nerf deliverability and stifle competition.
Nowadays, if you want to build services on top of email, you have to pay an email sending API which has been blessed by others in the industry. One of them.
This concept may sound familiar to you. It's called a racket.
It's only a matter of time that regulators realize that internet email is a for-profit oligopoly. And we should avoid that.2
The industry must self-establish clear rules which are harsh on spammers but give everybody a fair chance.
A simple proposal where everybody wins
Again, I understand spam is a problem which cannot be ignored. But let's do better.
We already have the technology in place but the industry has no incentives to move in this direction. Nobody is making a great fuss when small servers are being discriminated against, so they don't care.
But I believe the risk of facing external regulation should be a big enough incentive.
I'm not asking for a revolution. Please hear my simple proposal out:
- Let's keep antispam measures. Of course. Continue using filters and crowdsourced/AI signals to reinforce the outputs of those algorithms.
- Change blacklisting protocols so they are not permanent and use an exponential cooldown penalty. After spam is detected from an IP, it should be banned for, say, ten minutes. Then, a day. A week. A month, and so on. This discourages spammers from reusing IPs after the ban is lifted and will allow the IP pool to be cleaned over time by legitimate owners.
- Blacklists should not include whole IP blocks. I am not responsible for what my IP neighbor is doing with their server.
- Stop blackholing. No need to bounce every email, which adds overhead, but please send a daily notification to postmaster alerting them.
- There should be a recourse for legitimate servers. I'm not asking for a blank check. I don't mind doing some paperwork or paying a fee to prove I'm legit. Spammers will not do that, and if they do, they will get blacklisted anyways after sending more spam.
These changes are very minor, they mostly keep the status quo, and have almost no cost. Except for the last item, all the others require no human overhead and can be implemented by just tweaking the current policies and algorithms.
Email discrimination is not only unethical; it's a risk for the industry
Big Tech companies are under serious scrutiny and being asked to provide interoperability between closed silos such as instant messaging and social networks.
Well, email usage is fifteen points above social networking.
Talk about missing the forest for the trees. Nobody noticed the irony of regulating things that matter less than email.
Right now institutions don't talk about regulating email simply because they take it for granted, but it's not.
In many countries politicians are forced to deploy their own email servers for security and confidentiality reasons. We only need one politician's emails not delivered due to poorly implemented or arbitrary hellbans and this will be a hot button issue.
We are all experiencing what happened when politicians regulated the web. I hope you are enjoying your cookie modals; browsing the web in 2022 is an absolute hell.
What would they do with email?
The industry should fix email interoperability before politicians do. We will all win.
[1] I didn't clarify this at first because I didn't want this article to turn into an instruction manual. This is what I implemented: DKIM, DMARC, SPF, reverse DNS lookup, SSL in transport, PTR record. I enrolled on Microsoft's JMRP and SNDS, Google postmaster tools. I verified my domain. I got 10/10 on mail-tester.com. Thanks to everybody who wrote suggesting solutions, but I did not have a configuration issue. My emails were not delivered due to blacklists, either public or private. Back
[2] Hey, I get it. Surely my little conspiracy theory is exaggerated. Some guy on Hacker News will tell me that they work as a SRE on Gmail and that I'm super wrong and that there are 100% legit reasons as to why things are this way. Okay. Do something for me, will you? Please unread this last section, I retract it. I just needed to get it out of my system. Thanks for indulging me. Done? Good. Everything else above is a fact. Email in 2022 is anti-competitive. The Gmail guy can go explain himself to the US Senate or the European Commission. Back
The top 13 actionable learnings to sail smoothly through this startup crisis
This week I attended Saastr Europa, the biggest SaaS event in Europe. Of course, everybody talked about the current SaaS "situation".
If you couldn't attend, don't worry. I got you covered.
Here are the top 13 actionable learnings to sail smoothly through this crisis.
1. The crash is real for public companies, not so real for early stage.
SaaS as a category is growing.
But none of that matters. Uncertainty and doubt trickles down. VCs are going to be very cautious for the next months.
Plan for that.
2. Bessemer benchmarked SaaS companies YoY growth
- $1-10M, average 200%. Top 230%+
- $10-25M, average 115%. Top 135%+
- $25-50M, average 95%. Top 110%+
Where are you located?
3. Increase runway!
- Promote yearly upfront payments with an attractive discount
- Improve collections and renegotiate with vendors
- Reduce paid mkt spend. Acquisition for the bottom 20% customers is inefficient, quit those
4. On international expansion
Don't think it's a silver bullet to improve your metrics.
Similar to an unhappy couple having a baby. You will not find PMF in country 2 if you haven't found it in country 1.
Do a lot of research with your early customers.
5. On providing professional services
The true value is not in software but in a solution.
Solution = SaaS + PS
Make PS recurring and pay attention to Gross Margin.
6. Logo retention > ARR Churn
Keeping big logos is important, not only strategically but also because it means you have stickiness and are doing things right.
A VP Sales should be obsessive about logo retention.
7. Transitioning from founder-led sales to a sales team is difficult
Early people are hungry and curious.
Later people are focused on results and process.
Move early people to "builder" projects even outside sales to keep them active or they will leave.
8. Measure Customer Success using an honest metric:
- Slack: messages sent
- Dropbox: files added
- Hubspot: features used
CS is the perimeter of your company. Pay close attention to it and you will see the future.
9. Increase your prices!
40% of companies have already done it.
Avg increase by ticket size:
- $11-25: 18%
- $500+: 34%
Increases in between follow a linear gradient.
10. Don't try to optimise your tech organisation too early.
Technical debt can kill your company after 10 years. But obsessing about practices and optimising processes too early will kill it BEFORE you make it to 10.
Focus on PMF and iterate fast.
11. Let go of bottom 10% performers
If somebody is a clear underperformer it's a great time to let go of them.
Your team knows who's good and who's not. It will improve overall team morale.
12. Net New ARR > ARR
ARR is too big of a metric and can make slight deviations from the plan seem insignificant
NN ARR allows you to discover future cashflow problems much earlier.
13. USA ≠ EU
You cannot open the USA as "just another country". Reserve around $5M to start operations there.
"Looking too European" is a mistake, so is taking American resumes at face value.
Tags: startups
I didn't return my Apple Studio Display
The Apple Studio Display is, unquestionably, a very good monitor.
But the real question is, should you pay €1,800+ for it?
I have been asking that question myself for the past ten days.
Today I decided that while I can't recommend it, I will not return my new monitor. Here's why.
The Pros
I replaced my Benq 27" 2560x1440 IPS LCD display with the Studio Display. These are the main benefits:
The panel. It is good. It could be better, of course. It could have more modern features, too. But it is a good 5k retina panel.
As soon as I started using the Studio Monitor it felt like I had put glasses on. Every other screen looks blurry now.
The rest of the review is accessory to this experience. It is something that you have to, quite literally, see with your own eyes.
The speakers and microphone. They are fantastic. A big step up from any other speakers I've used on any computer.
True Tone. All other displays look bluish now. Great underrated feature.
It is the best-in-class. Simple as that. Its only competitor is the LG UltraFine, which is not much cheaper and lacks other features. If you want a 5K retina display the Studio Display is the best choice.
The Cons
Like everything with recent Apple, there is no progress without compromise.
The stand is too low. Asking users to pay an extra €460 for an adjustable stand is an insult. Sorry but there is no other way to put it. Therefore, this beautiful piece of hardware now stands on top of an ugly PHP reference manual.
Apple, if you ship a monitor with a non-adjustable stand in 2022, please make sure that the default height is at an ergonomic level. It should be at least 5-8 centimeters higher. Since Apple is the company which cares the most about accessibility, no sarcasm here, we can only conclude that this was either a punitive or aesthetic decision.
The screen. It is glossy like all Apple displays. For me this is the first glossy display I've used, so it stands out.
I now notice distracting reflections when working with a dark app. I would have liked to test the nano-textured glass, but I am not going to pay an extra €250 on top of the €1,800.
The speakers. They have too much bass. All audio is artificially deep. For music this is not an issue, but for video conferences it makes every person sound like James Earl Jones.
Clearly Apple never tested for this use case, because they hate video conference users as we will see below.
It doesn't bother me too much, but I wonder why the speakers couldn't sound more natural. It is not a defect of the hardware. Somebody made an odd decision.
The Don't Cares
High brightness. 600 nits is really bright indoors, but if you need such a high brightness level it means you have other problems. If you are in an extremely lit room, maybe with direct sunlight, the reflections will overpower the display brightness.
Good feature, but unlike on a laptop, it doesn't make a difference.
USB-C hub. If you use an Apple laptop you need an external Thunderbolt dock anyway.
I still have six devices which require USB-A and only one which requires USB-C. Therefore, they are plugged to my dock. The extra USB-C ports on the Display remain unused.
It is a nice feature, don't get me wrong, but in 2022 we are not yet in a world where you can ignore USB-A. That is the truth, even if Apple doesn't like it. I know I am asking for an impossible, but if they wanted to make the USB hub useful, they should have included USB-A ports on this monitor.
Non-detachable power cable. I guess it should be user replaceable, but I've never, in my 30 years using computers, have a monitor power cable fail me. It's a non-issue.
That said, mark my words, I hope I don't have to eat my hat in three years.
No buttons or controls. I guess this is a good feature and it does make the display more beautiful, but I really don't care. Hey, this is my review!
The Cruelty
There is a product manager at Apple who, for some reason, hates webcam users with a burning passion.
They despise them so much that they wish they could personally slap each and every webcam user in their ugly, vassal faces.
Since that is physically impossible, they decided to incarnate that slap into the worst webcam Apple has ever shipped.
I compared the 2020 Macbook camera to the 640x480 VGA camera of my 2006 Nokia 5200. Believe it or not, this webcam is worse.
Apple picked up the worst lens SKU they had in stock and hopefully due to a bug they decreased the quality of the image processing pipeline. And this is on an expensive monitor with plenty of physical space to fit a big, quality lens.
WHY, APPLE, WHY?
Gruber found the perfect adjective: cadaveric.
I kid you not, the first call I did with my new monitor, the colleague at the other end, who sees me every day, asked me if I was sick.
For the love of all that's good, Apple, if your webcam can be even slightly fixed with software, please do it.
The Veredict
I was not only ready, I was eager to return this monitor.
I had been keeping some notes where I was compiling my thoughts to help me make a decision.
At the bottom I wrote down what I would do with the return money. "Surprise my wife with an expensive vacation". "Buy new water heater". "Get an electric bike". "Buy two 4K monitors".
But I knew that, below all layers of cynicism and anger, I had to be fair and make a rational decision.
Before putting the monitor back in its box, I asked myself the questions that really matter when making a purchase decision.
- Improvement: Is this monitor better than my current one? YES
- Value: Can I get a better one for that same price? NO
And I did something else: I plugged the Benq back and used it for ten minutes.
Nope.
- Need: Am I willing to go back to my 27" Benq? NO
It's either this or glasses
Once you try the Studio Display every other monitor makes you feel like you need glasses —Apple, feel free to use this benefit in your marketing copy.
I'm approaching 40 and I am starting to realize that my sight is deteriorating a bit. I can see very well, but I find it more difficult to read small text at a distance.
The Studio Display fixes that. I do not need to increase font size anymore. The text rendering is so good that I can continue reading 12pt websites at an arm's distance.
Had I not used it for ten days, my frame of reference would be the same, and I would not "miss" a feature I didn't know existed. I would be happy with my Benq, increasing font size when needed, adjusting to my diminished visual abilities.
Wait, I know what you're thinking. You could have shown me this article ten years ago and I would've dismissed it. "Old people problems", I would've said, "an exaggeration".
But I can't go back. Apple pundits often say that "Apple ruined them with Retina", and I understand it now. It is something you have to experience.
Apple, why do you make this so difficult?
Of course the price tag is expensive. But it's not about the money, it's Apple's bizarre design decisions.
I can understand a compromised monitor for €900. But I can't understand a compromised one for €1,800.
Therefore, my recommendation for you as a reader is that you do not buy it, unless:
- Money is no issue, you only go for quality, not value. In that case, get the adjustable stand too, and maybe the nano-textured glass.
- You really do need a retina screen because you are starting to notice blurry pixels on regular monitors.
Marques puts it very well. The Apple Studio Display is not a good deal.
Unfortunately, it is the only deal.
As for me, there is no salvation. I am ruined by Retina.
You win again, Apple.
I just wish you didn't make every new product a battle. Please, let me buy a product and be happy afterwards without reservations.
Here it is, on top of the ugly PHP book. This setup will remind me every day for the next ten years that an Apple PM decided that not suffering from neck pain should be an €460 upsell.
Do you feel like Google search results keep getting worse?
If you feel like your Google searches are less and less effective, you are not alone.
Michael Seibel, partner at YC and a very good technologist, wrote a Twitter thread which generated thousands of comments on HN
The Internet before Google
You may remember the pre-Google internet, where it was difficult to find content online. Information was spread between the web, gopher, BBSs, newsgroups, and more.
Most webs had a Links section where webmasters recommended similar sites. Thus, whenever you found an interesting page you could discover more like it.
Then directories appeared. Yahoo! started as an index which grouped webpages by topics. Geocities created communities based on interests.
A few years later, search engines as we know them today appeared. Altavista had pretty good search results for the era, but Google disrupted the industry very quickly.
You know the story: they were not the first, but they established themselves as the leaders thanks to the quality of their results. Their founders, Larry Page and Sergey Brin developed the PageRank algorithm at the University of Stanford.
Yes, Google is a successful spin-off from a research department, created by nerds.
The decline of search results
Google has continued advancing their technology, of course. So it seems like it wouldn't make sense that search results get worse instead of better.
What started with a "simple" algorithm which used hyperlinks to establish website authority has been getting more and more complicated.
There are two main reasons:
- The need to understand what the user means and not what they write
- SEO strategies have converted the first page of Google results into a global war
Internet gets popular. The common denominator
In the beginning most of the web users were technically inclined. That is no more, especially with the popularization of the smartphone.
People stopped searching by keywords, and started searching by natural language sentences in all languages in the world.
Therefore, Google must understand the intent of the search given a user query. They use Artificial Intelligence techniques, but that means sometimes they ignore important parts of the query.
For example, ignoring niche words, interpreting correct spellings as typos of a more popular word, changing the meaning of sentences, and more.
The search for this common denominator improves overall user experience at the expense of decreasing the quality of certain searches.
In summary, we all had to learn how to search by keywords many years ago. Google now has learned natural language, and some users will need to re-learn how to use search again.
Ecommerce and product searches
Ecommerce is on the rise. More and more users now search for products and services. Businesses have a great incentive to appear on top of the search results.
In 1998 we searched for information about our hobbies. In 2022 we search to shop. Regardless, our visits to websites are monetized in some way.
SEO techniques try to reverse engineer Google algorithms to appear on top of organic searches. Everybody is gaming the system in their favor.
It is a cat and mouse game where Google does its best to provide a good experience, but in the end, they are judge and jury. Because...
Google is also the top advertiser in the world. Business use SEM to promote their services, and the incentive for Google is to promote SEM results, as they are the ones bringing money to the table.
In the end, everybody is getting worse results. We see aggregator sites which add no value, webs optimized for Google instead of the visitors, and plain scams.
Are there alternatives?
That is a good question. What can we, as users do to improve this situation?
I have been researching alternative searchers and, unfortunately, I don't think they're as good as Google.
First of all, there are only two real alternatives: Bing and Yahoo!. Most of the so-called "alternative search engines" are providing results directly from one of the three above. They are just a layer of paint on top of the Big Three.
There are niche, 100% independent search services which try to replicate the Google of the 90s, but they are very limited. Try them!
- Marginalia
- Wiby
- Brave
- Siri can also provide website results, did you know that?
In another universe we can find regional search engines who actually are more popular than Google in specific geographies. Yandex (43% in Russia), Baidu (76% in China) and Naver (85% in South Korea).
They are not really useful for an American or a European, but it's good to know that they're there.
You may ask yourself, why are there not more alternatives? The truth is that building a search engine is a humongous task, especially in a mature market.
Re-learn how to use Google
My personal recommendation is that you re-learn how to best use Google.
Remember to use the advanced search options.
Log in when searching, because Google uses AI to improve your searches based on past history. The more you search, the better your results will be.
In summary, nobody can trump Google, at least in Western countries.
If you are not satisfied with the quality of search results try some alternatives, but don't expect anything revolutionary.
Get acquainted with the "new Google" and use it for your benefit.
Adapted from my Twitter thread. Follow me on Twitter or subscribe for more!
Tags: internet
Quantum computing keeps advancing, and it looks spectacular
About a month ago IBM introduced the Eagle, its first 127-qubit quantum computer.
And it's breathtaking, both on the inside...
... and the outside
Interestingly, this is not just frivolous design. Besides the futuristic looks, quantum computers require some very peculiar architectural designs.
What makes quantum computers special
Regular computers like the one you're using right now store data in bits. You know, zeros and ones. Bits are electric signals transmitted between electronic components, like transistors.
Quantum computers also use the binary system, but they store data on a different medium. They use particles such as electrons or photons, or superconductor cable loops.
These materials are chosen because they have two quantum features which are required, well, to make quantum computers work.
1. Superposition, or the ability to store different status at the same time. Two bits allow the storage of a small number between zero and four. Two qubits allow the storage of four simultaneous numbers. That's four times as much information.
2. Entanglement, or sharing "data" between qubits. Regular bits are independent, but the status of one qubit can influence another qubit.
Interesting applications
The math and physics are complex, but in summary, quantum computers can handle a huge amount of data. They make current supercomputers look like pocket calculators.
That makes them especially useful to solve problems which can only be solved by testing multiple combinations of numbers. For example, drug discovery, cryptography, planning and routing, weather forecast, etc.
You may realize that those are the same problems where we are applying Artificial Intelligence nowadays. That's no coincidence. AI is a technique to solve complex problems with a bit of intelligence, while quantum computers can bruteforce the solution. And both methods can be useful and complementary depending on the situation.
Quantum is the future, but not the present
While the technology is still immature, scientists are preparing for a world with widespread quantum computing capabilities.
In this world, traditional computing will become obsolete, a lot of problems will need to be reassessed, and others will appear.
Who knows? Maybe in thirty years you will be reading my blog on a quantum cellphone... or whatever it is we will use then.
If you want to learn more, I recommend this article in Nature, this introduction to quantum computing in NewScientist and the very enjoyable TV drama about quantum computers Devs
Adapted from my Twitter thread