doctorow
2016-09-08 12:52
Here's the two principles, with elaboration @haroldfeld
We must set agreements and principles that allow us to resist the song of the Sirens in the future moments of desperation. And I want to propose two key principles, as foundational as life, liberty, and the pursuit of happiness or the First Amendment:
1) When a computer receives conflicting instructions from its owner and from a remote party, the owner always wins.
Systems should always be designed so that their owners can override remote instructions and should never be designed so that remote instructions can be executed if the owner objects to them. Once you create the capacity for remote parties to override the owners of computers, you set the stage for terrible things to come. Any time there is a power imbalance, expect the landlord, the teacher, the parent of the queer kid to enforce that power imbalance to allow them to remotely control the device that the person they have power over uses.
You will create security risks, because as soon as you have a mechanism that hides from the user, to run code on the user's computers, anyone who hijacks that mechanism, either by presenting a secret warrant or by breaking into a vulnerability in the system, will be running in a privileged mode that is designed not to be interdicted by the user.
If you want to make sure that people show up at the door of the Distributed Web asking for backdoors, to the end of time, just build in an update mechanism that the user can't stop. If you want to stop those backdoor requests from coming in, build in binary transparency, so that any time an update ships to one user that's materially different from the other ones, everybody gets notified and your business never sells another product. Your board of directors will never pressurize you to go along with the NSA or the Chinese secret police to add a backdoor, if doing so will immediately shut down your business.
Throw away the Oreos now.
Let's also talk about the Computer Fraud and Abuse Act. This is the act that says if you exceed your authorization on someone else's computer, where that authorization can be defined as simply the terms of service that you click through on your way into using a common service, you commit a felony and can go to jail. Let's throw that away, because it's being used routinely to shut down people who discover security vulnerabilities in systems.
2) Disclosing true facts about the security of systems that we rely upon should never, ever be illegal.
We can have normative ways and persuasive ways of stopping people from disclosing recklessly, we can pay them bug bounties, we can have codes of conduct. But we must never, ever give corporations or the state the legal power to silence people who know true things about the systems we entrust our lives, safety, and privacy to.
These are the foundational principles. Computers obey their owners, true facts about risks to users are always legal to talk about. And I charge you to be hardliners on these principles, to be called fanatics. If they are not calling you puritans for these principles you are not pushing hard enough. If you computerize the world, and you don't safeguard the users of computers form coercive control, history will not remember you as the heroes of progress, but as the blind handmaidens of future tyranny.