March 1st, 2015 — Business and Clients, Programming, Web Development
Here’s a list of things I know, or at least know about, as a web developer. I’m sure I’ve left a lot of things out. Web development is a large and complex collection of technologies, tools, languages, protocols, and services. I started programming for the web back in 1995, so I’ve been able to adapt to changes and learn new tools as they were released. If I had to learn web development from scratch today I’m sure it would take me a long time to master even a few of these things.
If you are doing a new project from scratch you’ll get to pick and choose your tools and “stack.” If you are coming into a team that has already made these decisions you’ll have to adapt. If you work with legacy code you’ll have to get familiar with more languages, frameworks, libraries, technologies, and programming styles.
Web applications usually have two big pieces: the front-end code that runs in the browser, and the back-end code that runs on the server.
The front end (browser side)
You’ll need to know HTML, in detail. HTML5 is the latest but you’ll probably have to know how it’s different from HTML4. You also need to know how styles work: CSS, the box model, inline styles, inheritance, specificity, resets, the differences across browsers and browser versions. You will probably need to know about grid systems, and responsive layouts. Most likely you’ll need to learn one or more of the popular front-end frameworks: Bootstrap, Foundation, Toast, Yaml, etc. You may have to know about CSS preprocessors, like Less and Sass.
If you’re lucky you have some design skills, or you are working with a designer who understands the web. Otherwise you’ll be responsible for layout, typography, colors, image formats and optimization, for a start. Customers often don’t understand that a web page is not a Photoshop proof, that they can’t put a 40MB TIFF file on their home page, or why their favorite font is not available to everyone else.
You probably have to know something about the different kinds of devices people use to browse the web and how that affects design and front-end implementation. Desktop and laptop computers, tablets, and phones have obvious and subtle differences. There’s no hovering on most touchscreen devices, for example. High-resolution Retina or 4K displays may give you grief. Mobile users often have slow connections and data rate caps and limits, so you need to think about that and test on all kinds of devices, with wi-fi and cellular connections.
Somewhere along the way you need to understand the HTTP protocol, at least what request/response and statelessness means. You need to understand cookies, browser caching, expiration, content types and MIME. You have to know about mixing content retrieved with HTTP and HTTPS. You will probably need to understand synchronous vs. asynchronous requests.
The back end (server side)
The web server runs code that may be written in any number of languages, and many web applications use more than one language. PHP is widely used, so is Ruby with the Rails framework, Java, the various .NET languages (C#, VB.net, F#). You may run into code written in Perl, C, C++, Go, Erlang, Python, or something else. All of the big server-side programming languages have extensive libraries and components you will encounter or want to use.
Almost all web applications use a database or some kind of datastore accessible by the web server. MySQL, MS SQL Server, PostgreSQL, SQLite, Oracle, DB/2 are all relational database management systems (RDBMSs) that use the relational model and some dialect of the SQL language. You may use a non-relational (so-called NoSQL) database, and there are many of those: MongoDB, CouchDB, Cassandra, Redis, Memcached. Many web applications use more than one database or data storage tool. The database management stuff may be partially hidden behind an ORM provided by your application framework, but it’s likely you will eventually have to understand the details of how your data is stored and retrieved, how data integrity is assured, and how the data is backed up and secured.
External services give you access to all kinds of things: sending and receiving email, credit card and check payment processing, analytics, maps and location services — the list is endless. If your web application uses external services you will have to learn about APIs, REST, authentication, JSON, XML, RPC, SOAP. You may have to use or write a scraper or crawler that extracts data from sites that don’t offer an API. You’ll probably have to work with curl/libcurl at some point when getting your server to talk to an external service.
Besides the languages there are programming techniques, styles, and workflows you will have to understand: object-oriented programming (OOP), maybe functional programming, certainly imperative programming, test-driven development (TDD), Agile, SCRUM, etc. You may need to use a debugger for the server-side code.
The web server has to be hosted somewhere. It may be a server in the office, or co-located at a hosting facility. More likely it will be a shared, dedicated, or managed cloud instance at Amazon, Rackspace, Google, Microsoft, or any number of cloud hosting providers. Hosting decisions can limit your choice of languages and tools, how much control you have over the server configuration, and things like email handling, DNS, bandwidth limits, and so on.
Security and legal/regulatory issues
As soon as your application goes live on the public internet it will be attacked by bots and hackers. You have to understand common vulnerabilities like SQL injection, cross-site scripting (XSS), cross-site request forgery (CSRF), cross-domain request issues, CORS, session hijacking, spambots. If your employer or customer doesn’t have a system administrator you will have to know about firewalls, keypair authentication, and securing your servers. You have to keep up with security threats and patches to the server operating system and every part of your stack.
If you accept credit cards or electronic payments on your site you will be subject to PCI compliance audits. Other types of personal data you collect and store may be subject to other compliance and audit rules (HIPAA, COPPA). Every country your site does business in will have its own legal and regulatory requirements.
You will need to master at least one text editor. You will have to know how to get your code uploaded to a server with FTP, SCP, or some other deployment tool. You should be using source code version control: git, subversion, Mercurial. Eventually you will have to learn your way around UNIX, the file system, the shell, tools like find, grep, vim, emacs, nano, ssh, cron.
You will have to understand domain name registration, DNS, probably SSL certificates. You will probably want to monitor your server for crashes and outages and spikes in activity than can signal a denial of service attack or something else wrong.
All of the languages, libraries, frameworks, and tools you choose will be updated and patched frequently, so you have to stay on top of that while making sure updates don’t break your application. Even moderately-complex web applications can have tens or hundreds of dependencies at different layers in the stack. Dependency management and managing versions and updates can turn into a risky and complicated problem all on its own, and of course there are tools to learn for that.
A large number of web sites are based on content-management systems: WordPress, Drupal, etc. You may have to know the advantages and shortcomings of those platforms, along with a lot of the stuff listed above.
If your application get significant traffic and users you will need to think about bandwidth (especially costs), performance, scaling. Decisions that made sense during development or with little traffic can come back to plague you when your site gets loaded.
August 23rd, 2014 — Programming
I was asked today on Twitter about how to find work as a digital nomad. My comments are too long for a Twitter reply and may be interesting to people who don’t follow me on Twitter, so here goes.
How I became location-independent
After almost thirty years as a professional programmer working in offices and cubicle farms I started taking on freelance projects on the side, while I was still working full-time at an office job. I found small web site and database projects through word of mouth, by contacting small businesses with broken web sites, by writing articles on this blog, and by presenting at local user group meetings. I got referrals from programmer friends, web site design firms, and even from recruiters. It didn’t take long to build up a steady stream of work.
About the same time I started concentrating on fixing broken web sites and the underlying code, rather than developing new applications or sites from scratch. I explained why that works for me in my article The joys of maintenance programming. My decision to focus on debugging and maintenance was informed by what I saw as high demand and less competition, my ability to figure out and fix legacy code, and by the increasing difficulty I encountered trying to get hired at startups, which I attribute to my age and unwillingness to work crazy hours for pizza and beer.
I took a contract with a company that had a profitable and almost-working web-based business. They had fired their original developers over a contract dispute. The company needed immediate help fixing and enhancing their application, they didn’t make me work at the office, and that turned into a five-year relationship. With a steady income I was able to live anywhere I could get an internet connection.
I was living in Portland and happy to just work from home, but I wanted to travel more. With my kids growing up and moving out I had fewer reasons to stay in Portland, or to stay anywhere. A little over two years ago I gave up my apartment, sold almost everything, put some stuff in storage, and took off on my motorcycle. I was able to stay with friends and family, used airbnb, and sometimes splurged on hotels. I lived in Palm Springs for a month at a corporate condo no one was using during the baking summer. I went to Belize for a few weeks, and loved it, but internet connectivity is poor there and I didn’t get much work done. I stayed in Las Vegas for a couple of months, taking advantage of cheap hotel rooms and food. I was looking for someplace to go to learn to scuba dive when a Thai bartender suggested Thailand. I did some research online and the next day bought a one-way ticket from San Francisco to Bangkok. I’ve been in SE Asia ever since.
Let me explain the specific suggestions from my Tweet:
Working on your own, whether you freelance or have a regular job, requires both discipline and flexibility. You need to get organized and make sure you meet your commitments. You have to adapt to what your clients (or boss) needs so they don’t worry about you working remotely. That might mean staying in one place for a while to concentrate on a project, and taking phone calls at 3am.
You also have to be flexible about traveling and living in a foreign country. Not everything will go as planned. You may not be able to stay somewhere you like because of visa restrictions. High-speed internet at your hotel may be terrible, and in some places reliable electricity is not a sure thing.
Be flexible about the kind of work you will do, and the work conditions. If you are the guy who only wants to write brand-new Python code and you need three monitors and a special chair you will find the digital nomad lifestyle tough. I work from coffee shops, hotel rooms and lobbies, beds and sofas, lounge chairs on the beach, trains, and airplanes. I have an apartment in Bangkok so I am mainly working from home, but I travel on average two weeks every month. I work on whatever web site or application I can fix, whether it’s decaying PHP 4 or brand-new C++ that went into the weeds.
Be flexible about where you go. You may hear about some great place or read about an expat paradise, but find out that it’s not for you. Maybe the weather sucks, or it’s too expensive, or you don’t make friends. Don’t get stuck. Go somewhere else. The world is a big place and you will find places that are interesting and fun to visit and work from.
Some people can’t adapt when things don’t go their way. Some people are easily overwhelmed by not understanding the local language, immigration bureaucracies, getting money from banks, or missing family and friends. You will meet these people everywhere you go, complaining and worrying instead of adapting and enjoying. Working remotely in a foreign country is not like going on vacation. Be honest with yourself, and start out slow with an escape plan in case you don’t like the location-independent lifestyle.
Line up work in advance
If you have a job already find out if you can work remotely. Start out working from home, then go on a trip somewhere you can work from. If your boss doesn’t have a problem with you visiting friends or family and working remotely it will probably not be a big deal to work from Panama or Thailand for a few weeks.
If you don’t have a job or you do freelance work you don’t need permission to work remotely, but you do need clients who don’t care where you are, and you need to have a reasonably steady stream of work. I get work through an agency now but before I signed up with them I had a good roster of clients who sent work my way.
It will be harder to get clients when you are traveling. You won’t legally be allowed to work in the countries you travel to, and even if you find local work the pay will probably be terrible. You can find work online through freelancer sites, but you’ll be competing with people who can live on a lot less than you can. That’s why I advise lining up work in advance. How you do that depends on the kind of work you do. I wrote an article about freelancing as a programmer: Tips for successful freelancing.
There are agencies and recruiters who specialize in matching freelancers with projects, seek them out and establish a relationship.
My clients are mostly small business and non-profits that don’t have an IT staff, and don’t need full-time programming or IT support. Put yourself out there and start asking people you know about small organizations that need some work done. I’ve even found new clients by sending emails to companies that have obviously broken or old web sites, telling them I can fix their problems.
Get banking set up, and other practical considerations
If you work for American or European companies and get paid in dollars or Euros, and live somewhere with a weaker currency and lower cost of living (like Thailand) you can work fewer hours and maintain your standard of living. Or you can work more hours and build up savings. It’s easy to spend too much on travel — airfare, hotels, etc. get expensive, so plan to stay in one place for a while and then move on. In many countries it’s easy to rent an apartment for a month or two, with furniture, kitchen, and internet service.
Getting paid, paying bills back home, and getting money in a foreign country can get complicated, so sort that out before you go. Pay off your bills if you can, and simplify what you can’t. Set up your bills so you get statements by email or online and arrange online bill payment with your bank. Open a bank account that won’t charge you crazy foreign transaction or ATM fees (I recommend the Charles Schwab Investor Checking account). If your bank offers chip and pin debit cards, get one, though you can still use a magnetic stripe card in most places. Get a backup credit card that doesn’t charge foreign transaction fees. I get paid by some clients by wire transfer or direct deposit to my US bank account, and others pay through PayPal. If you open a PayPal Business account you can accept credit card payments (for a fee) and you can get a PayPal Business debit card that comes in handy. I talk to quite a few travelers and expats who have chronic problems getting money from their clients into their hands, so sort that out before you leave, because it’s close to impossible to open a US or European bank account remotely.
If you are an American citizen you will have to file Federal tax returns. Learn about that, information is free on the IRS web site. Depending on your state of residence you may have to file a state return. If you plan to go nomadic you might want to change your state of legal residence to a state that doesn’t have an income tax.
You will need a mailing address in your home country. I use Traveling Mailbox. For a small monthly fee you get a US mailing address, scanning and online mail management, forwarding (including overseas), depositing checks, and package receiving. Many expats use parents, family, trusted friend for handling mail. You may want to get a lawyer in your home country and give them power of attorney, in case you need to have something signed and notarized. Sending documents back and forth by FedEx is expensive and you will have to go to a consulate or embassy to get notary service.
If your passport is close to expiring, renew it before you leave. Many countries won’t issue a visa or admit you if your passport expires in six months or less. If you need a visa for your destination, or you want to stay longer than a visa on arrival allows, get a visa before you leave. Research visa requirements online at official government sites, and don’t trust guide books or postings in online forums. You don’t want to be denied entry because you don’t have correct and current information.
Check if your health insurance covers you overseas. For Americans the answer is probably no. You can buy a travel insurance policy for catastrophic problems. In most countries paying out of pocket for routine and minor medical or dental procedures is practical because costs are a fraction of what you pay in the US. For long-term travel look into expat medical insurance through companies such as Cigna and BUPA, and pay attention to what is covered and if you can use private hospitals. If you have a medical condition or need medications get your prescriptions, on paper, and ask your doctor about generic or equivalent versions sold overseas.
If you need immunizations for wherever you go get those at a travel clinic or from your doctor, and be sure to get documentation. In some countries, such as Thailand, immunizations are advised but not required for entry, and it’s cheaper to get them once you’re here.
Take photos of everything: passport, visa, prescriptions, credit cards, birth certificate, and other important papers and keep them on your smartphone and backed up somewhere like Dropbox.
You can find lots of opinions on what to take, what to leave behind, how to pack, the best luggage, and so on. Everyone has different requirements and preferences. Traveling very light works for me. I don’t want to have to check luggage or carry a huge backpack around in tropical heat and humidity. I don’t want to be a theft target. I’ve been minimalist for a long time and I don’t own a lot of things or get attached to stuff. When I first planned to go nomadic I bought a big 42 liter travel backpack at REI, but when I had it filled up with stuff I found it too heavy and bulky. I returned the big backpack and determined to use the 25 liter Ogio Drifter backpack I used for commuting on my motorcycle. I still use that pack for traveling, though I can no longer fit everything I own into it.
I use a 13″ MacBook Air. It’s light, reliable, and has great battery life. I don’t use a lot of applications: Chrome and Safari, Terminal for remote work, the command line (I’m a long-time Unix user), Vim for editing. I use Google Docs for spreadsheets, drawings, and word processing, with offline editing enabled. I use Parallels for running Windows, mostly for Internet Explorer testing and for clients who have Windows-specific applications. I use LastPass for managing the hundreds of passwords I need for multiple clients. I have StrongVPN for watching Netflix and dealing with the occasional web site that cares that I’m not in the US (bank and credit card sites sometimes do IP geolocation). I carry a USB ethernet adapter and retractable ethernet cable, and an HDMI adapter and cable for plugging into a TV.
I have a Nexus 7 tablet I mainly use for reading books and magazines and playing games. I don’t think of the tablet as an essential piece of gear. I carry a my MacBook power adapter, a small USB charger, a fairly hefty battery backup, and a couple of USB cables. I normally use the camera in my phone, and I have a GoPro camera I use sometimes.
I don’t carry a lot of clothes. Travel clothes can look silly and mark you as a tourist. On the other hand good travel clothes are comfortable, rugged, and easy to wash and dry in a hotel room. Before you leave go to REI or equivalent for some basic items like shorts, a pair of presentable trousers or a dress, walking shoes, a couple of t-shirts, a long-sleeved casual shirt or blouse, and a hat or head covering. It depends on where you go but in SE Asia it’s hot, humid, and rains frequently. I recommend Icebreaker shirts, ExOfficio underwear and clothing, and REI’s house brand. Jeans are heavy, take a long time to dry, and I think they are too hot for tropical climates. Shoes are important, bad shoes will make you miserable. After some trial and error I’ve settled on Keen Clearwater CNX sandals because they’re comfortable and waterproof (and sold in Bangkok). You can buy t-shirts, shorts, swim trunks, flip-flops anywhere and in most places dressing like the locals is cheap and you will blend in better.
I carry an inflatable travel pillow, a travel laundry line for hanging clothes up to dry, a small flashlight, small quick-dry towel. Hotel shampoo works fine for washing clothes. You can buy soap, shampoo, toothpaste, insect repellent, sunscreen, etc. almost anywhere, don’t bother packing a lot of that stuff — you can’t take large bottles of anything in your carry-on anyway. If you have an electric shaver or clippers or any other electrical appliances make sure they work on both 220 and 110 volts, don’t hassle with a bulky, heavy converter. Power plug adapters are not always necessary, and they are available locally for less than a dollar, you don’t need to buy those in advance.
Clean out your wallet. Loyalty cards, airline cards, business cards won’t be needed. Take photos of cards you will need and keep them on your phone or computer. Split your debit/credit cards up so they aren’t all in one place in your luggage.
Don’t worry too much about what to take. You won’t need as much stuff as you think you will, and you can buy most things you need at your destination. Haircuts, shaves, hair wash and blow-dry, laundry may be cheaper to pay for as necessary — that’s certainly true of Thailand and most of SE Asia. For comparison, a haircut costs 100 baht (about $3) in Bangkok, and a straight razor shave is 80 baht.
Stay in touch
When you work remotely it’s very important to stay in constant contact with your clients. You need to answer emails right away and be able to take phone calls. Clients will lose confidence in you if they don’t get fast responses, and the time difference is not their problem. I can handle a lot of issues with my cellphone. Just replying quickly to assure your clients that you got their email and you’re working on the issue is enough to keep them from worrying that you’ve disappeared.
I use a Nexus 5 phone. It works everywhere in the world and I can swap SIM cards easily. Prepaid phone and internet plans are fairly cheap in SE Asia, and easy to set up — you can do it 24 hours a day right at the Bangkok airport. The Nexus 5 also serves as my internet connection when I don’t have decent wi-fi: I can use it as a wi-fi or Bluetooth hotspot, or plug it in to my MacBook and charge the phone while I have it tethered over USB.
If you have a smartphone get it unlocked before you leave. Don’t worry if your phone provider won’t unlock it, though: it’s easy enough to unlock your American smartphone in Bangkok, Singapore, Hong Kong, etc.
I use Skype for calling the US, they have a cheap plan that allows me to call any US landline or cell number, including toll-free numbers, from my computer or phone. I use Google Hangouts for voice and video chat and instant messages to my G+ contacts. I have a Google Voice number that goes to voicemail, for banks and credit cards and other companies that need a phone number. Google Voice is also my outgoing SMS provider so I can send free SMS to phones in the US. I use Twilio for very cheap forwarding of a US phone number to my Thai cell number, that usually has better voice quality than Skype since it works over the cell network.
Going nomadic is easier than most people think, but it’s not for everyone. You need to have skills that you can sell. You need the discipline to work from anywhere and not get distracted. You may get lonely living in a strange place where you can’t speak the language. You will meet all kinds of people, some of whom will try to take advantage of you. Finding work is a constant challenge, and you have to manage your finances carefully.
Be prepared for friends, family, and co-workers to dismiss your plans as silly and unworkable. Don’t worry, you will meet people who are traveling and working and you will make friends and learn from them.
You can find hundreds of digital nomad web sites and bloggers, there’s plenty of information and advice out there. Research as much as you can, but try not to overthink or get worried about minor details.
I’ve learned a lot and had ups and downs since I went nomadic, but I don’t imagine ever going back to a desk job or giving up on seeing as many new things as I can. I’ve been able to reduce the hours I work while improving my quality of life. I keep my skills sharp. Best of all I’ve met some amazing people and made friends I never would have known if I was sitting in an office.
February 10th, 2014 — Business and Clients, Programming
I’ve worked on big projects, small projects, in huge teams and by myself, in fossilized federal agencies and cool Silicon Valley companies. I have learned and used at least twenty programming languages. I’ve lived through waterfall/BDUF (big design up front), structured programming, top-down, bottom-up, modular design, components, agile, Scrum, extreme, TDD, OOP, rapid prototyping, RAD, and probably others I’ve forgotten about. I’m not convinced any of these things work.
[Edit: Let me explain what I mean by writing methodologies “don’t work.” I mean they don’t deliver a predictable or repeatable software development process in and of themselves. I don’t mean that using a methodology dooms the project. Most software development methodologies try to make programming a more engineering-like process, and in that regard they fall short.]
How can you tell if a methodology works?
Whether a methodology works or not depends on the criteria: team productivity, happiness, retention, conformity, predictability, accountability, communication, lines per day, man-months, code quality, artifacts produced, etc. Every methodology works if you measure the right thing. But in terms of the only measurement that really matters—satisfying requirement on time and within budget—I haven’t seen any methodology deliver consistent results.
My own experiences are anecdotal, but they are shared by almost every programmer I know. It turns out that anecdotes are all that anyone has: rigorous studies of software development methodologies haven’t been done because it’s impossible to control for all of the variables.
Try this thought experiment: Imagine two teams of programmers, working with identical requirements, schedules, and budgets, in the same environment, with the same language and development tools. One team uses waterfall/BDUF, the other uses agile techniques. It’s obvious this isn’t a good experiment: The individual skills and personalities of the team members, and how they communicate with each other, will have a much bigger effect than the methodology.
In his 2003 thesis People and methodologies in software development Alistair Cockburn concludes “People’s characteristics, which vary from person to person and even from moment to moment, form a first-order driver of the team’s behavior and results. Such issues as how well they get along with each other and the fit (or misfit) of their personal characteristics with their job roles create significant, project-specific constraints on the methodology. This result indicates that people’s personal characteristics place a limit on the effect of methodologies in general.”
Our own worst enemy
When I started programming in the 1970s software development was tightly controlled by management through a hierarchy of project managers, business analysts, and senior programmers. We did not get to pick the language or tools. I worked on some big, complex projects in companies that worked this way. Some succeeded, some didn’t. Now it’s common for the programmers to choose the methodology and style of working, along with their languages and tools. Analysts are not part of most programmer’s experience anymore, and project managers have been reduced to “team leads” and “Scrum masters,” neutered of any managerial authority and not really in control of anything other than rituals dictated by team consensus.
Strict management, in love with Gantt charts and schedules and documentation, was reduced to “stakeholders” and “users,” and then abstracted away into “user stories.” It’s common now for me to get involved in a project that seems to have no adult supervision. Surprisingly, left to themselves programmers don’t revert to cowboy coding—they adopt or create methodologies stricter and more filled with ritual than anything I experienced in 1980. In fact programmers today can be much more rigid and religious about their methodologies than they imagine a 1970s-era COBOL shop was. I now routinely get involved with projects developed by one or two people burdened with so much process and “best practices” that almost nothing of real value is produced.
Once a programming team has adopted a methodology it’s almost inevitable that a few members of the team, or maybe just one bully, will demand strict adherence and turn it into a religion. The resulting passive-aggression kills productivity faster than any methodology or technology decision.
Does anything work?
My own experience, validated by Cockburn’s thesis and Frederick Brooks in No Silver Bullet, is that software development projects succeed when the key people on the team share a common vision, what Brooks calls “conceptual integrity.” This doesn’t arise from any particular methodology, and can happen in the absence of anything resembling a process. I know the feeling working on a team where everyone clicks and things just get done. What I don’t understand is why I had that feeling a lot more in the bad old days of BDUF and business analysts than I do now.
I think programmers should pay much more attention to listening to and working with their peers than to rituals and tools, and that we should be skeptical of too much process or methodologies that promise to magically make everyone more productive. Maybe social skills come harder to programmers than to other people (I’m not convinced that’s true), but developing those skills will certainly pay off a lot more than trying yet another development methodology.
[Giles Bowkett published an excellent take-down of Scrum that ties in with this article. He covers the shortcomings of Scrum and how “agile” practiced in real life in more (and funnier) detail than I got into here. Why Scrum Should Basically Just Die In A Fire]