Scaling Web Applications on PostgreSQL pgConfSV 2015 presentation

pgConf 2015 PostgreSQL Developer Conference in the Bay Area

I was very happy to have my submission accepted at the recent pgConfSV conference (where SV is for Sillicon Valley). For various reasons I was unable to cover everything I wanted during the talk, which is a note to self for future public speaking engagements! Time your talk! :). Well, below is an updated version of that presentation, which shows an incremental and methodical path to scaling web applications to millions of users using PostgreSQL, all the while covering a very range of material.

Audience

In general, the ideal audience for this is operationally and architecturally minded full stack engineers, building web apps that either are already serving a ton of traffic, or will be soon.

But on the broader scale, I was intending for this presentation to be helpful to anyone trying to get a grasp on how to evolve their web application to where it's able to serve a rather high throughput of 5K-50K requests per second. This range is still far below what the internet "giants" such as Facebook, Google, or Twitter get (if I had to guess, it would be in 1M/sec). But, it is also far from an early naive web application with just a few users.

Turns out it is possible to achieve high scalability on the cheap, and using PostgreSQL, which is what we did at Wanelo, and it turned out great.

Presentation

Thanks!

↳ Keep reading …

Serial Console Hacks with Arduino

Battling Console

I wanted to share a method that I use to connect to a Serial port of any Arduino I am using at any given moment. This method has a caveat, in that if you have more than one Arduino connected, it will pick one of them at random.

Motivation

I always hate Serial port windows. They do not automatically reconnect, and if they try (Eclipse) they don't always work (Teensy). So I went searching for a reliable solution that will automatically reconnect after loosing a connection.

I found it! It's called minicom!

↳ Keep reading …

Music, DJ-ing, and Origins of Weird Names

– So, did you know?

if (my.question.style_of(blog.post.style) == FALSE) {
  omg() || exit() || return; || run(SUPER_FAST);
}

Me: – "So, are you prepared to listen to some music?"

You: – "What the hell are you talking about, this is Electronics Blog, and I came here to read about my goddamn chip that's not working... You are breaking my brain!!!"

Me: – "I knew you would love it! Then let's begin :)"

You: – "Grrrrr...."

/* music_info.ino  */

curtains = new TheaterCurtain();
if (!curtains.open()) {
  /* proceed without the curtains */
  return ("true" || 1 || TRUE);
}

↳ Keep reading …

How to use cheap Chinese Arduinos that come with with CH340G / CH341G Serial/USB chip (Windows & Mac OS-X)


  • Updated Oct, 2016 with the new signed driver for OS-X Sierra
  • Updated Nov 22, 2015 with the new signed driver for OS-X El Capitan and Yosemite
  • Updated Jan 9, 2016 with Windows Drivers

My golden rule is that if something took me longer than 15 minutes to figure out, then it's worth documenting in a tiny blog post so that it would save time to others, just like many other similar posts saved me million hours by providing simple clear instructions.

Introduction – What is CH340G?

↳ Keep reading …

Announcing LaserCutter ruby gem and MakeABox.IO web site

My day job, @ Wanelo, requires cranking on all cylinders, and so I've been pretty busy with life, the universe, and everything, although it mostly means Work, the Wife, and the Cat). In all this constructive chaos, I totally forgot to mention, and properly introduce to the world something I've been working on during many sleepless nights this September :)

So without further ado, LaserCutter –– ruby gem (a library and a CLI) for making PDF designs of laser-cut boxes, which fit/snap in together at all edges using tabs that go in and out. The output of the library is a PDF document. Typically next step would import that PDF into Adobe Illustrator for additions and touch ups, and then sent off to a laser cutter for the actual, well, cutting.

In addition to the ruby gem, there is also simple web front-end, called MakeABox.IO, which can get you started creating boxes without installing any gem, or running anything on the command line.

If you've made any enclosures of your own, like either for your electronics gear, or a lighting show, or your cat auto-flushable toilet, you probably used one of the existing and free tools out there. I did not find them all when I started making boxes, but over time the choice seemed to come down to three options:

↳ Keep reading …

BORAT: Bathroom Occupancy Remote Awareness Technology with Arduino

A great company where I work, Wanelo, with all of it's 35 people, recently moved to a new office which has only two single-occupancy bathrooms, each on a separate floor.

A few times a day, when the need was strong, I'd go to the downstairs bathroom only to find its door locked. I'd then run upstairs and find the other one locked too... Damn! I'd come back down only find out that someone else grabbed the first bathroom while I was upstairs. Argh!

You can likely see how this could be a frustrating and disruptive experience even just once or twice. Now multiply that by 35 employees and every work day of the year, and you end up with an actual productivity problem.

Given my foray into Arduino over the last few months, I knew I could come up with a solution. I got approved for a small budget of about $200, and started looking around.


The problem was very simple: people needed to know when each bathroom was occupied, or not. Just like on an airplane you can see bathroom light on/off, I wanted something similar for our two bathrooms. Something everyone could see.


↳ Keep reading …

Choosing The Right Arduino IDE (And The Winner… Is Eclipse!)

As a relative late comer to Arduino world, I went through the beginner tutorials and examples using the provided Arduino IDE. I faithfully downloaded it from arduino.cc web site, took a quick tour and was pretty unimpressed with the set of features. For one, I am very particular about the color scheme of my programming editors, and not having a choice was an immediate downer.

Arduino IDE is simple to use, and I think this was one of the key design goals for this software. It is commonly used to introduce many folks to programming electronics, and just programming, notably in C/C++. As such, this tool is pretty limited, and limiting too. Very quickly I found myself very stuck unable to browse through external symbols by clicking through them, using auto-complete features, refactor code, and so on: all the "basic" features I so got spoiled with, by tools such as JetBrain's RubyMine, AppCode, IDEA, open source (but originally IBM's) Eclipse, and not to mention Apple's XCode.

Arduino IDE

After going through several example projects using the IDE I became more and more frustrated with its limitations. Programming hardware is hard enough (pun intended), and the IDE is supposed to make life easier.

For example,

  • How do I explore the source code of the included libraries that are being used by my sketch?
  • Why does "Import Library" insert a new #include into my sketch when I already have one?
  • Why is tab indentation not maintained as you move to the next line while editing?
  • Why is the board and port selection remain global, and are not assigned to each sketch? Can't I be working on multiple boards at the same time, especially, say, when they talk to each other?
  • And of course... why can't I change the goddamn colors :)

So I became worried that my foray into electronics would die young if I didn't find a more capable programming alternative, and so I started exploring.

Arduino IDE Alternatives

First off – a tiny disclaimer: this is not a feature by feature comparison of Arduino-capable IDEs. It's a personal opinion of an experienced software engineer, who recently entered this domain. I work on Mac OSX, and so I only briefly mention the Windows options. But on a Mac I did end up trying most of the options listed here.

When I talk about software IDE, I generally do not mean a fancy text editor. While I know that plenty of great developers love and use text editors on complicated software projects, the convenience of a true IDE on a small to mid-sized software project is hard to dispute. So while I do, when appropriate, use a combination of VIM or TextMate for exploring or quickly editing, I prefer to actually write code in a true IDE. If I had to define what true IDE means for me, I would list the following features:

  • full indexing with ability to click on a symbol or use a key shortcut to go to the definition or source file where that symbol is defined
  • in-place documentation lookup for APIs
  • code auto-complete (configurable, and not automatic – key press invoked)
  • code auto-format with automatic tab positioning while editing
  • refactor (extract method, etc)
  • symbol rename (automatic renaming of functions, class names with file name change, variables, etc across many files)
  • built-in debugger
  • automated test integration
  • arguably less important are global project search, global find and replace, project-level symbol lookup.

My absolute favorite over the years have been the series of IDEs for all popular languages produced by JetBrains –– a company that singlehandedly dominated the IDE nitche for over a decade now, including winning over droves of professional java programmers away from Eclipse.

But anyway, here is the list of what's available for proper software development of the code, with Arduino as the final destination:

Arduino IDE for Visual Studio

(screenshot courtesy VisualMicro)

  • embedXCode – use XCode 4 or XCode 5 to write Arduino sketches. I tried to install this, and was able to pretty quickly compile a sketch. But to be honest, I never loved XCode to begin with. It's gotten infinitely better over the years, but something about it's Preferences screens is so incredibly daunting, that I never got really good at XCode. Perhaps someday :)

Arduino IDE for Visual Studio

(screenshot courtesy StackOverflow)

  • This brings us to the last contender: Eclipse. Eclipse has been around for a long time, and is an amazing platform for so many things, including software development. Having used Eclipse on several Java projects in the past, it seemed the most natural fit. So I downloaded the Eclipse Arduino Plugin, and that's what the rest of this post is about.

Eclipse Arduino Plugin Saves the Day

Having now used this IDE for several weeks straight, I am pretty happy with the Eclipse Arduino Plugin. The project is pretty active on GitHub, and the author is also quite nice :)

The Case for Open Source

One of the first things I bumped into, with the nightly build of the plugin and with my Arduino Esplora board, was a pretty major problem: I couldn't upload any sketches. So I posted a bug report on GitHub, and author replied with a quick note on how to get the source of the plugin, and which class to look at, so that I could fix the problem. It sounded like a challenge. Of course I took it.

For the next few hours instead of working on my Arduino sketch I was fixing the Eclipse plugin. To my surprise, it was relatively easy to get setup with the environment where I imported the entire plugin source into JetBrains IDEA (haha, sorry Eclipse! You are still number two :) and was able to diagnose and fix the issue with the timing of opening serial port and uploading the sketch. A few hours later my pull request was merged, and the nightly build of Eclipse Plugin started working for everyone with Arduino Esplora! That, my friends, is the true power of open source.

While I was at it, I also updated the README with proper markdown and (perhaps) slightly better English. And of course I couldn't stop there either, and continued going slightly crazy, massively refactoring serial communications of the plugin deep into the night, and then submitting a beautiful pull request. However, at that point the plugin author probably had gotten pretty annoyed that I was making his code look and work a bit better, and sadly rejected the PR, explaining that another rewrite of serial comms is happening. Oh well, at least I can keep using my fork on my own machine, where I get to see pretty error messages that actually explain what's going on :)

But I digress.

Installing Eclipse Plugin for Arduino

Option 1: probably the simplest way to get started is by downloading the nightly build, in my experience they've been pretty stable. The single-file download will already contain a compiled binary (called something like "arduinoEclipse.app" or similar), and you can just run it.

Option 2: But my preferred way to install the plugin is to first install a full version of Eclipse Luna for C/C++, and then add Arduino plugin to it via software installer.

↳ Keep reading …

Books I Am Reading

This post is about the books I am reading right now, with short reviews.

↳ Keep reading …

How It All Started

I've been a software engineer or technical leader for most of my career, which approaches on 20 years (I just recently hit 40 earlier this year). I started coding when I still lived in Kharkov, Ukraine, not too far from where the war is raging right now (a fact I am still finding hard to believe). I wrote my first program sometime in 1988, which was a keyboard driven version of "Paint": you could move a cursor that drew a line on your screen, and save and recall your creations. This was all before the mouse made it over there.

Just around my 40th birthday I wanted to fix a subwoofer that I intended to use for the birthday party that day. It was a Mackie powered subwoofer, about 500W (plenty loud), but it's been broken since the last Burning Man, when it quietly died in the 110'F heat. I opened it up, found some wires that seem inappropriately disconnected, and soldered loose ends to what I thought was the right receptacle with a crappy giveaway soldering iron I received at some Java conference more than a decade ago. I wasted two hours of my life aimlessly trying to "fix" the subwoofer, and while I was doing that a realization dawned on me: I have absolutely zero idea about how any of this electronics actually works... I did remember some facts about resistors and capacitors, but I surely couldn't tell what was what in front of me, how it all fit together, and more importantly: how was I going to fix this thing.

In the end I was not successful in my quest. I've assembled the subwoofer back to it's original form, and it's currently sitting in my garage waiting to be delivered someone with trained hands at JK Sound SF, someone who can actually fix it. But in the meantime, I signed up for a soldering class at the TechShop, and purchased a "pro" level soldering iron – Weller WES51 on Amazon. This is how it all began.

↳ Keep reading …