2014 Mofo Availability

The yardstick we use in DevOps is how often our sites go down. I wanted to share how we did in 2014 with you!

Some of these sites say 100%…and that isn’t always precisely accurate. We were just down for so little time that it rounded up!

Adhoctribution-Production — 100.000%
Appmaker-Production — 99.992%
Badgekit.Openbadges.Org — 100.000%
Badges.Webmaker.Org-Production — 100.000%
Bakery.Openbadges.Org — 94.861%
Bleach-Production — 100.000%
Blog.MozillaIgnite.Org — 99.999%
Build.Webmaker-Production — 100.000%
Discourse.Mozilla-Community.org — 99.982%
Discovery-Openbadges — 100.000%
Drumbeat-Production — 100.000%
Events-Production — 100.000%
FireFox10.org — 99.993%
GameOn-Production — 99.929%
Gear.Mozilla.Org — 100.000%
Goggles-Webmaker-Prod — 99.982%
HTTP-Redirects-Production — 100.000%
Kitbuilder-Webmakerprototypes-Production — 100.000%
Login-Webmaker-Prod — 99.999%
MakeAPI-Production — 99.991%
Makedrive-Dev — 99.997%
Makerparty.webmaker.org — 99.993%
MakesOrg-Production — 99.999%
MozillaFestival-Production — 100.000%
MozillaOpenNews-Production — 100.000%
MozillaScience.Org — 99.972%
Node-Hubble-Production — 99.996%
OpenBadges-Production — 99.999%
Openbadger-CSOL-Production — 99.991%
Openbadges-Backpack-Production — 99.975%
Opennews.org-Production — 100.000%
Popcorn-Production — 99.964%
Soundcloud — 99.952%
Source.Opennews.Org — 100.000%
Studiomofo-Prod — 99.998%
Thimble-Production — 99.991%
TogetherJS-Hub-Production — 99.965%
TogetherJS-Production — 99.990%
Training-Webmakerprototypes-Production — 99.991%
WMLogin-Dev — 100.000%
Weblitmapper-Webmakerprototypes-Production — 99.997%
Webmaker-App-Dev — 99.992%
Webmaker-HeatRemix — 100.000%
Webmaker-Production — 100.000%
Webmaker-Publisher-Dev — 99.989%
advocacy.mozilla.org — 99.975%
api.badgekit.org — 100.000%
badgekit-api-mozilla.mofoprod.net — 100.000%
badgekit-mozilla.mofoprod.net — 99.998%
badgekit.org — 99.999%
chicagosummeroflearning.org — 99.999%
fundraising.mozilla.org — 100.000%
hiveberlin.org — 99.932%
hivechicago.org — 99.917%
hivekc.org — 100.000%
hivelearningnetworks.org — 100.000%
hivenyc.org — 99.967%
mozillawebmakerproxy.mofoprod.net — 100.000%

LGBT and Our New CEO at Mozilla

I, along with a number of other Mozilla employees, have just tweeted that the decision to appoint Brendan Eich to CEO has caused us concern. Why?

I think this deserves a longer form to explain my personal thoughts on the subject.

Mozilla’s culture is one of openness, which is why I am not in fear for my job by expressing my concerns and opinion here or on Twitter. Being incredibly international, Mozilla culture is also exemplified by unconditional tolerance for other views, opinions, religions, and cultures.

I used to work for the Obama campaign, and I work happily alongside Republicans and many libertarians alike. I’m agnostic, but I work perfectly alongside very religious people.

You may be asking yourself right now “But wait…I thought you said that Mozilla was so open to other views, yet you have concerns with the appointment of Brendan Eich as CEO…where is the disconnect?”

Let me explain.

Brendan has been CTO for a very long time, and has been extraordinarily successful. After all, he invented Javascript and co-founded Mozilla in the early days. To say that he is a capable and intelligent person is to understate it. (Note: I’ve not had the opportunity to work directly with him.)

So why wouldn’t I have a problem with Brendan as CTO, while I do have concerns about his role as CEO?

From Wikipedia’s definition of CEO:

(Note: this is not necessarily Mozilla’s definition of CEO.  Not much of the outside-of-Mozilla world knows that, however)
“The communicator role can involve the press and the rest of the outside world, as well as the organization’s management and employees; the decision-making role involves high-level decisions about policy and strategy. As a leader of the company, the CEO/MD advises the board of directors, motivates employees, and drives change within the organization.”

That is a very different role than that of a CTO:
” As a corporate officer position, the CTO typically reports directly to the chief executive officer (CEO) and is primarily concerned with long-term and “big picture” issues (while still having deep technical knowledge of the relevant field).”

A CEO is publicly seen as one of the most visible faces of an organization, and is quite a bit about image, partnerships, and culture, rather than the largely technical role of a CTO.

Unfortunately, right now Brendan’s public image (which is also now in part Mozilla’s public image in his new role) is one showing that he donated money to deny equal rights to the LGBT community during Prop 8 in California.  Note: I fully support Brendan’s right to hold these views and support them financially as he sees fit, even while I vigorously disagree with his views on this issue.

In California, donations to these political groups and causes are public record. Thus, this is no longer just a personally held view, it is now a publicly held view.

As it is a publicly held view it has the potential to reflect on Mozilla. My concern is that it could come across to potential funders, contributors, volunteers, and users of our products that maybe Mozilla *doesn’t* really believe in all the inclusiveness we have prided ourselves in, even if we know we still do.

I am concerned that having Brendan in the role of CEO could send the wrong message to some of the people we want to build the open web with.  Personally, I have had at least 15 former colleagues and friends engage me about how Mozilla could make this move, many of whom have volunteered for Mozilla in the past.

In the opinion of many(myself included), things like equal access to vote, to marry, to have opportunity, or the ability to visit a spouse in the hospital are not matters of politics, but rather human liberty and freedom, things which Mozilla stands for.

Had there been no public donations record indicating he supported Prop8, neither you nor I would have known, and his image as CEO and the image of Mozilla could not suffer. After all, as many articles on the subject have pointed out, Mozilla has an internal culture and benefits package that couldn’t be friendlier to LGBT individuals.

I am very trusting and respectful of Mitchell Baker and the rest of Mozilla management.  Mitchell has worked alongside Brendan for a decade and never knew about their differing opinions on the issue of LGBT rights. I am proud that today, we had a large town hall call to voice our concerns…this is what Mozilla is about.   I don’t personally believe for a second that any management would stand by and allow any policy in our Community Participation Guidelines to regress.

I’m grateful that I have the right to express my concerns on who represents Mozilla both to management and to you.  In the words of Mark Surman, Exec Director of Mozilla Foundation:

“Our culture of openness extends to letting our staff and community be candid about their views on Mozilla’s direction. We’re proud of that inclusiveness and how it distinguishes Mozilla from most organizations. We expect and encourage Mozillians to speak up when they disagree with management decisions, and carefully weigh all input to ensure our actions are advancing the project’s mission.”

This affirms to me why Mozilla is the organization to which I have pledged my heart.

This all warrants discussion, so let us keep this conversation going.  Blog, tweet, and post about what you think.

Other posts worth noting on the subject:

Brendan Eich’s Post

Mitchell Baker’s Post 

Geoffrey MacDougall’s Post

OpenMatt’s Post

Erin Kissane’s Post

 Chris McAvoy’s Post

Matthew MacPherson’s Post 


Even more Webmaker frontend data

Over the last few days, I’ve thrown a bunch of charts at you newly available from NewRelic.

We’ve been adding more and more apps with this capability, so here is some more fun data.

First, thimble.webmaker.org

Next, goggles.webmaker.org

Now, popcorn.webmaker.org

We need a CDN, and multiregion to get these underserved countries loading our apps faster.

Follow-Up: Webmaker.org browser performance and more in New Relic

In a previous post, I showed you the performance of our node.js application servers for the Webmaker suite.

As my boss has noted, though, app server performance is not the same thing as user experience or browser performance.

Though we recently added monitoring of AWS resources in New Relic and now had information about load balancer utilization, RDS(Amazon Web Services database service) utilization, disk volume health, and even SES bounce rates, he was right…

We still had no idea how users actually experienced our applications.

My last DevOps gig got me used to having RUM(real user measurement), and I’ve been telling coworkers and boss “Soon, soon we’ll have browser performance data like these other languages”. While I’ve been telling them that, I’ve been experimenting with other solutions like Stackdriver, Pingdom RUM, bucky, and Google Analytics, despairing that node.js was a second class citizen in New Relic.
New Relic just announced v1.4.0 of its node.js agent, which finally includes browser performance. You can read about it here.

Today, we put this into production on webmaker.org

I have wondered how our location in Virginia (AWS datacenter) and lack of CDN impacted our international. After all, the webmaker team just spent a ton of effort localizing webmaker pages. Side note: We’re ranked in the top five or ten on transifex for localization projects. (HIGH FIVE AALI/TEAM)

So then, how do we do internationally on webmaker.org pages?
Highlight countries to see data

Well, those results are pretty expected. A lot of overseas traffic, including places with underserved internet, is far slower than North American traffic.

So, overall then, how much time is being spent in the browser? Is it all just network, as the previous map might imply? Let us check the overall time spent.


This tells me we’re spending a lot of time rendering, which we can resolve with some work on a CDN and more diversity geographically with our infrastructure (Context: We’re in us-east-1 on AWS)

One other thought…what kind of data can I find for client type (mobile vs. desktop, etc)?

Here’s what the last 24h look like for webmaker.org

So another thing I can share with developers is javascript errors on the client. I was surprised at our error rates in our first day of having visibility.


Another interesting thing to look at in the new views of data is browser prevalence and performance.

Chrome Load Times and Usage

Firefox Load Times and Usage

Being able to have all of these views for all languages and frameworks we use, combined with the monitoring AWS resources (Load balancer stats, DB server stats, disk volume stats, SES stats) the New Relic plugins make it a pretty complete package for monitoring and alerting. Even better, plugins for Mongo, Elasticsearch, and tons of other apps and utilities fill yet another gap.

Overall, I’m digging the new RUM in New Relic as well as the wide array of plugins on the marketplace, but here are a few things I still yearn for.

Separation of app server activities

Below is an example of first a python (django, specifically)app perf chart in production and a then one for a node.js app.

Python App Server Chart

Node.js App Server Chart

I, as an avid fan of color charts, am a bigger fan the more colors it has.

Needs Moar Dashboard

I make heavy use of custom dashboards in New Relic, giving every app a detail page and every project an overview page.
With the advent of all of these plugins, such as those for Elasticsearch or Mongo, and the RUM and browser data, the limit of 15 charts per dashboard just isn’t gonna cut it.

Dashboards showing application server performance, database response and throughput, error rates, top transactions, scalability analyses, load balancer latency/throughput/response stats, RDS and Elasticache stats(CPU %, Free space, Mem), Elasticsearch index performance, and Mongo performance can really help speed up correlation and problem resolution.

We need AWS plugins to be able to associate names with tags, not just hostnames
One awesome thing in Stackdriver was the ease with which I could make a group.
“OK, um, all “webmakerorg-production”
Magic! There were my autoscale groups with instances in them, aggregating CPU and Memory by autoscale group. This is one place New Relic falls short of Stackdriver.

All said, we’ve had amazing progress for us node.js production shops from New Relic and the community developing plug-ins. From engineering here at Mozilla Foundation, thanks!

Newrelic Node.js Agent Is Maturing!

Today, we added newrelic node.js agent v1.4.0, which was just recently released and finally supports browser monitoring.

There is a ton more that it can show too.

Overall browser performance for webmaker.org

Pages with js errors for webmaker.org

This is going to be so informative for our developers!

Webmaker Apps Performance

Curious how fast a Node.js app running in production is? These are our graphs for application performance on webmaker.org applications.

You can even see where we’ve deployed a new version.
Webmaker.org App Performance

Popcorn App Performance

Thimble App Performance

Goggles App Performance

Login Performance

MakeAPI App Performance

Autoscale group aggregate CPU utilization

Autoscale group aggregate CPU utilization

Database IOPS and latency

Database IOPS and latency

Database Utilization

Database Utilization