Setting up StackExchange's Opserver

Nick Craver from StackExchange recently posted StackExchange's homegrown monitoring system. The system, named Opserver is available on GitHub.

In this post I hope to gloss over some of the issues and/or configuration steps necessary to run this slick monitor.

So lets try this.


  1. Grab the code from GitHub either, for the lazy here is the zip file link
  2. If using Visual Studio 2010 or previous, make sure you have ASP.NET MVC installed. Grab that here and make sure to run the installer as administrator [Windows 7+]. 
  3. Crack open the solution file (either from your cloned repo OR the unzipped package downloaded in step ). There should be 2 projects visible and loaded. If you get a "The project file '.....csproj' cannot be opened. The project type is not supported by this installation." error, see this StackOverflow answer.
  4. Compile. You should have no errors and everything succeeds. 
Side note: This codebase was built using Visual Studio 2012 and recently updated to use Visual Studio 2013.


All configuration is handled by .json files in the Opserver/Config directory. Nick has a .example file for each of the different types of dashboards there are. Some of these appear to be very centric to StackExchanges build, so out of the box you may not be able to use all the configurations without reworking some areas of your codebase.


This is the first file you'll need to copy/paste in order to get into the system. 
Edit this file to match the security settings you want.

The example looks like this:
<?xml version="1.0" encoding="utf-8"?> <SecuritySettings provider="AD"> <!-- Optional, these networks can see the overview dashboard without authentication -->     <InternalNetworks>         <Network name="SE Internal" cidr="" />     </InternalNetworks> </SecuritySettings> <!-- Example of global access for everyone:<SecuritySettings provider="alladmin" />-->
which limits access to StackExchange's internal network, yours will need to be changed

Possible "provider" attributes are: "activedirectory", "ad", "alladmin", or no attribute in which everyone is read only. "ad" and "activedirectory" are analogous.

For my case I have everyone being an admin, since this is mainly for our internal development and QA teams.

Mine looks like this:
<?xml version="1.0" encoding="utf-8"?>
<SecuritySettings provider="alladmin">

Note, even if everyone is an admin, they will have to login initially using an empty username/password combination to get into the system.


If you do not know what Redis is or do not use it skip this.
Adding a RedisSettings.json file allows you to monitor Redis memory use as well as perform some health checks.

The configuration for Redis is a tad cryptic for me still, so bear with me.

Here is our config file, modified to remove server names and specifics:
 "allServers": {
  "name": "All",
  "instances": [
    "name": "Core",    
    "analysisRegexes": {
     "**Dev**": "^dev-",
     "**Live**": "^prod-",
 "Servers": [
  { "name": "server1" },
  { "name": "server1Slave1" },
  { "name": "server2" },
  { "name": "server2Slave1" },
  { "name": "server2Slave2" }

Under the instances array we have a "Core" name and two children that define regular expressions to match Redis key names for categorizing. That is, if a key starts with "dev-", it will be categorized as "**Dev**" under the "Core" instance when running analysis.

The Servers array defines the server names your Redis instances are running on. 
Here's a screenshot of what the Redis monitor home page looks like:
You can see the "Core" name in use on the left.


The entries in this file define the connection strings for any SQL server instances you wish to monitor. There are two main arrays to populate, "clusteres" and "instances".

"clusters" refers to SQL Server 2012 cluster configurations. If the SQL instance you put in here are not SQL Server 2012 you can get memory use spark charts only. Outside of that it will appear offline.

"instances" refers to standalone SQL server instances, and at least in our internal setup, make up a majority of our SQLSettings configuration.

Our configuration, truncated some for brevity:
    "defaultConnectionString": "Data Source=$ServerName$;Initial Catalog=master;Integrated Security=SSPI;",
    "clusters": [
         "name": "SDCluster01",
         "nodes": [
          { "name": "SDCluster01\\SDCluster01_01" },
   { "name": "SDCluster02\\SDCluster01_02" },
    "instances": [        
        { "name": "SDDB01" },
 { "name": "SDDB02" },
 { "name": "SDDB03" },
 { "name": "SDDB04" },

Any one of the JSON objects that have a "name" property, can also specify a "connectionString" entry in case it differs from the "defaultConnectionString"'s entry.

Here is a display of our non 2012 cluster and information:
As you can see, the top "cluster" is highlighted in red, even tho the spark chart is coming in. 
Also you can see under the standalone grouping we have 1 instance that's not up at all.

Each one of these instances you can drill into to obtain more detailed information, for example:


To use this monitor panel, you pretty much need to use StackExchange.Exceptional. This, again, is StackExchange's error logging system, which writes unhandled exceptions to a central database. 

If you happen to not use this, skip this for now.

The configuration for this is pretty straight forward for a quick setup, the only necessary item is the connection string of the database your Exceptional logging system is writing to.

    "warningRecentCount": "100",
    "criticalRecentCount": "200",
    "viewGroups": "StatusExceptionsRO",
    "applications": [
        "Customer Support",
    "stores": [
            "name": "SanDiego",
            "queryTimeoutMs": 2000,
            "pollIntervalSeconds": 10,
            "connectionString": "Server=SDDB_01;Database=Exceptions;Integrated Security=SSPI;"

String items underneath "applications" defines what is displayed by default on the Exception pages header. If an exception is written to the database that does not fit any of those application names, a new category will display in the pane.

I was going to post a screenshot, but after blurring out anything from our stack, it was useless. I would like to add however, that both Exceptional and the Exception page are just beautiful.


So far my coworkers and I love this, and I can see from the code its only getting better (hello there sp_WhoIsActive, sp_BlitzIndex!).

I hope this helps someone. Together with Exceptional I've personally found 3 issues with our codebase not previously known, scary but also eye opening. And I think that is the goal, make your errors eye opening and easy to see, so you can fix them.



Here's a post I found today that is just awesome.

It reads almost like a poem.



Using .Net usings...

In .Net to make use of other namespaces (libraries of classes) you do so by the 'using' keyword. On the other hand you can fully qualify the namespace if you wish to avoid including 'using' methodology. Whatever the case this isn't a lesson on the using keyword.

The real point of this post is to attempt a practical ordering included namespaces for all classes. The pattern I'm suggesting (hell, maybe it's even actual recommendation already?) is the following:

(Core namespaces)

(Your [company/project/school] namespace)

(3rd Party namespace)

Essentially giving organization to something that is usually a mess.

Lets take a peek at an example.

using System;
using System.Collections;
using System.Collections.Generic;
using System.Data;
using System.Drawing;
using System.IO;
using Dundas.Charting.WebControl;
using MyCompany.Core.Framework;
using MyCompany.Core.Utils;
using MyCompany.Project.Common.Analytics;
using MyCompany.Project.Common.Utils;
using MyCompany.Project.Core.Emissary;

Look at this mess, no order, its not immediately obvious which 3rd party components are being used.

Now using my suggestion:

using System;
using System.Collections;
using System.Collections.Generic;
using System.Data;
using System.Drawing;
using System.IO;

using MyCompany.Core.Framework;
using MyCompany.Core.Utils;
using MyCompany.Project.Common.Analytics;
using MyCompany.Project.Common.Utils;
using MyCompany.Project.Core.Emissary;

using Dundas.Charting.WebControl;

Doesn't that look better? It gets across everything immediately and in an standard fashion.

Oh also, always always always get rid of unused namespaces. More clutter to junk up your files.

Quick easy fashion to do this. Right click on your class definition -> Organize Usings -> Remove and Sort. Now there is no excuse to have unordered, non-used namespaces in your files, and its built in.


Unknown Windows shortcuts

Most keyboards these days have the Microsoft logo'ed Windows key. This key is simply magic.

For example hold that key down and tap the 'E' key.
My computer just pops up, bam! Navigate away.

Windows key + 'R' : The run prompt shows up ready to do your bidding.

Couple this with the different service consoles or applications (services.msc or notepad++ for example) and you are just getting around without even taking your hands off the keyboard.

Windows key + 'D' : Minimize all visible applications, OR re-display visible applications as their previous display. Try it, its amazing.

Most of these are known and used everyday, however there are a few people I show are still wowed by. And recently Lifehacker posted some Windows 7 tips that wowed me.

If you're using Windows 7 with multiple monitors you just have to use these.

Windows + Up Arrow = Maximizes active window
Windows + Down Arrow = Un-maximizes active full screened window OR minimizes active windowed application.
Windows + Left (or Right) Arrow = Cycles window in direction of arrow key across monitors. Amazing!

I've been loving this since I learned about it, just changes everything!


Lull point

I've not posted in so long, my massive audience has now probably stopped following. Sorry loyal followers!

I must admit, I feel I just don't offer enough as a developer to post compared to other avid/interesting bloggers. And as of recently I've been in a development lull.
No interesting discoveries, no new tools I've dabbled with, nothing really.

Recent work has changed from a News harvester to a rewrite of a very high traffic site.
While it is exciting/new/challenging, I'm just tanking in enthusiasm.

Ah well, pity me huh?

Recent things I can blurb about:

-Read Don't Make Me Think: A Common Sense Approach to Web Usability, 2nd Edition By Steve Krug.
Thoughts: Very easy read and sensible book on usability. Life changing? No, but sometimes I think you need to get smacked in the face with obviousness to realize things. For example the chapter on web usability I found really informative.

-Going to Spain! As we try to do every few years, we are getting out of the U.S. for a duet of weeks to see exactly what Spain is about. Very excited about this, and the excitement is growing as the days counting continue downward.

That's about all I can dump off the mind at the moment. I believe it is a glimpse at my lull.

Ah well.... as Johnny Cash said, Drive on...


Dear Microsoft

Ever since I was a gaming noob or programming wanna-be (I guess I still am this though) I've been interested in computing ergonomics. Ergo mouse, ergo keyboard, ergo desk, ergo monitor configuration etc. While this puts me out as the dude at work with the weird setup, it is surprising to see how many career developers don't even consider it. The risks are real.

I currently use the Microsoft Natural Ergonomic Keyboard 4000+ v1.0

This is a great keyboard. Honestly...

But there is one thing that I can totally do without. Take a gander at that gigantic number and arrow pad section. It's basically 1/3 of the keyboard in total. Imagine an ergonomic style keyboard (with wavy keys) without a number pad. This gigantic number pad requires me to move my right arm an additional 8ish inches to the right causing some discomfort after long periods of use. Now when I'm whipping out design elements on a windows form dragging and dropping things all over; the additional mobility cost spent to rename default name properties is fatiguing.

I imagine what it would be like to have my keyboard not off set from the front of me or my mouse in an ideal location. The first keyboard maker that comes out with this will have real market for this. With the continued research into repetitive stress injury and computer related injuries (I realize that sounds crazy) will come the increased knowledge and want for extremely ergo items.