Collecting Haze Data with Arduino, Raspberry Pi, and Amazon DynamoDB

I finally decided to put those Sharp airborne dust sensors which I bought during the last haze epidemic (in 2013) to good use, along with the small mountain of parts I have at home.

I used this very simple Arduino circuit and code snippet to read the Sharp sensor.  I ended up using a 100uF capacitor and 220-ohm resistor because those are what I had on hand; performance seems unimpaired. I used an Arduino Uno, because the dust sensor needs 5V. The sensor is very noisy and jumpy, so I used an interquartile mean and 200 measurements (discarding the bottom 25% and top 25%) to get a more robust reading. The raw value still jumps around by 1 LSB.


I also modified the slope function so that a full-range reading is 250 ug/m^3 (the original formula is here). Here is the modified Arduino code:


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
// program to read a Sharp GP2Y1010AU0F dust sensor
// this sensor produces a dust value with a maximum value of 250 ug/m3

#define NUMSAMPLES 200
#define dustPin 0
#define ledPower 2
#define delayTime 280
#define delayTime2 40
#define offTime 9680

void setup() {
  Serial.begin(9600);
  pinMode(ledPower, OUTPUT);
  pinMode(4, OUTPUT);
}

// swap sort algorithm
void swapsort(int *sorted, int num) {
  boolean done = false;    // flag to know when we're done sorting              
  int j = 0;
  int temp = 0;

  while(!done) {           // simple swap sort, sorts numbers from lowest to highest
    done = true;
    for (j = 0; j < (num - 1); j++) {
      if (sorted[j] > sorted[j + 1]){     // numbers are out of order - swap
        temp = sorted[j + 1];
        sorted [j+1] =  sorted[j] ;
        sorted [j] = temp;
        done = false;
      }
    }
  }
}


// read the dust sensor, implementing an interquartile mean
// thanks to STMicro Application Note 3964 "How to design a simple temperature measurement application using the STM32L-DISCOVERY"
int readRawDustValue() {
  int i = 0;
  int rawVal[NUMSAMPLES];

  for (i = 0; i < NUMSAMPLES; i++) {
    // ledPower is any digital pin on the arduino connected to Pin 3 on the sensor
    digitalWrite(ledPower, LOW); // power on the LED
    delayMicroseconds(delayTime);

    rawVal[i] = analogRead(dustPin); // read the dust value via pin 5 on the sensor
    delayMicroseconds(delayTime2);

    digitalWrite(ledPower, HIGH); // turn the LED off
    delayMicroseconds(offTime);
  }

  // now we have the raw values, sort them
  swapsort(rawVal, NUMSAMPLES);
  
  // drop the lowest 25% and highest 25% of the readings
  long dustVal = 0;
  for (i = NUMSAMPLES / 4; i < (NUMSAMPLES * 3 / 4); i++) {
    dustVal += rawVal[i];
  }
  dustVal /= (NUMSAMPLES / 2);
  return (dustVal);  
}


// convert the raw count to a dust value
// the full-range signal is 771 counts = 3.76V
// based on http://www.howmuchsnow.com/arduino/airquality/
//
// dust density (mg/m3) = 0.172 * V - 0.1

float calcDustDensity (int rawVal) {
  float calcVoltage = rawVal * (5.0 / 1024);
  
  // I use a different figure so that 771 counts = 248 ug/m3
  float dustDensity = ((calcVoltage * 0.688) - 0.1) * 100;
  
  if (dustDensity < 0) dustDensity = 0;
  return (dustDensity);
}


void loop() {
  int dustValue = readRawDustValue();
  float dustDensity = calcDustDensity(dustValue);
  
  Serial.print("Raw Dust Value = ");
  Serial.println(dustValue);
  Serial.print("Dust Density = ");
  Serial.println(dustDensity);
}

Unfortunately the Arduino Uno only has 2K of RAM and mine did not have an Ethernet or Wi-Fi shield, so I decided to use a Raspberry Pi (Model B, the old single-core one) to read the Arduino Uno and talk to DynamoDB.

Basically, the Arduino appears as a serial port to the RasPi and I read the values using pySerial and uploaded them using boto.

Surprisingly the RasPi behaves exactly like a "real" Linux box, installing the AWS CLI and Python SDK (boto3 and boto) is exactly the same as on a large machine. There were no hiccups during the installation (albeit the installation took a long time).


The Python script is as follows (my first Python program!) note the hard-coded AWS credentials which is really a terrible practice.


  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
#!/usr/bin/python

import boto
from boto import dynamodb2
from boto.dynamodb2.table import Table

import datetime
import time
import serial
import re
import sys

# monkey hacking to work around "inexact numeric" issue in boto
import decimal
from boto.dynamodb.types import DYNAMODB_CONTEXT
# Inhibit Inexact Exceptions
DYNAMODB_CONTEXT.traps[decimal.Inexact] = 0
# Inhibit Rounded Exceptions
DYNAMODB_CONTEXT.traps[decimal.Rounded] = 0


conn = dynamodb2.connect_to_region(
 'ap-southeast-1',
 aws_access_key_id='AKIAxxxx',
 aws_secret_access_key='R0KIxxxx',
)
table = Table(
 'dustValues',
 connection=conn
)

# open the serial port (we need to be setuid root for this)
serialport = serial.Serial("/dev/ttyACM0", 9600, timeout=0.5)

rawDustValue = 0
dustDensity = 0

oldDustValue = 0
oldDustDensity = 0

# to put a blank line..
print "\n"

while True:
 command = serialport.readline()
 matchObj = re.match( '^(.*) = (.*)$', command, re.M | re.I)
 if (matchObj):
  # hash is unixTimestamp
  unixTimestamp = int(time.time())
  timestamp = time.strftime("%Y%m%d%H%M%S")

  # Var =  Raw Dust Value
  # Val =  137
  # Var =  Dust Density
  # Val =  81.51
  if (matchObj.group(1) == 'Raw Dust Value'):
   rawDustValue = int(matchObj.group(2))
   # print "raw dust value = ", rawDustValue

  if (matchObj.group(1) == 'Dust Density'):
   dustDensity = int(float(matchObj.group(2)))
   # print "dust density = ", dustDensity

 if ((rawDustValue != oldDustValue) and (dustDensity != oldDustDensity)):
  oldDustValue = rawDustValue
  oldDustDensity = dustDensity

  # print "hash = ", str(unixTimestamp)

  # calculate the PSI (this is a very approximate value)
  # based on dustDensity (in ug/m3) and this
  # http://www.haze.gov.sg/docs/default-source/faq/computation-of-the-pollutant-standards-index-(psi).pdf
  # we only use the 24-hour PM2.5
  # note that the Sharp sensor can't distinguish particle size
  # so PM10 particles are also falsely measured

  psi = 0
  if (dustDensity <= 12):
   psi = dustDensity * 4.17
  elif (dustDensity > 12 and dustDensity <= 55):
   psi = 51 + ((dustDensity - 13) * 1.17)
  elif (dustDensity > 55 and dustDensity <= 150):
   psi = 101 + ((dustDensity - 56) * 1.05)
  elif (dustDensity > 150):
   psi = 201 + (dustDensity - 105)

  psi = int(psi)

  # shorten timestamp so it fits on the tiny PiTFT screen
  ts = int(unixTimestamp) - 1444116000

  print ts, ":",
  print "raw=", rawDustValue,
  print " ug/m3=", dustDensity,
  print " PSI=", psi, "   \r",
  sys.stdout.flush()

  # write to the table
  try:
   table.put_item(data={
    'unixTimestamp': int(unixTimestamp),
    'timestamp': timestamp,
    'rawDustValue' : int(rawDustValue),
    'dustDensity' : int(dustDensity),
    'psi' : int(psi)
   })
  except:
   # do nothing
   pass

I then placed the Python script in /etc/rc.local (making sure to append an ampersand so that booting would complete).

And voila:

Takahashi EM-11 Temma 2 Jr. Review

I have had the misfortune of going through very many mounts through the years: two EQ-1's (one Orion and one Barska), two Vixen Polaris, one Vixen GP, a Celestron CGEM, Astro-Physics 600E QMD, Astro-Physics Mach1, Sky Watcher Star Adventurer, and now the subject of this review - the Takahashi EM-11 Temma 2 Jr.

And the one takeaway from these years of experience (and wasted money) is to buy the very best mount you can afford.  To that  I will now add the caveat: that you can afford and can carry.

To cut a long story short: the EM-11 is very smooth mechanically, is not sensitive to balance so far as I can tell, the polar scope is perfectly aligned, and it can turn out 10-minute guided subs like clockwork.  I can tell from the guiding graphs that it is not as smooth as the Mach1, and it definitely has much less capacity, but I can't carry the Mach1 and tripod with one hand.

The EM-11 has an amazingly low periodic error of about 7" peak-to-peak, which is quite an achievement given the small diameter of its worm wheel (Takahashi only guarantees 20" peak-to-peak).  Of course this isn't so amazing compared to the 3" peak-to-peak (< 1" with PEM) of my Mach1 - but the EM-11 is in the same capacity class as the Celestron AVX or Vixen GP, which normally have around 30" peak-to-peak periodic error.

This periodic error is of the same level as my Astro-Physics 600E QMD, but the EM-11 does not have periodic error correction.  However, the EM-11 has lots of ball bearings, which means its declination guiding behavior is good. Again - not as good as the Mach1, but good enough that I don't lose subs. Which is more than I could say for the AP600 (or the CGEM, but that goes without saying).

To repeat: this particular EM-11 performs better than my old AP600E QMD, albeit with a much lower payload.

I'm using the EM-11 with an old Gitzo G1340 Mark 2 Systematics tripod, which is rated for 20 lb.  Strictly speaking, the tripod is overloaded with the EM-11, about 10 lb of counterweight, and the William-Optics Lomo 80mm APO triplet, but the system is still stable.  Much more stable than the Star Adventurer on the same tripod: hence as an aside, it is the Star Adventurer wedge which is the weak point of that system.

A useful fact: this particular EM-11 has the low-latitude base.  It still cannot reach 1 degree latitude, but by slightly extending the two south legs of the tripod, I was able to get it down to 1 degree without need of a wedge. The additional tilt required is slight, and does not seriously compromise the stability of the tripod.

Another useful insight: the Gitzo tripod is overloaded with everything on it, but settling times are still under 2 seconds.  I tried removing the rubber tips on the feet and use the spikes, but settling times actually got worse.  The rubber feet act like Celestron vibration suppression pads, so should be left on.

Overall, the EM-11 meets my expectations of what a premium mount should be, i.e. it performs very well within its limits and does not require tweaking. It is not quite as refined as the Mach1, but there are no major concerns.

Note that all of my observations thus far are in using the EM-11 for imaging.  When imaging, you generally point it at one or two targets a night, and hammer away with a camera.  In fact, at this time, I have been using the EM-11 with its setting circles to find things (because I can't get the Temma 2 GoTo system working just yet). There's a bit of a retro feel to using the setting circles (kind of like using a film camera or manual-focus lens) but these circles work surprisingly well, so I'm happy.   This is OK, because you can spend five minutes finding an object if you will be spending the next four hours imaging it.

However, as a visual mount, to show the wonders of the night sky to people, the EM-11 is a terrible choice. When I moved from the Celestron Nexstar hand controller to the Astro-Physics GTO, it was like going from the 20th century to the medieval period.  The AP hand controller has a reputation for being built like a Russian tank - sturdy, but primitive.  The Takahashi hand controller is like the Stone Age.  There is no way to command GoTo from the hand controller.

I do have a Roving Networks Bluetooth serial adapter, and a SkyWire, so I can command GoTo's on the EM-11 using an Android or iOS device, but the mount as-is is incapable of GoTo unless you connect it to a computer or smart phone somehow.

My motivation for acquiring this mount is because the Mach1, for all its mechanical and electronic perfection, is a bit too heavy to be set up and taken down on a regular basis.  I would really like to expand the scope of my observation activities and the Mach1 is simply too heavy for casual observing.  I wanted a mount that weighed as little as possible, had GoTo (I have the Star Adventurer, and finding anything with that mount is a headache), and would be as nice as possible.

Of course in the lightweight mount category there is the Celestron AVX and iOptron ZEQ25, as well as the more upmarket Vixen Sphinx SXP and SX2.  I did try the ZEQ25 for some time, but it proved to be extremely sensitive to balance and I never liked the strange worm tension adjustment mechanism that also served as the clutch. I could never get the tension just right.

The EM-11 is way too expensive if purchased new (70% of the price of a Mach1, with one-third the payload) but I was able to acquire this one for a significant discount off the price of a new one.  It still is a rather expensive, low-payload mount, and I would not recommend it as a first (or only) mount, but for someone who already has that "perfect" mount and just wants more portability, it is a good choice.

My main concern was that no mount could ever match up to the performance of the Mach1, and I would be annoyed at the degraded user experience. Well, the EM-11 doesn't match up to the Mach1, but within its payload limits, it comes close, and the significantly lower weight is priceless.

Highly recommended! But not to beginning users.  The combination of the steep price, extremely user-unfriendly hand control, and low payload make the EM-11 a good choice for second or third mount. It's the Lotus Elise of mounts, small, finely-engineered, does what it does very well, but not terribly practical.

Now if only Takahashi would bundle the EM-11 with an actual intelligent hand controller, lower the price to $3000, and include a PEC recording capability, it would be more approachable - the Mazda Miata of mounts, as it were.